Multivariate approach to quality control in clinical chemistry

Transcription

1 Clinical Chemistry 44: (1998) Laboratory Management Multivariate approach to quality control in clinical chemistry Jerry Dechert* and Kenneth E. Case When monitoring analyzer performance in the clinical setting, laboratories are required to test multiple concentrations of control material on a daily basis. Because of the nature of laboratory testing, there is the potential for correlation between the concentrations of control material being monitored. Although traditional clinical quality-control approaches make an underlying assumption of independence with respect to the control concentrations, this will not always be the case. The presence of correlation in some circumstances suggests the use of a new approach for evaluating clinical laboratory monitoring data: the multivariate control chart. Such a chart (the chart) is evaluated and compared with traditional quality-control approaches used in the laboratory setting. Results indicate that the multivariate approach provides an attractive alternative to many traditional methods of quality assurance when control concentrations are correlated. Many approaches have been used for the purpose of monitoring and controlling clinical laboratory testing over the years. In fact, the original application of statistical monitoring of clinical quality control dates back to the 1950s (1). This initial application by Levey and Jennings involved testing duplicates of each concentration of control material and plotting the results against 3 SD limits. Although a number of alternative methods have been proposed in the literature, they all focus on the individual concentrations of control material and fail to acknowledge the multivariate nature of the monitoring problem. This paper will explore the use of a multivariate qualitycontrol chart for application in the clinical laboratory and compare its performance with some traditional clinical monitoring approaches. Although the original application of statistical monitoring dates back to Levey and Jennings, a number of School of Industrial Engineering, University of Oklahoma, 0 West Boyd, Suite 14, Norman, OK * Author for correspondence. Fax ; dechert@mailhost. ecn.ou.edu. Received January 1, 1998; revision accepted May, alternative approaches have been identified over the years (, 3). Probably the most renowned alternative is Westgard s Multirule Procedure (4). This approach reduces the high false rejection rate that can accompany the strict use of SD limits and provides good sensitivity to clinically significant shifts, but it requires multiple quality-control testing points (i.e., quality-control batches) to achieve this sensitivity. Recent developments in clinical quality control have focused attention on using quality-control systems designed to detect clinically significant changes in the measurement system (5, 6). As the observed variability in assay systems continues to decrease, understanding clinically significant errors and designing quality-control systems to specifically detect these errors will become more and more important. Therefore, it is necessary to identify quality-control approaches that have predetermined average run lengths (ARLs) for detecting specifically defined changes in the quality-control system. A multivariate quality-control chart is an approach with that capability. Correlation in Clinical Quality-Control Data The most compelling reason to examine multivariate quality-control charts for monitoring in the clinical setting is the potential correlation structure in laboratory qualitycontrol data. When testing control materials, laboratories will typically test two or three concentrations of control materials at the same time (i.e., low, mid, and high). The result is that there is often correlation between control concentrations within an analytical run. The low, medium, and high controls are typically related because they are affected by essentially the same sources of variability at the same time. Examples of sources of variation leading to correlation within an analytical run include laboratory temperature and humidity, calibration curve shape, electrical fluctuations in analyzers, and technician or pipetting errors (particularly in pretreatment assays) to name a few. It is not our purpose here to perform a definitive study with respect to correlation in the clinical laboratory. Our experience, however, provides undeniable evidence of correlation in the clinical setting. Specifically, we have 1959

2 1960 Dechert and Case: Multivariate approach to quality control collected data from clinical chemistry and drug monitoring assays and evaluated correlation in these data sets. The data used in the evaluation included actual laboratory monitoring data along with some data generated according to NCCLS EP-5 protocol for evaluating precision. Estimated correlation coefficients between concentrations across four assays were 0.05, 0.40, 0.67, and 0.70, respectively. From this small number of assays, it is clear that the presence of correlation in testing data is real; however, the degree of correlation may vary from situation to situation. In addition to using the aforementioned data for evaluating correlation within an analytical run, the data were evaluated for the presence of correlation from run to run within a control concentration. Autocorrelation plots were analyzed for these data, and these plots indicate no significant correlation from run to run within one concentration of control material. Because there are many possible sources of variability leading to correlation, it is incumbent on laboratories to check quality-control data for the presence of correlation. By calculating the sample correlation coefficient r, a laboratory can readily quantify the degree of correlation between concentrations. If substantial correlation is evident, then the laboratory should consider the use of quality-control approaches that will properly account for correlation in the data. Given the possibility of correlation in control data, the next step is to determine its impact on the performance of quality-control methods. If one considers the instance of two control materials, then the difference between assuming independent and correlated control concentrations can be demonstrated graphically, as shown in Fig. 1. In the case where independence applies, the in-control region determined by UCL (mean k SD) and LCL (mean k SD) presents itself as a square. In the case where correlation is present, the true in-control region presents itself as an ellipse (7). This means that assuming independence of control concentrations and applying k SD limits as such when the control concentrations are truly correlated will produce areas of over-control and under-control, as shown in Fig. 1. An over-control area is the region in which independent control limits would indicate a shift in the control materials when no true shift has occurred (i.e., commit an alpha error). The undercontrol area is the region in which independent control limits would fail to indicate a true shift in the control materials (i.e., commit a beta error). As stated previously, the purpose here is not to definitively characterize the nature of correlation between control concentrations within a given analytical run. The purpose is to acknowledge the real existence of this correlation in many instances and to provide a means for dealing with this correlation. A method for dealing with correlation, the chart, is detailed in the next section. The Chart Although a number of multivariate approaches are available for quality-control application, the chart (8, 9) was used for this research. This selection was based on the fact that the chart assumes a known covariance matrix, the same assumption made for evaluating traditional qualitycontrol approaches. In practice, a T chart (10, 11) can be used in the case of an unknown covariance structure with results similar to the chart. Assuming a known covariance matrix for the control concentrations being monitored, application of the chart is very straightforward. There is only an upper limit for the chart, and it is based on the distribution. The upper limit for the chart is,p, where p is the number of parameters being monitored and is the probability of a type I error (i.e., the probability of the chart signaling a shift in the control materials when none has occurred). In the clinical application, p would be the number of control concentrations being monitored (typically two or three). Given the upper limit for the chart, the statistic plotted is simply n(x 0 )/ 1 (x 0 ), where n is the sample size (typically n 1 for the clinical application), x is the observed sample mean vector (observed values from the quality-control batch), 0 is the original mean vector (control concentration targets or historical means), and 1 is the inverse of the covariance matrix of the control concentrations. A simple algebraic formula for calculating this statistic for two concentrations of control material is as follows: n x 1 1 x 1 1 x 1 1 x 1 Fig. 1. Control region for two correlated control concentrations. where x 1 and x are observations for control concentrations I and II, 1 and represent historical means for control concentrations I and II, and 1 and represent the SDs of control concentrations I and II, respectively, with 1 the covariance between concentrations I and II. By plotting this calculated statistic against the upper limit

3 Clinical Chemistry 44, No. 9, of the chart, one can determine if the measurement system has shifted. Method Comparison To understand the performance of a multivariate approach, it is necessary to compare such an approach to traditional methods of clinical quality control. For the purposes of this research, traditional methods of quality control include the strict application of a single limit set at.75 SD limits (1.75s ), the use of an immediate retest for limits set at 1.91 SD ( 1.91s ), and Westgard s Multirule Procedure as originally published. Although typical approaches would be the strict application of SD limits (1 s ) and SD limits with an immediate retest ( s ), these rules have been replaced with 1.75s and 1.91s to match the in-control ARL of the Westgard Multirule Procedure. The multivariate approach chosen for application is the chart with alpha to also match the in-control ARL of the Westgard Multirule Procedure. The comparison of the three traditional methods and the chart is made for the typical clinical application of N (i.e., n 1, p from the multivariate perspective). However, different shifts in the control materials are considered in this research from those that generally appear in the literature. The comparisons are made for the usual instance when both concentrations of control material shift together (i.e., concentration I and concentration II both shift the same direction at the same time), denoted as N /. Additionally, comparisons are made for the case when a single concentration of control shifts while the other concentration of control remains on target (i.e., concentration I experiences a shift while concentration II stays centered or vice versa), denoted as N /1, and the case where the two control concentrations diverge (i.e., concentration I shifts upward while concentration II shifts downward), denoted as N /. Fig. graphically portrays the shifts investigated in this research. Although all these shifts are not typically evaluated in the literature, they may easily occur in practice and should be considered. In addition, the comparisons of the quality-control methods are made based on ARLs, where the ARL is simply the number of quality-control batches tested (on average) before the quality-control method will signal. In the case where no change has occurred in the measurement process, the ARL corresponds to the average number of quality-control batches until a false rejection. In the case where a true shift has occurred in the measurement process, the ARL corresponds to the average number of quality-control batches until a true detection. The reason for applying ARLs for comparison purposes as opposed to probabilities is that the Westgard Multirule Procedure involves information over multiple quality-control batches. Adding a run rule that requires four qualitycontrol batches (i.e., 4 1s ) for evaluation in no way improves the probability of detecting a true shift with the first sample evaluated after the shift occurs. It does, however, reduce the average number of quality-control batches required to detect the shift in the long run. Therefore, the ARL is used for comparison purposes in this research. To generate the ARL information, the calculations for 1.75s and 1.91s are straightforward probability calculations. The ARLs for the Westgard Multirule Procedure are determined using computer simulation. For the chart, ARLs are generated by integrating the non-central distribution (1). Additionally, a correlation structure must be specified to evaluate the chart, and the correlation between the concentrations is assumed to be 0.60 and held constant for all analysis in this research. Here, we consider only two concentrations of control material; however, the modeling can be extended to the instance of three concentrations of control with similar results. The results from the statistical modeling are shown in Figs Fig. 3 shows the ARLs for the case when two concentrations of control are both shifted together (N Fig.. Illustration of shifts evaluated. Fig. 3. Method comparison for N / mean shift.

4 196 Dechert and Case: Multivariate approach to quality control /). Note that both means are shifted in terms of SDs (i.e., both control materials experience a 1.0 SD shift in their means). Figs. 4 and 5 show the ARLs when the control concentrations are shifted independently (i.e., N /1 and N / ). Examination of the graphs yields some valuable insights. The first is that the Westgard Multirule Procedure is the most sensitive to changes in both concentrations of control materials in the same direction (N /). The other traditional approaches show performance close to the Westgard Multirule Procedure, with the chart being the least sensitive to this type of shift. However, the is more sensitive to the other two types of shifts considered in the research, N /1 and N /. In fact, the chart substantially outperforms the other approaches for the case of diverging control concentrations. Although the traditional methods have reduced sensitivity to a shift in a single concentration of control or two diverging control concentrations, the performance of the multivariate approach actually improves. This is because of the nature of the in-control ellipse for the chart. For Fig. 4. Method comparison for N /1 mean shift. a shift common to two concentrations of control, the direction of the shift moves along the axis of the in-control ellipse. However, for a shift in a single concentration of control, the direction of the shift moves away from the axis of the ellipse, making the approach more sensitive to these kinds of shifts. For two diverging concentrations of control material, the chart is even more sensitive to the change because of the direction of the mean vector shift vs the in-control ellipse. These results are also important from the standpoint of the perceived error protection for the traditional qualitycontrol approaches. When the ARLs are examined for the different types of mean shifts, it is clear that the error protection capabilities for the traditional methods decrease. For the multivariate case, the error protection for a shift in a single concentration or diverging concentrations actually improves. This means that a chart developed for the case of a shift in both materials will provide that minimum degree of error protection regardless of the type of shift encountered. This is not the case for traditional quality-control approaches. To further evaluate the performance of the chart, the sensitivities to changes in precision for all the qualitycontrol approaches are evaluated in Fig. 6. The chart shows the ARLs for detection of changes in the control concentration SDs by multiples of the SD. Only the case where both SDs are inflated is considered. From an examination of the chart, it is clear that all the methods considered have essentially the same sensitivities to changes in random error or precision. Therefore, none of the approaches has a particular advantage over the other methods in terms of detecting changes in random error. Conclusion The existence of correlation between control concentrations in quality-control data is very real. Although this correlation may not always be large and certainly varies from assay to assay, its presence can play a large role in the performance of a selected quality-control approach. To deal with correlated control concentrations in clinical quality control, a multivariate approach has been proposed. This approach utilizes the correlation structure of multiple concentrations of control materials to determine Fig. 5. Method comparison for N / mean shift. Fig. 6. Method comparison for changes in precision in both control concentrations.

5 Clinical Chemistry 44, No. 9, the appropriate in-control region for the monitoring application. There are a number of advantages to the use of the multivariate approach for monitoring clinical quality control. The first is that the approach provides the appropriate control region for the application. If the assumption of independence does not hold, then the assumed performance of traditional approaches can be misleading. The multivariate approach, however, can guarantee error protection for a variety of different types of shifts in the control materials. Another advantage of the chart is that it moves away from the application of runs rules. There is an inherent uneasiness with detecting shifts that occurred four or five quality-control batches ago and shutting down the system. What about all the patient samples in between that were subjected to the same shift in the measurement process? With the chart, each quality-control batch would be either acceptable or unacceptable on its own merit. There is still a risk that a shift could go undetected; however, that risk does not change from quality-control batch to quality-control batch, and that risk is considered explicitly in the design of the chart. Given the potential for correlation in clinical qualitycontrol data, it would be advisable for laboratories to check for correlation in their data. If the assumption of independence holds, then traditional methods should be continued. If substantial correlation is evident, then a multivariate approach, such as the chart, should be applied to the quality-control application. References 1. Levey S, Jennings ER. The use of control charts in the clinical laboratory. Am J Clin Pathol 1950;0: Westgard JO, Groth T, Aronsson T, de Verdier C.-H. Combined Shewhart-Cusum control chart for improved quality control in clinical chemistry. Clin Chem 1977;3: Parvin CA, Gronowski AM. Effect of analytical run length on quality-control (QC) performance and the QC planning process. Clin Chem 1997;43: Westgard JO, Barry PL, Hunt MR. A multi-rule Shewhart chart for quality control in clinical chemistry. Clin Chem 1981;7: Linnet K. Choosing quality-control systems to detect maximum clinically allowable analytical errors. Clin Chem 1989;35: Petersen PH, Fraser CG. Setting quality standards in clinical chemistry: can competing models based on analytical, biological, and clinical outcomes be harmonized? Clin Chem 1994;40: Jackson JE. Quality control methods for two related variables. Ind Qual Control 1956;1: Alt FB. Encyclopedia of statistical sciences. New York: John Wiley & Sons, 1985: Tracy ND, Young JC, Mason RL. Multivariate control charts for individual observations. J Qual Technol 199;4: Alt FB. Multivariate quality control: state of the art. Am Soc Qual Control Qual Congr Trans 198; Jackson JE. Multivariate quality control. Commun Statistics 1985; 14: Anderson TW. An introduction to multivariate statistical analysis. New York: John Wiley & Sons, 1958:7 7.