THE MERICN JOURNL OP CLINICL PTHOLOGY Cpyright 968 by The Williams & Wilkins C. l. 5, N. 3 Printed in U.S.. QULITY CONTROL BY THE REFERENCE SMPLE METHOD ERROR DETECTION S FUNCTION OF THE RIBILITY OF THE CONTROL DT ELIS MDOR, M.D. Institute f Pathlgy, Case Western Reserve University, and Divisin f Clinical Pathlgy, University Hspitals f Cleveland, Cleveland, Ohi 446 The reference sample methd f Levey and Jennings is widely endrsed t cntrl the daily peratin f clinical labratries. 4 The methd emplys a reference sample, such as serum, which is analyzed nce a day tgether with the clinical specimens. 4 The cntrl limits (2 S.D. abve and belw mean) are established after 2 daily reference samples have been analyzed. ny value which thereafter falls utside these limits is cnsidered t be in errr. Despite the widespread use f the reference sample methd, there is scant dcumentatin f its capacity t disclse errrs. Thus, an Peenen and Lindberg 7 fund that the methd failed t disclse significant analytical errrs in flame phtmetry; they suggested that errrs may be masked by peratr bias. But Weinberg and Barnett, 8 as well as Sax and clleagues, 5 have failed t find such bias. The cefficient f variatin, that is, the standard deviatin expressed as a per cent f the mean, has been prpsed by Sparapani and Berry 6 as a useful cmmn denminatr t relate the day t day variability f different reference samples, as well as f different methds, ne t anther. Thrugh the use f the cefficient f variatin, the authr has bserved that significant analytical errrs are readily detectable when the cefficient f variatin is small and the cntrl limits narrw but may be hidden when the cefficient f variatin is large. MTERILS ND METHODS Quality cntrl charts were set up accrding t Levey and Jennings fr bld sugar and serum glutamic-xalacetic transaminase (SGOT) activity. Bld sugar cncentratins were measured by the ptassium ferricyanide methd f Hffman 9 adapted t the utnalyzer (Technicn Instruments Cr- Received Octber 9, 967. 36 pratin, rdsley, N. Y.), and SGOT activities were measured by the spectrphtmetric methd f madr and assciates. liquts f pled serum were stred at 2 C. and thawed n the day f use, as recmmended by Levey and Jennings. The data f 4 cnsecutive days were emplyed in this study (but nly the data fr the first 2 days are shwn t simplify the charts). In the nrmal curve 3.6% f a large sample f numbers will be fund in the area between and 2 S.D. frm the mean. Therefre, if a systematic errr equal t S.D. is added t these numbers, 3.6 % f the results will exceed 2 S.D. If an errr equal t.5 S.D. is added, then 28.6% f the results will be greater than 2 S.D. In practice, the distributin f reference sample values apprximates the nrmal distributin, but the number f reference samples being dealt with at any given time is quite small. It is nt feasible t predict accurately the number f results which will be utside the cntrl limits when a systematic errr ccurs frm day t day; this restrictin is even mre applicable when randm errrs are being sught. Fr these reasns, a simulatin study based n systematic and randm errrs added t real life reference sample values was cnsidered t be necessary in rder t define the sensitivity t errrs f the reference sample methd. Mdels based n synthetic data were als used t set up quality cntrl charts. The use f synthetic data permits a greater flexibility with respect t the chice f values fr the parameter and eliminates any pssible ambiguity assciated with the ccurrence r detectin f errr. 2 ls, the use f synthetic data minimizes the effects f bserver errr. The synthetic reference samples were based n grups f sequential nrmal deviates f zer mean and unit standard deviatin. Therefre the synthetic reference sample Dwnladed frm https://academic.up.cm/ajcp/article-abstract/5/3/36/76942 n 3 March 28
Sept. 968 QULITY CONTROL 36 appear t be drawn frm a nrmal ppulatin with a unifrm cefficient f variatin. Because the data are nrmal, they are symmetrically distributed abut the mean, and therefre cntrl limits f ±2 S.D. are applicable. 7 ' 4 Synthetic reference sample values (SRS) with a mean f and a standard deviatin f were generated by the frmula: SRSi = + ND,- () Nrmal deviates (ND) were taken sequentially frm tables f randm nrmal numbers. 5 The systematic prprtinal errrs f variable per cent which were added t r subtracted frm the real life synthetic reference sample values were generated by the frmula: SPEi = RS,- %E/ (2) where SPE stands fr systematic prprtinal errr, RS,- fr each reference sample, and %E fr percentage f errr.* The situatin where %E is expressed as a multiple f the cefficient f variatin was als examined: %E = / C.. where C.. stands fr cefficient f variatin and the factr, /, has the values f.5,,.5, r 2. T each f 2 daily measured values btained by assay f the reference samples fr bld glucse and SCOT were added randm nrmal numbers with a S.D. equal t.5,,.5, and 2 times that f the actual measured values: RS/ = RS ; + ND,- S.D. / (3) where RS/ stands fr the mdified reference sample value and S.D. stands fr the standard deviatin f the riginal reference sample values. * Systematic errrs may be cnstant, r they may be prprtinal t the magnitude f the measured value. 3 The difference between systematic cnstant and prprtinal errrs is quite.small in the case f a set f reference sample values btained n aliquts f ne cntrl serum, and hence this difference was nt cnsidered here. Systematic unci randm errrs are defined in the "Discussin." Finally, randm errrs were als superimpsed n the synthetic reference samples with the frmula: SRS/ = SRS,- + ND,- S.D. / (4) ll calculatins were perfrmed with an electrnic digital cmputer. RESULTS Systematic Errrs. Bld glucse and SGOT cntrl charts. Systematic errrs were added t the cntrl reference samples t simulate the effect f departures frm accuracy. The reference samples emplyed t cntrl the determinatin f bld glucse cncentratin had a mean and standard deviatin f 4.4 ±.S7 mg. per ml., with a cefficient f variatin f.64%. When a systematic errr f +2% was added t the cntrl reference sample values, the resulting values shwed a trend in that mst f the values were abve the mean and tw fell utside the cntrl limits (Fig., middle chart). When a systematic errr f +3% was added t the cntrl reference sample values, eight f the 2 values were abve the upper cntrl limit; all but ne f the values were larger than the mean (Fig., bttm chart). Similar findings were bserved when negative systematic errrs were added t the bld glucse cntrls. The reference samples analyzed as daily cntrls fr the determinatin f SGOT activity had a mean and standard deviatin f 2.2 ±.94 units per ml., with a cefficient f variatin f 4.7%. When a systematic errr f 5% (abut C..) was added t the cntrl values, mst f the resulting values were abve the mean, reflecting the psitive systematic errr, and three f the 2 values were utside the upper cntrl limit (Fig. 2, middle chart). When an errr f % (abut 2 C..) was added t the cntrl values, 4 f the 2 values were utside the upper cntrl limit and the remainder f the values were abve the mean (Fig. 2, bttm chart). 2. Cntrl charts based n synthetic data. s illustrated by the abve results, the size f the errrs detected by the reference sample methd is a functin f the cefficient f variatin. In rder t illustrate this further Dwnladed frm https://academic.up.cm/ajcp/article-abstract/5/3/36/76942 n 3 March 28
362 MDOR l. 5 in the absence f technical artifacts, synthetic data with a mean f and a cefficient f variatin f, 2, 4, 8, and 2% were emplyed. s the results were cmparable fr all f these grups f synthetic data, nly ne grup is described in detail. In the cntrl grup, three f the 4 synthetic values were utside the limits f ±2 S.D. frm the mean. When systematic errrs f +.5 C.. were intrduced, the errrs were nt readily apparent by visual inspectin (Fig. 3, secnd chart). Errrs equal t + C.. appeared as a clear trend, as mst f the values were greater than the mean and nine f the 4 values fell utside the upper cntrl limit (Fig. 3, middle chart). Errrs f +.5 and +2 C.. were even mre readily apparent, as mre than ne-third f the values fell utside the upper cntrl limit and almst all f the remaining values were abve the mean (Fig. 3, lwer tw charts). Similar results were btained with the intrductin f systematic negative errrs (Fig. 4). Randm Errrs. Bld glucse and SGOT cntrl charts. Randm psitive and negative errrs were added t the cntrl reference samples in rder t simulate the effect f departures frm precisin. When randm errrrs f ±.5 and ± 8 4 ~~b ""b <5"~ DYS 5 2 ERROR O 8 LLI c M4-H" => i i H I _i CD 8 4 n DYS 5 2 DYS 5 2 + 2% + 3% COEFFICIENT OF RITION =.6% FIG.. Bld glucse quality cntrl chart. The day t day variability (mean ± S.D.) fr 2 cnsecutive daily reference serum samples was 4.4 ±.87 mg. per ml. The cntrl limits are the mean ± 2 S.D. (tp chart). Systematic psitive errrs f 2% (~ C..) added t each reference sample are apparent as a trend (middle chart). Systematic errrs f 3% (~2 C..) are readily apparent (bttm chart). Dwnladed frm https://academic.up.cm/ajcp/article-abstract/5/3/36/76942 n 3 March 28
Sept. 968 QULITY CONTROL 363 C.. were added t the bld glucse and SGOT reference samples, the errrs were nt detectable by visual inspectin (Tables and 2). Hwever, randm errrs f ±.5 and ±2 C.. were readily detected by the reference sample methd (Tables and 2). 2. Synthetic data cntrl charts. The synthetic data were emplyed t test the sensitivity t errrs f the reference sample methd in the absence f technical variables. gain, errrs f ±.5 and ± C.. were nt visually apparent in the cntrl chart (Fig. 5, secnd chart frm the tp), but randm errrs equivalent t ±.5 and ±2 C.. were increasingly bvius upn inspectin f the cntrl charts (Fig. 5, lwer tw cntrl charts). DISCUSSION The results f the present study illustrate that the sensitivity t errrs f the reference sample methd is a functin f the size f the C.. Thus, when the day t day C.. f the cntrl days is small, subsequent day t days errrs are readily apparent. Cnversely, when the C.. is large, significant errrs may be masked. practical rule t describe this finding seems t be that errrs.5 t 2 times larger than the C.. will be readily shwn by the reference sample methd. Hwever, fr the ne value btained each day n a reference sample, the Levey-Jennings methd inherently cannt differentiate between systematic and ran- 22. 2. 82 E v. in "c. >- > 22. O < I- cr> 2. 8.2 -» -t «- DYS 5 2 -i DYS 5 2 ERROR - + 5% 3 rr CO 22. 2. 8.2 DYS 5 2 + % COEFFICIENT OF RITION = 4.7% FIG. 2. SGOT cntrl chart. The day t day variability (mean ± S.D.) was 2. ±.94 unit/ml. fr 2 daily reference serum samples (tp chart). Systematic psitive errrs f 5% (~ C..) added t each reference sample are apparent as a trend (middle chart). Systematic errrs f % (~2 C..) are readily apparent (bttm chart). Dwnladed frm https://academic.up.cm/ajcp/article-abstract/5/3/36/76942 n 3 March 28
364 MDOR l. 5 dm errrs; that is, it cannt separate lack f accuracy frm lack f precisin in each day's results. T clarify the limitatins f the reference sample methd, it appears pertinent t discuss the peratinal definitins f accuracy and precisin. ccuracy has been defined as the extent t which a measured value agrees with the assumed r accepted value. In this cntext, systematic errrs may be cnsidered t be deviatins frm the true r assumed value with a bias in the same directin fr a given set f determinatins. Precisin has been defined as the extent +2CT -2 + 2a *..... " -P -r> 2 2 rnnr LnnUn U --- -O-, 3 4. 3 4 + U.O L... + 2-2 a; * * " ' a" 2 «-" 4- P \ / + U 3. 4 + 2 '.. B- - > «_ "- * - ' "" i R r\/.. + i. u. v. -2CT 2 3 ' 4 + 2a * * w 2 T» T * * 4. O P \/ + U 3 4 FIG. 3. Cntrl chart fr synthetic data based n sequential nrmal deviates (tp chart). Systematic errrs f +.5, +, +.5, and +2 C.. were added t the cntrl data. Errrs greater than C.. are disclsed by the three lwer charts. Dwnladed frm https://academic.up.cm/ajcp/article-abstract/5/3/36/76942 n 3 March 28
Sept. 968 QULITY CONTROL 365 t which a set f determinatins deviate measure f the deviatin f test results frm frm their wn mean as frequently measured their wn mean, all determinatins being by the standard deviatin. Precisin is de- perfrmed by ne peratr and withut termmed by the repeatability r the repr- change f appai'atus when apparatus can be ducibility f test results. Repeatability is a significant. Reprducibility measures the 9 rt C U rn>n>nn LKKUI\ _ " - ^ ^ U n i i 9 i 2 3 4 + d. -.- -«---.- ^ -. -.-" - -"- - - -" - -. - -.5 C.. 2 ' * 4 T C. - c. " 2 *. 4 T -2 --"--- -. -.5 c. _. i " i. it. -2 ' "" 4 + 2 9rt- T ' * " T "2'».,3'»* FIG. 4. Cntrl chart fr synthetic data (tp chart). Systematic errrs f.5,,.5, and 2 C.. were added t the cntrl data. Errrs larger than.5 C.. are clearly apparent (tw lwer charts). Dwnladed frm https://academic.up.cm/ajcp/article-abstract/5/3/36/76942 n 3 March 28
366 MDOR l. 5 TBLE SENSITIITY OF THE REFERENCE SMPLE METHOD TO RNDOM ERRORS DDED TO 2 BLOOD GLUCOSE bve mean + 2 S.D. Belw mean 2 S.D. Ttal REFERENCE SMPLES Cntrl N. f Reference Samples at Randm Errr f ±.5 2 3 ±i 2 3 ±.5 4 5 ±2 C..* * Cefficient f variatin, standard deviatin /mean. TBLE 2 SENSITIITY OF THE REFERENCE SMPLE METHOD TO RNDOM ERRORS DDED TO 2 SGOT REFER ENCE SMPLES bve mean + 2 S.D. Belw mean 2 S.D. Ttal Cntrl N. f Reference Samples at Randm Errr f ±.5 ± ±.5 4 5 ±2 c.v. 2 deviatin f test results btained in different labratries. Randm errrs therefre may be cnsidered t be deviatins frm the true r assumed mean value which fluctuate unpredictably fr a given set f determinatins. Significant randm errrs usually indicate a lack f adequate cntrl ver repetitive technical manipulatins.! Therefre, it is legitimate t ask what the reference sample methd des d. It has been suggested that the methd measures the day t day variability. 4 Day t day variability des nt prvide a reliable measurement f precisin, as pinted ut by f The abve distinctin between the tw main grups f analytical errrs is in accrd with the peratinal definitins prpsed by Benedetti- Pichler, 3 later refined by Linnig and clleagues. 3 detailed analysis f the cncepts f accuracy and precisin has been made by Pwer 4 as well as by Eisenhart. 6 2 4 Linnig and c-wrkers. 3 There are tw reasns fr this. First, day t day variability cmprises multiple uncntrlled variables such as several analysts, the use f several instruments, and variable reagents and analytical glassware. Secnd, in the currently endrsed versin f the reference sample methd based n nly ne reference value fr each day, it is nt pssible t calculate a mean and standard deviatin, nr t state cnfidence limits fr this single value; hence, neither accuracy nr precisin can be determined fr each batch f assays. Despite the abve limitatins, it shuld be recgnized that at present the reference sample methd is the methd f quality cntrl mst easily implemented in the clinical labratry. Barnett and Pint 2 have shwn that the reference sample methd is mre sensitive t errrs than is the methd f "mixed patient samples," and we have fund that it is at least twice as sensitive as the "average f nrmals" methd. The main advantage f the reference sample methd is that it emplys reference samples with a matrix which clsely resembles that f the test specimens. The cntent f the substance being measured can be readily ascertained by each labratry using several mutually cmplementary assay methds, as well as by reference labratries. Finally, the reference sample methd can be readily expanded t include multiple reference samples 5 as well as primary standards. It can als be expanded t prvide data fr analysis f variance. 8 In cnclusin the reference sample methd may have a paradxical ability t detect errrs. Thus, when the assay methd is precise the initial cntrl samples will have a small cefficient f variatin and bth systematic and randm errrs will be detectable. Cnversely, when the assay methd is quite variable the cntrl samples will have a large cefficient f variatin and subsequent significant errrs may remain undetected; the analyst may therefre receive a false sense f security. It appears reasnable t cnclude that mre sensitive methds f quality cntrl which can disclse errrs at the mment that they ccur, and which can distinguish between systematic and randm errrs, are needed t maintain an acceptable standard f labratry perfrmance. Dwnladed frm https://academic.up.cm/ajcp/article-abstract/5/3/36/76942 n 3 March 28
Sept. 968 QULITY CONTROL 367 +2 5 ERROR. _ *. _ ^ ^_ ^ _ _ ^ «H ^ -,? 2,. 2 3 +2a. '.. *,.. _ > - - - ±.5 c... *... r-i i H + 2 3 -^ " i i - + 2a + 2a -2 2» " - - - - - -* - D - - - - - - - - - - - - - - ±.5 C. D a a a D n - D I 2 3 a a JL ^. -_' -_^_»j - ',_ ±2 c.» i -. 2 3 FIG. 5. Cntrl chart fr synthetic data based n sequential nrmal deviates (tp chart). Randm errrs equal t ±.5, ±, ±.5, and ±2 C.. were added t the cntrl data. Errrs greater than ±.5 C.. are readily apparent (tw lwer charts). SUMMRY cmmunity hspital labratry, as well as The present study was designed t test synthetic data, were emplyed fr this purthe sensitivity t errrs f the reference P s e- Cntrl limits were set as the mean sample methd f quality cntrl. Data b- ± 2 S.D. Simulated errrs were then imtained during the day t day peratin f a psed n these data. Dwnladed frm https://academic.up.cm/ajcp/article-abstract/5/3/36/76942 n 3 March 28
368 MDOR l. 5 The results demnstrate that the sensitivity t errrs is a functin f the size f the cefficient f variatin. Errrs had t be.5 times larger than the cefficient f variatin befre they became bvius. The results suggest that the reference sample methd may have a paradxical ability t detect errrs. Thus, when assay methd is precise, the initial cntrl samples will have a small cefficient f variatin, and subsequent systematic and randm errrs will be detectable. Cnversely, when the assay methd is quite variable, the cntrl samples will have a large cefficient f variatin, subsequent large errrs may escape ntice, and the analyst may therefre receive a false sense f security. REFERENCES. madr, E., Massed, M. F., and Franey, R. F.: Reliability f glutamie-xalacetic transaminase methds. m. J. Clin. Path., 47: 49-428, 967. 2. Barnett, R. N., and Pint, C. L.: Evaluatin f a system fr precisin cntrl in the clinical labratry. m. J. Clin. Path., 48: 243-247, 967. 3. Benedetti-Pichler,..: The applicatin f statistics t quantitative analysis. Indust. & Eng. Chem., nal. Ed., 8: 373-377, 936. 4. Cpeland, B. E. (Ed.): Quality Cntrl Manual. Chicag: The merican Sciety f Clinical Pathlgists, 96. 5. Dixn, W. J., and Massey, F. J., Jr.: Intrductin t Statistical nalysis, Ed. 2. New Yrk: McGraw-Hill Bk Cmpany, 957. 6. Eisenhart, C: Reliability f measured values. Part I. Fundamental cncepts. Phtgram. Eng., 8: 542-554, 952. 7. Fisher, R..: Statistical Methds fr Research Wrkers, Ed.. Edinburgh: Oliver and Byd, Ltd., 948, p. 44. 8. Gszen, J.. H.: The use f cntrl charts in the clinical labratry. Clin. Chini. cta, 5: 43-438, 96. 9. Hffman, W. S.: rapid phtelectric methd fr the determinatin f glucse in bld and urine. J. Bil. Chem., 2: 5-55, 937.. Hughes, H. K.: Suggested nmenclature in applied spectrscpy. nal. Chem., 34: 349-354, 952.. Levey, S., and Jennings, E. R.: Use f cntrl charts in the clinical labratry. m. J. Clin. Path., 2: 59-66, 95. 2. Lingane, P. J.: Critique f chrnptentimetry as a tl fr study f adsrptin. nal. Chem., 39: 485-494, 967. 3. Linnig, F. J., Mandel, J., and Petersn, J. M.: plan fr studying the accuracy and precisin f an analytical prcedure. nal. Chem., 26: 2-, 954. 4. Pwer, F. W.: ccuracy and precisin f micranalytical determinatin f carbn and hydrgen. statistical study. Indust. & Eng. Chem., nal. Ed., : 66-673, 939. 5. Sax, S. M., Drman, L., Libensn, D. D., and Mre, J. J.: Design and peratin f an expanded system f quality cntrl. Clin. Chem., IS: 825-833, 967. 6. Sparapani,., and Berry, R. E.: n evaluatin f standard deviatins in clinical chemistry. m. J. Clin. Path., 43: 59-593, 965. 7. an Peenen, H. J., and Lindberg, D.. B.: The limitatins f labratry quality cntrl with reference t the "number plus" methd. m. J. Clin. Path., 44: 322-33, 965. 8. Weinberg, M. S., and Barnett, R. N.: bsence f analytic bias in a quality cntrl prgram. m. J. Clin. Path, 38: 468-472. 962. Dwnladed frm https://academic.up.cm/ajcp/article-abstract/5/3/36/76942 n 3 March 28