Pass-Fail Testing: Statistical Requirements and Interpretations

Transcription

1 Joural of Researh of the Natioal Istitute of Stadards ad Tehology [J. Res. Natl. Ist. Stad. Tehol. 4, (2009)] Pass-Fail Testig: Statistial Requiremets ad Iterpretatios Volume 4 Number 3 May-Jue 2009 David Gilliam, Stefa Leigh, Adrew Rukhi, ad William Strawderma Natioal Istitute of Stadards ad Tehology, Gaithersburg, MD david.gilliam@ist.gov stefa.leigh@ist.gov adrew.rukhi@ist.gov william.strawderma@ist.gov Performae stadards for detetor systems ofte ilude requiremets for probability of detetio ad probability of false alarm at a speified level of statistial ofidee. This paper reviews the aepted defiitios of ofidee level ad of ritial value. It desribes the testig requiremets for establishig either of these probabilities at a desired ofidee level. These requiremets are omputable i terms of futios that are readily available i statistial software pakages ad geeral spreadsheet appliatios. The statistial iterpretatios of the ritial values are disussed. A table is iluded for illustratio, ad a plot is preseted showig the miimum required umbers of pass-fail tests. The results give here are appliable to oe-sided testig of ay system with performae harateristis oformig to a biomial distributio. Key words: biomial distributio; ofidee bouds; ofidee oeffiiet; ritial value; probability of detetio; probability of false alarm. Aepted: April 27, 2009 Available olie: Itrodutio I evaluatig the effiay of equipmet that is meat for detetio of hidde otrabad or dagerous substaes, the istrumet is ofte subjeted to testig that measures its performae agaist requiremets set forth i protools set by atioal or iteratioal stadards orgaizatios. Performae requiremets i these stadards ilude those for probability of detetio (PD) ad probability of false alarm (PFA) at a speified level of statistial ofidee. The detetio systems osidered i this paper are all assumed to behave aordig to a biomial distributio. Oly two outomes are osidered for idepedet trials with otrabad preset: the detetio system either orretly reports detetio or does ot. Furthermore, the probability of detetio must remai ostat durig the period of the testig. Otherwise, it may be meaigless to perform biomial model based tests to determie estimates of this quatity. Similarly, for tests with otrabad abset, the detetio system either orretly reports o detetio, or it falsely reports the presee of otrabad: ad the probability of a false alarm is presumed to remai fixed throughout the period of testig. For a detetio system, PD or PFA a oly be determied aurately by a suffiiet umber of trials. However, there is a umber alled the ofidee level (CL) that gives some sese of adequay of the results from a series of trials of a give size. CL is defied i terms of the biomial probability mass futio, also alled the biomial disrete desity futio, b(m;,p), bm ( ;, p) Pr(BIN(, p) m)! m m p ( p), m! m! ( ) () 95

2 Joural of Researh of the Natioal Istitute of Stadards ad Tehology where m 0,,...,, deotes the umber of suessful detetios or false alarms) i idepedet trials with p PD, or p PFA, 0 p (see Johso, Kotz, ad Kemp, 992.) The umber of suesses i repeated idepedet trials oforms to this futio if eah trial a be sored as either suess or failure ad the probability for suess is fixed. I Se. 2 we disuss the defiitios of CL ad related ritial values i detetio problems. Setio 3 gives statistial iterpretatio of these values i terms of hypothesis testig ad ofidee bouds. The ote is oluded with Se. 4 otaiig some examples. 2. Defiitios ad Test Requiremets The quatity CL a be loosely iterpreted as the likelihood that ay suh system oformig to a biomial distributio with m suesses i a series of idepedet trials will have a true PD value greater or equal to a hose value, PD. More formally, the aepted defiitio of CL i settig testig requiremets is stated i terms of the equatio below. The usage of this term is osoat with that of ASTM stadard C (2005). For a umber m of suesses foud i a series of pass-fail trials, with a fixed value of PD, desigated PD, the ofidee level CL(m,, PD ) is defied by the equatio CL( m,, PD) b( j;, PD). j 0 I other words, if for x 0,,...,, 0 p, BINCDF( xp,, ) Pr(BIN(, p) x) x ( ) k p ( p) k 0 k (2) (3) deotes the biomial umulative distributio futio, the (2) a be expressed as CL( m,, PD ) BINCDF( m,, PD ). (4) Note that uder this defiitio CL (m,, PD ) aot exeed PD. To fid the ritial value m, i.e., the miimum value of m establishig the PD of iterest with a preseleted, fixed level of ofidee, CL, oe must ivert the iequality, BINCDF( m,, PD ) CL. k (5) It follows that m is well defied oly if BINCDF (,, PD ) CL, i.e., if PD CL. (6) Sie BINCDF(x,, p) is a step-futio i x (i.e., is ot stritly ireasig), it does ot have a proper iverse futio. If we set m, m to be the least iteger suh that BINCDF(m,, PD ) exeeds CL, the m INVBINCDF ( CL,, PD ) +, (7) where INVBINCDF(CL,, p) is the iverse umulative biomial distributio futio (i.e., is the smallest oegative iteger suh that the umulative distributio futio evaluated at this value equals or exeeds CL.) Versios of this futio are available i may statistial software pakages, iludig MATLAB (bioiv), R(qbiom), NAG, GAMS, IMSL, S-PLUS, ad SAS ad i geeral spreadsheet appliatios, suh as EXCEL (futio CRITBINOM(, p, CL).) l The biomial umulative distributio futio a be expressed through the iomplete beta-futio, BINCDF( m,, p) I ( m, m+ ) m x ( x) dx p m x ( x) dx 0 (8) m >0, m + > 0, (Abramowitz ad Stegu, 972), so that for fixed m ad, BINCDF(m,, p) is a dereasig futio of p, 0 p. This formula allows oe to defie BINCDF(m,, p) for ay real (oiteger) values m ad suh that 0 < m < +. A aalogous defiitio of CL applies to testig for PFA i systems where o otrabad or dagerous substae is preset. For ay hose value of PFA, desigated PFA, the ofidee level CL (m,, PFA ), equals the probability that the umber of false alarms ourrig i a series of idepedet biary trials exeeds m. Thus, this level is defied by the equatio CL CL( m,, PFA ) b( k;, PFA ) (9) Ay metio of speifi ommerially available statistial software pakages or geeral spreadsheet appliatios does ot imply edorsemet of preferee for these produts by the NIST., p k m+ BINCDF( m,, PFA ). 96

3 Joural of Researh of the Natioal Istitute of Stadards ad Tehology Similarly to the PD ase, (9) To fid the maximum value M of M, M 0,,...,, establishig the PFA of iterest with a preseleted, fixed level of ofidee CL, oe must ivert the iequality BINCDF( M,, PFA ) CL. () To express M through the futio INVBINCDF (,, p), i.e., to establish the largest value m satisfyig (), the formula, INVBINCDF( p,, ) (2) max{ x: BINCDF( x,, p) }, a be employed. To prove (2), otie that for x 0,...,, BINCDF( x,, p) (3) BINCDF( x,, p), so that INVBINCDF(,, p) mi{ x:bincdf( x,, p) } mi{ x:bincdf( x,, p) } max{ x: BINCDF( x,, p) }. (4) Therefore, M INVBINCDF(,, PFA ), CL ( PFA ). so that M ad M is ot defied whe INVBINCDF ( CL,, PFA ), (5) otaiig the ukow p with a give probability alled ofidee oeffiiet (see Hah ad Meeker, 99). Assume that for the give CL, a lower ofidee boud for PD p of ofidee oeffiiet CL is desired: that is for a biomial observatio X BIN (, p), oe requires a futio p p (X,, CL) suh that (7) Pr( p( X,, CL) p) CL. The well kow solutio of this problem for X, is px (,, CL) max{ p:bincdf( X,, p) CL}. (8) (e.g, Casella ad Berger, 2002.) Whe X 0, p (0,, CL) 0. Thus with m defied by (7), the iequalities p < p (strit iequality) ad X m (o-strit iequality) are equivalet. Therefore, the ritial value m has the iterpretatio of the largest value of the biomial BIN(, p) variable suh that the lower ofidee boud for p does ot exeed PD. A related iterpretatio is provided by the statistial hypothesis testig problem, H 0 : p PD uder the alterative: H : p < PD. The most powerful test of level CL rejets H 0 whe the observed value X exeeds the ritial value m, X > m (whih meas the same as p (X,, CL) PD ). The ritial value for PFA has a similar statistial iterpretatio, amely, M is the largest value of the biomial variable for whih the upper ofidee boud for the biomial probability does ot exeed PFA. Ideed, a upper ofidee boud of ofidee oeffiiet CL has the form, i.e., whe ( PFA ) > CL. Thus (5) ad (7) show that uder the same value of CL, whe PD PFA, a simple formula, m (6) + M, relates m ad M. 3. Hypothesis Testig ad Cofidee Bouds o Biomial Probability p( XCL,, ) p ( XCL,, ). Idetity (3) shows that px (,, CL) mi{ p:bincdf( X,, p) CL}. Thus, p( M,, CL) PFA, but p ( M +,, CL) > PFA. (9) (20) We give here two statistial iterpretatios of Eq. (7) ad Eq. (5). The first of these is related to a (lower) ofidee limit for biomial probability p. Suh limits are supposed to provide a data-depedet iterval I terms of the hypothesis testig with H 0 : p PFA ad the alterative: H : p > PFA, the most powerful test of level -CL rejets H 0 whe the observed value X exeeds the ritial value M, X > M. 97

4 Joural of Researh of the Natioal Istitute of Stadards ad Tehology 4. Examples Cosider a example i whih oe fids twety-ie orret results i a sigle set of thirty trials. If the system uder test oforms to a biomial distributio, the based o the result of twety-ie out of thirty orret resposes i that oe set of tests, oe a make multiple orret iferees, suh as: the PD > 0.95 with 44 %, ofidee, the PD > 0.90 with 8 %, ofidee, or the PD > 0.85 with 95 % ofidee. Oe a easily ostrut a table whih simultaeously iludes requiremets for both PD ad PFA. Table gives the ritial value M ad m for 68 % ofidee to show the geeral harateristis of these quatities. These are the maximum permissible umbers of iorret results that may be tolerated i establishig the speified PD or PFA values at this level of ofidee. If the tabulated value is idiated as *, the the umber of trials i that set is isuffiiet to establish the orrespodig PD or PFA at this ofidee level. Oe may geerate tables of this kid for ay CL, PD, ad PFA usig Eq. (7) ad Eq. (5) by usig the previously metioed futios like bioiv or CRITBINOM from statistial software pakages or spreadsheet appliatios. The atual value of M ad m give by these futios i the ases marked by * is. The symmetry of testig requiremets whe PFA PD permits tabulatig the results for PFA ad PD i a sigle table, but it does ot imply that PFA should or must always be hose equal to PD. The PD ad PFA values may be assiged idepedetly i ay testig protool. I fat, to avoid disruptio of the stream of ommere by large umbers of false alarms, it is ofte eessary to require ispetio equipmet to have PFA smaller tha PD. By solvig (6) or (0), we obtai a formula for the miimum umber of required trials k eeded to establish a give value of PD or PFA for the same CL, with k a, log( CL) log( CL) a. log PD log( PFA) (2) (22) Here a deotes the smallest iteger exeedig a. This formula is useful i desigig test protools that give the most satisfatory requiremet with the least amout of testig. Figure shows a plotted as a futio of PD ad CL. This futio ireases muh more rapidly for PD approahig tha for CL. Table. Maximum permissible umbers of iorret results for verifyig a lower boud o PD or a upper boud o PFA with 68 % ofidee PD PFA * * * * * * * 0 3 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * Similarly k i (2) would irease muh more rapidly for PFA 0 tha for CL. Whe oly the miimum umber of trials k is performed, the system must give 00 % orret results to establish the speified PD or PFA at, the desired ofidee CL. I statistial terms, k is the smallest umber of trials with 00 % orret detetios suh that the CL-lower ofidee boud for detetio probability exeeds the give value PD. The same is true whe there are o false alarms with the CL-upper ofidee boud o the false alarm probability beig less tha PFA. A table suh as Table will show how may errors may be permitted if a larger umber of trials are arried out, while still establishig the speified PD or PFA at the desired CL. 98

5 Joural of Researh of the Natioal Istitute of Stadards ad Tehology Fig.. The miimum required umber of tests to establish a give value of PD (or -PFA) for a give CL. 5. Disussio ad Colusios The formula for k shows that requirig either PD or CL to be too ear uity a result i impossibly large umbers of pass-fail tests. If suh rigorous riteria are i fat required the oe should searh for some method of verifiatio differet from pass-fail testig. The results preseted here make it possible to desig pass-fail testig protools based o futios readily available i statistial software pakages ad geeral spreadsheet appliatios. About the authors: David Gilliam is a ulear egieer/physiist i the Neutro Iteratios ad Dosimetry Group, Ioizig Radiatio Divisio, Physis Laboratory. Stefa Leigh ad Adrew Rukhi are mathematial statistiias i the Statistial Egieerig Divisio, Iformatio Tehology Laboratory. Bill Strawderma a professor i the Departmet of Statistis at Rutgers Uiversity. He is also a Faulty Appoitee at NIST. The Natioal Istitute of Stadards ad Tehology is a agey of the U.S. Departmet of Commere. 6. Referees [] M. Abramowitz ad I. Stegu, Hadbook of Mathematial Futios, Dover, New York (972) p 263. [2] ASTM Iteratioal, Stadard Guide for I-Plat Performae Evaluatio of Automati Vehile SNM Moitors: C , W. Coshohoke, PA (2005) pp -4. [3] G. Casella ad R. Berger, Statistial Iferee, 2d editio, Duxbury, Paifi Grove (2002) pp [4] G. J. Hah ad W. Q. Meeker, Statistial Itervals: A Guide for Pratitioers, Wiley, New York (99) p 25. [5] N. Johso, S. Kot, ad A. Kemp, Uivariate Disrete Distributios, New York: Joh Wiley (992) pp