Estimating The Number of Residual Defects

Size: px

Start display at page:

Download "Estimating The Number of Residual Defects"

Hillary Robertson
6 years ago
Views:

1 Etimating Th Numr o Ridual Dt Yahwant K. Malaiya Jaon Dnton Computr Sin Dpt. Colorado Stat Univrity Fort Collin, CO 8523 malaiya j dnton@.olotat.du Atrat Ridual dt i on o th mot important ator that allow on to did i a pi o otwar i rady to rlad. In thory, on an ind all th dt and ount thm, howvr it i impoil to ind all th dt within a raonal amount o tim. Etimating dt dnity an om diiult or high rliaility otwar, in rmaining dt an xtrmly hard to tt or. On poil way i to apply th xponntial SRGM and thu timat th total numr o dt prnt at th ginning o tting. Hr w how th prolm with thi approah and prnt a nw approah ad on otwar tt ovrag. Sotwar tt ovrag dirtly maur th thoroughn o tting avoiding th prolm o variation o tt tivn. W apply thi modl to atual tt data to projt th ridual numr o dt. Th rult how that thi mthod rult in timat that ar mor tal than th xiting mthod. Thi mthod i air to undrtand and th onvrgn to th timat an viually orvd. 1. Introdution Th timatd numr o rmaining dt in a pi o od i otn ud a an aptan ritrion. Som rnt puliation uggt that lading dg otwar dvlopmnt organization typially ahiv a dt dnity o aout 2. dt/kloc [1]. Som organization now targt vn lowr dt dniti. Th NASA Spa Shuttl Avioni otwar with an timatd dt dnity o.1 dt /KLOC i rgardd to an xampl o what an urrntly ahivd y th t mthod [5]. A low dt dnity an quit xpniv to ahiv, th Spa Shuttl od ha n rportd to hav ot aout $1, pr lin o od. Th ot o ixing a dt latr an vral ordr o magnitud highr than during dvlopmnt, yt a program mut hippd y om dadlin ditatd y markt onidration. Thi mak th timation o dt dnity a vry important hallng. On onival way o knowing th xat dt dnity o a program i to atually ind all rmaining dt. Thi i oviouly inail or any ommrial produt. Evn i rour ar availal, it will tak a prohiitiv amount o tim to ind all ug in a larg program [2]. Sampling ad mthod hav n uggtd or timating th numr o rmaining dt. M- Connll [14] ha givn a mthod that involv uing two indpndnt tting ativiti, prhap y two dirnt tam o ttr. Thi and othr ampling thniqu aum that ault ound hav th am ttaility a ault not ound. Howvr, in atual prati, th ault not ound rprnt ault that ar hardr to ind [13]. Thu uh thniqu ar likly to yild an timat o ault that ar rlativly air to ind and thu l than th tru numr. Fault ding mthod [14] ur rom imilar prolm. It i poil to timat th dt dnity ad on pat xprin uing mpirial modl lik th Rom La modl [7] or th modl propod y Malaiya and Dnton [9]. Th timat otaind y uh modl an vry uul or initial planning, howvr th modl ar not xptd to aurat nough to ompar with mthod involving atual tt data. Anothr poil way to timat th numr o ault i y uing th xponntial SRGM. It aum that th ailur intnity i givn y (t) = E E 1,E 1 t (1) It an hown that th paramtr E and 1 E dpnd on ytm and tt pro haratriti. Spiially, E rprnt th total numr o dt that would vntually ound. W an timat th numr o rmaining dt y utrating th numr o dt ound rom th valu o E otaind y itting th modl to th data. W applid thi thniqu to vral data t, igur 1 how rult typial o thi thniqu. Svral thing ar

2 worth noting aout thi plot. Firt, tting ontinud to rval dt vn atr thi thniqu prditd that thr whr no rmaining dt. Sond, th valu o E nvr tailiz. Toward th nd o tting E tak on a valu vry lo to th numr o dt alrady ound, and thu provid no uul inormation. An SRGM rlat th numr o dt ound to tting tim pnt. In atual prati, th dt inding rat will dpnd on tt tivn and an vary dpnding on th tt input ltion tratgy. A otwar tt ovrag maur (lik lok, ranh, and P-u ovrag t.) dirtly maur th xtnt to whih th otwar undr tt ha n xrid. Thu w xpt that a uitaly hon tt ovrag maur will orrlat ttr with th numr o dt nountrd. Th rlationhip twn tt ovrag and th numr o dt ound ha n invtigatd y Piwowarki, Oha and Caruo [16], Huthin, Goradia and Otrand [6], Malaiya t al [11], Lyu, Horgan and London [8] and Chn, Lyu and Wong [3]. In th nxt two tion, a modl or dt dnity in trm o tt ovrag i introdud and it appliaility i dmontratd uing tt data. Stion 4 prnt a twoparamtr approximation o th modl. Finally w prnt om orvation on thi nw approah. 2. A Covrag ad approah Rntly a modl wa prntd y Malaiya t al that rlat th dnity o ridual dt with tt ovrag maur [11]. Thi modl i ad on th Logarithmi Poion SRGM whih ha n ound to hav uprior prditiv apailiti [1]. W aum that th Logarithmi modl i applial to th total numr o dt ound. A jutiiation or why atual tting otn rult in thi modl an prntd y onidring th ditriution o ttaility o th dt and th non-randomn o tting approah [12]. Tt tivn an vary dpnding on th tt ltd. Th t o thi variaility an rmovd y mauring not th tting tim ut th atual tt ovrag ahivd. Uing th Logarithmi Poion modl w an rpla tim with tt ovrag to otain a nw modl. W aum that ovrag unit lik ranh t. alo ollow a Logarithmi Poion modl with rpt to tting tim. Th aumption i ad on th at that dirnt ranh t. hav dirnt proailiti o gtting ovrd jut lik dt. Hr w u th uprript D or dt and i = 1,2,.. or variou tt unit lik lok, ranh t. Th Logarithmi Poion modl ha n hown to a good hoi or an SRGM modl. It upriority ovr om othr modl o th am omplxity ha n hown to tatitially igniiant [1]. Uing th Logarithmi Poion modl w an xpr dt ovrag C D (t) a, C D (t) = D N D ln(1 + D 1 t); CD (t) 1 (2) Similarly or tt unit i ovrag, lt u aum, C D (t) = i N i ln(1 + i 1 t); Ci (t) 1 (3) whr N D i th total initial numr o dt and N i i th total numr o unit o typ i in th program undr tt. Hr D ;D 1 ;i ;i 1, ar appropriat paramtr or th applial Logarithmi Poion modl. W an olv or t uing quation 3 and utitut it in quation 2 to otain whr C D (C i )=a i ln[1 + ai 1 (ai 2 Ci, 1)]; C i 1 (4) a i = D N D a i 1 = D 1 i 1 a i 2 = N i Otn w want to u th numr o dt ound rathr than dt ovrag. I w indiat th total xptd numr o dt ound in tim t y (t), w an writ C D (t) = (t) N D. Hn rom quation 4, i (5) D (C i )=a i 3 ln[1 + ai 1 (ai 2 i, 1)]; C i 1 (6) whr a i 3 = ai _ N D = D Equation 6 an ud whn th initial numr o dt i not availal. W mut not that quation 4 and 6 ar applial only whn C i i l than or qual to on. Not that quation 6 allow or l than 1% dt ovrag whn numral ovrag i at 1%. Thi i au xriing all ranh do not xri th program xhautivly. Ahiving a 1% ovrag uing a mor trit maur, uh a P-u, would xri th od mor thoroughly. Figur 2 how a plot illutrating th hap o th urv drid y quation 4. At th ginning, dt ovrag grow only lowly with tt unit ovrag. Howvr, at highr tt ovrag, thr i a linar rlationhip. Th valu around whih th urv xhiit a kn ha a igniian a w will low. Th important point i to not that at 1% ovrag o unit i, w xpt to ind th numr o dt givn y th point whr th urv intrt th vrtial lin orrponding to 1% ovrag. W an xpt that at lat thi many ault wr initially prnt. A lowr ound on th numr o rmaining ault i otaind y utrating th numr o ault atually ound rom th numr o ault xptd at 1% ovrag. Th ound i

3 3 Dt Found 25 D t Data St: Paquini Tt Ca Figur 1. Etimatd total dt uing xponntial modl (Paquini data) D t Et o Dt Rmoval ? 2? 15 1? 5? Enumral Covrag Figur 2. Dt v. Tt Covrag Modl

Figur 3. ROBUST: Blok Covrag data (Paquini t al.) lor to th atual numr i w u a mor trit tt ovrag maur. Th numr aro th top o th plot in Figur 2 giv th numr o tt a applid or th top urv.

4 Figur 3. ROBUST: Blok Covrag data (Paquini t al.) lor to th atual numr i w u a mor trit tt ovrag maur. Th numr aro th top o th plot in Figur 2 giv th numr o tt a applid or th top urv. Thy how that dpit th linar growth o dt ovrag, mor tting ort i ndd to otain mor ovrag and thu ind mor ault. Th ond urv how th plot that would otaind i om o ault wr alrady rmovd in a prviou tt pha whn urrnt tting wa initiatd. Th ond plot how that whn th initial dt dnity i lowr, tting tart inding dt at a highr ovrag lvl. Thi orrpond to to hiting o th urv downward. 3. Applying th Nw Approah Appliaility o thi modl i illutratd y th plot in igur 3, 4, and 5. Thi data wa olltd xprimntally y Paquini t al rom a 61 lin C program y applying 2, tt a [15]. Th tt ovrag data wa olltd uing th ATAC tool. Figur 3 how a rn rom RO- BUST, a tool whih implmnt our thnqiu; and that ha n dvlopd at CSU [4]. Furthr dvlopmnt o thi tool i undrway to inlud additional apailiti. For th 2, tt, th ovrag valu otaind wr: lok ovrag : 82.31% o 297 lok, diion ovrag : 7.71% o 1171 diion, and p-u ovrag 61.51% o 2546 p-u. Thi i to xptd in p-u ovrag i th mot rigorou ovrag maur and lok ovrag i th lat. Complt ranh ovrag guarant omplt lok ovrag, and omplt p-u ovrag guarant omplt diion ovrag. Th plot uggt that 1% lok ovrag would unovr 4 dt, 1% ranh ovrag would unovr 47 dt, whra 1% p-u ovrag would rval 51 dt. I w aum that all o th otwar omponnt ar rahal, than w an xpt that th atual total numr o ault to lightly mor than 51. In atual prati, otn larg program ontain om unrahal od, otn alld dad or oolt od. For C program it an in th nighorhood o 5%. For aurat timat thi nd to takn into aount. For impliity o illutration, hr w will aum that all o th od i rahal. Unrahal od an minimizd y uing ovrag tool and making ur that all modul and tion ar ntrd during tting. Th data t ud in thi papr ar or program that ar not volving and thu th atual dt dnity i ontant. W not that th ittd modl om vry linar atr th kn in th urv. Lt u din Ck i a th kn, whr th linar part intrt th x-axi. For lok, ranh and p-u ovrag it our at aout 4%, 25% and 25% rptivly. Blow w th igniian o thi valu. Th ovrag modl in Equation 4 and 6 provid u a nw way to timat th total numr o dt. A w an in Figur 3, 4 and 5, whih u th data otaind y Paquini t al., th data point ollow th linar part o th modl rathr loly. Both th xprimntal data and th modl uggt that th 1% ovrag vntually ahivd hould unovr th numr o ault a givn in Tal 1 low. Numr in th lat olumn hav n roundd to nart intgr. It hould notd that or thi projt 124 tt rvald 28 ault. Anothr 18,76 tt did not ind any additional

5 D t Covrag Data Modl Data St: Paquini Branh Covrag Figur 4. Dt v. % Branh Covrag Tal 1. Projtd numr o total dt with 1% ovrag Dt Covrag Dt Covrag Maur ound ahivd xptd Blok Covrag 28 82% 4 Branh Covrag 28 7% 47 P-u Covrag 28 67% 51 C-u Covrag 28 74% 43 ault, vn though at lat 5 mor ault wr known. Th data uggt that th numral (lok, ranh t.) not ovrd y th irt 124 tt wr vry hard to rah. Thy prhap long to tion o th od intndd or handling pial ituation. Thr i a uumption rlationhip among lok, ranh and P-u. Covring all P-u aur ovring all lok and ranh. Covring all ranh aur ovrag o all lok. Thu among th thr maur th P- u ovrag maur i mot trit. Thr i no ovrag maur uh that 1% ovrag will aur dttion o all th dt. Thu in th aov tal, th ntry in th lat olumn i a low timat o th total numr o dt atually prnt. Uing a mor trit ovrag maur rai th low timat lor to th atual valu. Thu th timat o 51 ault uing P-u ovrag hould lor to th atual numr than th timat providd y lok ovrag. Two ovrag maur, DU-path ovrag and all-path ovrag ar mor trit than P-u ovrag, and may uital or a whr ultra-high rliaility i rquird. Howvr onidring th at that otn vn otaining 1% ranh ovrag i inail, w ar unlikly to dtt mor ault than what th P-u ovrag data provid vn with airly rigorou tting. It hould notd that C-u ovrag do not it in th uumption hirarhy and thror it i hard to intrprt th valu otaind y uing C-u ovrag data. Furthr appliation o thi nw mthod i illutratd y xamining th data providd y Vouk [17]. Th thr data t wr otaind y tting thr parat implmntation o a nor managmnt program or an inrtial navigation ytm. Eah program i aout iv thouand lin o od. In th irt program, 1196 tt ound 9 dt. For th othr two program 796 tt rvald 7 and 9 dt rptivly. Figur 6 how th plot o P-u ovrag ahivd vru th numr o dt ound. Tal 2 how th timat or th total numr o ault that would ound with 1% ovrag. Tal 2. Projtd numr o total dt with 1% ovrag (Vouk data t) Fault Exptd Fault Data St Found Blok Branh P-u Vouk Vouk Vouk Tal 2 again how that th timat otaind ar onitnt with th uumption hirarhy.

6 6 Data St: Paquini 5 4 Covrag Data Modl D t P-u Covrag Figur 5. Dt v. % P-u Covrag 4. Signiian o Paramtr Figur 2, whih i a dirt plot o th modl, and igur 3, 4, and 5 whih plot atual d ata, uggt that at highr ovrag valu, a linar modl an a vry good approximation. Thi lad u to an analytial intrprtation o th havior. Thi intrprtation an ud or otaining a impl prliminary modl or planning purpo. W will otain a linar modl rom quation 4. Lt u aum that at highr valu o C i quation 4 an impliid a and C D (C i ) = a i ln[ai 1 ai 2 Ci = a i ln(ai 1 )+ai ai 2 Ci = A i + Ai 1 Ci whr C i >C i n (7) A i = ai ln(ai 1 )= D N D D ln 1 i 1 A i 1 = ai ai 1 = D N D i (9) Not that thi impliiation i applial only or valu o Cn i gratr than whr th kn our. Th xprimntal paramtr valu or thi modl an not otaind until a lar linar havior yond th kn ha n talihd. Auming that th kn our whr th linar part o th modl intrt th x-axi, uing quation 7, th kn i at N i (8) Ckn i =, Ai A i (1) 1 Hr w an mak a uul approximation. For a trit ovrag maur, or C i =1, C D 1. Hn rom quation 7, w hav A i + Ai 1 1: (11) Rplaing A i y 1, A i 1 in quation 1, and uing 8, w an writ, Ckn i =1, 1 a i ai 1 Thi an writtn a??, C i kn =1, ( Di min D min Di (12) )D (13) Whr Dmin i, D min, Di ar paramtr and D i th initial dt dnity. Thu or lowr dt dniti, th kn our at highr tt ovrag. Thi ha a impl phyial intrprtation. I a program ha n prviouly ttd rulting in a lowr dt dnity, it i likly that th numral with highr ttaility hav alrady n xrid. Thi man that tting will tart inding a igniiant numr o additional dt only atr highr tt ovrag valu ar ahivd. 5. Conluion Dt dnity i an important maur o otwar rliaility, iguring prominntly in th rliaility amnt o

7 14 Data St: Vouk Covrag Data Modl D t P-u Covrag Figur 6. Dt v. % P-u Covrag many quality auran nginr and managr. Exiting mthod or timating th numr o dt an undrtimat th numr o dt au o a ia toward aily ttal ault. Th xponntial modl tnd to gnrat untal projtion and otn yild a numr qual to dt alrady ound. W hav prntd a modl or dt ound ad on ovrag, and hown that thi modl provid a vry good dription o atual data. Our mthod provid tal projtion arly in th dvlopmnt pro. Exprimntal data uggt that our mthod an provid mor aurat timat and provid dvlopr with th inormation thy rquir to mak an aurat amnt o otwar rliaility. Th hoi o ovrag maur do ha an t on th projtion mad, and w uggt that a trit ovrag maur uh a P-u hould ud or vry high rliaility program. Hr w hav aum that no nw dt ar ing introdud in th program. Furthr invtigation ar ndd to xtnd thi mthod or volving program. 6. Aknowldgmnt Thi work wa upportd in part y a BMDO undd projt monitord y ONR. Rrn [1] R. V. Bindr. Six igma: Hardwar i, otwar no! [2] R. W. Butlr and G. B. Finlli. Th inaiility o quantiying th rliaility o li-ritial ral-tim otwar. IEEE Tranation on Sotwar Enginring, 19(1):3 12, Jan [3] M. Chn, M. R. Lyu, and W. E. Wong. An mpirial tudy o th orrlation twn ovrag and rliaility timation. In IEEE Third Intrnational Sympoium on Sotwar Mtri, Brlin, Grmany, Mar [4] J.A.Dnton. ROBUST, An Intgratd Sotwar Rliaility Tool. Colorado Stat Univrity, [5] L. Hatton. N-vrion dign vru on good dign. IEEE Sotwar, pag 71 76, Nov./D [6] M. Huthing, T. Goradia, and T. Otrand. Exprimnt on th tivn o data-low and ontrol-low ad tt data adquay ritria. In Intrnational Conrn on Sotwar Enginring, pag 191 2, [7] P. Laky and A. Nuldr. Sytm and Sotwar Rliaility Auran Notook. Rom Laoratory, Rom, Nw York, [8] M. R. Lyu, J. R. Horgan, and S. London. A ovrag analyi tool or th tivn o otwar tting. In Proding o th IEEE Intrnational Sympoium on Sotwar Rliaility Enginring, pag 25 34, [9] Y. K. Malaiya and J. A. Dnton. What do otwar rliaility paramtr rprnt? In 8th Intrnational Sympoium on Sotwar Rliaility Enginring, pag , Aluqurqu, NM, Nov [1] Y. K. Malaiya, N. Karunanithi, and P. Vrma. Prditaility o otwar rliaility modl. IEEE Tranation on Rliaility, pag , D [11] Y. K..Malaiya, N. Li, J. Biman, R. Karih, and B. Ski. Th rlationhip twn tt ovrag and rliaility. In Proding o th Intrnational Sympoium on Sotwar Rliaility Enginring, pag , Nov

8 [12] Y. K. Malaiya, A. von Mayrhaur, and P. Srimani. An xamination o ault xpour ratio. IEEE Tranation on Sotwar Enginring, pag , Nov [13] Y. K. Malaiya and S. Yang. Th ovrag prolm or random tting. In Proding o th Intrnational Tt Conrn, pag , Ot [14] S. MConnll. Gauging otwar radin with dt traking. IEEE Sotwar, 14(3), May/Jun [15] A. Paquini, A. N. Crpo, and P. Matrlla. Snitivity o rliaility growth modl to oprational proil rror. IEEE Tranation on Rliaility, pag , D [16] P. Piwowarki, M. Oha, and J. Caruo. Covrag maurmnt xprin during untion tt. In Proding o th 15th Intrnational Conrn on Sotwar Enginring, pag 287 3, May [17] M. A. Vouk. Uing rliaility modl during tting with non-oprational proil. In Proding o th 2nd Bllor/Purdu Workhop on Iu in Sotwar Rliaility Etimation, pag , Ot

Computer Science Technical Report

Computer Science Technical Report Computr Scinc Tchnical Rport Etimating Dfct Dnity Uing Tt Covrag Yahwant K. Malaiya Jaon Dnton Computr Scinc Dpt. Colorado Stat Univrity Fort Collin, CO 8523 malaiya dnton@c.colotat.du Tchnical Rport CS-98-14