An Input Domain-Based Reliability Growth Model and Its Applications in Comparing Software Testing Strategies

Transcription

1 CENTRE NATIONAL DE LA RECHERCHE SCIENTIFIQUE LABORATOIRE D'ANALYSE ET D'ARCHITECTURE DES SYSTÈMES A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Software Testig Strategies Yiog Che ad Jea Arlat LAAS REPORT 9505 April 995 LIMITED DISTRIBUTION NOTICE This report has bee submitted for publicatio outside of CNRS. It has bee issued as a Research Report for early peer distributio.

2 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Software Testig Strategies Yiog Che * ad Jea Arlat LAAS-CNRS, 7 Aveue du Coloel Roche 3077 Toulouse Cedex - FRANCE Abstract Existig iput domai-based models do ot accout for software growth, because they do ot cosider fault correctios. This paper proposes a iput domai-based growth model with fault correctio history beig take ito accout. Both partitio ad radom testig ca be used to geerate iput cases for test rus. It is geerally cosidered i the model that iput case geeratio, fault detectio ad fault correctio ca all be imperfect. As a applicatio of the proposed iput domai-based growth model, the efficiecy of radom ad partitio testig i terms of growth is studied ad compared aalytically. The impacts o the efficiecy of testig strategies due to the umber of faults i the program, the distributio of fault i the iput domai of the program, as well as the imperfectios of iput case geeratio, fault detectio ad fault correctio are studied. Through sophisticated aalysis we obtai some ew results which explai why ad uder which coditios radom testig has the same efficiecy as, a higher or lower efficiecy tha partitio testig. As a further applicatio of the proposed iput domai-based growth model, the efficiecy of variats of partitio testig are studied ad compared. Keywords software fault model, imperfect fault correctio, software model, testig coverage, compariso betwee testig strategies * O leave from the Departmet of Computer Sciece, Uiversity of the Witwatersrad, Johaesburg. He is curretly a postdoctoral fellow at LAAS-CNRS, Toulouse, uder the Europea Commissio Huma Capital ad Mobility Programme, CaberNet.

3 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies Itroductio Software has bee defied as the probability that o failure occurs i a specified eviromet durig a specified (cotiued) exposure period. For some programs the appropriate time uit of exposure period is the caledar or CPU time, ad for some other programs the appropriate time uit of exposure period is a applicatio ru correspodig to a selectio of a iput case (a vector of iput values eeded to make a applicatio ru) from the iput case domai (ICD) of the programs [Musa&87]. For the evaluatio of software there exist differet kids of models. [Ramamoorthy&Bastai82] classifies software models accordig to the developmet phases of software life-cycle, while [Goel85] divides the them accordig to the ature of failure process ito times-betwee-failures models, failure-cout models, fault seedig models, ad iput domai based models, as outlied i Fig.. Sice fault correctios are ecessary i the testig ad debuggig phase of life-cycle, growth models which take fault correctios ito accout are maily used i this phase. The models without dealig with fault correctios, say iput domai models, ca oly be applied i this phase by treatig the program after each correctio as a ew program. Withi the frame of Goel's classificatio, times-betwee-failures models ad failure-cout models belog to growth models. Reliability growth models usually defie the software as the probability R(T) = Prob{o failure withi time period [0, T] } where T is the exposure period whose time uit is the caledar or CPU time. It is usually assumed that R(T) follows certai probability distributio, for example, R(T) = e -z(t), where z(t) is the failure rate fuctio. The mai cocer of these growth models is to estimate the value of the failure rate fuctio z(t). Thus, [Ramamoorthy&Bastai82] classified the growth models ito three types accordig to the strategies evaluatig the failure rate fuctio: discrete chages at correctios of faults, e.g., JM model [Jelisky&Morada72]; cotiuous chages i elapsed time, e.g., [Goel&Okumoto79, Laprie84, Laprie&9, Musa&Okumoto83]; ad icorporatig both discrete chages at correctios of faults ad cotiuous chages i elapsed time [e.g., Littlewood8, Littlewood&Verrall73]. These three types of models are plotted i the bottom left of Fig.. O the other had, iput domai-based models [e.g., MacWilliams73, Brow&Lipow75, Nelso78, Tsoukalas&93, Weiss&Weyuker88], defie software as the probability R(N) = Prob{o failure over N applicatio rus} where N is the exposure period whose time uit is the umber of applicatio rus. Assumig that iput cases are selected idepedetly, the R(N) ca be expressed by R(N) = (R()) N = R N where, R R() is the per applicatio ru, or for short. Sice R(N) ca be simply calculated from R, R becomes the mai objective to be estimated by iput domaibased models.

4 2 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies Reliability Models ( evaluatio) [Ramamoorthy Bastai 82] Desig phase ( predicatio) Testig & debuggig phase ( estimatio) Validatio phase ( validatio Operatio phase ( assessmet) [Goel 85] Time-betwee-failures models Failure-cout Models Fault seedig models Iput domai models (without error correctios) Reliability growth models with error correctios R(t) = e -λt λ is to be estimated by testig λ = f(t) R() s f [MacWilliams73] R(t) = R() N R() m Σ ( j= f j s p(cj j )) [Brow&Lipow 75] f(t) λ [Ramamoorthy Bastai 82] [Littlewood 90] Jeliski- Morada t, testig time λ Musa, Goel, Okumoto Laprie, Kaou Tohma, Jacoby λ HE, HPP, NHPP models Littlewood Littlewood-Verrall radom testig Partitio based Iput domai models With error correctios imperfect testig partitio testig PB-SRGM imperfect detectio modified partitio testig imperfect correctio Fig. Classificatios of software models Obviously, the R is related to the applicatio iput profile of the program. However, i the testig phase, the applicatio iput profile may be ukow. I this case, a uiform distributio ca be assumed [Hamlet94] to be the applicatio iput profile. More cocretely, the per test ru ca be defied by the ratio of the umber of test rus i which failures are observed ad the total umber of test rus whe ifiite umber of differet iput cases are applied for test rus: R = F = lim s ( f s) where, s is the total umber of test rus carried out ad f is the umber of test rus i which failures have occurred. F = lim( f s) is called failure rate per applicatio ru, or error size of the program s uder test [Ramamoorthy&Bastai82]. Because of test time limit oly a subset of the etire ICD ca be applied to test the program i practice. Thus the per test ru, R, is usually estimated by ˆR = ˆF = ( f s) I the primary form of iput domai based model [MacWilliams73], the s iput cases are supposed to be selected radomly from ICD. [Brow&Lipow75] suggested a alterative model i which the ICD is supposed to be partitioed ito m classes. If s i iput cases are selected from class C i ad f i failures are observed, the ca be calculated by ( ) i= ˆR = ˆF m = ( f i s i ) P(C i )

5 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies 3 where P(C i ) is a probability fuctio reflectig the iput profile i terms of classes. Further extesios are studied i [Nelso78, Tsoukalas&93, Weiss&Weyuker88], where the iput profile is take ito accout by addig differet weights to differet iput cases. However, o iput domai based models so far take the fault correctios ito cosideratio. Thus they are maily used i the validatio phase of software life-cycle, where fault correctios are ot ecessary. I fact, if faults are foud i this phase, the program uder test may be refused by the users. If they are used i the testig ad debuggig phase, a program after ay fault correctio has to be cosidered as a ew program. Fault correctio history is simply igored. I this paper, a iput domai-based growth model, with fault correctio history beig take ito accout, is proposed to evaluate the R. It is assumed that the applicatio iput profile of the programs follows a uiform distributio. Partitio ad radom testig are used to geerate iput cases from ICD. They are geerally assumed to be imperfect, i.e., they caot geerate iput cases uiformly from the etire ICD, ad therefore their fault coverage is reduced. Fault detectio ad correctio are also assumed to be imperfect, that is, the probability to detect ad fix a fault decreases with icreasig umbers of test rus. This copes with the fact that the remaiig faults after a period of testig are more difficult to detect ad to fix tha the earlier faults. Efficiecy of partitio ad radom testig has bee studied ad compared by a umber of researchers [e.g., Dura&Ntafos84, Hamlet89, Hamlet94, Hamlet&Taylor90, TChe&Yu94, Tsoukalas&93, Théveod-Fosse&9, Weyuker&Jeg9]. Partitio testig previously was believed to be better tha radom testig. However, experimetal observatios repeatedly show that radom testig is as good as or better tha partitio testig. A similar state-of-affairs cocers radom ad partitio testig of fault-tolerat systems by meas of fault ijectio [Avresky&92, Arlat&90, Che93, Echtle&Che9, Powell&93]. Sice extremely high is required by fault-tolerat systems, the efficiecy of testig strategies is eve more cocered i testig of fault-tolerat systems. Recetly, it is show aalytically that partitio testig ca be better or worse tha radom testig: partitio testig ca be a excellet testig strategy or a poor oe, depedig o how the failurecausig iput cases are distributed i the classes of the partitio [e.g., TChe&Yu94, Weyuker&Jeg9]. These results help to explai the cotroversial experimetal results. As a applicatio of the iput domai-based growth model proposed i this paper, the efficiecy of radom ad partitio testig strategies are studied ad compared aalytically. I previous aalytic approaches, the cotrol-flow coverage ad fault detectio efficiecy of partitio testig ad radom testig are maily compared [TChe&Yu94, Dura&Ntafos84, Hamlet89, Hamlet94, Hamlet&Taylor90, Weyuker&Jeg9]. Recetly, [Hamlet94] idicated that software depedability should be a more direct criterio to judge the test efficiecy of testig strategies. He defied three depedability measures: program for uiform iput distributio, program i the worst case, ad the probability that a program has zero faults. I this paper we use for uiform iput distributio as the criterio to compare imperfect partitio testig ad imperfect radom testig. To our kowledge, this is the first approach which falls i the frame suggested by Hamlet. Usig this ew evaluatio criterio, multiple faults, imperfectio i testig, fault detectio ad correctio are all take ito cosideratio. Through sophisticated aalysis we obtai results which cofirm the results from [TChe&Yu94, Dura&Ntafos84, Weyuker&Jeg9] ad gai ew kowledge why ad uder which

6 4 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies coditios radom testig has the same efficiecy as, a higher or lower efficiecy tha partitio testig. As a secod applicatio of our iput domai-based growth model, differet variats of partitio testig are evaluated ad compared. I our previous work, we used growth as a criterio to compare the efficiecy of testig strategies for fault-tolerat protocols, a special kid of software i highly reliable distributed systems [Che&9, Che93, Che&Görke94, Echtle&Che9]. This paper exteds this primary idea ito testig strategies for covetioal programs. Nevertheless, fudametal ehacemets have bee made i this work to cope the more geeral requiremets of testig strategies for covetioal programs. The remaiig part of this paper is orgaised as follows. Next sectio defies the models for testig, fault, fault detectio ad fault correctio. How to collect failure data ad obtai the parameter values of all these models will be discussed i Sectio 3. The iput domai-based growth model, the mai work of this paper is preseted i Sectio 4. Applicatios of the model i comparig the efficiecy of testig strategies are discussed i Sectio 5. Fially, a summary of mai cotributios ad results of this paper is give i Sectio 6. 2 Testig Strategies ad Fault Model This sectio gives basic cocepts ad defies the compoet models eeded for the proposed iput domai-based growth model. 2. Partitio ad Radom Testig Strategies Defiitio Partitio A partitio of the iput case domai ICD is defied by a set {C, C 2,..., C m }, where C i ICD for i m; m 2 U C i = ICD ; i= 3 C i C j = φ (empty set) if i, j =, 2,..., m ad i j ; 4 C i = C j for i, j =, 2,..., m, where S expresses the umber of elemets i set S. This defiitio assumes that the classes of a partitio are disjoit ad have equal size. These assumptios are based o the cosideratio that we will ot actually partitio the ICD, but defie classes accordig to the test criterio applied, for istace, the iput cases which will exercise the same program path belog to a class. As log as all classes cosist of a ifiite umber of iput cases (this is true i may cases), the assumptios are proper. The pathologic partitios i which some classes cosist of oly very few iput cases caot be icluded i this defiitio. A extesio of this defiitio is beyod the scope of this paper. Defiitio 2 Partitio testig Partitio testig cosists of (test) rouds of test rus. A test ru is a executio of the program uder test by applyig a iput case. If the ICD of the program is partitioed ito m classes, a test roud cosists of exactly m test rus, aimed at coverig (geeratig iput cases from) all the m classes of the partitio. If partitio testig covers some classes more tha oce i a test roud,

7 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies 5 some other classes will be ucovered. Let u be the expected umber of ucovered classes i a test roud, the portio of covered classes, (m u) m, is called partitio testig coverage. If u = 0, (m u) m =, the coverage is perfect, otherwise, it is imperfect. Partitio testig with u ucovered classes per roud ca be viewed as cosistig of two parts of test rus. I the first part, m u test rus will select iput cases from m u differet classes (we say these rus are determiistic rus), ad i the secod part, u test rus will radomly select iput cases from these m u classes which have bee covered by the first part of test rus. This defiitio gives a primary form of partitio testig scheme. Some variats of partitio testig will be discussed i Subsectio 5.2. Differet from partitio testig which selects iput cases followig a give partitio, radom testig selects iput cases followig a certai probability distributio. Defiitio 3 Radom testig Radom testig cosists of (test) rouds of test rus. A test ru is a executio of the program by applyig a iput case. A (radom) test roud cosists of exactly m test rus, aimed at followig a give probability distributio. If radom testig caot exactly follow the give distributio, ICD is imperfectly covered. Let υ be the probability fuctio which reflects the imperfectio of radom testig, 0 < υ. This probability fuctio υ is called the radom testig coverage. If υ =, the coverage is perfect, otherwise, it is imperfect. Radom testig coverage ad partitio testig coverage are differet, but they both affect test efficiecy. Whe we compare the efficiecy of the two testig strategies, we will assume they take the same value, i.e., υ = (m u)/m. I the followig discussio, we will assume that the give probability distributio is a uiform distributio, because we have assumed that the applicatio iput profile of the program uder test follows a uiform distributio. 2.2 Fault Model I this sectio we tur to discuss the faults i the program uder test. Defiitio 4 Failure, fault, failure-causig iput case, error ad detectio set A wrog output of a program is a failure. A program cotais faults, if it will give a failure whe certai iput cases are applied. A iput case is a failure-causig iput case related to a fault if its applicatio ca detect the fault by causig a failure. The set of failure-causig iput cases related to a fault is called a error related to the fault. The set of all failure-causig iput cases related to all faults is the detectio set (DS) of the program. I other words, a error is the complete maifestatio of a fault i the iput domai, while a failure is a maifestatio of a fault i the output domai. Defiitio 5 Error size ad error class For a give subset S ICD, the ratio S DS is a error class, if its error size is o-zero. ICD is called the error size of subset S. A class

8 6 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies Evidetly, if all iput cases are equally likely, the error size of a class is the probability that a failurecausig iput case is geerated from this class per test ru ad the error size of ICD, DS ICD, is the probability that a failure-causig iput case is geerated from ICD per test ru. Thus, DS ICD is also called failure probability. Accordig to the defiitio of software for iput domai-based model, the of a program is the R = DS ICD Defiitio 6 Error group ad error distributio Assumig that there exist faults g, g 2,..., g, i a program ad C, C 2,..., C m are classes of a ICD partitio of the program. If the error correspodig to fault g k is distributed i d k classes, d k m, the the uio of these d k classes is called a error group, deoted by G k ad the error size of G k is deoted by Θ k = G k DS ICD. The parameter set (m,, d k, Θ k ) for k =, 2,...,, forms a error distributio. Remark: Error, error group ad error size are related but differet. Differece: a error E k is the set failure-causig iput cases related to a fault g k, a error group G k is the uio of the error classes related to g k, ad the error size Θ k of G k is a probability, istead of a set. Relatios: E k = G k DS, the umber of error groups is equal to the umber of errors i a program, Θ k = G k DS ICD = E k ICD. Note, accordig to Def. 5, Θ k G k DS G k. As a example Fig.2 shows the structure tree of a program with 0 paths. If the structure tree is used as the criterio to partitio the ICD, we ca obtai a partitio with 0 classes. If there are three fault g, g 2 ad g 3 i the program, the correspodig error groups are: G = C C 2 C 3 C 4, G 2 = C 7 ad G 3 = C 8, as show i the lower part of Fig.2. 0 g g g C C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 C 0 Fig.2 A partitio with a error distributio A fault correctio is a behaviour aimed at localisig ad removig the faults i the program which are detected. The effect of a fault correctio ca be so large that the program after the correctio has o relatio at all with the program before correctio. I this case, either the failure history or the fault correctio history of the old program is useful for ay growth model. Therefore, i our growth model we oly cosidered fault correctios which esure that the partitio after the correctios still satisfies the give partitioig criterio, so that we ca cotiue to test the program without havig to make a ew partitio.

9 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies 7 Defiitio 7 Fault correctio A fault correctio is a chage (decreasig, icreasig, or oe) of the error sizes of some classes of a give partitio. Accordig to the umber of correctios eeded to remove a fault (chage its error size to zero) we ca iformally distiguish two kids of faults: easy faults, the error sizes of the faults are chaged to zero after a limited umber of correctios; ad difficult faults, the error sizes of the faults are ot chaged to zero after a large umber of correctios, because they are either difficult to be detected or difficult to be corrected. The easy faults are usually removed after a period of testig. The remaiig faults are all the difficult faults, which are the mai problem to be studied i testig ad evaluatio of software. With fault correctio i cosideratio, the error sizes ad the error distributios will be chagig with fault correctios. Defiitio 8 Fault correctio rate Let (m, j-, d k(j-), Θ k(j-) ) be the error distributio before the j th fault correctio, ad it is expected to be chaged ito (m, j, d kj, Θ kj ) after the j th fault correctio, where j Θ k = kj = ρ j Θ k = k( j ) Parameter ρ is called the error size chage rate ad ρ is called the correctio rate. 3 Parameter Estimatio So far we have characterised testig strategies, error distributio, ad fault correctio by a set of parameters u, j, d kj, Θ kj, ρ. Now we discuss how to orgaise the testig process ad how to use the failure data collected durig testig to estimate the values of these parameters. For this purpose, we divide partitio testig ito two stages: a phase-correctio stage ad a roud-correctio stage. The phase-correctio stage cosists of q test phases, ad each test phase cosists of p test rouds, as show i Fig.3. If faults are detected durig testig, testig will ot be iterrupted for fault correctio. After the p rouds of testig, all the faults detected i the phase are corrected together. Fault correctio is viewed as a part of a test phase. Roud-correctio stage cosists of t test rouds. If ay failure is detected, fault correctio will be performed after a roud. Fault correctio is viewed as a part of a test roud. A failure table, show i Tab., is maitaied durig the phase-correctio stage, which records how may failure-causig iput cases have bee geerated (how may failures have bee observed) from each class i each of the q test phases. Of course, p ad q must be umbers large eough to obtai eough statistic data for parameter estimatio. Havig completed the q test phases i the phase-correctio stage, testig will be cotiued with the roud-correctio stage, i which fault correctio will be performed after each roud (if ay fault is detected). The fial parameter values estimated i the first stage will be used as the iitial parameter Partitio testig will be maily studied, ad radom testig will be hadled as a special case of partitio testig.

10 8 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies values of the secod stage. Chages of parameter values caused i the secod stage will be predicted accordig to certai probability distributios. () phase-correctio stage (parameter estimatig) a test roud = m rus class (2) roud-correctio stage (parameter predicatig) a test roud = m rus + error correctio the test stage = q test phases Error Correctio Error Correctio a test phase = p rouds + error correctio u q d kq δ kq ρ EC EC EC EC EC EC EC EC EC the test stage = t test rouds Class o. of failures / o. of test rus i test phase Fig.3 Illustratio of the two testig stages o. of failures / o. of failures / o. of test rus o. of test rus... i test phase 2 i test phase 3 o. of failures / o. of test rus i test phase q C f s f 2 s 2 f 3 s 3... f q s q C 2 f 2 s 2 f 22 s 22 f 23 s f 2q s 2q C i f i s i f i2 s i2 f i3 s i3... f iq s iq C m f m s m f m2 s m2 f m3 s m3... f mq s mq Tab. The failure frequecy table I this sectio, we will briefly study the phase-correctio stage ad the parameter estimatio. The roud-correctio stage will be studied i full details i the ext sectio. Let s ij be the total umber of test rus which select iput cases from C i i the j th test phase, i which f ij failures are observed. For i =, 2,..., m ad j =, 2,..., q, f ij ad s ij are recorded i Tab.. Based o these data ad the specificatio ad/or implemetatio of the program uder test, the values of the parameters u, j, d kj, Θ kj, ρ ca be estimated as follows. Sice p test rouds are performed i each test phase, each classes should be tested exactly p times, i.e., s ij = p, if the testig coverage is perfect. Otherwise, s ij p. If s ij p, class C i ca be cosidered to be tested at least oce i average i each test roud. If s ij < p, class C i is ot tested i at least p s ij rouds, which reflects the imperfect coverage of this test phase. The the parameter u ca be estimated by the average umber of classes which are tested less tha p times i the test phase: u j = m m i= s ij < p ( p s ij )

11 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies 9 The above estimatio is oly based o the statistic data i the j th test phase. Sice the umber of ucovered classes is a global parameter idepedet of the test phases, we ca further build the average amog all the value u, u 2,..., u q to get the fial estimate of u: q q m u j ( p s ij )) = q m ( p s ij ) u = q j = = q j = ( m i= s ij < p q m j = i= s ij < p Now let's briefly discuss how to group error classes, so that the classes which are affected by a sigle fault belog to a group. For a give partitio, there are usually some relatios amog the classes. The relatioship is determied by the specificatio or the implemetatio of the program. As a example, the structure tree i Fig.2 ca be used to explai the relatioship amog error classes. If the classes correspodig to paths 0, 0 2, 0 3, ad 0 4 have o-zero error sizes, it is more likely that a commo poit of these paths cotais a sigle fault tha that the private parts of these paths cotai four idepedet faults. Thus, classes C, C 2, C 3 ad C 4 should be viewed as a group. However, if the classes correspodig to paths 0 7 ad 0 8 have o-zero error sizes, they must be cosidered as two error groups, because the structure tree does't have a poit which affect these ad oly these two paths. How to determie the error groups will ot be further discussed i this paper. It will be assumed that after the j th test phase, we ca obtai j groups, G j, G 2j,..., G j j, ad the umber d kj of error classes i each group accordig to Tab. ad the specificatio ad/or implemetatio of the program. Based o the priciple of iput domai-based models, the error size Θ kj of group G kj after the j th test phase ca be estimated by the average ratio of the umber of failures ad the umber of test rus related to the classes i group G kj, where k =, 2,..., j, j =, 2,..., q, f Θ kj = ij p m = p m f ij C i G kj C i G kj Accordig to Def. 8 ad the estimate of Θ kj, the imperfect correctio rate ρ j of the j th correctio i the phase-correctio stage ca be estimated: j ρ j = Θ kj Θ k( j ) = f ij f i( j ) k = j k = j k = C i G kj j k = C i G k( j ) Further build the average value amog all ρ, ρ 2,..., ρ q, we obtai the fial estimatio of the fault correctio rate: ρ = q q ρ j j = = q q j = j j f ij f i( j ) k = C i G kj k = C i G k( j ) The fial parameter values, u, q, d kq, Θ kq, ρ, estimated i this stage will be used as iitial values i the ext stage. 4 The Iput Domai-based Reliability Growth Model I the previous sectio, the values of the set of ecessary parameters are estimated. As the result we obtai a set of iitial values of parameters: m, u, q, d kq, Θ kq, ρ, k =, 2,...,.

12 0 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies Sice the roud-correctio stage will use the same partitio ad the same testig algorithms, m ad u will remai uchaged i this stage. Moreover, sice test phases have bee repeated may times i the phase-correctio stage, most easy faults have bee removed. I this testig stage we maily cosider difficult faults which caot be removed eve after a large umber of correctios. Thus, we assume that the umber of error groups q ad the umber of classes d kq i group G k will remai uchaged, oly the Θ kq ad the ρ will chage with icreasig umber of test rouds ad fault correctios. As costats of this testig stage we will reame q ad d kq as ad d k, ad as iitial values, we will reame Θ kq ad ρ as Θ k0 ad ρ 0. The remaiig error size of ICD from the last stage (or iitial error size for this stage) is the Θ k = k 0. The values of Θ k0 ad ρ 0 after r th test roud i this stage will be deoted as Θ kr ad ρ r. Beside these parameters, we eed some more parameters to characterise the icreasig difficulty of fault detectio ad correctio i the ew testig stage, which will be discussed whe they are itroduced. I Subsectios 4. ad 4.2, the growth uder partitio ad radom testig with perfect testig coverage (u = 0) will be ivestigated, respectively. I Subsectio 4.3, a itegrated growth model with imperfect testig coverage for both radom ad partitio testig will be established. 4. Reliability Growth uder Partitio Testig Let G k be a error group with d k classes ad its error size is Θ k(r-) after the (r ) th test roud. Durig the r th test roud, oe iput case will be geerated from each class of G k. The probability that a failure-causig iput case is geerated from ayoe of the d k classes is m Θ k(r-) /d k. This is because Θ k(r ) is the probability that a failure-causig iput case is geerated from G k whe a iput case is equally likely selected from the etire ICD. If we do't select a iput case from the etire ICD, but oly from oe class of G k, the probability that a failure-causig iput case is geerated is the (m/d k )Θ k(r-) = m Θ k(r-) /d k. Sice there are d k classes i group G k, d k iput cases will be selected from G k. Thus, the probability ( d k ) d k that the fault correspodig to G k is detected is m Θ k(r ). I this equatio it is assumed that fault detectio probability is idepedet of the umber of test rouds performed. Geeral, we use a fuctio e λ p (r ) to reflect the fact that the detectio probability per ru may decrease with icreasig umber of test rouds, because easier faults are more likely to be detected ad removed i the earlier rouds. Decreasig of detectio probability is cotrolled by parameter λ p. If λ p = 0, e λ p (r ) =, the detectio probability will ot suffer from decreasig with test rouds' icreasig. This case refers to perfect fault detectio. Takig this factor ito accout, the detectio probability is geeralised as follows: ( ) d k (m Θ k(r ) d k ) e λ p (r ) First let's assume that the fault correctio is perfect (the etire error size related to a fault will be removed if the fault is detected), the error size Θ k(r-) of G k will be completely removed if the fault is detected. Therefore we have the error size removed from group G k by the r th test roud: (r ) ( ) d k (m Θ k(r ) d k ) e λ p Θ k(r ) ()

13 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies If fault correctio is imperfect, oly a portio of Θ k(r-) ca be removed. Accordig to the defiitio of fault correctio rate, the removed portio is Θ k(r ) ( ρ r ), where ρ r = ( ρ 0 ) e µ (r ), which reflects the fact that fault correctio rate will decrease with icreasig umber of test rouds, because easier faults are more likely to be removed i the earlier rouds. The decreasig stregth is cotrolled by parameter µ. As show i Fig.4, the fault correctio rate ρ r ca decrease very fast or very slow, depedig o the selectio of µ values. ρ 0 = µ = 0. ρ r = ( ρ 0 ) e - µr µ = 0.02 µ = µ = µ = r, the umber of test rouds Fig.4 The decreasig stregth of fault correctio rate ρ r is determied by parameter µ Sice detectio ad correctio probabilities decrease expoetially with icreasig of test rouds, the growth will actually stop after a log time of testig. This is true i practice. However, if the program is tested by aother test tool or aother testig team (the ed users ca also be viewed as aother testig team), testig ca be viewed as a ew roud-correctio stage ad thus the umber of test rouds performed ca reset. to zero, ad thus the expoetial decreasig detectio ad correctio probabilities ca beefit from a low umber of test rouds. This is because a fault which is difficult to detect or correct for oe test tool or testig team may ot be difficult for aother. All programs delivered to the ed users are usually tested for a log time without discoverig ay fault. The users however discover faults usually i the earlier applicatio stage of the programs. Therefore we suggest to test a program by combiatio of differet kids of testig strategies, differet implemetatios of the same kid of testig strategy, ad differet testig teams. Of course, such chages will also alter the parameter values related to testig coverage, detectio probability ad correctio rate. The, we have a more geeral form of the error size removed from group G k by the r th test roud: (r ) ( ) d k (m Θ k(r ) d k ) e λ p ( ) Θ ρ k(r ) r Sice there are error groups, ad if we have totally carried out t test rouds, the the error size removed after t test rouds, ESR(t), is t (r ) ( ) d k ESR(t) = (m Θ k(r ) d k ) e λ p r = k = Θ ρ k(r ) ( r ) After the r th correctio, the error size, Θ k(r-), of group G k will be chaged. Cosiderig all imperfect factors, the ew error size ca be calculated as follows: ( d k ) d k ( ) Θ ρ k(r ) r Θ kr = Θ k(r ) m Θ k(r ) (2) ( ) (3) Accordig to the defiitio of ad the ESR(t) i Equ. (2) we have the growth model for partitio testig with perfect testig coverage:

14 2 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies (r ) ( ) d k t R par (t) = Θ k 0 (m Θ k(r ) d k ) e λ p k = r = k = Θ ρ k(r ) ( r ) (4) where, Θ k = k 0 is the iitial error size of the program before the roud-correctio stage ad ESR(t) is the remaiig error size. Thus Θ k 0 ESR(t) k = Θ k 0 the program after the t th test rouds. 4.2 Reliability Growth Model uder Radom Testig ( k = ) is the of Firstly, we assume that the radom iput case geeratio will select iput cases from the etire ICD uiformly. Withi the r th radom test roud (m test rus), the probability that the fault correspodig to error group G k is detected is Θ k(r ) ( ) m. This expressio assumes that the detectio probability is idepedet of the umber of test rouds performed. As discussed i the last sectio, if we should more geerally assume that the detectio probability will decrease with icreasig umber of test rouds. Similar to partitio testig, we use a expoetial fuctio e λ q (r ) to model the decreasig detectio probability. Thus we obtai a more geeral expressio of the detectio probability to error group G k per radom test roud: ( ) m Θ k(r ) e λ q (r ) Accordig to Def. 3, if radom testig caot exactly follow the give probability distributio, the radom testig coverage will be: 0 <υ. I order to compare with partitio testig, we assume υ = (m u)/m. The the detectio probability to error group G k by the r th test roud is: Θ k(r ) m u m e λ q (r ) m Similar to partitio testig, we assume a fault correctio will be carried out after each test roud. The the error size removed from the error group G k the error correctio after the r th roud is Θ k(r ) m u m e λ q (r ) m Θ k(r ) ( ρ r ) Summig amog all error groups ad test rouds, we have the total error sizes removed from all groups after t radom test rouds: ESR ra (t) = t r = k = Θ k(r ) m u m e λ q (r ) m Θ k(r ) ( ρ r ) Similar to Equ. (3), the error size of group G k i the ext test roud will be chaged ito: Θ kr = Θ k(r ) Θ k(r ) m u m e λ q (r ) m Θ k(r ) ( ρ r ) The the geeral growth model for radom testig uder our fault, fault detectio ad fault correctio models is (5)

15 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies 3 R ra (t) = Θ k 0 k = t r = k = Θ k(r ) m u m e λ q (r ) m Θ k(r ) ( ρ r ) (6) 4.3 The Complete Model I this subsectios, we derive the iput domai-based growth model for both radom ad partitio testig. Accordig to Def. 2, if testig coverage is imperfect, a partitio testig roud ca be viewed as cosistig of two parts of test rus. Part : m u determiistic test rus will select iput cases from m u differet classes, as if the partitio testig coverage is perfect; Part 2: u test rus will radomly select iput cases from these m u classes which have bee covered by the first part of test rus. Now we discuss this two parts respectively. As discussed i Subsectio 4., if the partitio testig coverage is perfect, the detectio probability ca be calculated accordig Equ. (). If oly m u test rus are performed, the detectio probability i ( ) a k Equ. () will be chaged ito (m Θ k(r ) d k ) e λ p (r ), where a k = d k (m u) m is the actual umber of iput cases which is geerated from the error group G k, if oly m u, istead of m, test rus are performed. We deote is this detectio probability by ( ) a k P par (r, G k ) = (m Θ k(r ) d k ) e λ p (r ) O the other had, if h radom test rus are performed, the detectio probability i ay h 0 radom test rus ca be expressed, accordig to Equ. (5), by P ra (r, G k ) = Θ k(r ) m u m e λ q (r ) If h = u, the detectio probability i part 2 ca be calculated. There are two reasos we itroduce a ew parameter h i this equatio, istead of directly usig parameter u: I some implemetatios, more tha m test rus may be carried out i a test roud, i order to improve the imperfect testig coverage. Such implemetatios of partitio testig ca be modelled by settig h > u. Otherwise, h = u, see Subsectio 5.2. With this extra parameter h, the radom testig model i Equ. (6) ca be itegrated ito the partitio testig model, see special case i Subsectio 4.4. Evidetly, a fault ca be detected o the test rus i both part ad i part 2. I this case, the commo part must be removed. Sice P par (r, G k ) is the probability that the fault is NOT detected o the m u determiistic test rus, the o-commo part of detectio probability from the determiistic ad radom rus i the r th test roud ca be expressed by P par (r, G k ) + ( P par (r, G k )) P ra (r, G k ) h

16 4 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies where ( P par ((r, G k )) P ra (r, G k ) is the probability that the error group G k is NOT detected o the m u determiistic test rus, but o the u radom test rus. The the total error size removed by the m u determiistic test rus ad the u radom test rus i all the t test rouds ca be expressed by t =( ( ( ) P ra (r,g k )) Θ k(r ) ( ρ r )) r = k ESR(t) = P par (r,g k ) + P par (r,g k ) The fial after t rouds of imperfect partitio testig is the t R(t) = Θ k 0 P par (r,g k ) + P par (r,g k ) k = r = =( ( ( ) P ra (r,g k )) Θ k(r ) ( ρ r )) k (7) Equ. (7) is the geeral iput domai-based growth model which itegrates radom ad partitio testig strategies together. It ca be easily show that the space complexity of Equ. (7) is O(m) ad time complexity is O(t m), which are liearly complex regardig the size of partitio. The geeral model i Equ. (7) cosiders all the mai factors which cotribute to the imperfectio of the implemetatios of testig strategies. It will be show i the ext subsectio that the growth models i Equ. (4) ad (6), as well as other models for partitio ad radom testig with differet imperfect factors are all uified i this model. 4.4 Projectios of the Model This subsectio shows special cases of the model i Equ. (7), which ca be used to cope cocrete implemetatios, i which some mechaisms ca be viewed as perfect. Special case : Geeral radom testig ( ) m If the parameters h = m, ad λ p, the P ra (r, G k ) = Θ k(r ) ((m u) u) e λ q (r ) ad P par (r, G k ) = 0. Equ. (7) is degraded to the geeral growth model for radom testig i Equ. (6), i other words, Equ. (6) is a special case of Equ. (7). Special case 2: Perfect radom testig If testig coverage, fault detectio ad correctio of radom testig are all perfect, that is, u = 0, λ q = 0, ad ρ r- =, we obtai the growth model uder perfect radom testig: R(t) = Θ k 0 + Θ k(r ) (8) k = t r = k = ( ( ) ) m Θ k(r ) Moreover, if there is oly a sigle fault ( = ) ad oly oe test roud is carried out, the we have R() = Θ 0 + Θ 0 ( ( ) m ) Θ 0 = ( Θ 0 ) m Θ 0 (9) Special case 3: Partitio testig with perfect testig coverage If h = u = 0 i Equ. (7), partitio testig coverage is perfect, o radom test rus will be executed ad thus P ra (r, G k ) = 0. The Equ. (7) is degraded to the growth model for partitio testig with perfect testig coverage i Equ. (4), that is, Equ. (4) is a special case of Equ. (7).

17 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies 5 Special case 4: Perfect partitio testig If testig coverage, fault detectio ad correctio of partitio testig are all perfect, that is, u = 0, λ p = 0, ad ρ r- =, Equ. (7) is degraded to R(t) = Θ k 0 + m Θ k(r ) d k k = t r = k = ( ) d k Θ k(r ) Furthermore, if there is oly a sigle fault ad oly oe test roud is carried out, the we have R() = Θ 0 + ( m Θ 0 ) d ( ) Θ 0 = m Θ 0 ( ) d (0) Θ 0 () Comparig the Equ. (8) with (0) ad Equ. (9) with (), we ca see the elemetary differece betwee the growth models for partitio ad radom testig. 5 Applicatio of the Model This sectio applies the proposed iput domai-based growth model to compare the efficiecy of partitio ad radom testig strategies, as well as variats of partitio testig strategies i terms of growth. 5. Comparig Partitio ad Radom Testig Strategies Assume that ICD is partitioed ito m = 00 classes. After the phase-correctio stage the remaiig error size i ICD is DS ICD = 0.0. Now let's apply the growth model i Equ. (7) to study the effects of imperfect testig coverage, detectio probability, ad correctio rate, as well as the effects of error distributios i the partitio. Tab.2 summarises all cases we are goig to study, i which differet parameter values are selected to show the effects of parameters o the growth i Equ. (7). Observatio The effect of the umber of faults ad error distributios For these parameter values give i the first case i Tab. 2, we cosider three error distributios, respectively: (a) (b) (c) The iitial error size evely distributes i all the 00 classes, ad each error class forms a error group, i.e., (m,, d k, Θ k ) = (00, 00,, 0.0); The iitial error size evely distributes i all the 00 classes ad all the error classes form a sigle error group, i.e., (m,, d k, Θ k ) = (00,, 00, 0.0); The iitial error size cocetrates i oe class. Of course, there is oly oe error group which cosists of the uique error class, i.e., (m,, d k, Θ k ) = (00,,, ).

18 6 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies observ- rouds error distributios testig detectio correctio atios t m d k Θ k u h λ p λ q µ ρ r (Fig.5) 2 (Fig.6) 3 (Fig.7) 4 (Fig.8) 5 (Fig.9) 6 (Fig.0) u e µr 0-99 u 0.00 λ p m e µr u 0. λ p /m e µr rus rus u + u (u/m) e µr u + u (u/m) + u (u/m) e µr Tab.2 Selectio of parameters' values for observatios For each of these three error distributios ad for partitio ad radom testig, Fig.5 (a-c) plots respectively the curves for u = 0, u = 25, u = 50, u = 75, ad u = m = 99, calculated accordig to Equ. (7). The curve with u = 0 (the highest oe) correspods to the case with perfect testig coverage ad the curve with u = 99 (the lowest oe) correspods to the case with the worst testig coverage. It ca be see that: For the same testig coverage, if u = 0, the growth by partitio ad radom testig is the same for all three error distributios. If u > 0, curves from partitio testig are slightly higher tha radom testig. If partitio ad radom testig have differet testig coverages (this is more probable i practice), partitio testig ca be worse tha radom testig. For example, a very poor implemetatio of partitio testig with u = 99 is defiitely worse tha a implemetatio of radom testig with u = 50, as ca be see from Fig.5. Comparig Fig.5(b) ad (c) with (a): The umber of faults plays a sigificat role i growth. The larger the umber, the lower are the curves. Comparig Fig.5(a) ad (b) with (c): The distributios of error size affect the growth. The more cocetrate the error size, the higher are the curves. But the effect of error distributios is less sigificat tha the umber of faults.

19 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies error distributio: m errors distributes i m class (a) m errors distributes i m classes (left: partitio testig, right: radom testig) error distributio: error distributes i m class (b) Oe error distributes i m classes (left: partitio testig, right: radom testig) error distributio: error cocetrates i class (c) Oe error cocetrates i oe class (left: partitio testig, right: radom testig) Fig.5 Compariso betwee partitio ad radom testig (λ q = λ p ) Observatio 2 The effect of the imperfect fault detectio I Observatio we have assumed that the error distributio, the testig coverage, the detectio ad correctio parameters have the same values for partitio ad radom testig. I practical implemetatios, these parameters may have differet values, which will thus chage the compariso results. Sice partitio testig strategy uses more iformatio of the program uder test, it ca potetially gai a higher growth tha radom testig by more properly implemetig these mechaisms. I this observatio we use the same parameter values except settig λ p = 0.00 ad λ q = λ p m (λ p is m times lower tha λ q because the iput domai is partitioed ito m classes, which potetially improve the fault detectio probability) ad calculate the accordig to Equ. (7)

20 8 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies agai. The results are plotted i Fig.6 (a-c). It ca be see that there are quite big differeces betwee the curves achieved by partitio testig ad radom testig error distributio: m errors distributes i m class (a) m errors distributes i m classes (left: partitio testig, right: radom testig) error distributio: error distributes i m class (b) Oe error distributes i m classes (left: partitio testig, right: radom testig) error distributio: error cocetrates i class (c) Oe error cocetrates i oe class (left: partitio testig, right: radom testig) Fig.6 Compariso betwee partitio ad radom testig (λ q = λ p m ) Observatio 3 Radom testig ca be much better tha determiistic testig O the other had, a poor implemeted partitio testig ca be much worse tha radom testig. For example, for the same parameter values except assumig that λ p = 0. ad λ q = λ p /m, the radom testig will be much better tha partitio testig, eve if the implemetatio of radom testig has a lower testig coverage (a larger u value), as show i Fig.7. I this example, oly oe of the three error distributios is used for short.

21 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies error distributio: 0 errors, each cotais 5 classes Fig.7 A bad implemetatio of determiistic testig ca be worse tha radom testig Observatio 4 Compariso betwee perfect partitio ad radom testig I the previous comparisos we cosidered always some imperfect factors. Which strategy is better if all factors are perfect? I this case, u = λ q = λ p = µ = 0, ad ρ r =, ad Equ. (7) is degraded to Equ. (8) ad (0), for partitio ad radom testig, respectively. For the three error distributios used i observatios ad 2, Fig.8 shows the compariso. Sice u = 0, there is oly oe curve for each error distributio. It ca be see, that there is little differece betwee perfect partitio ad radom testig. The oly differece occurs whe there is oly oe fault ad all failure causig cases cocetrate i oe class. I this extreme error distributio, partitio testig ca detect ad correct the fault i the first roud ad thus the will achieve oe after the first roud. Radom testig however ca theoretically ever achieve oe. Such extreme curve will ot occur, if ay part of the implemetatio is imperfect..00 =, d k = =, d k = 00 = 00, d k = =, d k = =, d k = 00 = 00, d k = Fig.8 Compariso betwee perfect partitio ad radom testig uder three error distributios 5.2 Comparig Modified Partitio Testig Strategies Partitio testig specified i Def. 2 assumes that oly oe attempt shall be made to geerate a iput case from oe class i each test roud, o matter whether this attempt is successful or ot. Some implemetatios of partitio testig however allow further attempts if the iput case geeratio fails to select a iput case from a give class. Obviously, such variats will reduce the umber u of ucovered classes, but icrease the umber of test rus i a test roud. The questios are thus: Ca such a kid of modificatios improve the? If it ca improve the, how may attempts should be performed at best?

22 20 A Iput Domai-Based Reliability Growth Model ad Its Applicatios i Comparig Testig Strategies This subsectio will use the model i Equ. (7) to achieve these questios. Observatio 5 Modified partitio testig with up to two attempts Assume that the partitio testig coverage is (m u)/m ad oe more attempt (totally w = 2 attempts) will be performed if the first attempt fails to select a iput case from a give class. The, the parameter values i the model i Equ. (7) ca be assiged as follows to model this variat of partitio testig: istead of u, oly u ( u m) classes will ot be tested i a test roud, thus, m u ( u / m) a k = d k = d k u m m ( u m ) ; istead of u, h = u + u 2 / m radom test rus will be carried out i a test roud; istead of m, m + 2 u 2 / m test rus will be executed i a test roud. For u = 25, 50, 75, respectively (a complete parameter value assigmet is give i Tab.2), Fig.9 compares the growth uder three testig strategies: modified partitio testig with up to w = 2 attempts, primary partitio testig (w = ), ad radom testig. Sice a test roud of modified partitio testig cosists of m + 2 u 2 / m test rus, it is't meaigful to compare the i terms of the umber of test rouds performed. Istead, the umber of test rus is used i Fig.9. It ca be see that the modified partitio testig improves the i all three cases for the same umber of test rus. The improvemet is more sigificat for a lower testig coverage.. modified partitio testig with w = 2 modified partitio testig with w = 2 modified partitio testig with w = 2 partitio testig radom testig u = umber of test RUNs performed u = 55 partitio testig radom testig umber of test RUNs performed u = 75 partitio testig radom testig umber of test RUNs performed Observatio 6 Fig.9 Modified partitio testig with up to two attempts Modified partitio testig with more tha two attempts To aswer the secod questio, we geerally cosider that up to w attempts, w, will be performed i order to successfully select a iput case from a give class. The improved umber of ucovered classes i a test roud the ca be expressed by u = u ( u m) w. We ca further derive the ecessary parameter values as follows: a k = d k m u m = d u k m ( u m )w, the actual umber of iput cases which is geerated from the error group G k if oly m u', istead of m, determiistic test rus are performed;