What Do the Software Reliability Growth Model Parameters Represent?

Transcription

1 What Do the Softwae Reliaility Gowth Model Paametes Repesent? Yashwant K. Malaiya and Jason Denton Compute Science Dept. Coloado State Univesity Fot Collins, CO 8523 ABSTRACT Hee we investigate the undelying asis connecting the softwae eliaility gowth models to the softwae testing and deugging pocess. This is impotant fo seveal easons. Fist, if the paametes have an intepetation, then they constitute a metic fo the softwae test pocess and the softwae unde test. Secondly, it may e possile to estimate the paametes even efoe testing egins. These apioivalues can seve as a check fo the values computed at the eginning of testing, when the test-data is dominated y shot tem noise. They can also seve as initial estimates when iteative computations ae used. Among the two-paamete models, the exponential model is chaacteized y its simplicity. Both its paametes have a simple intepetation. Howeve, in some studies it has een found that the logaithmic poisson model has supeio pedictive capaility. Hee we pesent a new intepetation fo the logaithmic model paametes. The polem of a pioi paamete estimation is consideed using actual data availale. Use of the esults otained is illustated using examples. Vaiaility of the paametes with the testing pocess is examined. Intoduction A softwae eliaility gowth model (SRGM) can e egaded to e a mathematical expession which fits the expeimental data. It may e otained simply y oseving the oveall tend of eliaility gowth. Howeve some of the models can e otained analytically y making some assumptions aout the softwae testing and deugging pocess. Some of these assumptions ae simply to keep the analysis tactale. Othe ae moe fundamental in natue and constitute modeling of the testing and deugging pocess itself. This eseach was suppoted in pat y a BMDO funded poject monitoed y ONR and in pat y an AASERT funded poject. An analytically otained model has the advantage that its paametes have specific intepetations in tems of the testing pocess. An undestanding of the undelying meaning of the paametes gives us a valuale insight into the pocess.. If we know how a paamete aises, we can estimate it even efoe testing egins. Such apioi values when estimated using past expeience, can e used to do peliminay planning and esouce allocation efoe testing egins [3]. 2. The expeience with use of SRGMs suggests that in the eginning of testing, the initial test data yields vey unstale paamete values and sometimes the paamete values otained can e illegal in tems of the model. In such a situation, values estimated using static infomation can seve as a check. They can also e used to stailize the pojections adding to the infomation otained y the dynamic defect detection data. 3. Sometimes iteative techniques ae used to estimate the paamete values. The values otained can depend on the initial estimates that ae equied y numeical computation. Use of apioivalues as the initial estimate would initiate the seach in a egion close to the values sought. 4. Paametes that have an intepetation chaacteize the testing and deugging pocess quantitatively. Thei values can give us an insight into the pocess. They may help answe the questions aout how the inheent defect density can e educed o how testing can e made moe efficient. This pape examines the paametes of the exponential and the logaithmic models. We pesent a new model fo estimating the softwae defect density. A new intepetation fo the paametes of the logaithmic model is pesented. Techniques fo estimation of paametes ae pesented.

2 The quantitative pocess chaacteistic values used in this pape ae taken fom the data epoted y eseaches. The values depend on the pocess used and may e diffeent fo diffeent pocess. Thus the models pesented hee should e ecaliated using the pio expeience in a specific oganization using a specific pocess. Simila methods have een in use fo pojecting hadwae eliaility measues whee they have een found to e vey useful even though the esults ae only appoximate. The next section analytically pesents the intepetations of the paametes of the two models. Section 3 discusses estimation of paametes. Some osevations on paamete vaiations ae pesented next followed y the conclusions. 2 Exponential and Logaithmic SRGMs In this pape we will conside two two-paamete models. The exponential model, in the fomulation used hee is also temed Musa s asic execution model [7]. It is given y µ(t) =β E (,e,βe t ) () whee µ(t) is the mean value function and β E and βe ae the two model paametes. Fa mentions that this model has had the widest distiution among the softwae eliaility models [4]. Musa [7] states that the asic execution model geneally appeas to e supeio in capaility and applicaility to othe pulished models. Some of the othe models ae simila to this model. The logaithmic model is the othe model consideed hee. It is also temed Musa-Okumoto logaithmic poisson Model. Itisgiveny µ(t)=ln( + β L t) (2) whee and βl ae the two model paametes. Fa states that the logaithmic model is one of the models that has een extensively applied [4]. This is one of the selected models in the AIAA Recommended Pactice Standad [4]. Musa [7] wites that the logaithmic model is supeio in pedictive validity compaed with the exponential model. In a study using 8 data sets fom divese pojects, Malaiya et al. evaluated the pediction accuacy of five two-paamete models [4]. They found that the logaithmic model has the est oveall pediction capaility. Using ANOVA, they found that this supeioity is statistically significant. All softwae eliaility gowth models (SRGMs) ae appoximations of the eal testing pocess, thus none of the models can e egaded to e pefect. Howeve these two models possess simplicity and have een found to e applicale fo a vaiety of softwae pojects. Thus these two models have een chosen fo this study. 2. Deivation of the Exponential model Hee we give a deivation of the exponential model that gives its elationship with the test pocess. This will allow us to intepet the meaning of the two paametes of this model. Let N(t) e the expected nume of defects pesent in the system at time t. Let T s e the aveage time needed fo a single execution, which is vey small compaed with the oveall testing duation. Let k s e the expected faction of existing faults exposed duing a single execution. Then dn(t) T s =,k s N(t) (3) dt It would e convenient to eplace T s with something which can e easily estimated. Let T L e the linea execution time [7] which is defined as the total time needed if each instuction in the pogam was executed once and only once. It is given y T L = I s:q x whee I s is the nume of souce statements, Q x is the nume of oject (machine level) instuctions pe souce instuctions and is the oject instuction execution ate of the compute eing used. Let us define a new paamete K = k s T L T s whee the atio T L Ts will depend on the pogam stuctue. Using this, equation 3 can e ewitten as dn(t) =, K N(t) (4) dt T L The pe-fault hazad ate as given in equation 4 is K=T L. Thus K, temed fault exposue atio [7] diectly contols the efficiency of the testing pocess. If we assume that K is time invaiant, then the aove equation has the following solution: N(t) =N e,k T L t whee N is the initial nume of defects. This may e expessed in a moe familia fom as follows: N, N(t)=N (,e,t K T L) 2

3 The left side of this equation coesponds to µ(t), as given y equation. Thus the paametes β and β have the following intepetations: β E = N ; and β E = K T L (5) Expeimental data suggests that K actually vaies duing testing [5]. We will denote the constant equivalent as detemined y the application of the exponential model y ˆK. 2.2 Implications of the Logaithmic model The logaithmic model has een found to have vey good pedictive capaility in many cases. Howeve to deive it fom asic consideations equies one to make some assumptions as done in efeences [7], [6] and [5]. We show elow that if the logaithmic model descies the test pocess, the fault exposue atio is vaiale. We can assume that this vaiation depends on the test pocess phase which is given y the density of defect pesent at any time duing testing []. This leads us to an intepetation of the model paametes as shown in the next section. Reaanging equation 2 fo the mean value function µ(t), we can wite, Also, e µ(t) =(+β L t) (6) λ(t) = βl βl +β L t Sustituting fo ( + β L t) fom equation 6 e,µ(t) λ(t) = βl = βl e,n,isd(t) (7) Whee D(t) is the defect density at time t. Fom equation 4, the fault exposue atio is given y K(t) =T L λ(t) N(t) Using equation 7 to sustitute fo λ(t),weget Hee we have expessed the fault exposue ation K as a function of defect density D instead of time t. Hee the paametes α and α ae given y, α = βl βl Q x e, N () α I s = β L () The equations and ae used in the next section to pesent a new intepetation fo the logaithmic model. 2.3 Intepetation of the Logaithmic Model Paametes An intepetation of the paametes fo the exponential model is quite staightfowad. As t!, accoding to equation, µ(t)! β E. Musa states that duing deugging only aout 5% new faults ae intoduced. Thus β E is slightly geate than the initial nume of faults, and can e taken to epesent the total nume of faults that will e encounteed. The paamete β E is the time scale facto, o the pe fault hazad ate, as given y equation 5. A geate challenge is posed y the logaithmic model paametes. Hee we pesent a new intepetation ased on the analysis pesented in sec 2.2. Fom equation we can wite β L I = s α Sustituting this in equation and solving fo β L,we get β L α α = e Nα Is Q I s Let us now detemine the meaning of α and α ; in tems of the test pocess. Fig. gives the vaiation of the fault exposueatio K in tems of defect density. Let us denote y D min the density at which K min, the minimum valueofk, occus. Takinga deivativeofk with espect to D using equation 9 and equating it to zeo, we get T L K(t) = I s D βl β L e, N,IsD(t) (8) = We can ewite this as T L I s D βl βl e, K(D) = α D eα D! N e, IsD(T ) (9) 3 which yields, α D 2 eα D + α D eα D α = D min = α

4 K,7 5 K min logaithmic model? exponential model D min 5 Defect density D Figue : Vaiation of Fault Exposue Ratio with defect density and the coesponding value of K is given y K α e min = D min Thus oth α and α depend on the test pocess, α K mind min = ; and α = (2) e D min Using equations, and 2, we otain this intepetation of the logaithmic model paametes. = I s D min (3) β L K D min = Q x I s e e D min (4) Hee D is the initial defect density. Equation 3 states that is popotional to the softwae size and is contolled y how test effectiveness vaies with defect density. The paamete β depends on K min, the minimum value of the fault exposue atio. It is also dependent on the atio D D min It should e noted that β E and βl,andβe and βl have the same dimensions. The Tale elow compaes the intepetations of the paametes of the two models compaed hee. Value Scale Time Scale Dimension Defects Pe unit time Exponential β E N = D I s β E = ˆK Logaithmic = D mini s β L = K min T e L T L D,D min D min Tale : Compaison of model paamete intepetations 3 Factos affecting Defect Density Because the exponential model paametes ae explained in a simple way, the polem of a pioi estimation of its paametes is also easie. Assuming the nume of new faults intoduced duing the deugging pocess is small, β E can e taken to e appoximately equal to the initial nume of defects, N. It has een oseved that fo a specific development envionment fo the same softwae development team, the defect density encounteed is aout the same, fo the same development/testing phase [9]. This allows the initial defect density to e estimated with easonale confidence. Hee we pesent a facto multiplicative model to estimate the initial defect density and hence N. A facto multiplicative model assumes that the quantity to e estimated is influenced y seveal independent causes and the effect of each cause can e suitaly modeled y a multiplicative facto. Such models have also een used to estimate hadwae failue ates. Seveal linea additive models fo estimating the nume of defects have also een poposed, they have the disadvantage that they can poject zeo o negative nume of defects. The models y Agesti and Evanco [2], Rome La [22] and THAAD [6] ae facto multiplicative like ou model. A peliminay vesion of ou model [2] is eing implemented in the ROBUST softwae eliaility tool []. Ou model, pesented elow, has the following advantages:. It can e used when only incomplete o patial infomation is availale. The default value of a multiplicative facto is one, which coesponds to the aveage case. 2. It takes into account the phase dependence as suggested y Gaffney [5] 3. It can e ecaliated y choosing a suitale constant of popotionality and e efined y using a ette model fo each facto, when additional data is availale. The model is given y D = C:F ph :F pt :F m :F s :F (5) whee the five factos ae the phase facto F ph, modeling dependence on softwae test phase, the pogamming team facto F pt taking in to account the capailities and expeience of pogammes in the team, the matuity facto F m depending on the matuity of the softwae development pocess, the stuctue facto F s, depending on the stuctue of the softwae unde development and equiements volatility facto F, which depends on 4

5 the changes in the equiements. The constant of popotionality C epesents the defect density pe thousand souce lines of code (KSLOC). We popose the following peliminay su-models fo each facto. 3. Phase Facto (F ph ) The nume of defects pesent at the eginning of diffeent test phases is diffeent. Gaffney [5] has poposed a phase ased model that uses the Rayleigh cuve. Hee we pesent a simple model using actual data epoted y Musa et al. [7] (thei tale 5.2) and the eo pofile pesented y Piwowaski et al. [2]. In Tale 2 we take the default value of one to epesent the eginning of the system test phase. With espect to this, the fist two columns of Tale 2 epesent the multiplies suggested y the numes given y Musa et al. and Piwowaski et al.. The thid column pesents the multiplies assumed y ou model. Test Multiplie phase Musa et al. Piwowaski Ou Model Unit Susystem Insuf. data System (default) Opeation Tale 2: Phase Facto (F ph ) 3.2 The Pogamming Team Facto (F pt ) The defect density vaies significantly due to the coding and deugging capailities of the individuals involved [24] [25]. The only availale quantitative chaacteization is in tems of pogammes aveage expeience in yeas, given y Takahashi and Kamayachi [24]. Thei model can take into account pogamming expeience of up to 7 yeas, each yea educing the nume of defects y aout 4%. The data in the study epoted y Takada et al [25] suggests that pogammes can vay in deugging efficiency y a facto of 3. In a study aout the PSP pocess [2], the defect densities in a pogam witten sepaately y 4 pogammes wee evaluated. Fo aout 9% of the pogammes, the defect density anged fom aout 5 to 25 defects/ksloc. This suggests that defect densities due to diffeent pogamming skills can diffe y a facto of 5 o even highe. Thus we popose the model in Tale 3. The skill level may depend on factos othe than just the expeience. The PSP data suggests while thee may e some dependence on expeience, pogammes with the same expeience can have significantly diffeent defect densities. Team s Aveage Skill level Multiplie High.4 Aveage (default) Low 2.5 Tale 3: The Pogamming Team Facto (F pt ) 3.3 The Pocess Matuity Facto (F m ) This facto takes into account the igo of softwae development pocess at a specific oganization. This level, as measued y the SEI Capaility Matuity Model, can e used to quantify it. Hee we assume level II as the default level, since a level I oganization is not likely to e using softwae eliaility engineeing. Kolkhust [9] assumes that fo deliveed softwae, change fom level II to level V will educe defect density y a facto of 5. Howeve, Keene [3] suggests a eduction in the inheent defect density y a facto of 2 fo the same change. Jones [7] suggests an impovement y a facto of 4 in potential defects and a facto of 9 in deliveed defects fo changing fom level II to level V. Hee we use the numes suggested y Keene to popose the model given in Tale 4. SEI CMM Level Multiplie Level.5 Level 2 (default) Level 3.4 Level 4. Level 5.5 Tale 4: The Pocess Matuity Facto (F m ) 3.4 The Softwae Stuctue Facto (F s ) This facto takes into account the dependence of defect density on language type (the factions of code in assemly and high level languages), pogam complexity, 5

6 modulaity and the extent of euse. It can e easonaly assumed that assemly language code is hade to wite and thus will have a highe defect density. The influence of pogam complexity has een extensively deated in the liteatue [8]. Many complexity measues ae stongly coelated to softwae size. Since we ae constucting a model fo defect density, softwae size has aleady een taken into account. Thee is some evidence that fo the same size, modules with significantly highe complexity ae likely to have a highe nume of defects. Howeve, futhe studies ae needed to popose a model. It is known that module size influences defect density with a module [2]. Howeve in a softwae system consisting of modules, the vaiaility due to diffeent lock sizes may cancel out if we ae consideing the aveage defect density. The influence due to euse will depend on its extent, the defect-contents of eused modules and how well the eused modules implement the intended functionality. As this time, we popose a model fo F s depending on language use, and allow othe factos to e taken in to account y caliating the model. F s = + :4a (6) whee a is the faction of the code in assemly language. Hee we ae assuming that assemly code has 4% moe defects []. 3.5 The Requiements Volatility Facto (F ) It is common fo the equiements specification to change. If the equiementschange while the softwae is eing developed and deugged, the softwae will have a highe defect density with espect to the evised equiements. Musa [8] has suggested a new metic temed equiements volatility. Takahashi and Kamayachi [25] suggest that changes in the specifications can cause a 2-3% change in the defect density. An evaluation of the equiements volatility can lead us to an estimate of the oveall change in the equiements specification which may linealy affect the defect density. We ae looking fo suitale data to develop a model fo the F facto. 3.6 Caliating and using the defect density model The model given in equation 5 povides an initial estimate. It should e caliated using past data fom the same oganization. Caliation equies application of the models using availale data in the oganization and detemining the appopiate values of the supaametes. Since we ae using the eginning of the susystem test phase as the default, Musa et al. s data suggests that the constant of popotionality C can ange fom aout 6 to 2 defects pe KSLOC. Fo est accuacy, the past data used fo caliation should come fom pojects as simila to the one fo which the pojection needs to e made. Some of indeteminacy inheent in such models can e taken into account y using a high estimate and a low estimate and using oth of them to make pojections [23]. Example : Fo an oganization, the value of C has een found to e etween 2 to 6. A poject is eing developed y an aveage team and the SEI matuity level is II. Aout 2% of the code is in assemly language. Othe factos ae assumed to e aveage. Then the defect density at the eginning of the susystem test phase can ange etween 2 2:5 ( + :4:2)= 32:4 /KSLOC and 62:5 ( + :4:2) = 43:2 /KSLOC. 4 Estimation of SRGM Paametes 4. Estimation of β E and βe Since β E epesents the total nume of faults that will e detected, it can e estimated using the estimate fo the initial defect density, D. As suggested y Musa et al., we can assume that aout 5% new defects would e ceated duing deugging. Thus we can use this model fo β E. β E = :5D I s (7) Estimation of β E equies the use of the equation βe = ˆK T L whee ˆK is the oveall value of the fault exposue atio duing the testing peiod. The value of ˆK is some times appoximated y 4:2,7 failues pe fault, the aveage value detemined y Musa et al. [7]. Li and Malaiya [] have suggested that ˆK vaies with the initial defect density and have given this expession to estimate ˆK: ˆK = :2,6 D e :5D whee D is the defect density pe KSLOC. The paamete values have een computed hee y fitting the values fo fault exposue atio fo seveal pojects epoted y Musa et al. [7]. Example 2: Let us assume that the initial defect density fo a poject has een estimated to e 25 faults/ksloc and the softwae size is 54 lines. The pogam is tested on a CPU that uns at 4 MIPS and each souce instuction compiles into 4 ojects instuctions. Then the estimated values ae β E = :5255:4 = 4:7 (8) 6

7 ˆK = :2,6 e :525 = :675,7 (9) 25 β E = :675,7 = 3:,5 (2) 544 4;; 4.2 Estimation of Logaithmic Model Paametes Estimating the paamete values fo the logaithmic model is a significant challenge. We can take one of two possile appoaches. In the fist appoach we can fist estimate the paametes of the exponential model and then compute and βl. In the second appoach we can calculate and βl fom the intepetation intoduced in section Estimation though β E and βe The paametes of the exponential model β E and βe ae easily intepeted and estimated. Hee we use the osevation that fo a given data set, thee is some elationship etween β E and βl,andβe and βl [3]. This elationship can e used to estimate the paametes of the logaithmic model once the exponential model paametes have een estimated. To otain this elationship, let us assume that oth models poject the same µ(t f ) whee t f is the end of the testing peiod. Let the nume of defects emaining at time t f e N α ;α >. Fo example, if testing finds and emoves 9% of all the faults, then α =. Then µ(t f )=N, N α =N (, α ) (2) Fo the exponential model equation 2 will give, β E (,e,βe t f )=N (, α ) since N β E, we can ewite this equation as t f = ln(α) β E (22) using the logaithmic model we can wite equation 2 as ln(+ βl t f )=N (, α ) which can e eaanged as 2 =, α β E t (, β f = 4e L α ), 5 (23) Equating the ight hand side of equations 22 and 23, and eaanging we get β E β L ln β E ln(α)+ (24) 3 Let us now assume that in time t f the failue intensity also declines y facto α. Thus accoding to the exponential model, β E β E e,βe t f = βe βe α which can e solved fo to give β E = t f ln(α) (25) Similaly the logaithmic model gives βl + β L t f which can e witten as = βl βl α Fom equation 25 and 26 we otain β L = t f (α,) (26) β L β E = α, ln(α) Using equation 27, we can ewite equation 24 as β E =, ln(α) α (27) (28) Thus if we know α and the values fo β E and βe,we can calculate β L using equation 27 and βl using equation 28. Example 3: Fo a softwae system unde test, the paametes β E and βe have een estimated to e 42 and :35,4 espectively. Testing will e continued until aout 92% of all faults have een found. That gives α = = 2:5 (29),92 The equation 27 gives β L β E = 4:55 i:e: β L = 4:55:35,4 = :59,4 and equation 24 gives β E (3) = 2:75 i:e: β L 42 = = 5:6 (3) 2: Diect Estimation of and βl An altenative to the aove method is to use the intepetation of and βl in tems of D min and K min as given y equations 3 and 4. A easonale estimate fo K min 7

8 is :5,7 as suggested y the data given y Musa et al. [7] (thei Tale 5.6). As estimation of D min, the defect density at which the minimum value of K occus is hade to estimate. Fist the cuve fo K, as shown in figue has a vey flat minimum. That can make exact detemination of D min had in the pesence of nomal statistical fluctuations. Secondly, the vaiation in K depends on the testing stategy used. Availale data sets suggest the following.. If the initial defect density D is less than pe KSLOC, the value of D min is in the neighohood of 2 defects/ksloc. 2. Howeve if D is highe, the esulting value of Dmin is also highe. in many cases, taking D min = D =3 yields a suitale fist estimate. Example 4: Fo the T2 data [7], the initial defect density is 8.23 defects/ksloc and the size is appoximately 6.92 KSLOC (27.7K oject lines). The instuction execution ate is not given in [7], howeve we can otain the value of T L using availale infomation. Since Musa et al. have given the value of ˆK as 2:5,7 and the value of β E can e calculated to e :42,5, the value of T L is 2:5,7 =:42,5 = :5,2. We will estimate the values of the logaithmic model paametes assuming D min = 2andK min = :5,7. Fom equations 3 and 4 we have these estimates, and = I sd min = 6:922 = 3:84 (32) β L = K min e e Q x I s = :5,7 2:72 = 2:24,4 D D min (33) :5 8:23 e 2,2 Fitting of actual test data yields the two values as 7.26 and 2:,4. Consideing the fact that the few ealy points in the test data can often yield values that can e easily off y an ode of magnitude o can e illegal (negative), the estimates ae quite good. 4.4 Vaiaility of the paamete values Fo a give data set, if we use the patial data set fom eginning to some intemediate point in testing, the paamete values ae found to e diffeent fom the final values. We have investigated the incemental vaiation of the values detemined as testing continues. In the eginning the values can change apidly ut late they 6 β L 4 β E Defects found Figue 2: Vaiation of β E and βl stat settling towads the final value. Fo pactically all data sets, the values of β E and βl ise with testing time wheeas fo β E and βl the values fall. The typical ehavio is illustated y the plots fo the T data-set [7]. Figue 2 shows that while the value of β E keeps ising, βl appeas to stailize in the late phases of testing. This suggests that the logaithmic model descies the undelying pocess ette. Figue 3 shows how β E and βl vay as testing pogesses. Both show a downwad tend, howeve the cuve fo β L appeas to e stailizing. Figue 4 shows the peaks in and β L which ae lagely due to changes in the eliaility gowth ehavio. They ae often caused y changes in the testing stategy o y switching to a diffeent test suite. Fotunately often the two paametes ae petued in the opposite diections, thus minimizing the effect. The pesence of a significant tend in the plots fo the exponential model seems to suggest that it does not model the testing pocess as well as the logaithmic model. All SRGMs ae simplified models and hence descie the eliaility gowth appoximately. The a pioi estimates of these models can e ette than the values otained in the ealy phases of testing, ut can not e expected to e as accuate as the final values otained using actual test data. 5 Concluding Remaks In this study we have pesented methods to estimate the paametes of the exponential and logaithmic models. We have poposed an empiical model fo estimating the defect density, one that woks when complete data is not availale and can e easily efined as moe is leaned aout the softwae development pocess. A new intepetation fo the paametes of the logaithmic model has een poposed and we have shown how it can e used to estimate the values. An altenative ap- 8

9 β L β E Defects found Figue 3: Vaiation of β E and βl poach is to fist estimate the paametes fo the exponential model and then use them to estimate the logaithmic model paametes. The methods pesented hee can significantly impove the accuacy of the pojections duing the ealy phases of testing. The accuacy of the esults will depend on caeful caliation of the models using data fom ealie pojects that have used a simila pocess. Futue wok includes a detailed analysis of the specific esults fo the the data sets availale. Two methods fo the estimation of the logaithmic model paametes have een pesented and futhe eseach is needed in ode to make ecommendations as to the pedictive aility of each. We also need to investigate the sensitivity of the pojections due to vaiation in the paamete values. 6 Acknowledgment We would like to thank John Musa fo his suggestion to include the equiements volatility facto β L Defects found Figue 4: Vaiation of and βl (escaled) Refeences [] J.R. Adam, Softwae Reliaility Pedictions ae Pactical Today, Poc. IEEE Ann. Symp. on Softwae Reliaility, Coloado Spings, May 989. [2] W. W. Agesti and W. M. Evanco, Pojecting Softwae Defects fom Analyzing Ada Designs, IEEE Tans. Softwae Engineeing, Nov. 992, pp [3] G.F. Cole and S.N. Keene, Reliaility and Gowth of Fielded Softwae, Reliaility Review, Mach 994, pp [4] W. Fa, Softwae Reliaility Modeling Suvey, in Handook of Softwae Reliaility Engineeing, Ed. M. R. Lyu, McGaw-Hill, 996, pp [5] J. Gaffney and J. Pietolewicz, An Automated Model fo Ealy Eo Pediction in Softwae Development Pocess, Poc. IEEE Softwae Reliaility Symposium Coloado Sping, June 99. [6] M. Gechman and K. Kao, Tacking Softwae Reliaility and Reliaility with Metics, Poc. IS- SRE Industy Repots, 994. [7] C. Jones, Softwae Benchmaking We Document, IEEE Compute, Oct softwae//softwae.htm. 9

10 [8] T. M. Khoshgofta and J. C. Munson, The Line of Code Metic as a Pedicto of Pogam Faults: a Citical Analysis, Poc. COMPSAC 9, pp [9] B.A. Kolkhust, Pespectives on Softwae Reliaility Engineeing Appoaches found in Industy Poc. ISSRE Industy Repots, 994. [] N. Li and Y.K. Malaiya ROBUST: A Next Geneation Softwae Reliaility Engineeing Tool Poc. IEEE Int. Symp. on Softwae Reliaility Engineeing, pp , Oct [] N. Li and Y.K. Malaiya, Fault Exposue Ratio: Estimation and Applications Poc. IEEE Int. Symp. Softwae Reliaility Engineeing 996 pp [2] N. Li, Measuement and Enhancement of Softwae Reliaility Though Testing, Ph.D. dissetation, Coloado State Univesity, 997. [22] Rome La, Methodology fo Softwae Reliaility Pediction and Assessment, Tech Repot RL-TR , Vol. and 2, 992. [23] N.F. Schneidewind, Minimizing isk in Applying Metics on Multiple Pojects, Poc. IEEE Int. Symp. Softwae Reliaility Engineeing 992, pp [24] M. Takahashi and Y. Kamayachi, An Empiical Study of a Model fo Pogam Eo Pediction, in Softwae Reliaility Models, IEEE Compute Society, 99. pp [25] Y. Tokada, K. Matsumoto and K. Toii, A pogamme Pefomance Measue ased on Pogamme State Tansitions in Testing and Deugging Pocess, Poc. Intenational Confeence of Softwae Engineeing, 994, pp [3] Y. K. Malaiya, Ealy Chaacteization of the Defect Removal Pocess, Poc. 9th Annual Softwae Reliaility Symposium, May 99, pp [4] Y. K. Malaiya, N. Kaunanithi and P. Vema, Pedictaility of Softwae Reliaility Models, IEEE Tans. Reliaility, Deceme 992, pp [5] Y. K. Malaiya, A. von Mayhause and P. K. Simani, An Examination of Fault Exposue Ratio, IEEE Tans. Softwae Engineeing, Nov. 993, pp [6] J. D. Musa and K. Okumoto, A Logaithmic Poisson Execution Time Model fo Softwae Reliaility Measuement, Poc. 7th Int. Conf. on Softwae Engineeing, 984, pp [7] J. D. Musa, A. Iannino, K. Okumoto, Softwae Reliaility - Measuement, Pediction, Applications, McGaw-Hill, 987. [8] J. D. Musa, pesonal communications, 997. [9] G.A. Kuge, Validation and Futhe Application of Softwae Reliaility Gowth Models, Hewlett- Packad Jounal, Apil 989, pp [2] Pesonal Softwae Pocess We Document, Canegie Mellon Univesity, Results.htm, Rev. 5 Sept. 997 [2] P. Piwowaski. M. Oha and J. Causo, Coveage measuement Expeience duing Function Test, Poc. ICSE, 993, pp