ADVANCED SOFTWARE ENGINEERING COMP 3705 Exercise Usage-based Testig ad Reliability Versio 1.0-040406 Departmet of Computer Ssciece Sada Narayaappa, Aeliese Adrews Versio 1.1-050405 Departmet of Commuicatio Systems Thomas Theli, Caria Adersso
1. Itroductio I this laboratory assigmet you should try to estimate the remaiig umber of faults i a software program based o the umber of faults that has bee foud i operatio. More iformatio about software reliability predictio may be foud i [Wood96], although it should be possible to solve these assigmets without further studies. You will use failure cout reliability models ad models which measure time betwee failure. The secod part of the laboratory, you will implemet a usage model of a program ad compare the test cases derived from a use model with test cases derived with other testig techiques. 2. Learig Objectives The exercise aims at givig a uderstadig of how usage models ca be desiged ad the basics of reliability models i software egieerig. 3. Preparatio Assigmet 1: Read the followig: Chapter 12.0-12.7 i [Burstei03]. Appedix. Slides from the lecture Reliability. Optioal: pp. 470-476 i [Sommerville01] or pp. 568-574 i [Sommerville04]. Assigmet 2: Choose a program you have developed ad tested previously, for example, i aother course. It is importat that you have tested the program before. Recall the mai fuctioality of the program ad: a) if you have the test cases, brig them to the lab sessio. b) if you do ot have the test cases, desig at least 10 test cases usig a method of your choice. Assigmet 3: Desig a usage model o paper for the software program you tested i Assigmet 2. The usage model should cosist of states ad trasitios. Each state should have a ame ad each trasitio should have a label ad a probability value. If the model becomes too complex or too large, you oly have to develop a part of the model. 4. Reliability Goel-Oumoto Goel-Oumoto Subjective estimatio Assigmet 4 (computer): Use the file godata1.txt (load godata1.txt). The data is from a Brazilia Electroic Switchig System, TROPICO R-1500 for 1500 subscribers. Assigmet 5 (computer): Plot the cumulative umber of faults that has bee foud for every wee (e.g. for wee 3, plot the sum of the faults that has bee foud i wee 1, 2, ad 3). Which type of tred ca you idetify? 2
Assigmet 6: Imagie that you are resposible for the product. Estimate subjectively how may failures will be foud i the future. More specifically, estimate the followig: N future : the total umber of failures that will be foud i a period of time that is 50% loger tha the time that has bee plotted. (That is, the failures i the data set that you have plotted should be icluded i this figure). N tot : the total umber of umber of failures that will have bee foud after a ifiite period of time (icludig the failures i the data set you have plotted). Fill i the table i Sectio 6. Goal-Oumoto Estimatio The Goel-Oumoto model says that the expected umber of faults foud at time t could be described by mt () a( 1 e bt ), see Appedix A. The model is based o a o-homogeous poisso process. It assumes that the distributio of the umber of faults foud at time t is poisso with mea value m(t). Assigmet 7: Plot m(t) with pe ad paper or with Matlab. Assigmet 8: Give iterpretatios of a ad b. Assigmet 9 (computer): Use the file go.m This is a Matlab file which ca be used to estimate a ad b i the Goel-Oumoto model. The commad should be used i the followig way: load godata1.txt [a b] go(godata1(:,2)), where data(:,2) is a colum vector with the umber of failures foud i every time period (of equal legth), e.g. [1 7 2 1 3 28...]. The estimatio is doe as a Maximum lielihood estimatio, see Appedix A. Assigmet 10 (computer): Estimate a ad b with the Matlab commad. Assigmet 11: Estimate N future ad N tot based o the Goel-Oumoto model. Fill i the table i Sectio 6. Goal-Oumoto Real data Assigmet 12 (computer): Use the data file godata2.txt. The data set you received the last time was a trucated versio of this data. That is, this data set cotais more data poits ad it is ow possible to compare the estimates to real data. 3
Assigmet 13: Determie 1 the real values of N future ad N tot. Fill i the table i Sectio 6. 5. Reliability Jelisi-Morada Jelisi-Morada Subjective estimatio Assigmet 14 (computer): Use the file jmdata1.txt. Assigmet 15: Imagie that you are resposible for the product. Estimate subjectively how may failures will be foud i the future. More specifically, estimate the followig: N future : the total umber of failures that will be foud i a period of time that is 50% loger tha the time that has bee plotted. (That is, the failures i the data set that you have plotted should be icluded i this figure). N tot : the total umber of umber of failures that will have bee foud after a ifiite period time (icludig the failures i the data set you have plotted). Fill i the table i Sectio 6. Jelisi-Morada Estimatio The Jelisi-Morada model says that the failure itesity after the (i-1):th failure has occurred could be described by λ i φ( N ( i 1) ), see Appedix B. A assumptio of the model is that times betwee failures are idepedetly expoetially distributed. Assigmet 16: Plot λ i with pe ad paper or with Matlab. Assigmet 17: Give iterpretatios of N ad φ. Assigmet 18 (computer): Use the file jm.m This is a Matlab file which ca be used to estimate N ad φ i the Jelisi-Morada model. The commad should be used i the followig way: load jmdata1.txt [N φ ] jm(jmdata1(:,2) ), where data(:,2) is a colum vector with the times betwee failures (executio time i secods), e.g. [1 3139 2 10 3 4...]. The estimatio is doe as a Maximum lielihood estimatio, see Appedix B. Assigmet 19 (computer): Estimate N ad φ with the Matlab commad. 1. It is ot really possible to decide N tot (why?), but you ca estimate it. 4
Assigmet 20: Estimate N future ad N tot based o the Jelisi-Morada model. Fill i the table i Sectio 6. Jelisi-Morada Real data Assigmet 21 (computer): Use the data file jmdata2.txt. The data set you received the last time was a trucated versio of this data. That is, this data set cotais more data poits ad it is ow possible to compare the estimates to real data. Determie the real values of N future ad N tot. Fill i the table i Sectio 6. 6. Estimatio tables N future N tot Subjective estimatio GO estimatio Actual values N future N tot Subjective estimatio JM estimatio Actual values 7. Report The purpose of the report is to dicuss the result of the exercise ad related topics. Write the ame of exercise, your ames, group umber ad email addresses o the first page of the report. The size of the report should be 4-5 A4 pages (icludig the first page). 1. I the first part of the report (1-2 A4-pages), discuss the results of the reliability ad the usage modellig. Discuss the assigmets ad: Reliability: Decide which method that was best i this case - subjective estimatio or estimatio based o the reliability models. How do you compare the methods? What decisios durig software developmet ca be tae based o the estimatio results? Compare ad discuss the Goal-Oumoto model with the Jelisi-Morada model. What are the problems of usig these id of models? 2. I additio, the discuss the followig related topics i the report (1-2 A4 pages): [Burstei03, chapter 12, pages 436-437] Exercise 2. [Burstei03, chapter 12, pages 436-437] Exercise 6. [Burstei03, chapter 12, pages 436-437] Exercise 8. 5
8. Appedix Appedix A: The Goel-Oumoto model Nohomogeeous Poisso process models have over the years bee successfully used i studyig hardware reliability problems, ad durig the last decades the applicatio of NHPP models to software reliability aalysis has also bee implemeted. The cumulative umber of software failures up to time t, N(t), ca be described by a NHPP ad may existig software reliability models belog to this class. N(t), modelled by NHPP, follows a Poisso distributio with parameter m(t), that is, the probability that N(t) is a give iteger is expressed by [ mt ()] P{ N() t } -----------------e mt (),! 012,,, where m(t) is called the mea value fuctio ad describes the expected cumulative umber of failures i the iterval from 0 to time t. Usually, m(t) cotais some uow parameters. The estimatio of them is geerally carried out by usig the method of maximum lielihood or the method of least squares [Xie91]. If you deote by i the umber of faults detected i time iterval betwee s i-1 ad s i, where 0 s 0 < s 1 < s 2 <...< s ad s i, i 0, are the ruig times sice the software testig begis, the lielihood fuctio for the NHPP model with mea value m(t) is the followig L ( 1, 2,, ) [ ms ( i ) ms ( i 1 )] i exp( { ms ( i ) ms ( i 1 )}) --------------------------------------------------------------------------------------------------------------- i! Usually, umerical methods have to be used i solvig the lielihood equatios. The geeral assumptios of the Goel-Oumoto model are 1. The cumulative umber of faults detected at time t follows a Poisso distributio. 2. All faults are idepedet ad have the same chace of beig detected. 3. All detected faults are removed immediately ad o ew faults are itroduced. The Goel-Oumoto model assumes that the failure process is modelled by a NHPP model, ad says that the expected umber of faults foud at time t could be described by the mea value fuctio mt () a( 1 e bt ) a total umber of faults i the program. b costat of proportioality. 6
Isertig the value of mt ( i ) for all i ito the lielihood fuctio for the geeral NHPP model, you get that the lielihood fuctio of the parameters a ad b is L ( 1, 2,, ) ae bs i 1 e bs i i [ ( )] a e bs i 1 e bs i exp ( ) --------------------------------------------------------------------------------------------------------------------- i! Deoted by is the total umber of detected faults, that is i Estimatio of the model parameters a ad b ca be carried out by maximizig the lielihood fuctio. The atural logarithm of this lielihood fuctio is ll l ae bs i 1 e bs i i [ ( )] ae bs i 1 e bs i exp ( ) --------------------------------------------------------------------------------------------------------------------- i! l( i a ) i e bs i 1 + e bs i l( ) ae bs i 1 e bs i ( ) l(! ) i By taig the partial derivatives of the log-lielihood fuctio with respect to a ad b ad equatig them to zero, that is ll ----------- a ll ----------- 0 b you get two equatios for solvig a ad b. The partial derivatios are as follows ll ----------- a -- 1 e bs ( ) a ll s i e bs i s i 1 e bs i 1 ----------- b i ------------------------------------------------ e bs i 1 e bs as e bs i 7
The estimate of a may the be foud as a ------------------- 1 e bs ad after isertig this expressios ito the secod of the partial derivatios above, the estimate of b ca be calculated by solvig the followig s i e bs i s i 1 e bs i 1 ( ) s e bs i -------------------------------------------------- e bs ------------------- i 1 e bs 0 e bs i 1 Appedix B: The Jelisi-Morada model Time betwee failure models cocetrate o, as the ame idicates, modellig the times betwee occurred failures. The first developed time betwee failure model was the Jelisi- Morada model, which it is assumed that the times betwee failures are idepedetly expoetially distributed. This meas that, if X i deotes the time betwee the (i-1):th ad i:th failure, the probability desity fuctio of X i is defied as i equatio 1. f Xi () t λ i e λ i t (1) Where λ i is the failure itesity after the (i-1):th failure has occurred (ad before the i:th failure has occurred). I the Jelisi-Morada model, remaiig umber of failures ad is derived as i equatio 2. λ i is assumed to be a fuctio of the λ i φ( N ( i 1) ) (2) Where N is the iitial umber of faults i the program ad φ is a costat. The parameters i the model ca be iterpreted as follows. Let N be the iitial umber of faults i the program ad φ is a costat represetig the per fault failure itesity. The above formulas are together a model of the behaviour of the software with respect to failures. It is ot exactly represetig the real behaviour, merely a simplificatio of the real behaviour. To be able to use this model, for example for predictio, N ad φ must be estimated from the measured data. It is possible to mae a maximum lielihood estimate of the 8
parameters. The lielihood fuctio that ca be used to estimate N ad 3. φ is foud i equatio Lt ( 1, t ; N, φ) f Xi ( t i ) φ( N ( i 1) )e φ( N ( i 1) ) t i (3) Where t i is the measured values of X i, i.e. the measured times betwee failures, ad is the umber of measured times. By taig the atural logarithm of the lielihood fuctio ad simplifyig we obtai equatio 4. ll lφ + l( N i + 1 ) φ ( N i+ 1)t i (4) This fuctio should be maximized with respect to N ad φ. To do this the first derivative with respect to N ad φ ca be tae. The Nˆ ad φˆ, which satisfy that both the derivatives equals 0, are the estimates we are looig for. φ -------------------------------------- (5) ( N i+ 1)t i 1 ------------------------- ( N i + 1) t i -------------------------------------- (6) ( N i + 1)t i After the Jelisi-Morada model was published a umber of differet variatios of the model have bee suggested. Examples are: Failures do ot have to be corrected util a major failure has occurred. The failure itesity does ot have to be costat betwee to successive failures. Oe proposal i the literature is to itroduce a icreasig failure rate (IFR) derived as λ i φ( N ( i 1) )t, where t is the time elapsed sice the last failure occurred. A variat of the Jelisi-Morada model, which accouts for the probability of imperfect debuggig, i.e. the probability that a fault is ot removed i a repair activity has bee developed. With this model the failure itesity ca be expressed as λ i φ( N p( i 1) ), where p is the probability of imperfect debuggig. 9
9. Refereces [Burstei03] Burstei, I., Practical Software Testig A Process-Orieted Approach, Spriger-Verlag, 2003. [Sommerville01] Sommerville, I. Software Egieerig, 6th editio, Addiso-Wesley, 2001. [Sommerville04] Sommerville, I. Software Egieerig, 7th editio, Addiso-Wesley, 2004. [Wood96] A. Wood, Predictig Software Reliability IEEE Computer, pp. 69-77, Nov., 1996. 10