It is important to be able to. during the development. management models. Software Reliability Models

Transcription

1 SENG 637 Dependabiliy, Reliabiliy & Tesing of Sofware Sysems Sofware Reliabiliy Models (Chaper 2) Deparmen of Elecrical & Compuer Engineering, Universiy of Calgary B.H. ar hp:// p// / p / / / / Conens Basic eaures of he Sofware Reliabiliy Models Single ailure Model Reliabiliy Growh Model Weibull and Gamma ailure Class Models Bayesian Models Early Life-Cycle Predicion Models far@ucalgary.ca 1 far@ucalgary.ca 2 Reference Sofware Reliabiliy Engineering i Handbook Chaper 3: Sofware Reliabiliy Modeling Survey Chaper 2 Secion 1 Basic eaures of fsre Models far@ucalgary.ca 3 far@ucalgary.ca 4 Goal I is imporan o be able o Predic probabiliy of failure of a componen or sysem Esimae he mean ime o he ne failure Predic number of (remaining) failures during he developmen. Such asks are he arge of he reliabiliy managemen models. far@ucalgary.ca 5 Sofware Reliabiliy Models aul inroducion: Characerisics of he produc (e.g., program size) Developmen process (e.g., SE ools and echniques, saff eperiences, ec.) Reliabiliy Model Inpu daa Reliabiliy Predicion (ailure Specificaion) Esimaion Reliabiliy Model aul removal: ailure discovery (e.g., een of eecuion, operaional profile) Qualiy of repair aciviy Environmen (Usage) far@ucalgary.ca 6

2 ailure Specificaion /1 ailure Specificaion /2 1) Time of failure 2) Time inerval beween failures 3) Cumulaive failure up o a given ime 4) ailures eperienced in a ime inerval Time based failure specificaion ailure no. ailure imes (hours) ailure inerval (hours) 1) Time of failure 2) Time inerval beween failures 3) Cumulaive failure up o a given ime 4) ailures eperienced in a ime inerval ailure based failure specificaion Time(s) Cumulaive ailures ailures in inerval far@ucalgary.ca 7 far@ucalgary.ca 8 Two Reliabiliy Quesions Single failure specificaion: Wha is he probabiliy of failure of a sysem (or a componen)? Muliple failure specificaion: If a sysem (or a componen) fails a ime 1, 2,, i-1, Wha is he epeced ime of he ne failure? Wha is he probabiliy of he ne failure? Various Reliabiliy Models /1 Eponenial ailure Class Models Jelinski-Moranda model (JM) Nonhomogeneous Poisson Process model (NHPP) Schneidewind model Musa s Basic Eecuion Time model (BET) Hypereponenial model (HE) Ohers far@ucalgary.ca 9 far@ucalgary.ca 1 Various Reliabiliy Models /2 Weibull and Gamma ailure Class Models Weibull model (WM) S-shaped Reliabiliy Growh model (SRG) Infinie ailure Caegory Models Duane s model Geomeric model Musa-Okumoo Logarihmic Poisson model Various Reliabiliy Models /3 Bayesian Models Lilewood-Verrall Model Early Life-Cycle Predicion Models Phase-based sed model far@ucalgary.ca 11 far@ucalgary.ca 12

3 How o Choose a SRE Model? Collec failure daa (failure specificaion) Eamine daa (Densiy disribuion vs. Cumulaive disribuion) Selec a model Esimae model parameers Cusomize model using he esimaed parameers Goodness-of-fi es Make reliabiliy predicions Chaper 2 Secion 2 Background: Randomness & Probabiliy far@ucalgary.ca 13 far@ucalgary.ca 14 Randomness /1 Random acions in reliabiliy engineering: Inroducion of defecs ino he code and heir removal Eecuion of he es-cases, ec. We should lddefine some random processes o represen he randomness How o handle randomness? Collec failure daa hrough esing ind a disribuion funcion ha is a bes-fi for he colleced daa Make assumpions abou he presence of errors and reliabiliy far@ucalgary.ca 15 Randomness /2 Wha is a random variable? A random variable on a sample space S is a rule ha assigns a numerical value o each oucome of S (a funcion of S ino a se of real numbers) In reliabiliy modeling wha can be represened by random variable? Number of failures in an inerval Time of failure wihin an inerval ec. Previous Curren ime inerval Ne far@ucalgary.ca 16 Probabiliy Disribuion /1 Suppose ha a random variable X assigns a finie number of values o a sample space S Then X induces a disribuion funcion f ha assigns probabiliies o he poins in R R ={1, 2, 3,,n} f(k) = P( X=k) Probabiliy Disribuion /2 The se of ordered pairs [k, f(k)] is usually represened by a able or a graph (hisogram) requen ncy (ailure/im me uni) Time (days) The epeced value of X,, denoed by E(X) is defined by E(X) = 1 f(1) + 2 f(2) + + n f(n) far@ucalgary.ca 17 far@ucalgary.ca 18

4 Probabiliy Disribuion /3 Le M() be a random process represening he number of failures a ime The mean funcion μ() represens he epeced number of failures a ime μ() = E(M()) ailure inensiy is he rae of change of he epeced number of failures wih respec o ime λ() = d μ() / d () is he number of failures per uni ime () is an insananeous value Probabiliy Disribuion /4 Discree disribuions: Binomial disribuion Poisson disribuion Coninuous disribuions: Normal / Gaussian disribuion Lognormal disribuion Weibull disribuion Rayleigh disribuion Eponenial disribuion (Gamma) disribuion 2 (Kai square) disribuion far@ucalgary.ca 19 far@ucalgary.ca 2 Binomial Disribuion /1 Gives probabiliy of eac number of successes in n independen d rials, when probabiliy of success p on single rial is a consan. Siuaions wih only 2 oucomes (success or failure) Probabiliy remains he same for all independen rials (Bernoulli rials) Jakob Bernoulli ( ) 175) Binomial Disribuion /1 Probabiliy of eacly successes: f n!!! n p q n n : number of rials f() : probabiliy of successes in n rials p : probabiliy of success q : probabiliy of failure p + q =1 n p q n far@ucalgary.ca 21 far@ucalgary.ca 22 Binomial Disribuion /2 Probabiliy of having upo r successes: n : number of rials f() : probabiliy of successes in n rials p : probabiliy of success q : probabiliy of failure p+ q =1 (r) : probabiliy of obaining r or fewer successes in n rials r r n p q n Calculaing (r) becomes increasingly difficul as n (sample se) ges larger I is possible o find an approimae soluion by means of a normal disribuion Binomial Disribuion /3 Common shapes of binomial disribuion igure from: Mongomery e al. Engineering Saisics far@ucalgary.ca 23 far@ucalgary.ca 24

5 Eample: Binomial Disribuion Accepance sampling: A lo is acceped if no more han 2 defecives are found in a sample of 6. The defec probabiliy is 25%. Probabiliy of having eacly 2 defecs in he lo is: f Probabiliy of having more han 4 defecs in he lo is: r Eample 2 In a company here are 4 file servers each having he eac replica of a d daa se. The probabiliy bili of failure for each server is 1%. Probabiliy of having 2 servers fail: f Probabiliy of having more han 3 servers fail: r far@ucalgary.ca 25 far@ucalgary.ca 26 Poisson Disribuion /1 Some evens are raher rare, hey don happen ha ofen. Sill, over a period of ime, we wan o say somehing abou he naure of rare evens. Poisson disribuion is special case of binomial disribuion (eiher p or q is very small and n very large) Condiions under which a Poisson disribuion holds couns of rare evens, i.e. small probabiliy all evens are independen average rae (usually denoed by ) does no change over he period of ineres Simeon Poisson ( ) Poisson Disribuion /2 Poisson disribuion is a special case of binomial disribuion (eiher p or q is very small and n very large): =np μ f() e! μ 12,1,2,... : mean rae of occurrence (in saisics lieraure is usually denoed by ) : observed number of failures far@ucalgary.ca 27 far@ucalgary.ca 28 Poisson Disribuion /3 Common shapes of Poisson disribuion Eample: Poisson Disribuion Suppose ha he defec rae is only 2% find he probabiliy ha here are 3 defecive iems in a sample of 1 iems. np f e 3!. 18 3! 2 igure from: Mongomery e al. Engineering Saisics far@ucalgary.ca 29 far@ucalgary.ca 3

6 Hardware Reliabiliy Models Chaper 2 Secion 3 Single ailure Model far@ucalgary.ca 31 Uniform model: f Warrany Probabiliy of failure is fied. 1 T f T Eponenial model: Probabiliy of failure changes eponenially over ime f e f T T far@ucalgary.ca 32 Release ime Single ailure Model /1 Probabiliy Densiy uncion (PD): depicing changes of he probabiliy of failure up o a given ime. A common form of PD is eponenial disribuion Usually we wan o know how long a componen will behave correcly before i fails, i.e., he probabiliy of failure from ime up o a given ime. Eponenial PD: e f far@ucalgary.ca 33 Single ailure Model /2 Cumulaive Densiy uncion (CD): depicing cumulaive failures up o a given ime. f or eponenial disribuion, CD is: 1e igure from Musa s book far@ucalgary.ca 34 Single ailure Model /3 Reliabiliy funcion R(): defined as a componen funcioning i wihou failure unil ime, ha is, he probabiliy ha he ime o failure is greaer han. R f 1 or eponenial ldisribuion, ib i wih a consan failure rae : R e Single ailure Model /4 Wha is he epeced value of failure a ime T? I is he mean of he probabiliy bili densiy funcion (PD), named mean ime o failure (MTT) E T f d or eponenial disribuion, MTT is: 1 MTT far@ucalgary.ca 35 far@ucalgary.ca 36

7 Single ailure Model /5 Median ime o failure (m): a poin in ime ha he probabiliy of failure before and afer m are equal. m 1 1 f d or 2 m 2 ailure (hazard) Rae z(): Probabiliy densiy funcion divided by reliabiliy funcion. f z R or eponenial disribuion, z() is: far@ucalgary.ca 37 Single ailure Model /6 Sysem Reliabiliy: is he muliplicaion of he reliabiliy of is componens. (for serial sysems, i.e., wih no redundancy) n Rsysem Ri i1 or eponenial ldisribuion: ib i 1 2 n 1 2 n R e e e e sysem n i i 1 1 R e sysem far@ucalgary.ca 38 Single ailure Model /7 Sysem Cumulaive ailure (hazard) Rae: is he sum of he failure rae of is componens. n zsysem zi i 1 or eponenial disribuion: Chaper 2 Secion 4 Reliabiliy Growh Model n sysem 1 2 n i i1 z far@ucalgary.ca 39 far@ucalgary.ca 4 Reliabiliy Growh Models /1 or hardware sysems one can assume ha he probabiliy of failure (probabiliy densiy funcion, PD) for all failures are he same (e.g., replacing a fauly hardware componen wih an idenical working one). In sofware, however, we wan o fi he problem, i.e., have a lower probabiliy for he remaining failures afer a repair (or longer Δi = i-i-1). Therefore, we need a model for reliabiliy growh (i.e., reliabiliy change over ime). Reliabiliy Growh Models /2 In reliabiliy growh models we are assuming some effor of faul removal. This leads o a variable failure inensiy (). Every reliabiliy growh model is based on specific assumpions concerning he change of failure inensiy () hrough he process of faul removal. far@ucalgary.ca 41 far@ucalgary.ca 42

8 Reliabiliy Growh Models /3 Common sofware reliabiliy growh models are: Basic Eponenial model Logarihmic Poisson model The basic eponenial model assumes finie failures () in infinie ime. The logarihmic Poisson model assumes infinie failures. Validiy of he Models Sofware sysems are changed (updaed) many imes during heir life cycle. The models are good for one revision period raher han he whole life cycle. Revision Period 1 Revision Period 4 igure from Pressman s book far@ucalgary.ca 43 far@ucalgary.ca 44 Reliabiliy Growh Models /4 Variables involved in reliabiliy growh models: 1) ailure inensiy (): number of failures per naural or ime uni. 2) Eecuion ime (): ime since he program is running. Eecuion ime may be differen from calendar ime. 3) Mean failures eperienced d( (): mean fil failures eperienced in a ime inerval. Reliabiliy Growh Models /9 ailure(s) Probabiliy in ime Elapsed ime Elapsed ime period (1 hour) (5 hours) igure from Musa s book Mean far@ucalgary.ca 45 far@ucalgary.ca 47 Reliabiliy Growh Models /6 ailure inensiy () versus eecuion ime () Reliabiliy Growh Models /7 ailure inensiy () versus mean fil failures eperienced () (B) (P) e 1 Iniial failure inensiy Toal failures Decay parameer (B) (P) 1 e igure from Musa s book far@ucalgary.ca 48 igure from Musa s book far@ucalgary.ca 49

9 Reliabiliy Growh Models /8 Mean failures eperienced d( () versus eecuion ime () How o Use he Models? Release crieria (ime): ime required o es he sysem o reach a arge failure inensiy: (B) (P) 1 ln 1 ( ) ln P B ( P) P : Presen failure inensiy P : Targe failure inensiy igure from Musa s book far@ucalgary.ca 5 igure from Musa s book far@ucalgary.ca 51 How o Use he Models? Release crieria (failure): ime required o es he sysem o reach a arge number of failures: (B) (P) P 1 ln P P : Presen failure inensiy : Targe failure inensiy Reliabiliy Merics Mean ime o failure (MTT): Usually calculaed by dividing he oal operaing ime of he unis esed by he oal number of failures encounered (assuming ha he failure rae is consan). Eample: MTT for Windows 2 Professional is 2893 hours or 72 workweeks (4 hours per week). MTT for Windows NT Worksaion is 919 hours or 23 workweeks. MTT for Windows 98 is 216 hours or 5 workweeks. Mean ime o repair (MTTR): mean ime o repair a (sofware) componen. Mean ime beween failures (MTB): MTB = MTT + MTTR igure from Musa s book far@ucalgary.ca 52 far@ucalgary.ca 53 Reliabiliy Merics: Availabiliy Sofware Sysem Availabiliy (A): 1 1 A A or 1 m m A is failure inensiy m is downime per failure Anoher definiion of availabiliy: MTT MTT A MTT MTTR MTB Eample: If a produc mus be available 99% of ime and downime is 6 min, hen is abou.1 failure per hour (1 failure per 1 hours) and MTT=594 min. Reliabiliy Merics: Reliabiliy Sofware Sysem Reliabiliy (R): ln R 1R or for R.95 is failure inensiy R is reliabiliy is naural uni (ime, ec.) Eample: for =.1 or 1 failure for 1 hours, reliabiliy (R) is around.992 for 8 hours of operaion. far@ucalgary.ca 54 far@ucalgary.ca 55

10 Eample In reliabiliy growh esing of a sofware sysem he failure daa fis o boh he Eponenial model and Logarihmic Poisson model. The model parameers are given below: Eponenial model λ = 2 failures/cpu hour ν = 12 failures Logarihmic Poisson model λ = 5 failures/cpu hour θ =.25 failures Assume for boh cases below ha you sar from he iniial failure inensiy. Eample (con d) Deermine he addiional failure and addiional eecuion ime required o reach a failure inensiy objecive of 1 failure/cpu hour for boh models. or Basic model: 12 P failures 2 P 12 2 ln ln 4.16 CPU hour 2 1 or Poisson model: 1 P 1 5 ln ln failures CPU hour P far@ucalgary.ca 56 far@ucalgary.ca 57 Eample (con d) Repea he same calculaion for a failure inensiy objecive of 1 failure/cpu hour for boh models. or Basic model: 12 P failures 2 P 12 2 ln ln 18 CPU hour 2 1 or Poisson model: 1 P 1 5 ln ln 156 failures CPU hour P Eample (con d) Based on he eperimen s resuls compare he wo model resuls and eplain wha will happen o he addiional eecuion ime required when he failure inensiy objecive ges smaller. As failure inensiy ges smaller he addiional failures and eecuion ime required o reach hem become subsanially larger for Logarihmic Poisson model han he Basic model. Therefore he Basic model gives an opimisic esimaion while Logarihmic Poisson model offers a pessimisic i one. far@ucalgary.ca 58 far@ucalgary.ca 59 Chaper 2 Secion 5 Wibll Weibull and dg Gamma ailure Class Models Gamma ailure Class Models The generalized gamma disribuion (oal 3 parameers, i.e., scale, shape, locaion) includes oher disribuions as special cases based on he values of he parameers. far@ucalgary.ca 6 Reference: hp:// far@ucalgary.ca 61

11 Weibull Model /1 ailure disribuion is Weibull disribuion raher han eponenial Depending on he values of he parameers (oal 3 parameers, i.e., scale, shape, locaion), he Weibull disribuion can be used o model a variey of behaviours Weibull Model /2 Assumpions: Toal number of fauls is bounded The ime o failure is disribued as Weibull disribuion The number of fauls deeced in each inerval are independen for any finie collecion of imes Daa requiremen: aul couns on each esing inerval: f1,f2,,fn f2, fn Compleion ime of each period: 1, 2,, n far@ucalgary.ca 62 far@ucalgary.ca 63 Weibull Model /3 The Weibull disribuion has he probabiliy densiy funcion Weibull Model /4 Reliabiliy where = scale parameer = shape parameer (i.e., slope) = locaion parameer Cumulaive densiy funcion, cdf : ailure inensiy Model parameers: (scale parameer), (shape or slope parameer) and (locaion parameer) far@ucalgary.ca 64 far@ucalgary.ca 65 Weibull Model /5 Chaper 2 Secion 6 Bayesian Models Reference: hp:// far@ucalgary.ca 66 far@ucalgary.ca 67

12 Bayesian Models /1 The previous models allow change in he reliabiliy only when an error occurs. Assumpions: If no failures occur while he sofware is esed hen he reliabiliy should increase, reflecing he growing confidence in he sofware by he user. Differen fauls have differen impacs on reliabiliy. The number of fauls is no as imporan as heir impacs. Reliabiliy is a reflecion of boh he number of fauls ha have been deeced and he amoun of failure-free operaion. Uses prior disribuion represening he pas daa and a poserior disribuion o incorporae pas and curren daa. Bayesian Models /2 Takes more subjecive view of failure Seps of Bayesian approach far@ucalgary.ca 68 far@ucalgary.ca 69 Chaper 2 Secion 7 Early Life-Cycle Predicion Models Early Life-Cycle Models /1 Mos of he reliabiliy growh models predic reliabiliy a he laer sages of developmen life cycle, i.e., during esing sage. However, if a sofware organizaion wais unil all necessary daa is available o collec, wha usually happens is ha i is already oo lae o make proper improvemens in sofware reliabiliy o achieve he reliabiliy goal. Quesion: Would i be possible o predic reliabiliy a he earlier sages? Models ha relae early sage daa o reliabiliy are needed. Typical early sage daa: requiremens, man-power build-up, developmen process, error injecion rae and rends, code size esimaes, ec. far@ucalgary.ca 7 far@ucalgary.ca 71 The Raleigh-Punam Model I is usually assumed ha over he life of he projec he fauls deeced (per monh) will resemble a Raleigh curve The rae of epending effor is proporional p o he rae of commiing errors. Punam assumes ha he rae of effor ependiure is a Raleigh curve. Source: QSM Web page hp:// ELCM: The Process /1 Obain he milesones for he schedule: Sar dae and oal monhs in projec Dae of epeced full operaional capabiliy - d Esimae he oal number of fauls over he life of he projec - Er. A Raleigh curve can be generaed by solving for each monh using his equaion: E m 3 2 6E r d 2 e d 2 SENG421 (Winer 26) far@ucalgary.ca 73 far@ucalgary.ca 74

13 ELCM: The Process /2 Use his profile o gauge he faul deecion process during each phase of developmen. In paricular, his profile can be used o gauge he original schedule esimae and he predicion for he oal number of defecs. or eample, he esimaed number of defecs impacs he heigh of he curve while he schedule impacs he lengh of he curve. If he acual defec curve is significanly differen from he prediced curve hen one or boh of hese parameers may have been esimaed incorrecly and should be brough o he aenion of managemen. far@ucalgary.ca 75 Reliabiliy Growh Model Assumpions vs. Pracice Assumpions In pracice Sofware does no change and defecs are fied immediaely Tesing Operaional Profile (OP) Consan es effor and independen failure inervals All failures are observable Sofware changes rapidly and cerain defecs are scheduled o be fied in a laer dae ocus on funcional esing. I is difficul o define OP and perform operaional ess. Varying es effor (due o holidays and vacaions) and failure inervals may vary Tesing in a conrolled environmen may be differen from running sofware in live environmen Collec defec repors. Any kind of defec observed is recorded. No all repors address a failure Remaining failures may acually increase due o improper bug fies [Sringfellow & Andrews 22] Cll Collec fil failure repors Cll df A kid fdf b d Remaining failures are eiher consan or decreasing far@ucalgary.ca 76 far@ucalgary.ca 77