B.H. Far

Transcription

1 SENG 521 Software Reliability & Software Quality Chapter 6: Software Reliability Models Department of Electrical & Computer Engineering, University of Calgary B.H. Far 1

2 Contents Basic Features of the Software Reliability Models Single Failure Model Reliability Growth Model Exponential Failure Class Models Weibull and Gamma Failure Class Models Infinite Failure Category Models Bayesian Models Early Life-Cycle Prediction Models 2

3 Reference Software Reliability Engineering Handbook Chapter 3: Software Reliability Modeling Survey 3

4 Goal It is important to be able to Predict probability of failure of a component or system Estimate the mean time to the next failure Predict number of (remaining) failures during the development. Such tasks are the target of the reliability management models. far@ucalgary.ca 4

5 Software Reliability Models Input data (Failure Specification) Reliability Model Prediction Estimation Fault introduction: Characteristics of the product (e.g., program size) Development process (e.g., SE tools and techniques, staff experiences, etc.) Reliability Model Fault removal: Failure discovery (e.g., extent of execution, operational profile) Quality of repair activity Environment (Usage) 5

6 Failure Specification /1 1) Time of failure 2) Time interval between failures 3) Cumulative failure up to a given time Time based failure specification Failure no. Failure times (hours) ) Failures experienced in a time interval Failure interval (hours) far@ucalgary.ca 6

7 Failure Specification /2 1) Time of failure 2) Time interval between failures 3) Cumulative failure up to a given time 4) Failures experienced in a time interval Failure based failure specification Time(s) Cumulative Failures Failures in interval far@ucalgary.ca 7

8 Two Reliability Questions Single failure specification: What is the probability of failure of a system (or a component)? Multiple failure specification: If a system (or a component) fails at time t1, t2,, ti-1, What is the expected time of the next failure? What is the probability of the next failure? far@ucalgary.ca 8

9 Model Classification Time domain: Calendar or execution time Category: Number of failures is finite or infinite. Type: Distribution of the number of failures experienced by the time specified. Class (only finite category): Functional form of the failure intensity over time. Family (only infinite category): Functional form of the failure intensity in terms of the expected number of failures experienced. Reference: Musa s Book far@ucalgary.ca 9

10 Various Reliability Models /1 Exponential Failure Class Models Jelinski-Moranda model (JM) Nonhomogeneous Poisson Process model (NHPP) Schneidewind model Musa s Basic Execution Time model (BET) Hyperexponential model (HE) Others far@ucalgary.ca 10

11 Various Reliability Models /2 Weibull and Gamma Failure Class Models Weibull model (WM) S-shaped Reliability Growth model (SRG) Infinite Failure Category Models Duane s model Geometric model Musa-Okumoto Logarithmic Poisson model far@ucalgary.ca 11

12 Various Reliability Models /3 Bayesian Models Littlewood-Verrall Model Early Life-Cycle Prediction Models Phase-based model 12

13 How to Choose a SRE Model? Collect failure data (failure specification) Examine data (Density distribution vs. Cumulative distribution) Select a model Estimate model parameters Customize model using the estimated parameters Goodness-of-fit test Make reliability predictions far@ucalgary.ca 13

14 Another Categorization /1 Classification based on failure data 1. Time between failure models Jelinski-Moranda (Exponential Failure Model) Musa-Basic (Exponential lfailure Model) NHPP (Exponential Failure Model) Geometric (Infinite Failure Model) Musa-Okumoto (Infinite Failure Model) Littlewood-Verrall (Bayesian Model) 14

15 Another Categorization /2 Classification based on failure data 2. Failure count models Generalized Poisson Shick-Wolverton Yamada S-shaped (Gamma Failure Class) NHPP (Exponential Failure Model) Schneidewind (Exponential Failure Model) 15

16 Section 2 Background: Randomness & Probability far@ucalgary.ca 16

17 Randomness /1 Random actions in reliability engineering: i Introduction of defects into the code and their removal Execution of fthe test-cases, t etc. We should define some random processes to represent the randomness How to handle randomness? Cll Collect tfil failure data dt through htestingti Find a distribution function that is a best-fit for the collected data Make assumptions about the presence of errors and reliability far@ucalgary.ca 17

18 Randomness /2 What is a random variable? A random variable x on a sample space S is a rule that assigns a numerical value to each outcome of S (a function of S into a set of real numbers) In reliability modeling what can be represented by random variable? Number of failures in an interval Time of failure within an interval etc. Previous Current time interval Next far@ucalgary.ca 18

19 Probability Distribution /1 Suppose that a random variable X assigns a finite number of values to a sample space S Then X induces a distribution function f that assigns probabilities to the points in Rx Rx ={x1, x2, x3,, xn} f(xk) = P( X=xk) far@ucalgary.ca 19

20 Probability Distribution /2 The set of ordered pairs [xk, f(xk)] is usually represented by a table or a graph (histogram) Freq quency (Failure /time unit) Time (days) The expected value of X, denoted by E(X) is dfi defined dby E(X) = x1 f(x1) + x2 f(x2) + + xn f(xn) far@ucalgary.ca 20

21 Probability Distribution /3 LtM(t) Let be a random process representing the number of failures at time t The mean function μ(t) represents the expected number of failures at time t μ(t) = E(M(t)) Failure intensity is the rate of change of the expected number of failures with respect to time λ(t) = d μ(t) / dt (t) is the number of failures per unit time (t) is an instantaneous value far@ucalgary.ca 21

22 Reliability Growth Models /5 No. of failures in Probability interval (n) (p) n p n i pi i Mean failures experienced () for a given time period (e.g., 1 hour execution time) is calculated as: p i : probability of occurrence i : number of failures Mean failure (μ) far@ucalgary.ca 22

23 Reliability Growth Models /9 Failure(s) in time period Elapsed time (1 hour) Probability Elapsed time (5 hours) Figure from Musa s book Mean far@ucalgary.ca 23

24 Probability Distribution /4 Discrete distributions: Binomial distribution Poisson distribution Continuous distributions: Normal / Gaussian distribution Lognormal distribution Weibull distribution Rayleigh distribution Exponential distribution (Gamma) distribution 2 (Kai square) distribution far@ucalgary.ca 24

25 Binomial Distribution /1 Gives probability bilit of exact number of successes in n independent trials, when probability of success p on single trial is a constant. Situations with only 2 outcomes (success or failure) ) Probability remains the same for all independent trials (Bernoulli trials) Jakob Bernoulli ( ) far@ucalgary.ca 25

26 Binomial Distribution /1 Probability of exactly x successes: f x x n!!! p x q nx n n n x x p x q nx n : number of trials f(x) : probability of x successes in n trials p : probability of success q : probability of failure p + q =1 far@ucalgary.ca 26

27 Binomial Distribution /2 Probability of having upto r successes: F n : number of trials f(x) : probability of x successes in n trials p : probability of success q : probability of failure p + q =1 F(r) : probability of obtaining r or fewer successes in n trials r r x 0 n x p x q n x Cl Calculating l F(r) becomes increasingly i difficult as n (sample set) gets larger It is possible to find an approximate solution by means of a normal distribution far@ucalgary.ca 27

28 Binomial Distribution /3 Common shapes of binomial distribution Figure from: Montgomery et al. Engineering Statistics 28

29 Example: Binomial Distribution Acceptance sampling: A lot is accepted if not more than 2 defectives are found in a sample of 6. The defect probability is 25%. Probability of having exactly 2 defects in the lot is: f Probability bilit of having more than 4 defects in the lot is: FF r 4 F 5 F far@ucalgary.ca 29

30 Example 2 In a company there are 4 file servers each having the exact replica of a data set. The probability of failure for each server is 10%. Probability of having 2 servers fail: f Probability of having more than 3 servers fail: Fr 2 F3 F far@ucalgary.ca 30

31 Poisson Distribution /1 Some events are rather rare, they don t happen that often. Still, over a period of time, we want to say something about the nature of rare events. Poisson distribution is special case of binomial distribution ib ti (either p or q is very small and n very large) Conditions under which a Poisson distribution holds counts of rare events all events are independent average rate (usually denoted by ) does not change over the period of interest Simeon Poisson ( ) far@ucalgary.ca 31

32 Poisson Distribution /2 Poisson distribution is a special case of binomial distribution (either p or q is very small and n very large): =np f(x) x μ x! e μ x 0,1,2,... : mean rate of occurrence (in statistics literature is usually denoted by ) x : observed number of failures far@ucalgary.ca 32

33 Poisson Distribution /3 Common shapes of Poisson distribution Figure from: Montgomery et al. Engineering Statistics 33

34 Example: Poisson Distribution Suppose that the defect rate is only 2% find the probability that there are 3 defective items in a sample of 100 items. np f 2 e x ! far@ucalgary.ca 34

35 Section 3 Single Failure Model far@ucalgary.ca 35

36 Hardware Reliability Models Uniform model: Probability of failure is fixed. f t 1 t T 0 t T Exponential model: Probability of failure changes exponentially over time f t e t 0 0 f Warrantyant t T f time t T Release far@ucalgary.ca 36

37 Single Failure Model /1 Probability bilit Density Function (PDF): depicting changes of the probability of failure up to a given time t. A common form of PDF is exponential distribution Usually we want to know how Exponential lpdf PDF: Usually we want to know how long a component will behave correctly before it fails, i.e., the probability of failure from time 0 up to a given time t. f t t e t 0 0 t 0 far@ucalgary.ca 37

38 Single Failure Model /2 Cumulative Density Function (CDF): depicting cumulative failures up to a given time t. F t t 0 f t For exponential distribution, CDF is: F t tt 1 e t 0 0 t 0 Figure from Musa s book far@ucalgary.ca 38

39 Single Failure Model /3 Reliability function R(t): dfi defined das a component functioning without failure until time t, that is, the probability bilit that t the time to failure is greater than t. R t f t 1 F t t For exponential distribution, with a constant failure rate : R t tt e far@ucalgary.ca 39

40 Single Failure Model /4 What is the expected value of failure at time T? It is the mean of the probability density function (PDF), named mean time to failure (MTTF) E T t f t dt For exponential distribution, ib ti MTTF is: MTTF 0 1 far@ucalgary.ca 40

41 Single Failure Model /5 Median time to failure (tm): a point in time, tm, that the probability of failure before and after tm are equal. t m 0 tf tdt 1 or Ft 1 2 m 2 Splits data into two groups, each with the same number of items Handles outliers well often the most accurate representation of a group far@ucalgary.ca 41

42 Single Failure Model /6 Failure (hazard) Rate z(t): () Probability bili density function divided by reliability function. z t f t 1 F t f t R t Represents instantaneous probability of failure at time t. It is also known as the failure of the survivors because denominator is the population of survivors For exponential distribution, z(t) is: far@ucalgary.ca 42

43 Single Failure Model /7 System Reliability: is the multiplication of the reliability of its components. (for serial systems, i.e., with no redundancy) R system t Ri t n i1 For exponential distribution: 1t 2t nt 1 2 n Rsystem t e e e e R t e system t n i1 i far@ucalgary.ca 43 t

44 Single Failure Model /8 System Cumulative Failure (hazard) Rate: is the sum of the failure rate of its components. z system t zi t n i1 For exponential distribution: z t 1 2 system n i i1 n far@ucalgary.ca 44

45 Section 4 Reliability Growth Model far@ucalgary.ca 45

46 Reliability Growth Models exponential Random variables Time of failure Mean number of failures etc. constant Quadratic Gamma 46

47 Reliability Growth Models /1 One can assume that t the probability bilit of failure (probability bilit density function, PDF) for all failures are the same (e.g., replacing the faulty hardware component with an identical fixed one). In software, however, we want to fix the problem, i.e., have a lower probability bilit for the remaining i failures after a repair (or longer Δti = ti-ti-1). Therefore, we need a model for reliability yg growth (i.e., reliability change over time). far@ucalgary.ca 47

48 Reliability Growth Models /2 In reliability growth models we are assuming some effort of fault removal. This leads to a variable failure intensity (t). Every reliability growth model is based on specific assumptions concerning the change of ffailure intensity i (t) ) through hthe process of fault removal. far@ucalgary.ca 48

49 Reliability Growth Models /3 Common software reliability growth models are: Basic Exponential model Logarithmic Poisson model The basic exponential model assumes finite failures (0) in infinite time. The logarithmic Poisson model assumes infinite failures. 49

50 Validity of the Models Software systems are changed (updated) many times during their life cycle. The models are good for one revision period rather than the whole life cycle. Revision Period 1 Revision Period 4 Figure from Pressman s book far@ucalgary.ca 50

51 Reliability Growth Models /4 Variables involved in reliability growth models: 1) Failure intensity (): number of failures per natural or time unit. 2) Execution time (): time since the program is running. Execution time may be different from calendar time. 3) Mean failures experienced d( (): mean fil failures experienced in a time interval. far@ucalgary.ca 51

52 Reliability Growth Models /6 Failure intensity () versus execution time () (B) (P) 0 e Initial failure intensity 1 0 Total failures 0 Decay parameter Figure from Musa s book far@ucalgary.ca 52

53 Reliability Growth Models /7 Failure intensity () versus mean failures experienced () (B) (P) e Figure from Musa s book far@ucalgary.ca 53

54 Reliability Growth Models /8 Mean failures experienced () versus execution time () (B) 0 0 e (P) ln 0 1 Figure from Musa s book far@ucalgary.ca 54

55 How to Use the Models? Release criteria (time): time required to test the system to reach a target failure intensity: 0 ( B) ln P ( P ) 0 F P t f il i t it F P P F : Present failure intensity : Target failure intensity Figure from Musa s book far@ucalgary.ca 55

56 How to Use the Models? Release criteria (failure): time required to test the system to reach a target number of failures: 0 (B) P F (P) 0 1 ln P F ln : Present failure intensity it F P : Target failure intensity F Figure from Musa s book far@ucalgary.ca 56

57 Reliability Metrics Mean time to failure (MTTF): Usually calculated by dividing the total operating time of the units tested by the total number of failures encountered (assuming that the failure rate is constant). Example: MTTF for Windows 2000 Professional is 2893 hours or 72 workweeks (40 hours per week). MTTF for Windows NT Workstation is 919 hours or 23 workweeks. MTTF for Windows 98 is 216 hours or 5 workweeks. Mean time to repair (MTTR): mean time to repair a (software) component. Mean time between failures (MTBF): MTBF = MTTF + MTTR far@ucalgary.ca 57

58 Reliability Metrics: Availability Software System Availability (A): 1 1 A t At or t 1 t m t tm A t is failure intensity t tm is downtime per failure Another definition of availability: MTTF MTTF A MTTF MTTR MTBF Example: If a product must be available 99% of time and downtime is 6 min, then is about 0.1 failure per hour (1 failure per 10 hours) and MTTF=594 min. far@ucalgary.ca 58

59 Reliability Metrics: Reliability Software System Reliability (R): ln R t 1R t t or t t for R t is failure intensity R is reliability t is natural unit (time, etc.) 0.95 Example: for =0.001 or 1 failure for 1000 hours, reliability (R) is around for 8 hours of operation. far@ucalgary.ca 59

60 Example In reliability growth testing of a software system the failure data fits to both the Exponential model and Logarithmic Poisson model. The model parameters are given below: Exponential model λ 0 = 20 failures/cpu hour ν 0 = 120 failures Logarithmic Poisson model λ 0 = 50 failures/cpu hour θ = failures Let s assume for both cases that you start from the initial failure intensity. far@ucalgary.ca 60

61 Example (cont d) Determine the additional failure and additional execution time required to reach a failure intensity objective of 10 failure/cpu hour for both models. For Basic model: P F failures P ln ln 4.16 CPU hour 0 F For Poisson model: 1 P 1 50 ln ln 64 failures F F P CPU hour far@ucalgary.ca 61

62 Example (cont d) Repeat the same calculation l for a failure intensity it objective of 1 failure/cpu hour for both models. For Basic model: P F failures P ln ln 18 CPU hour F For Poisson model: ln P ln 156 failures F F P CPU hour far@ucalgary.ca 62

63 Example (cont d) Based on the experiment s results compare the two model results and explain what will happen to the additional execution time required when the failure intensity objective gets smaller. As failure intensity gets smaller the additional failures and execution time required to reach them become substantially larger for Logarithmic Poisson model than the Basic model. Therefore the Basic model gives an optimistic i estimation while Logarithmic Poisson model offers a pessimistic one. far@ucalgary.ca 63

64 Section 5 Exponential Failure Time Models far@ucalgary.ca 64

65 Exponential Failure Time Models Assumes finite failure models Functional form of failure intensity function (t) is exponential Binominal i type: All faults have a constant hazard rate z(t) = Hazard rate for the i-th fault is a function of the remaining number of faults t N e Failure intensity is t far@ucalgary.ca 65

66 Various Reliability Models Exponential Failure Time Models Jelinski-Moranda model (JM) Nonhomogeneous Poisson Process model (NHPP) Schneidewind model Musa s Basic Execution Time model (BET) Hyperexponential model (HE) Others far@ucalgary.ca 66

67 Jelinski-Moranda Model /1 Jli Jelinski-Moranda kim model lfollows exponential distribution but differs from the Geometric model in that the parameter used is proportional to the remaining number of faults rather than a constant. All faults equally contribute to the reliability of the system Hazard Rate z(t): Probability density function divided by reliability function. is a constant far@ucalgary.ca 67

68 Jelinski-Moranda Model /1 Assumptions: The rate of fault detection is proportional to the current fault content of the software. The fault detection rate remains constant over the intervals between faults. A fault is corrected instantaneously without introducing new faults into the software. Every fault has the same chance of being encountered within a severity class as any other fault in that class. The failures are independent. Data requirement: Either actual failure times: t1, t2,, tn Or elapsed time between failures: xi = ti ti-1 far@ucalgary.ca 68

69 Jelinski-Moranda Model /2 Meantime between failures at time t MTTF = 1/ (N-(i-1)) (i Mean failures experienced t N1e t Failure intensity function t N e t N total number of faults (constant) Parameters to estimate using collected data are: N and far@ucalgary.ca 69

70 Non Homogeneous Poisson Process Model /1 Assumes that the cumulative number of failures detected at any time follows a Poisson distribution. Time periods (intervals) can be unequal Data requirement: Fault counts on each testing interval: f1, f2,, fn Completion time of each period: t1, t2,, tn far@ucalgary.ca 70

71 Non Homogeneous Poisson Process Model /2 Assumptions: Every failure has the same chance of being detected as any other failure. The cumulative number of failures detected at any time follows a Poisson distribution with mean (t). ) That mean is such that the expected number of failures in any small time interval about t is proportional to the number of undetected t dfailures at ttime t. The mean is assumed to be a bounded non-decreasing function with (t) approaching in the limit, as the length of testing goes to infinity. Perfect debugging is assumed. far@ucalgary.ca 71

72 Non Homogeneous Poisson Process Model /3 Mean failures experienced bt t N 1 e Failure intensity tyfunction t Nbe bt Parameters to estimate using collected data are: N and b far@ucalgary.ca 72

73 Schneidewind Model Assumes that t the current fault rate might be a better predictor of the future behaviour than the observed rate in the distant past Suppose that there are n units of time all of fixed length Has 3 variations: Utilize all individual fault counts from all n time periods Ignore fault counts from 1 to s-1 periods (forget the distant past) Use cumulative fault count for 1 to s-1 periods and individual fault counts for s to n far@ucalgary.ca 73

74 Musa Basic Model /1 One of the most widely used software reliability models. Uses execution time rather than calendar time. 0 and 1 are parameters. 0 is equal to the number of faults in the system and 1 is a fault reduction factor. far@ucalgary.ca 74

75 Musa Basic Model /2 Assumptions: The detections of failures are independent of one another. The software failures are observed (i.e. the total t number of failures has an upper bound). The execution times (measured in CPU time) between failures are piecewise exponentially distributed. The hazard rate is proportional to the number of faults remaining in the program. The fault correction rate is proportional to the failure occurrence rate. Perfect debugging is assumed. far@ucalgary.ca 75

76 Musa Basic Model /3 Meantime between failures at time t MTTF(t) MTTF exp n Mean failures experienced e 0 0 C MTTF t t 1 e 1 0 Failure intensity function t t e t n 0 = total number of errors in the program MTTF 0 =MTTF at the start of testing C= test compression factor far@ucalgary.ca 76

77 Musa Basic Model /4 Number of errors at time t n(t) n Reliability 0 C 1 exp t n0 MTTF0 n 0 = total number of errors in the program R(t) exp t MTTF MTTF 0 =MTTF at the start of testing C= test compression factor far@ucalgary.ca 77

78 Hyper-exponential exponential Model /1 The basic idea is that t the different sections (or classes) of the software experience an exponential failure rate; however, the rates vary over these sections to reflect their different natures. This could be due to different programming groups doing the different parts, old versus new code, sections written in different languages, etc. We thus reflect the sum of these different exponential growth curves, not by another exponential, but by a hyper-exponential growth curve. far@ucalgary.ca 78

79 Hyper-exponential exponential Model /2 Data requirement: Fault counts on each testing interval: f1, f2,, fnf Completion time of each period: t1, t2,, tn Failure intensity function t N K i1 p i e i t i 0 p 1 K i p i i1 1 0 i 1 far@ucalgary.ca 79

80 Section 6 Infinite Failure Category Models far@ucalgary.ca 80

81 Infinite Failure Models In these models the software can never be fault free, i.e., the mean value function: lim t t Geometric model Musa-Okumoto model far@ucalgary.ca 81

82 Geometric Model /1 The time between failures has an exponential distribution whose mean decreases in geometric fashion (i.e., the earlier faults have larger impact) Hazard Rate z(t): Probability density function nctiondivided idedby reliability function. is a constant far@ucalgary.ca 82

83 Geometric Model /2 Assumptions: There is no upper bound on the total number of failures (i.e., the program will never be error free) All faults do not have the same chance of detection. The detections of faults are independent of one another. The failure detection rate forms a geometric progression and dis constant tbetween failure occurrences. Data requirement: Either actual failure times: t1, t2,, tn Or elapsed time between failures: xi = ti ti-1 far@ucalgary.ca 83

84 Geometric Model /3 The density for the time between failures of the ith and (i-1)st is exponential and takes the form i1 i1 D X f X i i D e Failure intensity (number of failures per unit time) De t where ln De t1 far@ucalgary.ca 84

85 Musa-Okumoto Model /1 Musa-Okumoto Logarithmic i Poisson Model uses Jelinski-Moranda model as a foundation Allows significant ifi changes to be made at the same time the software is being tested Differs from basic Musa Basic model, in that tit reflects the view that the earlier discovered failures have a greater impact on reducing the failure intensity function than those encountered later. Uses execution time to capture inter-failure times Widely accepted in the telecommunications far@ucalgary.ca 85

86 Musa-Okumoto Model /2 Assumptions Software is operated in a similar manner as the anticipated operational usage. Detections of failures are independent of one another. Expected number of failures is a logarithmic function of time. Failure intensity decreases exponentially with the expected number of failures experienced. There is no upper bound on the number of total failures (i.e., the program will never be error-free). Data requirement: Either actual failure times: t1, t2,, tn Or elapsed time between failures: xi = ti ti-1 far@ucalgary.ca 86

87 Musa-Okumoto Model /3 Failure intensity λ (t) λ0exp( θμ(t)) β0 β1 β t 1 1 λ 0 λ θt 0 1 Mean value function 1 lnλ θt 1 β β t 1 0 ln 1 μ(t) 0 θ is failure intensity decay parameter far@ucalgary.ca 87

88 Musa-Okumoto Model /4 Reliability R(t) Assuming the (i-1)th failure has happened Reliability R( t t i-1 ) β Hazard rate z(t) () β t 1 β 0 θt 1 1 i-1 0 i-1 1 (ti-1 Δt) 1 λ0 θ(ti-1 Δt) 1 λ 1 θ z(t) β0β1 λ 0 β (ti-1 Δt) 1 λ 0 θ(t i-1 Δt) 1 1 far@ucalgary.ca 88

89 Musa-Okumoto Model /5 Release criteria: time required to test the system to reach a target failure intensity: F P 1 p ln F P F : Present failure intensity it : Target failure intensity far@ucalgary.ca 89

90 Section 7 Weibull and Gamma Failure Class Models far@ucalgary.ca 90

91 Gamma Failure Class Models The generalized gamma distribution (total 3 parameters, i.e., scale, shape, location) includes other distributions as special cases based on the values of the parameters. Reference: far@ucalgary.ca 91

92 Weibull Model /1 Failure distribution is Weibull distribution rather than exponential Depending on the values of the parameters (total 3 parameters, i.e., scale, shape, location), the Weibull distribution can be used to model a variety of fbehaviours far@ucalgary.ca 92

93 Weibull Model /2 Assumptions: Total number of faults is bounded The time to failure is distributed as Weibull distribution The number of faults detected in each interval are independent for any finite collection of times Data requirement: Fault counts on each testing interval: f1,f2,,fn f2, fn Completion time of each period: t1, t2,, tn 93

94 Weibull Model /3 The Weibull distribution ib i has the probability bili density function where = scale parameter = shape parameter (i.e., slope) = location parameter Cumulative density function, cdf : far@ucalgary.ca 94

95 Weibull Model /4 Reliability Failure intensity Model parameters: (scale parameter), (shape or slope parameter) and (location parameter) 95

96 Weibull Model /5 Weibull model can take care of decreasing, constant and increasing failure intensity, depending on the value of β parameter, to account for the whole software life-cycle l (i.e., bathtub b model) far@ucalgary.ca 96

97 Yamada S-shaped Model /1 Weibull model does not model the initial increase of failure intensity (i.e., increase, decrease, constant, increase regions). Yamada model is suitable for data that shows an increasing failure rate followed by a decreasing failure rate (resembling reverse S). More realistic situation for the case of continuous debugging after release Assumes the mean value function and failure intensity follow a Gamma distribution far@ucalgary.ca 97

98 Yamada S-shaped Model /2 Assumptions: Cumulative number of faults follows a Poisson process Total number of ffaults is bounded d Time between failures of the (i-1)st and the ith depends on the time to failure of the (i-1)st When faults are detected they are removed immediately and no other fault is introduced Data requirement: Failure times: t1, t2,, tn Fault counts on each testing interval: f1, f2,, fn together with the length of the interval far@ucalgary.ca 98

99 Section 8 Bayesian Models far@ucalgary.ca 99

100 Bayesian Models /1 The previous models allow change in the reliability only when an error occurs. Assumptions: If no failures occur while the software is tested then the reliability should increase, reflecting the growing confidence in the software by the user. Different faults have different impacts on reliability. The number of faults is not as important as their impacts. Reliability is a reflection of both the number of faults that have been detected and the amount of failure-free operation. Uses prior distribution representing the past data and a posterior distribution to incorporate past and current data. far@ucalgary.ca 100

101 Bayesian Models /2 Takes more subjective view of failure Steps of Bayesian approach 101

102 Littlewood-Verrall Model /1 This model makes the assumption that t fault correction is imperfect, therefore new faults may be generated as ones discovered are fixed. With each fault correction, a sequence of software programs is generated. Each derived from its predecessor by attempting ti to fix the fault. Because of uncertainty, t the new version could be either better or worse than its predecessor. The failure times is assumed to have exponential distribution with a certain failure rate (similar to earlier models). The failure rate is also assumed to be random with Gamma distribution (opposite to earlier models). far@ucalgary.ca 102

103 Littlewood-Verrall Model /2 Assumptions: There is no upper bound on the total number of failures, i.e., the program will never be fault free. The debugging may not be perfect, ie, i.e., new bugs may be introduced during debugging. Successive execution times between failures are independent random variables with exponential distributions with parameter i, i=1,2,, n far@ucalgary.ca 103

104 Littlewood-Verrall Model /3 i s form a sequence of independent d random variables, each with a Gamma distribution with parameters and (i). The function (i) is taken to be an increasing function of i that describes the quality of the programmer and the difficulty of the task. A good programmer would have a more rapidly increasing function than a poorer one. The probability density function f x, i i x i i i 1 far@ucalgary.ca 104

105 Littlewood-Verrall Model /4 Linear and quadratic forms for (i) function i i ( linear form ) i 0 1i quadratic form Failure intensity function ( ) linear t t 1 far@ucalgary.ca 105

106 Section 9 Early Life-Cycle Prediction Models far@ucalgary.ca 106

107 Early Life-Cycle Models /1 Most of the reliability growth models predict reliability at the later stages of development life cycle, i.e., during testing stage. However, if a software organization waits until all necessary data is available to collect, what usually happens is that it is already too late to make proper improvements in software reliability to achieve the reliability goal. Question: Would it be possible to predict/track reliability at the earlier stages? Models that relate early stage data to reliability are needed. Typical early stage data: requirements, man-power build-up, development process, error injection rate and trends, code size estimates, etc. far@ucalgary.ca 107

108 Early Life-Cycle Models /2 The aim of prediction is to provide a quantitative forecast of the reliability that may be eventually achieved by any particular design. Prediction is therefore a fundamental activity in the overall design evaluation process. The prediction process does not in itself contribute directly to the reliability of a system, but the values produced constitute essential criteria for selecting courses of action that affect the reliability of a design. By carrying out prediction in a detailed and systematic manner, the process will help to identify potential reliability problems, including Misinterpretation of requirements. Sources of unreliability. Design imbalance. far@ucalgary.ca 108

109 The Raleigh-Putnam Model It is usually assumed that t over the life of the project the faults detected (per some unit of time) will resemble a Raleigh curve The rate of expending effort is proportional to the rate of committing errors. Putnam assumes that the rate of effort expenditure is a Raleigh curve. Design & Coding Source: QSM Web page Testing Release SENG421 (Winter 2006) far@ucalgary.ca 109

110 ELCM: The Process /1 It is usually assumed that t over the life of the project the failures detected (per month) resembles a Raleigh curve Obtain the milestones for the schedule: Start date and total months in project Date of expected full operational capability - td Estimate the total number of failures over the life of the project - Er. AR Raleigh curve can be generated dby solving for each month t (1 to number of months in project) using this equation: E m 2 2 6E t d r t 2 d t e 3t far@ucalgary.ca 110

111 ELCM: The Process /2 Use this profile to gauge the fault detection process during each phase of development. In particular, this profile can be used to gauge the original schedule estimate and the prediction for the total number of defects. For example, the estimated number of defects impacts the height of the curve while the schedule impacts the length of the curve. If the actual defect curve is significantly different from the predicted curve then one or both of these parameters may have been estimated incorrectly and should be brought to the attention of management. 111

112 Reliability Growth Model Assumptions vs. Practice Assumptions Software does not change and defects are fixed immediately Testing Operational Profile (OP) Constant test effort and independent failure intervals All failures are observable Collect failure reports Remaining failures are either constant or decreasing In practice Software changes rapidly and certain defects are scheduled to be fixed in a later date Focus on functional testing. It is difficult to define OP and perform operational tests. Varying test effort (due to holidays and vacations) and failure intervals may vary Testing in a controlled environment may be different from running software in live environment Collect defect reports. Any kind of defect observed is recorded. Not all reports address a failure Remaining failures may actually increase due to improper bug fixes [Stringfellow & Andrews 2002] far@ucalgary.ca 112

113 Exercise Let s assume that the MTTF for a software system is 2000 hours. The probability density function (PDF) for such system has an exponential form given by: t f t e for t 0 f t 0 for t 0 Draw the approximate plot of f(t) for the random variable t. Find the probability bilit that t the next failure happens in more than 1000 hours. Find the probability that the next failure happens in less than 1000 hours. Find the probability that the system fails in the interval from 1000 to 2000 hours. Find reliability of the system for 1,000 hours of operation. far@ucalgary.ca 113

114 Exercise (Answers) Draw the approximate plot of f(t) for the random variable t. f t T 114

115 Exercise (Answers) Find the probability that the next failure happens in more than 1000 hours. 1 1 t F T 1000 e dt e Find the probability that the next failure happens in less than 1000 hours. F T F t

116 Exercise (Answers) Find the probability that the system fails in the interval from 1000 to 2000 hours. 1 F t e dt e e t Find reliability of the system for 1,000 hours of operation tt 2000 R t 1000 e e

117 117