Software Reliability.... Yes sometimes the system fails...

Transcription

1 Software Reliability... Yes sometimes the system fails...

2 Motivations Software is an essential component of many safety-critical systems These systems depend on the reliable operation of software components How can we decide, whether the software is ready to ship after system test? How can we decide, whether an acquired or developed component is acceptable? G. Antoniol 2012 LOG6305 2

3 What is Reliability? IEEE-Std : Software reliability is defined as the probability of failure-free operation for a specified period of time in a specified environment ISO9126: Reliability is the capability of the software product to maintain a specified level of performance when used under specified conditions G. Antoniol 2012 LOG6305 3

4 Reliability Informal Probability of failure-free operation for a specified time in a specified environment for a given purpose This means quite different things depending on the system and the users of that system Informally, reliability is a measure of how well system users think it provides the services they require Reliability is a measure of how well the software provides the services expected by the customer. Quantification: Number of failures, severity G. Antoniol 2012 LOG6305 4

5 Software Reliability Cannot be defined objectively Reliability measurements which are quoted out of context are not meaningful Requires operational profile for its definition The operational profile defines the expected pattern of software usage Must consider fault consequences Not all faults are equally serious. System is perceived as more unreliable if there are more serious faults G. Antoniol 2012 LOG6305 5

6 Failure Fault A failure corresponds to unexpected run-time behavior observed (by a user) A fault is a static software characteristic which causes a failure to occur Faults need not necessarily cause failures. If a user does not notice a failure, is it a failure? Remember most users don t know the software specification G. Antoniol 2012 LOG6305 6

7 Input/Output Mapping Input set I e Inputs causing erroneous outputs Program Output set O e Erroneous outputs G. Antoniol 2012 LOG6305 7

8 How Can we Measure and Model Reliability The cumulative number of failures by time t Failure Intensity: The number of failures per time unit at time t Mean-Time-To-Failure MTTF: Average value of inter-failure time interval We do not know for sure when the software is going to fail next => model failure behavior as a random process G. Antoniol 2012 LOG6305 8

9 Modeling Cumulative # Failures? x x x x time Failure 1 Failure 2 Failure 3 Failure 4 G. Antoniol 2012 LOG6305 9

10 Reliability Growth Reliability is improved when software faults which occur in the most frequently used parts of the software are removed Removing x% of software faults will not necessarily lead to an x% reliability improvement In a study, removing 60% of software defects actually led to a 3% reliability improvement Removing faults with serious consequences is the most important objective G. Antoniol 2012 LOG

11 Problems with Quantification? US standard for equipment on civil aircraft: [A critical failure must be ] so unlikely that [it] need not be considered ever to occur, unless engineering judgment would require [its] consideration. Typical requirement for critical flight systems: 10-9 probability of failure per flying hour based on a flight mean duration of 10 hours. Problem 1: such a level of reliability cannot be demonstrated would require too much testing time Problem 2: Reliability is relative to the usage of the system G. Antoniol 2012 LOG

12 Usage Patterns Possible inputs User 1 Erroneous inputs User 3 User 2 G. Antoniol 2012 LOG

13 Reliability and Quality Reliability is usually more important than other quality aspects Unreliable software isn't used Hard to improve the quality of unreliable systems Software failure costs often far exceed system costs Costs of data loss are very high G. Antoniol 2012 LOG

14 Reliability Metrics Probability of failure on demand This is a measure of the likelihood that the system will fail when a service request is made POFOD = means 1 out of 1000 service requests result in failure Relevant for safety-critical or non-stop systems Rate of fault occurrence (ROCOF) Frequency of occurrence of unexpected behavior ROCOF of 0.02 means 2 failures are likely in each 100 operational time units Relevant for operating systems, transaction processing systems G. Antoniol 2012 LOG

15 Reliability Metrics Mean time to failure Measure of the time between observed failures MTTF of 500 means that the time between failures is 500 time units Relevant for systems with long transactions e.g. CAD systems Availability Measure of how likely the system is available for use. Takes repair/restart time into account Availability of means software is available for 998 out of 1000 time units Relevant for continuously running systems e.g. telephone switching systems G. Antoniol 2012 LOG

16 Reliability Measurement Measure the number of system failures for a given number of system inputs Used to compute POFOD Measure the time (or number of transactions) between system failures Used to compute ROCOF and MTTF Measure the time to restart after failure Used to compute AVAIL G. Antoniol 2012 LOG

17 Time Units Time units in reliability measurement must be carefully selected. Not the same for all systems Raw execution time (for non-stop systems) Calendar time (for systems which have a regular usage pattern e.g. systems which are always run once per day) Number of transactions (for systems which are used on demand) G. Antoniol 2012 LOG

18 Failure Consequences Reliability measurements do NOT take the consequences of failure into account Transient faults may have no real consequences but other faults may cause data loss or corruption and loss of system service May be necessary to identify different failure classes and use different measurements for each of these G. Antoniol 2012 LOG

19 Specification Reliability requirements are only rarely expressed in a quantitative, verifiable way. To verify reliability metrics, an operational profile must be specified as part of the test plan. Reliability is dynamic - reliability specifications related to the source code are meaningless. No more than N faults/1000 lines. This is only useful for a post-delivery process analysis. G. Antoniol 2012 LOG

20 A Classification Failure class Transient Permanent Recoverable Unrecoverable Non-corrupting Corrupting Description Occurs only with certain inputs Occurs with all inputs System can recover without operator intervention Operator intervention needed to recover from failure Failure does not corrupt sys tem state or data Failure corrupts system s tate or data G. Antoniol 2012 LOG

21 Specification Validation It is impossible to empirically validate very high reliability specifications No database corruptions means POFOD of less than 1 in 200 million If a transaction takes 1 second, then simulating one day s transactions takes 3.5 days It would take longer than the system s lifetime to test it for reliability G. Antoniol 2012 LOG

22 Compromises Because of very high costs of reliability achievement, it may be more cost effective to accept unreliability (and pay for failure costs) Social, business and political factors: A reputation for unreliable products may lose future business Depends on system type - for business systems in particular, modest reliability may be adequate G. Antoniol 2012 LOG

23 Costs Cost Low Medium High Very Reliability high Ultrahigh G. Antoniol 2012 LOG

24 Statistical Testing Testing software for reliability rather than fault detection Test data selection should follow the predicted usage profile for the software Measuring the number of errors allows the reliability of the software to be predicted An acceptable level of reliability should be specified and the software tested and amended until that level of reliability is reached G. Antoniol 2012 LOG

25 Testing Procedure Determine operational profile of the software Generate a set of test data corresponding to this profile Apply tests, measuring amount of execution time between each failure After a statistically valid number of tests have been executed, reliability can be measured G. Antoniol 2012 LOG

26 Difficulties Uncertainty in the operational profile This is a particular problem for new systems with no operational history. Less of a problem for replacement systems High costs of generating the operational profile Costs are very dependent on what usage information is collected by the organisation which requires the profile Statistical uncertainty when high reliability is specified Difficult to estimate level of confidence in operational profile Usage pattern of software may change with time G. Antoniol 2012 LOG

27 Reliability Growth Model Growth model is a mathematical model of the system reliability change as it is tested and faults are removed Used as a means of reliability prediction by extrapolating from current data Depends on the use of statistical testing to measure the reliability of a system version G. Antoniol 2012 LOG

28 Reliability Models Many different reliability growth models have been proposed No universally applicable growth model Reliability should be measured and observed data should be fitted to several models Best-fit model should be used for reliability prediction G. Antoniol 2012 LOG

29 Models and Prediction Reliability = Measured reliability Fitted reliability model curve Required reliability Estimated Time time of reliability G. Antoniol 2012 LOG6305 achievement 29

30 Hardware Reliability Well-developed reliability theories A hardware component may fail due to design error, poor fabrication quality, momentary overload, deterioration, etc. The random nature of system failure enables the estimation of system reliability using probability theory Can it be adapted for software? No physical deterioration, most programs always provide the same answer to the same input, the fabrication of code is trivial, etc. G. Antoniol 2012 LOG

31 Software Reliability Engineering No method of development can guarantee totally reliable software => important field in practice! A set of statistical modeling techniques Enables the achieved reliability to be assessed or predicted, quantitatively and objectively Based on observation of system failures during system testing and operational use Uses general reliability theory, but is much more than that G. Antoniol 2012 LOG

32 Applications How reliable is the program/component now? Based on the current reliability of a purchased/reused component: can we accept it or should we reject it? Based on the current reliability of the system: can we stop testing and start shipping? How reliable will the system be, if we continue testing for some time? When will the reliability objective be achieved? How many failures will occur in the field (and when) to help plan resources for customer support? G. Antoniol 2012 LOG

33 Random Variable A random variable X is a function defined over a sample space S X associates a real numbers x with each possible outcome in S: X(e) = x If S is finite or countable infinite we say that X is discrete If X can be any value in a given interval we say that X is a continuous random variable G. Antoniol 2012 LOG

34 Discrete Random Variable If S is countable X(e) is countable Event probability: { : ( ) =, } P e X e x e S Probability density function (pdf): p( xi ) 0 p( xi ) = 1 Cumulative distribution function (cdf): P { } x X x = i p( xi ) x i x G. Antoniol 2012 LOG

35 Mean and Variance [ ] = x E X x p( x) [( ) ] Var(X) = E X - E(X) 2 G. Antoniol 2012 LOG

36 Continuous Random Variables The pdf of a continuos random variable X is a function: f ( x) + Notice: x: f ( x) 0 f ( x) dx = 1 Cumulative distribution: x F(x) = f ( y) dy G. Antoniol 2012 LOG

37 Mean and Variance E[ X ] = + xf ( x) dx [ [ ]) 2 ] Var(X) = E (X - E X G. Antoniol 2012 LOG

38 Conditional Probabilities We are interested in a subset B of the universe set S, in other words the sample space is essentially B. Let P(B)>0 and let A be a subset of S we define the probability of A relative to the new sample space B: P(A B) = P(A B) / P(B) G. Antoniol 2012 LOG

39 Conditional Distributions Discrete: p( x x ) = P(X = x X = x ) = i j i j P(X = x X = i P(X = x j ) x j ) Continuous: f ( x y) = f ( x, y) f ( y) Notice: f + (x) = f ( x, y) dy G. Antoniol 2012 LOG

40 Stochastic Processes A stochastic process is a collection of random variables: X or X(t) t where t is a suitable index set. In other words t may be a discrete time unit the the index set is T=[0,1,2,3,...] or it can be a point in a continuos time interval, then the index set is equal to: T =[0, ) G. Antoniol 2012 LOG

41 Reliability The random variable of interest is the time to failure T. We focus our interest on the probability that the time to failure between is in some interval. The time to failure T in some interval: In other words: T (t, t + Δt) P(t T t + Δt) = f (t) Δt = F(t + Δt) - F(t) The pdf f is also called failure density function G. Antoniol 2012 LOG

42 Reliability Function R(t) Since T is defined only for positive values: F(t) = P(0 T t) = f ( y) dy is the probability of observing a failure before t. The probability of success at time t, R(t), i.e. the failure time is larger than t: R(t) = P(T > t) = 1- F(t) = t 0 + t f ( y) dy G. Antoniol 2012 LOG

43 Failure Rate Failure rate is the probability that a failure per unit time occurs in an interval, given that a failure has not occurred before t: P(t T < t + Δt T > t) Δt = P(t T < t + Δt) Δt P(T > t) = F(t + Δt) - F(t) Δt R(t) G. Antoniol 2012 LOG

44 Hazard Rate The hazard rate is the limit when the failure rate interval approaches to zero: F(t + Δt) - F(t) f ( t) z( t) = lim = Δt 0 Δt R(t) R(t) Given that the system survived up to t, z(t), is the instantaneous failure rate, i.e. z(t)dt is the probability that a system of age t will fail in the small interval t, t+dt G. Antoniol 2012 LOG

45 R(t) and z(t) By definition: z( t) = df(t) dt 1 R(t) Since: R(t) =1- F(t) which means dr(t) dt It can be derived: lg R(t) = - z (x)dx + c Since the system is usually good at t=0, R(0)=1: t 0 = - df(t) dt R(t) = exp - t z (x)dx 0 G. Antoniol 2012 LOG

46 R(t) z(t) f(t) R(t) = exp - t z (x)dx 0 t f (t) = z(t) exp - z(x)dx 0 G. Antoniol 2012 LOG

47 Hazard Rate Over Time Typical hazard rate: Hazard rate z(t) Region I: early failure Bathtub curve Region II: random failure Region III: wear-out failure Debugging phase Normal operating phase Do not apply to software G. Antoniol 2012 LOG time

48 Example Constant hazard rate z( t) = λ f ( t) = λ R(t) = e - λ t λ e - t Linearly increasing hazard rate z( t) = K λ f ( t) = Kt e -Kt 2 2 R(t) = e -K t2 2 G. Antoniol 2012 LOG

49 Expected Life It is often referred as Mean Time To Failure (MTTF): In term of reliability: MTTF = E[ T ] = 0 0 tf ( t ) dt MTTF = R( t) dt G. Antoniol 2012 LOG

50 MTTF Examples Constant hazard rate: MTTF = R(t)dt = e 1 - λ t dt = λ 0 0 Linearly increasing hazard rate: MTTF = R(t)dt = e K t 2 dt = 1 Γ 2 = 2 K 2 π 2K G. Antoniol 2012 LOG

51 Another Point of View Instead of the random variable T, the time to next failure, we can model with the number of failure a system experienced by time t. Clearly the number of failure experienced by time t is a random variable, actually it is a random process, since once fixed the time the number of experienced failures is a random variable. G. Antoniol 2012 LOG

52 Failure Intensity Let M(t) be a random process representing the cumulative number of failures by time t, its mean value is: µ (t) = E[ M(t) ] The failure intensity function is: µ λ (t) = d (t) dt = d dt ( E[ M(t) ]) G. Antoniol 2012 LOG

53 Failure Intensity m(t) = E[M(t)], with M(t) be the random process denoting the cumulative number of failures by time t, we denote m(t) as its mean value function l(t) : Failure intensity, average number of failures by unit of time at time t, Instantaneous rate of change of the expected number of failures (m(t)) Reliability growth implies: dl(t)/dt < 0 G. Antoniol 2012 LOG

54 Graphical Representation G. Antoniol 2012 LOG

55 Failure Intensity Properties The failure intensity represent the expected number of failures per unit of time. The number of failures that will occur between t and t + can be approximated by λ (t) Δt While µ (t) is a non decreasing function, it is the cumulative number of failures at t, the failure intensity, its derivative, must decrease in order to obtain an increase in R(t) Δt G. Antoniol 2012 LOG

56 Reliability Models Classification Time domain: wall clock versus execution time Category: the total number of failure that can be experienced in infinite time (finite or infinite) Type: the distribution of the number of failures experienced by time t (Poisson or Binomial) G. Antoniol 2012 LOG

57 Poisson Type The distribution of the failure experienced by time t is a Poisson process. In other words if: t = 0, t, t,..., t = t n is a partition of the interval [0,t] we have a Poisson process if f i i = 1,2,...,n the number of faults detected in the i-th interval are independent Poissons variables with [ ] E f = µ (t ) µ (t ) i i i-1 G. Antoniol 2012 LOG

58 Classification µ t ( ) If is a linear function of time, we say that M(t) is a Homogenous Poisson Process (HPP) µ t ( ) If is a non linear we say that M(t) is a Non-Homogeneous Poisson Process G. Antoniol 2012 LOG

59 Binomial Processes There is a fixed number of faults (N) in the software at the beginning of the time in which the software is observed When a fault is detected it is removed immediately If Ta is the random variable denoting the time to failure of fault a, then the Ta for a=1,2,3,...n are independently an identical distributed random variables (as well as the remaining faults!) G. Antoniol 2012 LOG

60 Software Reliability Engineering Process define reliability objective modeling expected system usage prepare test cases execute test collect failure data Perform: Reliability Certification Monitor: Reliability Growth Requirements Design/Code Test G. Antoniol 2012 LOG

61 Software Reliability Engineering Process define reliability objective modeling expected system usage prepare test cases execute test collect failure data Perform: Reliability Certification Monitor: Reliability Growth Requirements Design/Code Test G. Antoniol 2012 LOG

62 Model Expected System Usage Definition of reliability assumes a specified environment ð To make statements on reliability in field during system test, we must test in conditions that are similar to field conditions Model how users will employ the software: environment, type of installation, distribution of inputs over input space According to the usage model, test cases are selected randomly One example of usage model Operational Profile (Musa): Set of system operations and their probabilities of occurrence G. Antoniol 2012 LOG

63 Operational Profile 40% probability 30% 20% Operation1 Operation2 Operation3 Operation4 10% G. Antoniol 2012 LOG

64 Operations Major system logical task of short duration, which returns control to the system when complete and whose processing is substantially different from other operations major: related to functional requirement (similar to use cases) logical: not bound to software, hardware, users, machines short: 100s-1000s operations per hour under normal load conditions different: likely to contain different faults In OO systems, an operation ~ use case G. Antoniol 2012 LOG

65 Examples Command executed by a user Response to an input from an external system or device, e.g., processing a transaction, processing an event (alarm) Routine housekeeping, e.g., file backup, database cleanup G. Antoniol 2012 LOG

66 Develop Operational Profiles Identify who/what can initiate operations Users (of different types), external systems and devices, system itself Create a list of operations for each operation initiator and consolidate results Source: requirements, draft user manuals, prototypes, previous program versions, discuss with expected users 20 to several hundred operations are typical Determine occurrence rates (per hour) of the individual operations existing field data, record field operations, simulation, estimates Derive occurrence probabilities G. Antoniol 2012 LOG

67 Example: Fone Follower (FF) Requirements Forward incoming phone calls (voice, fax) anywhere Subscriber calls FF, enters phone numbers for where he plans to be as a function of time FF forwards incoming calls from network (voice, fax) to subscriber as per program. If no response to voice call, subscriber is paged (if subscriber has pager). If no response or no pager, voice calls are forwarded to voice mail Subscribers view service as standard telephone service combined with FF FF uses vendor-supplied operating system of unknown reliability G. Antoniol 2012 LOG

68 Initiators of Operations Event driven systems often have many external systems that can initiate operations in them Typically, the system under study may initiate itself administrative and maintenance operations FF: User types: subscribers, system administrators External system: Telephone network FF (audits, backups) G. Antoniol 2012 LOG

69 FF Operations List Subscriber System Administrator Network FF Phone number entry Add subscriber Delete subscriber Proc. voice call, no pager, answer Proc. voice call, no pager, no answer Proc. voice call, pager, answer Proc. voice call, pager, ans. on page Proc. voice call, pager, no ans. on page Proc. fax call Audit section of phone number database Recover from hardware failure G. Antoniol 2012 LOG

70 FF Operational Profile Operation Phone number entry Add subscriber Delete subscriber Proc. voice call, no pager, answer Proc. voice call, no pager, no answer Proc. voice call, pager, answer Proc. voice call, pager, ans. on page Proc. voice call, pager, no ans. on page Proc. fax call Audit section of phone number database Recover from hardware failure Occ.Rate (per hr) Occ.Prob ,1 0,10 0,0005 0,0005 0,18 0,17 0,17 0,12 0,10 0,15 0,009 0, G. Antoniol 2012 LOG

71 Statistical Testing Testing based on operational profiles is referred to as statistical testing This form of testing has the advantage of testing more intensively the system functions that will be used the most Good for reliability estimation, but not very effective in terms of finding defects Hence we differentiate testing that aims at finding defects (verification) and testing whose purpose is reliability assessment (validation). There exists research on techniques that combine white-box testing and statistical testing G. Antoniol 2012 LOG

72 Software Reliability Engineering Process define reliability objective modeling expected system usage prepare test cases execute test collect failure data Perform: Reliability Certification Monitor: Reliability Growth Requirements Design/Code Test G. Antoniol 2012 LOG

73 Can We Accept a Component? Certification Testing: Show that a (acquired or developed) component satisfies a given reliability objective (e.g., failure intensity) Generate test data randomly according to usage model (e.g., operational profile) Record all failures (also multiple ones) but do not correct G. Antoniol 2012 LOG

74 Procedure Use a hypothesis testing control chart to show that the reliability objective is/is not satisfied Reliability Demonstration Chart» Collect times at which failures occurred» Normalize data by multiplying with failure intensity objective (using same units!)» Plot each failure in chart» Based on region in which failure falls, accept or reject component G. Antoniol 2012 LOG

75 Reliability Demonstration Chart Reliability Demonstration Chart 16 Failure No Reject Continue Accept Fail.No Mcalls at failure Normalized measure => accept Failure Intensity Objective: 4 failures/mcalls Normalized failure time (failure time x failure intensity objective) John Musa, Software Reliability, 1998 G. Antoniol 2012 LOG

76 Creating Demonstration Charts Select discrimination ratio γ(acceptable factor of error in estimating failure intensity) Select consumer risk α (probability of accepting a system that does not satisfy failure intensity objective) Select supplier risk β (probability of rejecting a system that does satisfy failure intensity objective) Recommended defaults (γ=2, α=0.1, β=0.1) 10% risk of wrongly accepting component when failure intensity is actually >= 2 * failure intensity objective 10% risk of wrongly rejecting component when failure intensity is actually <=1/2 * failure intensity objective G. Antoniol 2012 LOG

77 Boundary Lines The next step is to construct boundary lines (between reject and Continue and Continue and Accept) according to the following formulae T N = A nlnγ 1 γ T N = B nlnγ 1 γ β A = ln 1 α Where T N = boundaries for normalized failure time (x axis) n = failure number (y axis) B = 1 ln β α G. Antoniol 2012 LOG

78 Software Reliability Engineering Process define reliability objective modeling expected system usage prepare test cases execute test collect failure data Perform: Reliability Certification Monitor: Reliability Growth Requirements Design/Code Test G. Antoniol 2012 LOG

79 Failure Data: Interfailure times G. Antoniol 2012 LOG

80 Can We Stop Testing? reliability objective Monitor: Reliability Growth execute test collect failure data Select appr. reliability models no Use models to compute current reliability reliability objective met? reliability objective yes Failure intensity measured failure intensity fitted reliability model curves estimated time of reliability achievement delivery G. Antoniol 2012 LOG

81 MUSA Model Developed by John Musa at ATT Bell Labs One of the first proposed and still widely applied It is based on execution time Execution time can be converted into calendar time ( in a second part of the model) G. Antoniol 2012 LOG

82 Basic Assumptions The rate of fault detection is proportional to the current number of faults in the code The fault detection rate remains constant over the interval between faults occurrence A fault is corrected instantaneously, without introducing new faults in the code The failure when the faults are detected are independent The software is operated in a similar manner as that in which reliability predictions are made Every fault has the same change of being encountered within a severity class as any other fault in that class G. Antoniol 2012 LOG

83 MUSA Assumptions Basic assumptions The cumulative number of failure by time t, M(t), follows a Poisson process with mean functions: The execution times between failure are piecewise exponentially distributes, i.e. the hazard rate is a constant [ β t ] 1 e µ (t) = β 1 with β > 0 0 0,1 G. Antoniol 2012 LOG

84 MUSA Assumptions... Resources available (testing personnel, debuggers, programmers) are constant over the segment of time for which the system is observed Fault-identification personnel can be fully utilized and computer utilization is constant Fault-correction personnel is established by the limitation of fault queue length for any fault correction person. G. Antoniol 2012 LOG

85 MUSA Required Data To apply MUSA model we need the actual time that software failed or the elapsed time between failures G. Antoniol 2012 LOG

86 MUSA Model Form Given the assumptions: [ β t ] 1 e µ (t) = β 1 with β > 0 0 0,1 λ (t) = β 0 β 1 e -β 1 t This is a finite failure model with the total number of faults: β 0 G. Antoniol 2012 LOG

87 Maximum Likelihood Estimation MLE chooses an estimator such that the observed sample is the most likely to occur among all possible samples. Assuming X has a pdf f ( x θ) where θ is unknown the joint pdf of the sample { X 1,..., X n } is called the likelihood function. L( θ) = n i=1 f ( x i θ) G. Antoniol 2012 LOG

88 MUSA MLE Suppose we observed n failure, at times: t1, t 2,..., t n Suppose from the last failure t n an additional time of x has elapsed which means the total time the system has been observed from the beginning is: t x with x 0 n + The likelihood function is: n n n 1 i L( β0, β1) = β0 β1 e e i= 1 [ -β ( ) ] - t e 1 + β β tn x G. Antoniol 2012 LOG

89 MUSA MLE Solution The parameters estimation for the betas give:! β 0 n = 1 exp! + [ β ( )] 1 t n x n! β n exp! 1 1 ( t ) n + x ( t x) [ β ] n n + i 1 t i= 1 = 0 Substitute the betas estimate into the equation of model to µ t, λ t,...,etc obtain an estimate of R(t), ( ) ( ) G. Antoniol 2012 LOG

90 Example Suppose the following times of failure were observed: 10, 18, 32 49, 64, 86, 105, 132, 167, 207 in hours and suppose an additional 15 CPU hours of no failures:! β 0 10 = 1 exp 222! ( β ) ! β exp 222! = 0 ( β ) 1 1 With the result of :! β = 136. and! β = G. Antoniol 2012 LOG

91 Parameters Interpretation By making the correspondence: β = φb and β = ν We represent the constant hazard rate φ factor B. and the fault reduction B models the imperfect bug fixing process; Military standards suggest B = 0.955, close to 1 but not 1 that means 5% of time when fixing a defect we really introduce a new one. G. Antoniol 2012 LOG

92 MUSA Simplest Solution Consider the failure intensities a function of the mean number of failure: µ λ( µ ) λ 1 λ( µ ) = β ( β µ ) = ν ou 0 find the least square solution of this line with the actual data. One parameter are available we can decide how much effort is still required. Let λ, λ be the final / present failure intensity F P ( ) Δµ = ν0 λ λ λ 0 P F Δt = ν 0 λp lg λ λ 0 F G. Antoniol 2012 LOG

93 Grouping of Data Chose a number of failure you wish to consider for each sub-interval say k (suggested k=5) Divide the data set into p independent sub-sets each containing exactly k failures (p is the smallest integer greater or equal the total number of failure divided by k) Estimate failure intensity as the ratio between the number of failure in the interval and the interval duration G. Antoniol 2012 LOG

94 Example k=3, first interval 2 failure, second interval 3, third 3... G. Antoniol 2012 LOG

95 Equations r l = t kl k t k( l 1) l = 1, 2,..., p 1 r p = me t m e total number of failure t e time when observations are stopped m l = k( l 1 ) cumulative failure up to l interval Build the couples: ( r m ) l l e k( p 1) t ( 1) k p, fit with the model λ ( µ ) µ = λ0 1 ν 0 G. Antoniol 2012 LOG

96 G. Antoniol 2012 LOG Line fitting ( ) i y i x, Remember: for a data set And an equation y=bx+a We have: bx y a x x n y x y x n b n i n i n i n i n i i ˆ ˆ ˆ = =

97 Remark The role of (r,m) are those of ( λ, µ ) But our equation is: λ µ 0 ν ou 0 ( µ ) λ 1 λ( µ ) = β ( β µ ) = 1 0 ν λ Th is to say we need to 0 µ = λ + ν invert the relation: 0 0 G. Antoniol 2012 LOG

98 Geometric Model Fault detection is a geometric Series, constant between two faults There are infinite faults in the system λ µ ( µ β ), = β β e 0 ( t) = β ln( 1+ β t) µ / β 1 Time between 2 faults is an exponential distributions λ ( t) = β 0 1+ β 1 β t 1 G. Antoniol 2012 LOG

99 G. Antoniol 2012 LOG Closed form from Geometric Model ( ) ( ) ( ) + = = = = = = = = = p l l p l l p l l p l l p l l l p l l p l l m r p r m r m p m m p ˆ 1 ln 1 exp ˆ ˆ ln ln ˆ β β β β

100 The Musa-Okumoto Model General Assumptions The system test reflects the intended usage of the system The failures are independent No new faults introduced during corrections Model-specific Assumptions The cumulative number of failures by time t follows a Poisson Process (NHPP) Failure intensity decreases exponentially with the expected number of failures experienced m: l(m)=l 0 exp(-qm) q>0 is failure intensity decay parameter, l 0 is initial failure intensity. Data Requirements Actual times when failures occurred or failure time intervals (execution time) G. Antoniol 2012 LOG

101 Poisson Processes k m(t) 2 1 x x x x Time t P k ( t) = e αt ( αt) k! k G. Antoniol 2012 LOG

102 The Musa-Okumoto Model (2) y:=10*exp(-0.02*x) λ( µ ) = λ e λ( µ ) 0 θµ = λ e 0 θµ 5 Failureintensity(lambda) Meanvaluesexperienced(my) Failure intensity versus failures experienced G. Antoniol 2012 LOG

103 Deriving m(t) and l(t) λ( µ ) = λ e 0 θµ λ ( τ ) = dµ dτ ( τ ) θµ ( τ ) = λ 0 e = λ λ θτ µ ( τ ) = ln( λ θτ + 1) 0 θ G. Antoniol 2012 LOG

104 The Musa-Okumoto Model (3) Meanfailuresexperienced(my) y:=(1/0.02)*log(10*0.02*x+1) µ ( τ ) = ln( λ0 θτ + 1) 80 θ Executiontime(tau) Failures experienced versus execution time t G. Antoniol 2012 LOG

105 The Musa-Okumoto Model (4) Failureintensity(lambda) Failure intensity versus execution time t y:=10/(10*0.02*x+1) λ0 λ( τ ) = λ θτ Executiontime(tau) 0 Collect failure data: time when failure occurred Use tools to determine model parameters l 0 and q Determine whether reliability objective is met or how long it might take to reach G. Antoniol 2012 LOG

106 The Musa-Okumoto Model (5) Once the parameters l 0 and are estimated, it can be used to predict failure intensity in the future, not only to estimate its current value. From this, we can plan how much additional testing is likely to be needed The model allows for the realistic situation where fault correction is not perfect (infinite number of failures at infinite time) When faults stop being corrected, the model reduces to a homogeneous Poisson process with failure intensity l as the parameter the number of failures expected in a time interval then follows a Poisson distribution G. Antoniol 2012 LOG

107 Reliability Estimation Probability of 0 failures in a time frame of length t (reliability): P τ ) 0 ( = e λτ Additional time/failures to reach, from present failure intensity l P, the required failure intensity l F : Dt 1 Δτ = 1 1 θ λ F λ P 1 Δµ = = θ λ P λ F G. Antoniol 2012 LOG

108 Conceptual View of Prediction G. Antoniol 2012 LOG

109 Time Measurement Execution (CPU) time: may be difficult to collect but accurate Calendar time: Easy to collect but makes numerous assumptions (equivalent testing intensity with time periods) Testing effort: Same as calendar time, but easier to verify the assumptions Specific time measurement may often be devised: #Calls, Lines of code compiled, #transactions G. Antoniol 2012 LOG

110 Selection of Models Several Models exist (ca. 40 published) => Which one to select? Not one model is consistently the best. Assumptions of Models: Definition of testing process Finite or infinite number of failures? No faults introduced during debugging? Distribution of data (Poisson, Binomial)? Data requirements? (inter-failure data, failure count data) Assessing the goodness-of-fit Kolmogorow-Smirnov, Chi-Square Trends in Data (prior to model application) Usually, Reliability Models assume reliability growth. Laplace test can be used to test whether we actually experience growth G. Antoniol 2012 LOG

111 Motorola zero failure Model derived from Brettschneider 89 ae bt gives: ln [ fd /(0.5 + fd) ] hours to lg[ ( fd) /( tf + fd) ] fd: failure density needed tf: total failure last failure G. Antoniol 2012 LOG

112 Pro s and Con s Pro s Con s Reliability can be specified Objective and direct assessment of reliability Prediction of time to delivery Usage model may be difficult to devise Selection of reliability growth model difficult Measurement of time crucial but may be difficult in some contexts Not applicable to safety-critical systems as very high reliability requirements would take years of testing to verify. G. Antoniol 2012 LOG