Interpreting a successful testing process: risk and actual coverage

Transcription

1 Interpreting a successful testing process: risk and actual coverage Mariëlle Stoelinga and Mark Timmer Faculty of Computer Science, University of Twente, The Netherlands {marielle, timmer}@cs.utwente.nl Abstract. Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e. the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e. the fraction of possible faults that has been covered. We provide algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion. 1 Introduction Software becomes more and more complex, making thorough testing an indispensable part of the development process. The U.S. National Institute of Standards and Technology has assessed that software errors cost the American economy almost sixty billion dollars annually. More than a third of these costs could be eliminated if testing occurred earlier in the development process [14]. An important fact about testing is that it is inherently incomplete, since testing everything would require infinitely many input scenarios. On the other hand, passing a well-designed test suite does increase the confidence in the correctness of the tested product. Therefore, it is important to assess the quality of a test suite. Two fundamental concepts have been put forward to evaluate test suite quality: (1) coverage metrics determine which portion of the requirements and/or implementation-under-test has been exercised by the test suite; (2) riskbased metrics assess the risk of putting the tested product into operation. Although existing coverage measures, such as code coverage in black-box testing [3, 12] and state and/or transition coverage in white-box testing [8, 13, 19], give an indication of the quality of a test suite, it is not necessarily true that higher coverage implies that more, or more severe, errors are detected. This is because these metrics do not take into account where in the system errors are most likely to occur. Risk-based testing methods do aim at reducing the expected number of errors, or their severity. However, these are often informal [15], based on heuristics [2], or indicate which components should be tested best [1], but seldomly quantify the risk after a successful testing process in a precise way.

2 This paper presents a framework in which test coverage and risk can be defined, computed and optimised. Key properties are a rigorous mathematical treatment based on solid probabilistic models, and the result that higher coverage (resp. lower risk) implies a lower expected number of faults. Starting point in our theory is a weighted fault specification (WFS), which consists of a system specification describing the desired system behaviour, a weight function that describes the severity of the errors, and an error function describing the probability that a certain error has been made. From the WFS, we derive its underlying probability model, i.e. a random variable that exactly describes the distribution of erroneous implementations. This allows us to define risk and actual coverage in an easy and precise way. Given a WFS, we define the risk of a test suite as the expected fault weight that remains after this test suite passes. We also introduce actual coverage for a test suite, which quantifies the risk reduction after passing the test suite. Whereas the risk is based on errors contained within the entire system, actual coverage only relates to the part of a system we tested. This coincides with the traditional interpretation of coverage. In this way our methods refine [5], which introduces a concept we would call potential coverage, since it considers which faults can be detected during testing. Our measures, on the other hand, take into account the faults that are actually covered during a test execution. We build up our framework in three steps of increasing complexity. First, we define our theory for systems modelled as finite functions. While being an intermediate step, this model can be used for testing individual functions or modules. Second, the notions are generalised to nondeterministic functions. This adds a factor of uncertainty, since different executions of the same test case may yield different results. Thus, observing a correct response once does not imply correctness. We therefore add a failure function to the WFS, describing the probability that incorrectly implemented behaviour yields a failure. Finally, we generalise our notions to input-output labelled transition systems (IOLTSs) [18], which are a canonical model for embedded systems testing. This adds a second factor of complexity, since their behavior is state-dependent, which makes it necessary to consider system traces rather than simple input/output pairs. For optimisation an estimation of the output probabilities, describing the expected probability with which the system chooses between specified outputs, has to be provided. Finally, while error probabilities, failure probabilities and output probabilities are important ingredients in our framework, techniques for obtaining these probabilities fall outside the scope of this paper. However, there is extensive literature on factors that determine them. For instance, as described in [6], estimating the error probabilities can be based on the software change history. They can also be based on McCabe s cyclomatic complexity number [10], Halstead s set of Software Science metrics [7], and requirements volatility [9]. The failure probabilities can be obtained by applying one of the many analysis techniques described in [11] and [20], and the output probabilities might be obtained from the developers or derived by measurements (e.g. during earlier test executions).

3 Organisation of the paper. The measures for risk and actual coverage are discussed for deterministic functions in Section 2, for nondeterministic functions in Section 3, and for IOLTSs in Section 4. Finally, Section 5 provides conclusions and future work. This paper is based on the Master s thesis of Timmer [17]. 2 Finite functions Given a set L, the set of all sequences (traces) over L is denoted by L, and the set of non-empty traces by L +. If σ, ρ L, then σ is a prefix of ρ (denoted by σ ρ) if there is a τ L such that στ = ρ. If τ L +, then σ is a proper prefix of ρ (σ ρ). We use P(L) to denote the power set of L and Distr(L) for the set of all probability distributions over L, i.e. all functions f : L [0, 1] such that s S f(s) = 1. Given a sequence X = x 1,..., x n, we write x i for the i th element in X and y X if y = x i for some i. We call X = n the size of X. Definition 1 (Finite function). A function f : L I L O is finite if both its domain L I and its co-domain L O are finite. We often write f spec for a function representing a specification. A (potentially incorrect) implementation of f spec is a finite function f impl : L I L O with the same domain and co-domain as f spec. Example 1. A standard technique in testing is equivalence partitioning, that partitions the inputs of a function into equivalence classes such that one input is representative for all others in its class [12]. Using this technique, functions like sin, max, and abs can be modelled as finite functions. Consider the function abs : R R, mapping each input on its absolute value. We identify the three equivalence classes 0, a, and a, with a > 0. Using these classes, abs can be modelled by f spec : {0, 1, 1} {0, 1} with f(0) = 0, f( 1) = 1, and f(1) = 1. This technique could also be used for modules (sets of functions), by observing a module as a Cartesian product of functions. Since it is uncertain which faults are introduced, developing an implementation can be described by a random experiment. For each input, we specify the probability that it is implemented incorrectly; its error probability. These probabilities are assumed to be independent. Additionally, a fault weight is specified for each input, denoting the severity of an incorrect implementation with respect to this input. The higher a fault weight, the higher the severity. A specification together with fault weights and error probabilities constitutes a weighted fault specification. Definition 2 (Weighted fault specification). A weighted fault specification (WFS) is a tuple W = f spec, w, p err, with f spec : L I L O a finite specification, w : L I R 0 a weight function assigning a fault weight to each input, such that 0 < a L I w(a) <. This constraint allows us to define a coverage notion that is relative to the total weight a L I w(a),

4 p err : L I [0, 1] an error function assigning to each input a L I the probability p err (a) that it is implemented incorrectly. An implementation of W is an implementation of f spec. The fault weight of an implementation f impl is the accumulated fault weight of all incorrectly implemented inputs. Definition 3 (Fault weight). Let W = f spec, w, p err be a WFS and f impl an implementation of W. Then, the fault weight of f impl is defined by w(f impl ) = w(a). a L I f impl (a) f spec(a) A test case t for a specification consists of an input and its expected output. An execution of t consists of an input and the actually observed output. Obviously, an execution is correct if the obtained output is the expected output. Similar notions exist for test suites, i.e., sequences of test cases. Definition 4 (Test case, suite, execution, and verdict). (i) A test case for a finite specification f spec : L I L O is a pair t = (a, f spec (a)) with a L I. A test suite T (L I L O ) for f spec is a finite sequence of test cases for f spec. We often write a T as a shorthand for (a, f spec (a)) T. (ii) An execution of a test case t = (a, b) against an implementation f impl is a pair e = (a, f impl (a)). An execution E = e 1,..., e n of a test suite against an implementation is the sequence of its test case executions. (iii) The verdict functions v t : (L I L O ) {pass, fail} and v T : (L I L O ) {pass, fail} are defined by v t (e) = { pass fail if b = f spec (a) otherwise v T (E) = { pass fail if i : v t (e i ) = pass otherwise For a test suite T = (a 1, f spec (a 1 )),..., (a n, f spec (a n )), the verdict of executing T against f impl is given by v T (f impl ) = v T ( (a 1, f impl (a 1 )),..., (a n, f impl (a n )) ). 2.1 Underlying probability model Since we are only interested in which inputs are handled incorrectly and not in what their incorrect response is, we partition all implementations into classes of implementations that respond correctly to exactly the same inputs. Definition 5 (Classification relation). Let f spec : L I L O be a finite specification, and f and g two implementations of f spec, then the relation fspec is defined by f fspec g iff a L I : f(a) = f spec (a) g(a) = f spec (a). We use [f ] fspec to denote the equivalence class of f with respect to fspec, and leave out the subscript f spec whenever no confusion arises.

5 We can lift the fault weight and the verdict function to classes of implementations by saying w([f impl ]) = w(f impl ) and v T ([f impl ]) = v T (f impl ). The error function p err induces a random variable F W over the equivalence classes of fspec. Because of the independence of error probabilities, its probability distribution is straightforward. Definition 6 (F W ). Let W = f spec, w, p err be a WFS, then we define F W : Ω (L I L O )/ to be the random variable describing to which equivalence class (according to ) an arbitrary implementation of W belongs. Let f impl : L I L O be an implementation of W, then P[F W = [f impl ]] = p err (a) (1 p err (a)). a L I f impl (a) f spec(a) a L I f impl (a)=f spec(a) 2.2 Risk Having defining a formal framework, we can now define the measures we are actually interested in. First of all, when a test suite passes, we want to estimate how many faults have remained undetected. To also incorporate the severity of these faults, we define the risk of an implementation as the expected fault weight after a test suite passes. Definition 7 (Risk). Let W = f spec, w, p err be a WFS and T a test suite for f spec, then the risk of an arbitrary implementation of W after it passes T is defined by risk W (T ) = E[w(F W ) v T (F W ) = pass]. For this conditional probability to be defined, P[v T (F W ) = pass] needs to be nonzero, i.e. a T : p err (a) < 1. Since we only calculate the risk after testing was successful (so it is possible to pass T ), this imposes no extra restriction. The risk can be computed easily using the formula below. Intuitively, we are certain that the inputs that were tested by T are implemented correctly because of determinism. For the inputs that were not tested, the probability of their presence is still their error probability. Theorem 1 shows that our measure is indeed a form of risk-based testing, since risk is commonly defined as probability times impact. Theorem 1. Let W = f spec, w, p err be a WFS and T a test suite for f spec. Then risk W (T ) = w(a) p err (a). a L I a T Proof. risk W (T ) = E[w(F W ) v T (F W ) = pass]

6 = = = [f impl ] (L I L O)/ [f impl ] (L I L O)/ [f impl ] (L I L O)/ = a L I a T = a L I a T = a L I a T w([f impl ]) P[F W = [f impl ] v T (F W ) = pass] w(a) a L I f impl (a) f spec(a) a L I f impl (a) f spec(a) a T [f impl ] (L I L O)/ f impl (a) f spec(a) w(a) P[F W (a) f spec (a)] w(a) p err (a) w(a) P[F W = [f impl ] v T (F W ) = pass] w(a) P[F W = [f impl ]] P[F W = [f impl ]] Note that the initial risk is risk W ( ) = a L I w(a) p err (a). Test optimisation with respect to risk. Given a WFS W = f spec, w, p err we can derive optimal test suites with respect to two (related) criteria. First, we can compute a test suite T opt-rfw of size k with a minimal risk, i.e. we obtain a minimal expected fault weight after passing T opt-rfw. That is, T opt-rfw = argmin risk W (T ), T = t 1,...,t k where argmin x f(x) denotes the value of x such that f(x) is minimal. T opt-rfw can be computed by first ordering the inputs a L I descendingly based on the product w(a) p err (a), thus obtaining a sequence a 1,..., a n such that w(a i ) p err (a i ) w(a i+1 ) p err (a i+1 ) for all 1 i < n. Then, it immediately follows from Theorem 1 that T opt-rfw = (a 1, f spec (a 1 )),..., (a k, f spec (a k )). Hence, T opt-rfw is obtained by selecting the first k test cases in the sequence. Second, given a threshold x R 0, we can compute the smallest test suite such that its risk is below x, i.e. T opt-rfw T opt-rfw = argmin T (L I L O) risk W (T ) x T.

7 Clearly, T opt-rfw can be computed by a procedure similar to the one for T opt-rfw. To be precise, we order the inputs as above and, choosing the smallest l such that risk W (T opt-rfw ) x, we take T opt-rfw = (a 1, f spec (a 1 )),..., (a l, f spec (a l )). Note that since both methods are based on sorting the n inputs, their time complexity is in O(n log n). 2.3 Actual coverage Whereas risk gives us information about the entire system, coverage usually only relates to the part of a system we tested. The absolute actual coverage (abscov) of a test suite T is defined as the accumulated fault weight of the inputs that are known to be implemented correctly after T passes. Due to determinism these are just the inputs a T. To assess the quality of a test suite, we calculate its absolute actual coverage relative to the total amount of fault weight that could potentially be present in the system. This measure will be called its relative actual coverage (relcov). Definition 8 (Coverage measures). Let W = f spec, w, p err be a WFS, and T a test suite for f spec, then abscov W (T ) = a T w(a) totcov W = a L I w(a) relcov W (T ) = abscov W (T ) totcov W Note that since weight functions never sum up to zero or infinity, relative actual coverage is properly defined. Test optimisation with respect to relcov. Again, we can derive optimal test suites for a given WFS W based on a fixed test suite size or a threshold for the measure. Note, however, that for actual coverage we have to maximise instead of minimise. The computation procedures are similar to the ones for optimisation with respect to the risk (given in Section 2.2). To be precise, we can compute a test suite T opt-cov of size k with maximal relcov, i.e. T opt-cov = argmax relcov W (T ). T = t 1,...,t k As before, T opt-cov can be computed by ordering the inputs a L I as a sequence a 1,..., a n, but now such that w(a i ) w(a i+1 ). Then, T opt-cov = (a 1, f spec (a 1 )),..., (a k, f spec (a k )). Second, given a threshold x R 0, we can compute the smallest test suite such that its relative actual coverage is below x, i.e. T opt-cov T opt-cov = argmin T. T (L I L O) relcov W (T ) x

8 Again, T opt-cov can be computed by ordering the inputs as above and taking T opt-cov = (a 1, f spec (a 1 )),..., (a l, f spec (a l )) with l the smallest integer such that relcov W (T opt-cov ) x. The time complexity of these two methods is also in O(n log n). 3 Nondeterministic finite functions Specifications now map each input on a set of correct responses, whereas implementations map each input on a probability distribution over all outputs. Definition 9 (Nondeterministic finite function). A nondeterministic finite function is a function f : L I P(L O ) with L I and L O finite. We write f spec for a nondeterministic finite function representing a specification. An implementation of f spec is a finite function f impl : L I Distr(L O ). We use f impl [a] to denote the set of possible responses of f impl to the input a, i.e. f impl [a] = {b L O (f impl (a))(b) > 0}. We add a failure function to the weighted fault specification, stating the probability that an implementation that incorrectly implements an input a responds incorrectly to it. Definition 10 (Nondeterministic weighted fault specification). A nondeterministic weighted fault specification (NWFS) is a tuple f spec, w, p err, p fail, with w and p err as before, and f spec : L I P(L O ) a nondeterministic finite specification, p fail : L I [0, 1] a failure function assigning to each input a L I the probability p fail (a) that a random implementation f impl with f impl [a] f spec (a) responds incorrectly to a. The fault weight on an implementation, test executions, and the verdict function are defined as before, except that we now consider implementations correct if they produce a subset of the correct responses to an input [18]. Test cases and test suites remain unchanged. Definition 11 (Fault weight). Let W = f spec, w, p err, p fail be an NWFS and f impl an implementation of W, then the fault weight of f impl is defined by w(f impl ) = w(a). a L I f impl [a] f spec(a) Definition 12 (Test execution and verdict). An execution of a test case t = (a, f spec (a)) is a pair e = (a, b) with b f impl (a). The verdict function v t : (L I L O ) {pass, fail} is defined by { pass if b fspec (a) v t (e) =. fail otherwise We use nr(a, T ) for the number of times (a, f spec (a)) T.

9 3.1 Underlying probability model The classification relation also has to incorporate the fact that implementations are now considered correct if they produce a subset of the correct responses. Definition 13 (Classification relation). Let f spec : L I P(L O ) be a nondeterministic finite specification, and f and g two implementations of f spec. Then f fspec g iff a L I : f[a] f spec (a) g[a] f spec (a). The error function still induces the random variable F W (introduced in Definition 6), although its range is now (L I Distr(L O ))/ fspec. We use F W [a] f spec (a) as a shorthand for the event F W = [f impl ] such that f impl [a] f spec (a). For each test suite T the failure function and error function also induce a random variable V W,T ; the verdict of T against a random implementation of W. Definition 14 (V W,T ). Let W = f spec, w, p err, p fail be an NWFS, then we define V W,T : Ω {pass, fail} to be the random variable denoting the result of executing T against an arbitrary implementation of W. For all test suites T = t 1,..., t n and inputs a L I we use VW,T a as a shorthand for V W,T with T = t i T t i = (a, f spec (a)), performing only the test cases for a. Lemma 1. Let W = f spec, w, p err, p fail be an NWFS and a L I, then Proof. P[V a W,T = pass] = (1 p fail (a)) nr(a,t ) p err (a) + 1 p err (a). P[V a W,T = pass] = P[V a W,T = pass F W [a] f spec (a)] P[F W [a] f spec (a)] + P[V a W,T = pass F W [a] f spec (a)] P[F W [a] f spec (a)] = 1 (1 p err (a)) + (1 p fail (a)) nr(a,t ) p err (a) By the assumed independence of fault presence between the different inputs, the next proposition immediately follows. Proposition 1. Let W = f spec, w, p err, p fail be an NWFS and T a test suite for f spec, then P[V W,T = pass] = (1 p fail (a)) nr(a,t ) p err (a) + 1 p err (a). a L I a T 3.2 Risk We will now introduce the concept of risk for NWFSs. It is still the expected fault weight after a test suite passes.

10 Definition 15 (Risk). Let W = f spec, w, p err, p fail be an NWFS and T a test suite for f spec, then the risk of an arbitrary implementation of W after passing T is defined by risk W (T ) = E[w(F W ) V W,T = pass]. Before discussing how to calculate the risk, we first look at the probability that an input is implemented incorrectly, even though a test suite passes; its posterior error probability (PEP). Definition 16 (PEP). Let W = f spec, w, p err, p fail be an NWFS, T a test suite for f spec, and a L I. Then, the posterior error probability (PEP) of a after T passes is defined by PEP W (a, T ) = P[F W [a] f spec (a) V W,T = pass]. Using Bayes formula the next proposition can be obtained. Proposition 2. Let W = f spec, w, p err, p fail be an NWFS, T a test suite for f spec, and a L I. Then Proof. PEP W (a, T ) = (1 p fail (a)) nr(a,t ) p err (a) (1 p fail (a)) nr(a,t ) p err (a) + 1 p err (a). PEP W (a, T ) = P[F W [a] f spec (a) V W,T = pass] = P[F W [a] f spec (a) V a W,T = pass] = P[V a W,T = pass F W [a] f spec (a)] P[F W [a] f spec (a)] P[V a T = pass] = P[V a W,T = pass F W [a] f spec (a)] p err (a) P[V a W,T = pass] = (1 p fail (a)) nr(a,t ) p err (a) (1 p fail (a)) nr(a,t ) p err (a) + 1 p err (a) Since PEP W (a, T ) only depends on W, a and nr(a, T ), we use PEP W (a, n) to denote PEP W (a, T ) with nr(a, T ) = n. Because PEP W (a, T ) is exactly the probability that w(a) should be included in risk W (T ), the following theorem holds. Theorem 2. Let W = f spec, w, p err, p fail be an NWFS and T a test suite for f spec. Then, the risk of an implementation of W after T passes is given by risk W (T ) = a L I w(a) PEP W (a, T ).

11 Proof. risk W (T ) = E[w(F W ) V W,T = pass] = w([f impl ]) P[F W = [f impl ] V W,T = pass] = [f impl ] (L I L O)/ [f impl ] (L I L O)/ = w(a) a L I a L I f impl [a] f spec(a) [f impl ] (L I L O)/ f impl [a] f spec(a) = a L I w(a) P[F W [a] f spec (a) V W,T = pass] w(a) P[F W = [f impl ] V W,T = pass] P[F W = [f impl ] V W,T = pass] = a L I w(a) PEP W (a, T ) Note that in case of determinism p fail (a) = 1 for all a L I. Substituting this, risk W (T ) reduces to the formula of Theorem 1, as expected. Test optimisation with respect to risk. Given an NWFS W = f spec, w, p err, p fail we can again compute T opt-rfw = argmin T = t1,...,t k risk W (T ), i.e. a test suite of size k with minimal risk. Theorem 2 implies that, if an input a is included n times in T, it contributes w(a) PEP W (a, n) to the risk. Therefore, including it once more reduces the risk by w(a) (PEP W (a, n) PEP W (a, n + 1)). We let r(a, n) denote this value. We can now construct T opt-rfw recursively. Initially, T opt-rfw =. Then, we include the input a L I such that r(a, nr(a, T )) is maximal. After all, this means that including a in T decreases the risk as much as possible. We continue adding inputs a i with maximal r(a i, nr(a i, T )) until T opt-rfw = k. Each time an input is added to T opt-rfw, only one calculation has to be redone, yielding a time complexity of O(n log n). To obtain T opt-rfw = argmin T (LI L O),risk W (T ) x T we perform the same procedure, but now continuing adding inputs until risk W (T opt-rfw ) x. Obviously this procedure has also time complexity O(n log n). Note that this procedure is a generalisation of the procedure for deterministic systems, since in that case r(a, 0) = w(a) p err (a) and r(a, n) = 0 for n Actual coverage For WFSs, actual coverage denoted the amount of fault weight of which we know that it is absent after T passes. However, for NWFSs we often only reduce the

12 probability of the presence of faults because of nondeterminism. We introduce relative error probability reduction (REPR) to denote the extent to which the error probability of a fault decreases as a result of passing a test suite. Definition 17 (Coverage measures). Let W = f spec, w, p err, p fail be an NWFS and T a test suite for f spec. Then we define abscov W (T ) = a T w(a) REPR W (a, T ), with REPR W (a, T ) = p err(a) PEP W (a, T ) p err (a). Total coverage and relative coverage remain unchanged. We use REPR W (a, n) to denote REPR W (a, T ) with nr(a, T ) = n. Note that for deterministic systems PEP W (a, T ) = 0 for all a T, and obviously PEP W (a, T ) = p err (a) for all a T, obtaining Definition 8. Using the definition of PEP, basic calculus yields the next proposition. Proposition 3. Let W = f spec, w, p err, p fail be an NWFS and T a test suite for f spec, then abscov W (T ) = ( (1 p fail (a)) nr(a,t ) ) w(a) 1. 1 p a T err (a) + (1 p fail (a)) nr(a,t ) p err (a) Test optimisation with respect to relcov. Given an NWFS W = f spec, w, p err, p fail we can again derive T opt-cov = argmax T = t1,...,t k relcov W (T ) and T opt-cov = argmin T (LI L O),relCov W (T ) x T. The computations are similar to the ones discussed in Section 3.2. However, by Definition 17 it now holds that r(a, n) = w(a) (REPR W (a, n + 1) REPR W (a, n)). Clearly, these computations are still in O(n log n). 4 Labelled transition systems An IOLTS describes system behaviour by means of states and transitions. Transitions are always caused by either an input action or an output action. Definition 18 (IOLTS). An IOLTS A is a tuple (S, s 0, L, ), where S is a finite set of states, s 0 S is the initial state, L is a finite set of actions, partitioned into a set L I of inputs (suffixed by a question mark) and a set L O of outputs (suffixed by an exclamation mark), S L S is the transition relation, which is required to be deterministic, i.e., (s, a, s ) (s, a, s ) = s = s.

13 We write A spec to denote a specification. An IOLTS A impl is a (potentially incorrect) implementation of A spec if it has the same alphabet and it is input-enabled, i.e., for all s S and a L I, there exists an s S with (s, a, s ). The set of all possible implementations of A is denoted by IMPL A. Definition 19 (Paths and traces). Let A = (S, s 0, L, ) be an IOLTS, then a path in A is a finite sequence π = s 0 a 1 s 1 a 2... a n s n, with s 0 = s 0 and i [0..n 1] : (s i, a i+1, s i+1 ). The set of all paths in A is denoted by paths A. Each path π has a trace associated with it, denoted by trace(π) and given by the sequence of the actions of π. From all paths in A we can deduce all traces in A: traces A = {trace(π) π paths A }. We use A[σ] to denote the set of output actions that A can provide after σ, i.e. A[σ] = {b! L O σb! traces A }. Based on IOLTSs, we can again define a weighted fault specification. Definition 20 (IOLTS weighted fault specifications). An IOLTS weighted fault specification (IOWFS) is a tuple W = A spec, w, p err, p fail, with A spec = (S, s 0, L, ) an IOLTS, w : L R 0 a weight function assigning a fault weight to each trace σ L, such that 0 < σ L w(σ) <. The fault weight w(σ) denotes the severity of a possible erroneous output after σ, p err : L [0, 1] an error function assigning to each trace σ L the probability p err (σ) that an implementation can provide an incorrect output after σ, i.e. that for an arbitrary A impl it holds that A impl [σ] A spec (σ), p fail : L [0, 1] a failure function assigning to each trace σ L the probability p fail (σ) that an arbitrary implementation A impl with A impl [σ] A spec (σ) responds incorrectly to σ. An implementation of W is an implementation of A impl. Since w, p err and p fail all have an infinite domain, it is difficult to specify them. For this purpose fault automata can be used [5]. These are IOLTSs combined with fault weights, that can be transformed into a weight function by either discounting or truncation. It is straightforward to extend fault automata to also include p err and p fail (no discounting for them is necessary). Alternatively, if a test suite T is given, we only specify w, p err and p fail for the traces in T. Following ioco theory [18], we require test cases for IOLTSs to be fail fast; they stop directly after observing a failure. Test cases continually either perform an input action or observe which output action a system provides. Definition 21 (Test cases and test suites). (i) A test case t for an IOLTS A spec = (S, s 0, L, ) is a prefix-closed, finite subset of L, such that - if σa? t, then b L : b a? σb t, - if σa! t, then b! L O : σb! t, - if σ traces Aspec, then σ L + : σσ t.

14 A test suite T is a tuple of test cases, denoted by t 1,..., t n. (ii) An execution of t is a trace σ t such that there is no ρ t for which σ ρ. The set of all executions of t is exec t = {σ t ρ t : σ ρ}. An observing trace of t is a trace σ t such that b L O : σb t. (iii) The verdict function v t : exec t {pass, fail} is defined for all σ exec t by { pass if σ tracesaspec v t (σ) = fail otherwise (iv) For each execution E = σ 1,..., σ n and trace σ L, we define obs(σ, E) = {i b L O : σb σ i } to be the number of times E observes after σ. We use obs(σ, T ) to denote the maximum number of times an execution of T might observe after σ, i.e. obs(σ, T ) = {i b L O : σb t i }. The set of all observing traces of T is given by obs T = t i T {σ t i b L O : σb t i }. The definitions of the classification relation and the fault weight of an implementation can be lifted easily to IOLTSs (by substituting traces for inputs) Underlying probability model The function p err of an IOWFS induces a random variable A W over the equivalence classes of Aspec, similar to the random variable F W for finite specifications. Definition 22 (A W ). Let W = A spec, w, p err, p fail be an IOWFS, then we define A W : Ω IMPL Aspec / to be the random variable describing to which equivalence class an arbitrary implementation of W belongs. Let A impl be an implementation of W, then P[A W = [A impl ]] = p err (σ) (1 p err (σ)). σ L A impl [σ] A spec[σ] σ L A impl [σ] A spec[σ] Although the infinite number of equivalence classes implies a probability of 0 for each specific one, this will turn out not to be a problem. We again introduce a random variable denoting the execution of a test suite, similar to V W,T. For the current model, however, we need to know which execution it yields instead of only whether is passes. After all, because of nondeterminism the same test suite might test different traces during different executions. Definition 23 (R W,T ). Let W = A spec, w, p err, p fail be an IOWFS, then we define R W,T : Ω L to be the random variable denoting the result of executing T against an arbitrary implementation of W. Note that the distribution of R W,T depends on the distribution of outputs the system provides. However, we do not need the explicit distribution of R W,T here. A random variable whose distribution will be needed in some of the proofs to follow is R σ,n W,T : Ω {pass, fail}, denoting the result of observing n times after σ on an arbitrary implementation of W tested by T. Note that both its purpose and its probability distribution are very similar to those of VW,T a.

15 Lemma 2. Let W = A spec, w, p err, p fail be an IOWFS, T a test suite for A spec, σ L and n N, then Proof. P[R σ,n W,T = pass] = (1 p fail(σ)) n p err (σ) + 1 p err (σ). P[R σ,n W,T = pass] = P[Rσ,n W,T = pass A W [σ] A spec [σ]] P[A W [σ] A spec [σ]] + P[R σ,n W,T = pass A W [σ] A spec [σ]] P[A W [σ] A spec [σ]] = 1 (1 p err (σ)) + (1 p fail (σ)) n p err (σ) 4.2 Risk For IOLTSs the risk is defined for a test suite and an execution. Again, we look at the posterior error probability of a trace, but now after a correct execution. Definition 24 (Risk). Let W = A spec, w, p err, p fail be an IOWFS, T a test suite for A spec, and E a correct execution of T. Then, the risk of an arbitrary implementation of W after T yielded E is defined by risk W (T, E) = E[w(A W ) R W,T = E]. Definition 25 (PEP). Let W = A spec, w, p err, p fail be an IOWFS, T a test suite for A spec, E a correct execution of T, and σ L. Then, the posterior error probability (PEP) of σ after T yields E is defined by PEP W (σ, T, E) = P[A W [σ] A spec [σ] R W,T = E]. Some executions might have performed an input action after σ, that way not being able to detect potential incorrect behaviour directly after σ. Therefore, we have to count the number of executions reaching σ and observing afterwards. Keeping this in mind, it is easy to see that the next proposition holds. Proposition 4. Let W = A spec, w, p err, p fail be an IOWFS, T a test suite for A spec, E = σ 1,... σ n a correct execution of T, and σ L. Then Proof. PEP W (σ, T, E) = PEP W (σ, T, E) = P[A W [σ] A spec [σ] R W,T = E] = P[A W [σ] A spec [σ] R σ,obs(σ,e) W,T = P[Rσ,obs(σ,E) W,T (1 p fail (σ)) obs(σ,e) p err (σ) (1 p fail (σ)) obs(σ,e) p err (σ) + 1 p err (σ). = pass] = pass A W [σ] A spec [σ]] P[A W [σ] A spec [σ]] P[R σ,obs(σ,e) W,T = pass]

16 = P[Rσ,obs(σ,E) W,T = pass A W [σ] A spec [σ]] p err (σ) = pass] = P[R σ,obs(e,σ) W,T (1 p fail (σ)) obs(σ,e) p err (σ) (1 p fail (σ)) obs(σ,e) p err (σ) + 1 p err (σ) Since PEP W (σ, T, E) only depends on W, σ and obs(σ, E), we use PEP W (σ, n) to denote PEP W (σ, T, E) with obs(σ, E) = n. The value of risk W (T, E) can in principle again be computed by ranging over all traces, summing their fault weight multiplied by their PEP: risk W (T, E) = σ L w(σ) PEP W (σ, T, E). However, as there are infinitely many traces, this is not computable in practice. As a solution, we first compute the initial risk, formally risk W (, ) = σ L w(σ) p err(σ). The algorithm introduced in [5] to compute the total coverage σ L w(σ) can be used for this, by multiplying each fault weight in the fault automaton by the corresponding error probability. Then, for all traces σ obs T we subtract the risk reduction that was obtained by E. Theorem 3. Let W = A spec, w, p err, p fail be an IOWFS, T a test suite for A spec, and E a correct execution of T. Then Proof. risk W (T, E) = risk W (, ) risk W (T, E) σ obs T w(σ) (p err (σ) PEP W (σ, T, E)). = E[w(A W ) R W,T = E] = w([a impl ]) P[A W = [A impl ] R W,T = E] = [A impl ] IMPL Aspec / [A impl ] IMPL Aspec / = w(σ) σ L = σ L A impl [σ] A spec[σ] [A impl ] IMPL Aspec / A impl [σ] A spec[σ] σ L w(σ) P[A W [σ] A spec [σ] R W,T = E] w(σ) P[A W = [A impl ] R W,T = E] P[A W = [A impl ] R W,T = E]

17 = σ L w(σ) PEP W (σ, T, E) = σ L w(σ) (PEP W (σ,, ) (PEP W (σ,, ) PEP W (σ, T, E))) = risk W (, ) σ L w(σ) (PEP W (σ,, ) PEP W (σ, T, E)) = risk W (, ) σ obs T w(σ) (PEP W (σ,, ) PEP W (σ, T, E)) = risk W (, ) σ obs T w(σ) (p err (σ) PEP W (σ, T, E)) Test optimisation with respect to risk. Due to nondeterministic outputs, the risk has been defined for IOLTSs based on a test suite and an execution. However, to find an optimal test suite with respect to the risk, we need to estimate the risk without knowing the execution in advance. It is therefore necessary to estimate the output behaviour of a system. We introduce the random variable O σ W : Ω L O to denote the output of the system when observing after σ. We use p out (σ, b) as shorthand for P[O σ W = b]. Definition 26 (O σ W ). Let W = A spec, w, p err, p fail be an IOWFS and σ L, then we define O σ W : Ω L O to be the random variable describing the output obtained when observing after σ. Let a! L O and b? L I, then p out (σ, a!) = P[O σ W = a!] and for technical reasons p out(σ, b?) = 1. Using p out, it trivial to find the probability p reach (σ) that a trace σ is reached during a test case in which it is included. Proposition 5. Let W = A spec, w, p err, p fail be an IOWFS, then for all σ = a 1 a 2... a n L it holds that p reach (σ) = n p out (a 1... a i 1, a i ), i=1 using the convention that a 1... a 0 is the empty string ɛ. Now, letting risk W (T ) be the random variable denoting the risk of a random execution of T, the following proposition holds. Proposition 6. Let W = A spec, w, p err, p fail be an IOWFS and T a test suite for A spec. Then E[risk W (T )] = risk W (, ) obs(σ,t ) ( ) w(σ) obs(σ, T ) p reach (σ) i i σ obs T i=0 (1 p reach (σ)) obs(σ,t ) i (p err (σ) PEP W (σ, i)).

18 To obtain this formula, we started with Theorem 3 and replaced p err (σ) PEP W (σ, T, E) (the error probability reduction for E) by the expected error probability reduction for an arbitrary execution. This reduction depends on the number of times the execution observes after σ, which is by definition between 0 and obs(σ, T ). The probability of observing i times is equal to obtaining i successes in a binomially distributed experiment with n = obs(σ, T ) and p = p reach (σ). Using these observations, the familiar formula for the binomial distribution, and Theorem 3, Proposition 6 follows. Since each trace σ contributes w(σ) p err (σ) to risk W (, ), Proposition 6 implies that, if obs(σ, T ) = n, the contribution of σ to the expected risk is c(σ, n) = w(σ) n i=0 ( ) n p reach (σ) i (1 p reach (σ)) n i PEP W (σ, i). i Note that c(σ, n) only denotes the contribution of σ itself, not of its prefixes. Adding another test case to T that observes after σ yields an expected risk reduction (ERR) of r(σ, n) = c(σ, n) c(σ, n + 1). We can now construct T opt-rfw,d = argmin T = t1,...,t k E[risk W (T )], i.e. a test suite of size k with minimal expected risk. As an extra restriction, we limit the depth of each test case to d to obtain a finite test suite. To compute the best test case of depth d (i.e. the one with maximal ERR) we first derive a recursive equation for the maximal ERR obtained by such a test case. In order to express this as a function of the maximal ERR of a test case of depth d 1, we also have to keep track of the trace observed thus far. We therefore let M T (σ, d) denote the maximal ERR that can be obtained by adding a test case of depth d with a history σ to some test suite T. Note that we are interested in M T (ɛ, d). First of all, trivially M T (σ, 0) = 0. For an arbitrary depth d > 0, M T (σ, d) is calculated by looking at the first step. If we start by performing an input a? L I, we obtain no ERR directly and are left with M T (σa?, d 1). If we observe, we directly obtain an ERR of r(σ, obs(σ, T )). Then, with probability p out (σ, b!) the system performs a b!, leaving us with M T (σb!, d 1). Formalising these observations, we obtain M T (σ, d) 0 if d = 0 max M T (σa?, d 1), = a? L I max r(σ, obs(σ, T )) + (p out (σ, b!) M T (σb!, d 1)) if d > 0 b! L O To construct T opt-rfw,d, we start with T = and compute M T (ɛ, d): the maximum expected risk reduction to be obtained by a test case of depth d. Algorithmically, we start bottom-up by calculating M T (σ d 1, 1) for all σ d 1 L of length d 1. Based on these values, we can calculate M T (σ d 2, 2) for all σ d 2 L of length d 2. Working our way up, we arrive at M T (ɛ, d). Dur-

19 ing the calculations we record for each M T (ρ, l) whether it was obtained by observation or an input (and which one). Then, the first step of the best test case t 1 is just the action that was chosen for M T (ɛ, d). Then, suppose that a L was performed, we perform an input or observe, according to which was chosen M T (a, d 1), and so on. To find the best test case t 2 to add to T after if already contains t 1, we set T = {t 1 } are repeat the procedure described above. Since only a part of the state space changes, this can be calculated efficiently. We just continue in this way until T = k and then set T opt-rfw,d = T. The complexity of finding the first test case is in O( L k ). In worst-case (only output actions exist) the complexity of finding the second test case is also in O( L k ), but in practice this might be better. However, the complexity could be improved to polynomial if optimisation is done directly based on the fault automaton instead of the unfolded state space. Fore more details, we refer to [5]. 4.3 Actual coverage Actual coverage for IOLTSs is similar to actual coverage for functions. We reuse the concept of relative error probability reduction in the definition for absolute actual coverage. Total coverage and relative actual coverage are unchanged (see [5] for an algorithm to compute totcov for IOLTSs). Definition 27 (Absolute actual coverage). Let W = A spec, w, p err, p fail be an IOWFS, T a test suite for A spec, and E an execution of T. Then we define abscov W (T, E) = σ L w(σ) REPR(σ, T, E). Now again we can easily see that the following proposition holds. Proposition 7. Let W = A spec, w, p err, p fail be an IOWFS, T a test suite for A spec, and E a correct execution of T. Then abscov W (T, E) = ( (1 p fail (σ)) obs(e,σ) ) w(σ) 1. 1 p err (σ) + (1 p fail (σ)) obs(e,σ) p err (σ) σ obs(t ) Optimisation with respect to relcov can be done in exactly the same way as optimisation with respect to risk. 5 Conclusions and future work While testing is an important part of today s software development process, little research has been devoted to the interpretation of a successful testing process. In this paper, we introduced a weighted fault specification (WFS) to describe the required behaviour of a system and the estimation of its probabilistic behaviour. Based on such an WFS, we presented two measures: risk and actual coverage.

20 Risk denotes the confidence the system after testing is successful, whereas actual coverage denotes how much was tested. We presented optimisation strategies with respect to both measures, enabling the tester to obtain a test suite of a given size that will obtain minimal risk or maximal actual coverage. Our work gives rise to several directions for future research. First, it is crucial to validate our framework by means of creating tool support and applying this in a series of case studies. Second, it might be useful to include fault dependencies in our models. Third, it is interesting to investigate the possibilities of on-the-fly test derivations (as e.g. performed by the tool TorX [4]) based on risk or actual coverage. This could yield a tool that, during testing, calculates probabilities and decides which branches to take for an optimal result. References [1] S. Amland. Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study. Journal of Systems and Software, Volume 53, Issue 3, pages , [2] J. Bach. Heuristic risk-based testing. Software Testing and Quality Engineering Magazine, November [3] T. Ball. A Theory of Predicate-Complete Test Coverage and Generation. In Proc. of the 3rd Int. Symp. on Formal Methods for Components and Objects (FMCO 04), volume 3657 of LNCS, pages 1 22, [4] A. Belinfante, J. Feenstra, R.G. de Vries, J. Tretmans, N. Goga, L.M.G. Feijs, S. Mauw, and L. Heerink. Formal test automation: A simple experiment. In Proc. of the 12th Int. Workshop on Testing Communicating Systems (IWTCS 99), volume 147 of IFIP Conference Proceedings, pages Kluwer, [5] L. Brandán Briones, E. Brinksma, and M.I.A. Stoelinga. A semantic framework for test coverage. In Proc. of the 4th Int. Symp. on Automated Technology for Verification and Analysis, volume 4218 of LNCS, pages Springer, [6] T.L. Graves, A.F. Karr, J.S. Marron, and H. Siy. Predicting fault incidence using software change history. IEEE Trans. on Softw. Eng., 26(7): , [7] M.H. Halstead. Elements of Software Science. Elsevier Science, [8] D. Lee and M. Yannakakis. Principles and methods of testing finite state machines - a survey. In Proc. of the IEEE, volume 84, pages , [9] Y.K. Malaiya and J. Denton. Requirements volatility and defect density. In Proc. of the 10th Int. Symp. on Software Reliability Engineering (ISSRE 99), pages IEEE Computer Society, [10] T. J. McCabe. A complexity measure. IEEE Transactions on Software Engineering, 2(4): , [11] K. W. Miller, L. J. Morell, R. E. Noonan, S. K. Park, D. M. Nicol, B. W. Murrill, and M. Voas. Estimating the probability of failure when testing reveals no failures. IEEE Transactions on Software Engineering, 18(1):33 43, [12] G. J. Myers, C. Sandler, T. Badgett, and T. M. Thomas. The Art of Software Testing, Second Edition. Wiley, [13] L. Nachmanson, M. Veanes, W. Schulte, N. Tillmann, and W. Grieskamp. Optimal strategies for testing nondeterministic systems. SIGSOFT Software Engineering Notes, 29(4):55 64, 2004.

21 [14] M. Newman. Software errors cost u.s. economy 59.5 billion annually, NIST assesses technical needs of industry to improve software-testing. Press Release, http: // [15] Felix Redmill. Exploring risk-based testing and its implications. Software Testing, Verification and Reliability, 14(1):3 15, [16] M. Timmer. Evaluating and predicting actual test coverage. Master s thesis, University of Twente, The Netherlands, [17] G. J. Tretmans. Test Generation with Inputs, Outputs and Repetitive Quiescence. Software Concepts and Tools, 17(3): , [18] H. Ural. Formal methods for test sequence generation. Computer Communications, 15(5): , [19] J. Voas, L. Morell, and K. Miller. Predicting where faults can hide from testing. IEEE Software, 8(2):41 48, 1991.