Specialised Test Strategies

Transcription

1 Grant Agreement: Comprehensive Modelling for Advanced Systems of Systems Specialised Test Strategies Technical Note Number: D34.2 Version: 2.1 Date: September 2013 Public Document

2 Contributors: Wen-ling Huang, UB Jan Peleska, UB Uwe Schulze, UB Editors: Jan Peleska, UB Reviewers: Stefan Hallerstede, IHA Adrian Larkham, ATEGO Paolo Casoto, UFPE 2

3 3

4 Document History Ver Date Author Description Jan Peleska Initial document version Jan Peleska Overview completed, Part I improved Jan Peleska Final version of Part I, Chapter 2, prepared for publication Jan Peleska Exhaustive equivalence class testing for systems with finite state and finite output completed. Foundations of Reactive Kripke Structures and equivalence class grey-box testing in work. Table of contents extended to whole report, in order to get a writing plan Jan Peleska Completion of the following chapters. 1 Overview, Chapter 7 Exhaustive Model- Based Equivalence Class Testing, Chapter 8 Foundations of Reactive Kripke Structures, Chapter 9 Grey Box Equivalence Class Partition Testing Jan Peleska Part III on testing distributed systems added Jan Peleska Original Part I removed, because this material is now covered by the other parts (in particular, the new Parts II and III). Chapters 1,2,3,4,5,6,7 are stable. With respect to the previous release, Sections are new. Chapters 8 12 are still under construction Jan Peleska First complete version. Abstract extended by listing the main contributions. Chapter 8 12 completed Jan Peleska Improvements due to review remarks from Arian Larkham and Paolo Casato Jan Peleska Document completed Jan Peleska Small improvements. 4

5 Abstract This technical report describes test strategies developed by the authors within the COMPASS project for the purpose of Systems of Systems (SoS) testing. The strategies take the test-related characteristics of SoS into account, such as distribution, complexity of the overall state space, size of input and output interfaces, and dynamic change of configurations. The following main results have been achieved. Elaboration of two novel exhaustive equivalence class testing strategies for testing constituent systems with large data types one black box, one grey box allowing to prove I/O equivalence between the system under test and its specification model. A novel exhaustive test strategy for testing emergent properties of distributed constituent systems cooperating in the SoS. This strategy is based on the equivalence class principle investigated before, property checking techniques known from bounded model checking and adapted and extended for the purpose of SoS testing, and partial order reduction known from global model checking. A novel heuristic for reducing the amount of SoS system test cases in a justifiable way; this strategy applies a new interpretation of the well-known pairwise testing principle with orthogonal arrays. A new strategy for testing dynamic changes of SoS configurations. 5

6 Contents 1 Overview 11 I Equivalence Class Partitions 13 2 Exhaustive Model-Based Equivalence Class Testing Introduction Reactive Kripke Structures Notation and Definitions Quiescent Reduction Traces Input Traces I/O Equivalence Input Equivalence Class Partitionings Over Reactive Kripke Structures With Finite Outputs Test Hypotheses and Proof of Exhaustiveness Test Strategy Complexity Considerations Related Work Conclusion and Future Work Foundations of Reactive Kripke Structures Introduction of Reactive Kripke Structures Definition Term Replacement Functions Quiescent Reduction Traces Input Traces I/O Equivalence Equivalence Class Partitions Over Reactive Kripke Structures 49 6

7 3.4 Parallel Composition of Reactive Kripke Structures Quiescent Reduction of Parallel Compositions Property Specification for Reactive Kripke Structures With LTL Property Specifications on Finite Traces Safety Properties and I/O-Equivalence Finite Trace Semantics for LTL Safety Formulas the Fixpoint Evaluation Encoding Grey Box Equivalence Class Partition Testing White Box and Grey Box Testing Test Hypotheses Finite Testability of Polynomials Exhaustive Test Strategy Example: Grey Box Tests for Ceiling Speed Monitoring Example: Grey Box Tests for Train Braking Supervision II Strategies for Testing Distributed Systems 84 5 Testing Distributed Systems Challenges and Solutions Challenges Overview of Solutions Formal Approach to Distributed Systems Testing A Sufficient Condition on Exhaustive SoS Test Suites Local Versus Emergent SoS Properties A Test Strategy Based on Equivalence Classes and Partial Order Reduction Interoperability Testing Interoperability Testing Objectives Encoding Interoperability Testing Objectives in LTL Potentially Lossy Communication Lossless Communication Pairwise Testing and Orthogonal Arrays N-Wise Testing Application of Pairwise Testing to SoS System Testing Assessment of the Pairwise Testing Strategy With Orthogonal Arrays

8 III Testing Against Dynamic SoS Configurations Dynamic SoS Configurations Autonomy, Evolution, Dynamicity, and Their Impact on SoS Testing Models, Contracts, and SoS Configurations The SoS Configuration Tree A Test Strategy for Testing Dynamic SoS Configurations Basic Approach to Testing Dynamic Configuration Changes Disabling and Enabling Constituent Systems Specifying Emergent Properties Depending on Configuration Changes The Test Strategy Runtime Acceptance Testing Problem Statement Built-in Test Equipment Runtime Acceptance Testing Based on Simulations Conclusion Summary of Results Future Work on Tool Support

9 List of Figures 2.1 Interface of the ETCS ceiling speed monitoring function (simplified) State machine of the ETCS ceiling speed monitoring function Input classes I Initial states in quiescent reduction Q(K) and Q(K ) Parallel composition of Reactive Kripke Structures Lifted inter-class transitions for ceiling speed monitor from Example Interface of a brake supervision controller Behaviour of brake supervision controller SoS Configuration SoS Configuration Tree

10 List of Tables 3.1 Loop Constraints Equivalence classes for the ceiling speed monitoring Brake supervision controller interface description Equivalence classes for the brake supervision controller Equivalence classes for the brake supervision controller Trace testing class B All test cases of a system with 3 parameters and 2 values of each parameter Test cases of a pairwise testing of a system with 3 parameters and 2 values of each parameter All test cases for 4 factors with level Test suite of a pairwise testing of 4 factors and level Orthogonal array L 8 (2 4 )

11 Chapter 1 Overview Purpose of this document. This technical reports describes test strategies for the purpose of Systems-of-Systems (SoS) testing. They have been developed by the authors within the context of the COMPASS project. The strategies described are currently being implemented in the RT-Tester test automation tool for the purpose of automated model-based SoS testing: the tool parses SoS models specified in SysML/CML and identifies symbolic test cases in the abstract syntax tree (AST) of the model, solves the constraints representing symbolic test cases by means of an SMT solver, thereby generating concrete test data, and creates test procedures running these concrete test cases in the RT- Tester test execution environment against the system under test (SUT). The purpose of test strategies is to guide the test case identification process such that both requirements and structural coverage of the SoS are achieved in a comprehensive and at the same time efficient way, and the test suite strength, that is, its capability to uncover deviations of the SUT from the specified behavior, is as high as possible. Document structure. This document is structured into several parts. In Part I we introduce novel strategies based on the well-known equivalence class partitioning technique. These strategies are meant to be applied lo- 11

12 cally, that is, on constituent system level. They help to identify effective test cases for dealing with large data types processed by algorithms occurring, for example, in system control tasks where calculations have to be performed on numerical variables with big word length (both integral and floating point numbers). In these situations it would be infeasible to enumerate the input and state spaces comprehensively and to test all possible combinations of data vector components. The strategies developed in Part I identify test cases dealing with relatively small collections of data representatives, though the data types involved may be large. At the same time application of the strategies may result in exhaustive test suites, capable of proving the consistency between specification model and SUT. Finally, it is shown in Part I, how local mission threads, that is, specific end-to-end tests of constituent systems can be categorised by means of the equivalence class principle. As a result of these considerations, a relatively small collection of mission thread classes can be selected from each constituent system, and representatives of these classes from all constituent systems can be combined in SoS system-level testing. Part II is focused on distributed systems testing: we develop strategies for identifying test cases investigating the cooperation capabilities of the constituent systems, and the resulting emergent properties. On an abstract level, these strategies identify effective concurrent executions of constituent system computations, each representing a sequence of local equivalence classes. Even after having categorised local mission threads in traces of equivalence classes, the potential combinations of local mission threads to global ones may still lead to combinatorial explosions when testing complex SoS. Therefore we elaborate strategies how to identify relevant combinations of local mission threads. The well-known orthogonal array principle can be applied for constructing global combinations with high error detection capabilities. In Part III the challenge of model-based test design for dynamic SoS configurations is considered. The state of the are of current model-based testing only considers static networks of cooperating constituent systems. We therefore analyse how constituent systems dynamically entering and leaving SoS configurations can be taken into account during test case design and associated test data elaboration. Dynamic changes of SoS configurations may require runtime acceptance testing, where tests are executed during SoS operation, in order to check whether constituent systems entering or leaving the current SoS configuration leave the system in a consistent state. The elaboration of such runtime acceptance tests is also investigated in this part. 12

13 Part I Equivalence Class Partitions 13

14 Chapter 2 Exhaustive Model-Based Equivalence Class Testing Note. This chapter is an extended version of the publication [HP13]. 2.1 Introduction Motivation. Equivalence class testing is a well-known heuristic approach to testing software or systems whose state spaces, inputs and/or outputs have value ranges of a cardinality inhibiting exhaustive enumeration of all possible values within a test suite. The heuristic suggests to create equivalence class partitions structuring the input or output domain into disjoint subsets for which the behavior of a component or system is assumed to be the same, based on the specification [SLS06, p. 228]. If this assumption is justified it suffices to test just a few values from each class, instead of exploring the behavior of the system under test (SUT) for each possible value. In order to investigate that the SUT respects the boundaries between different equivalence class partitions boundary values are selected for each class, so that equivalence class and boundary value testing are typically applied in combination. As an alternative to deriving equivalence class partitions from the specification, the structure of the SUT or its model can be analyzed: classes are then defined as sets of data leading to the same execution paths [ECfES01, B.19]. For testing safety-critical systems the justification of the equivalence class 14

15 partitions selected is a major challenge. It has to be reasoned why the behaviour of the SUT can really be expected to be equivalent for all values of a class, and why the number of representatives selected from each class for the test suite is adequate. While being quite explicit about the code coverage to be achieved when testing safety-critical systems, standards like [ECfES01, RTC92, ISO09] do not provide any well-defined acceptance conditions for equivalence class partitions to be sufficient. Main Contributions. In this chapter we present rules for generating input equivalence class partitions, whose justification is given by the fact that they lead to an exhaustive 1 test suite: under certain hypotheses the generated classes and the test data selected from them prove conformance between a specification model and its implementation, if the latter passes all tests of this suite. The algorithm is applicable in a model-based testing context, provided that the behavioural semantics of the modelling formalism can be expressed using Kripke Structures. The equivalence class partitioning strategy is elaborated and proven to be exhaustive on Kripke Structures. As an example of a concrete formalism, we illustrate how the strategy applies to SysML state machine models [Obj10]. To our best knowledge, this is the first formal justification of the well-known equivalence class testing principle (see Section 2.7 for a discussion of related work). Example 1. The following example describes a typical system of the class covered by our input equivalence class partition testing strategy. It will be used throughout the chapter for illustrating the different concepts and results described in this chapter. The example is taken in simplified form from the specification of the European Train Control System ETCS and describes the required behaviour of the ceiling speed monitoring which protects trains from overspeeding, as specified in [UNI12, ]. The interface is shown in Figure 2.1. The I/O variables have the following meaning. 1 Sometimes the term complete test suite is used instead of exhaustive test suite. 15

16 Interface V est V MRSP W EB Description Current speed estimation [km/h] Applicable speed restriction [km/h] (MRSP = Most Restrictive Speed Profile) Warning to train engine driver at driver machine interface (DMI) (1 = displayed, 0 = not displayed) Emergency brake (1 = active, 0 = inactive) V est V MRSP Ceiling Speed Monitoring W EB Figure 2.1: Interface of the ETCS ceiling speed monitoring function (simplified). The behaviour of the ceiling speed monitoring function is specified by the UML (or SysML) state machine shown in Figure 2.2. The function gives a warning to the train engine driver if the currently applicable speed limit V MRSP is not observed, but the actual estimated speed V est does not exceed the limit too far. If the upper threshold for the warning status is violated (this limit is specified by guard conditions g ebi1 or g ebi2 ), the emergency brake is activated. After such an emergency brake intervention has occurred, the brakes are only released after the train has come to a standstill. Overview and Proof Sketch. In Section 2.2 the basic concepts about Reactive Kripke Structures a specialisation of general Kripke Structures which is suitable for application in a reactive systems context are introduced. In Section 2.3 it is shown how input equivalence class partitionings for Reactive Kripke Structures are constructed. In Section 2.4, two test hypotheses are presented, whose validity 16

17 [V est apple V MRSP ] Normal Status [V est >V MRSP ] Warning Status entry/ W=0; EB=0; entry/ W = 1; [V est = 0] Intervention Status (EBI) entry/ EB=1; g ebi1 V MRSP > 110 ^ V est >V MRSP + 15 [g ebi1 _ g ebi2 ] [g ebi1 _ g ebi2 ] Guard condition of faulty SUT g ebi2 V MRSP apple 110 ^ V est >V MRSP +7.5 g ebi1 V MRSP > 110 ^ V est >V MRSP + 20 In this example, condition [g ebi1 g ebi2 ] guarding the state machine transition from Warning Status to Intervention Status is the one used in the specification model. The SUT, however, is assumed to be faulty; it applies guard condition [g ebi1 g ebi2 ] instead of the correct one used in the specification model. Otherwise the SUT behaves just as the specification model. Note that while this example only handles a transition guard failure, the test strategy presented in this chapter uncovers all violations of I/O-equivalence. This is described in detail in the sections below. Figure 2.2: State machine of the ETCS ceiling speed monitoring function. allows us to prove that our equivalence class partitioning and test data selection principle leads to an exhaustive test suite (Theorem 1). While this theorem states that I/O equivalence can be established using a finite input alphabet only (though the input data types may be infinite), it does not state whether the number of input traces needed is finite. In Section 2.5 we therefore show by means of this theorem, that Reactive Kripke Structures associated with input equivalence partitionings can be abstracted to deterministic finite state machines. Then the well-known W-Method can be applied to establish a finite exhaustive test suite proving I/O equivalence between specification model and SUT. Section 2.7 discusses related work, and we conclude with a discussion of the results obtained and a conjecture about an extension of the main theorem s validity in Section

18 2.2 Reactive Kripke Structures Notation and Definitions Let K = (S, S 0, R, L, AP ) a Kripke Structure (KS) with state space S, initial states S 0 S, transition relation R S S and labelling function L : S P(AP ), where AP is a set of atomic propositions. We specialise on state spaces over variable valuations: let V a set of variable symbols for variables v V with values in some domain D = v V D v. The state space S of K is the set of all variable valuations s : V D. 2 It is required throughout this document that the labelling function shall be consistent with, and determined by these variable valuations, in the sense that AP contains propositions with free variable in V and 3 s S : L(s) = {p AP s(p)} Since s satisfies exactly the propositions contained in L(s), is satisfies the negation of all propositions in the complement, that is, p AP L(s) : s(p). K is called a Reactive Kripke Structure (RKS) if it satisfies the following additional properties. 1. V can be partitioned into disjoint sets V = I M O called input variables, (internal) model variables, and output variables, respectively Transitions leave either the input vectors or the internal and output state vectors unchanged, that is, (s, s ) R : s I = s I s M O = s M O 3. The state space can be partitioned into states from where only input changing transitions are possible, and those from where only internal 2 The state space is always total in the sense that all s : V D are elements of S. This allows us to assume that specification models K and implementations K operate on the same state space S, possibly with differing subsets of reachable states. 3 We use notation s(p) for the Boolean expression p, where every free variable v var(p) has been replaced by its current value s(v) in state s. For example, s(x < y) is true if and only if s(x) < s(y) holds. Observe that this can be alternatively written as s = p, or p[s(v)/v v V ]. 4 Frequently we use input vectors c to the system, c is an element of D I = D x1 D x I, where x 1,..., x I are the input variables. Changing the valuation of all input variables of a state s 0 to c = (c 1,..., c I ) is written as s 1 = s 0 { x c}. State s 1 coincides with s 0 for all but the the input variables, and s 1 (x i ) = c i, i = 1,..., I. 18

19 and output changing transitions are possible. The former states are called quiescent, the latter transient. S Q, S T S : S = S Q S T S Q S T = (s, s ) R : s S Q s M O = s M O (s, s ) R : s S T s I = s I 4. All initial states have the same internal and output variable valuations, and all possible inputs are allowed in initial states 5. s : M O D : S 0 = {{ x c} s c D I } 5. The input vector may change without any restrictions. s S Q, s S : s M O = s M O (s, s ) R 6. Internal and output state changes are deterministic in the sense that they only depend on the current state valuation. T : S T S Q : s S T, s S : (s, s ) R s = T (s) Function T can be extended to the complete state space by defining s S Q : T (s) = s. 7. Unreachable states s are elements of S Q, so that the transition relation is also defined on R(s, ), but only input changes may occur. The rules above imply that the transition relation of a RKS can be written as R = {(s, s ) s S Q s M O = s M O } {(s, T (s)) s S T }. Observe that while transient states always have quiescent ones as post-states (this is stated in rule 7), quiescent states may have both transient and quiescent ones as post-states. Example 2. Consider the UML/SysML state machine described in Example 1. Its behavioural semantics can be described by a RKS K = (S, S 0, R, L, AP ) with variable symbols from V = I M O, I = {V est, V MRSP }, M = {l}, O = {W, EB}. Sets I and O contain the interface variable symbols with domains D Vest = D VMRSP = [0, 350] R 5 Observe that initial states may be quiescent or transient. 19

20 (maximum speed of ETCS trains under consideration is 350km/h). Symbol l ( location ) has values in D l = {NS, WS, IS} and its valuation signifies the current control state Normal Status, Warning Status, or Intervention Status, respectively. The output symbols have values in D W = D EB = B = {0, 1}. The state space S contains all valuations of these symbols, S = V D, D = [0, 350] {NS, WS, IS} Setting D I = [0, 350] [0, 350], the initial states are elements of S 0 = {s 0 S (c 0, c 1 ) D I : s 0 = {V est c 0, V MRSP c 1, l NS, W 0, EB 0}} Fixing the variable order to vector (V est, V MRSP, l, W, EB), we will from now on describe states s by their value vector (s(v est ), s(v MRSP ), s(l), s(w), s(eb)), so that an initial state s 0 is written as (c 0, c 1, NS, 0, 0). The transition relation R is specified by the predicate 6 R((V est, V MRSP, l, W, EB), (V est, V MRSP, l, W, EB )) 7 i=0 ϕ i((v est, V MRSP, l, W, EB), (V est, V MRSP, l, W, EB )) ϕ 0 (l = NS V est V MRSP l = NS W = W EB = EB) ϕ 1 (l = NS V est > V MRSP (g ebi1 g ebi2 ) l = WS W = 1 EB = EB V est = Vest V MRSP = V MRSP) ϕ 2 (l = NS (g ebi1 g ebi2 ) l = IS W = 1 EB = 1 V est = Vest V MRSP = V MRSP) ϕ 3 (l = WS V est > V MRSP (g ebi1 g ebi2 ) l = WS W = 1 EB = EB) ϕ 4 (l = WS V est V MRSP l = NS W = 0 EB = 0 V est = Vest V MRSP = V MRSP) ϕ 5 (l = WS (g ebi1 g ebi2 ) l = IS W = W EB = 1 V est = Vest V MRSP = V MRSP) ϕ 6 (l = IS V est > 0 l = IS W = W EB = 1) ϕ 7 (l = IS V est = 0 l = NS W = 0 EB = 0 V est = Vest V MRSP = V MRSP) 6 See [CGP99] about how to express transition relations as first order predicates: unprimed variables x denote their values in the pre-state of a transition, that is, the variable value of x before the transition is executed. Primed variables x denote their value after the transition has been executed. For example, ϕ 6 specifies the transition that can fire from any pre-state where the state machine resides in basic state IS and the estimated speed V est is greater that zero. The effect of this transition is that the state machine remains in basic state IS, the warning light output remains unchanged and the emergency brake output is set to 1. 20

21 The quiescent states are characterised by the pre-conditions ( unprimed conjuncts ) in ϕ 0, ϕ 3, ϕ 6, the transient states by the pre-conditions in ϕ 1, ϕ 2, ϕ 4, ϕ 5 and ϕ 7. Observe that in order to enforce the RKS rule 6 (transient states are followed by quiescent states), ϕ 2 specifies the direct transitions from control state NS to IS. Initial state s 0 = (0, 90, NS, 0, 0), for example, is quiescent; this follows from ϕ 0. In contrast to this, s 1 = (95, 90, NS, 0, 0) S 0 is transient (ϕ 1 applies). The latter initial state applies in a situation where the ceiling speed monitoring controller is re-booted while the train is driving (V est = 95), and the state is immediately left, since V est exceeds the allowed speed V MRSP. The atomic propositions AP and the labelling function L will be discussed in the examples below Quiescent Reduction The notion of transient states in RKS is semantically redundant. They only help to facilitate the mapping of concrete modelling formalisms (such finite state machines or UML/SysML state machines) into RKS. The redundancy of transient states is captured in the following definition. Definition 1 Given a Reactive Kripke Structure K = (S, S 0, R, L) the Kripke structure Q(K) defined by Q(K) = (Q(S), Q(S 0 ), Q(R), Q(L)), Q(S) = S Q Q(L) = L SQ : S Q P(AP ), Q(S 0 ) = {T (s 0 ) s 0 S 0 } Q(R) = {(s, s ) s, s S Q (R(s, s ) ( s S T : R(s, s ) s = T (s )))} is called the quiescent reduction of K. The state space of Q(K) consists of quiescent K-states only, and its labelling function is the restriction of L to quiescent states. The initial states of Q(K) consist of the union of the quiescent initial K-states and the quiescent poststates of transient initial K-states (recall that T maps quiescent states to themselves and transient states to their quiescent post-states). The transition relation Q(R) relates quiescent states already related in K, and those pairs of quiescent states that are related indirectly in K by means of an intermediate 21

22 transient state. Example 3. For the RKS described in Example 2, the quiescent reduction Q(K) has initial states Q(S 0 ) = {(V est, V MRSP, l, W, EB) V est V MRSP l = NS W = 0 EB = 0} {(V est, V MRSP, l, W, EB) V est > V MRSP (g ebi1 g ebi2 ) l = WS W = 1 EB = 0} {(V est, V MRSP, l, W, EB) (g ebi1 g ebi2 ) l = IS W = 1 EB = 1} The transition relation is given by Q(R)((V est, V MRSP, l, W, EB), (V est, V MRSP, l, W, EB )) 7 i=0 ψ i((v est, V MRSP, l, W, EB), (V est, V MRSP, l, W, EB )) ψ 0 (l = NS V est V MRSP l = NS W = 0 EB = 0) ψ 1 (l = NS V est > V MRSP (g ebi 1 g ebi 2 ) l = WS W = 1 EB = EB) ψ 2 (l = NS (g ebi 1 g ebi 2 ) l = IS W = 1 EB = 1) ψ 3 (l = WS V est V MRSP l = NS W = 0 EB = 0) ψ 4 (l = WS V est > V MRSP (g ebi 1 g ebi 2 ) l = WS W = 1 EB = EB) ψ 5 (l = WS (g ebi 1 g ebi 2 ) l = IS W = W EB = 1) ψ 6 (l = IS V est > 0 l = IS W = W EB = 1) ψ 7 (l = IS V est = 0 l = NS W = 0 EB = 0) Traces Traces of K are finite sequences of states related by R, including the empty sequence ε. Traces(K) = {ε} {s 0... s n S n N 0 s 0 S 0 n 1 i=0 R(s i, s i+1 )} The last state of a finite sequence of states is denoted by last(s 0... s n ) = s n, and tail(s 0... s n ) = (s 1... s n ), tail(s 0 ) = ε. Given trace s 0... s n we define its restriction to symbols from X V by (s 0... s n ) X = (s 0 X )... (s n X ) Input Traces Given a RKS K = (S, S 0, R, L, AP ), we consider the effect of input traces on states in K s quiescent reduction Q(K): an input trace ι = c 1... is a finite 22

23 sequence of input vectors c i D I, that is, ι (D I ). The application of an input trace ι to a quiescent state s Q(S) is written as s/ι and yields a trace of Q(K) which is recursively defined by where ι = tail(ι). s/ε = s s/( c 1.ι ) = s.(t (s { x c 1 })/ι ) As in the definitions above, T denotes the function mapping quiescent K- states to themselves and transient ones to their quiescent post-states. Obviously each pair of consecutive states in trace s/ι is related by transition relation Q(R). We will be frequently interested in the last element of an input trace application to a state; therefore the abbreviation s//ι = last(s/ι) is used. Since RKS are deterministic with respect to their reactions on input changes, s//ι is uniquely determined I/O Equivalence Model-based testing always investigates some notion of I/O conformance: stimulating the SUT with an input trace ι, the observable behaviour should be consistent with the behaviour expected for ι according to the model. The following definitions specify aspects of I/O equivalence, as they are relevant in the context of Reactive Kripke Structures. Definition 2 Given the quiescent reduction Q(K) of some RKS K, and quiescent states s 0, s 1 Q(S). 1. If (s 0 /ι) O = (s 1 /ι) O holds for all input traces ι, s 0 and s 1 are called I/O equivalent, written as s 0 s 1, otherwise s 0 and s 1 are called I/O distinguishable. 2. If (s 0 /ι) O = (s 1 /ι) O for some input trace ι, s 0 and s 1 are called ι-equivalent, written as s 0 ι s 1, otherwise s 0 and s 1 are called ι- distinguishable. 3. If (s 0 /ι) O = (s 1 /ι) O for all input traces ι contained in a set W, s 0 and s 1 are called W -equivalent, written as s 0 W s 1, otherwise s 0 and s 1 are called W -distinguishable. 4. If (s 0 /ι) O = (s 1 /ι) O holds for all input traces ι of length k, s 0 and s 1 are called k-equivalent, written as s k 0 s 1, otherwise s 0 and s 1 are called k-distinguishable. 23

24 Lemma 1 I/O equivalence satisfies the following properties. 1. If s 0 s 1, then (s 0 //ι) (s 1 //ι) for all input traces ι. 2. s 0 k s 1 and 0 t k implies s 0 t s 1. Definition 3 Two RKS K, K over the same variable symbols V are called I/O equivalent (written K K ) if their quiescent reductions are equivalent in the sense that (s 0, s 0) Q(S 0 ) Q(S 0) : (s 0 I = s 0 I s 0 s 0). 2.3 Input Equivalence Class Partitionings Over Reactive Kripke Structures With Finite Outputs In the remainder of this chapter we study the special case where our specification models K and implementations K only have output and internal variables with finite domains. The term finite is to be interpreted here in the sense that these values can be enumerated with reasonable effort. This contrasts with the domains of input variables, which we allow to have infinite range (such as real values) or to have very large finite cardinality (such as floating point or large integer types), where an enumeration would be impossible, due to time and memory restrictions. As a consequence, it is possible to further restrict the sets AP of atomic propositions under consideration. Since all possible values of internal states and output variables can be explicitly enumerated, AP can be structured into disjoint sets AP = AP I AP M AP O AP I {p p is atomic and var(p) I} AP M = {m = α m M α D m } AP O = {y = β y O β D y } Example 4. Consider a SysML state machine transition C 0 [x<m+y] C 1 24

25 s 0, s 1 S : L(s 0 ) (AP M AP O ) = L(s 1 ) (AP M AP O ) s 0 M O = s 1 M O D Specialised Test Strategies (Public) with x I, m M, y O, where D x = R, D y = {0, 1}, D m = {10, 11}. When transforming this machine into a RKS, the atomic propositions AP can be strictly separated according to their free variables being from I, M, or O, respectively. For example, AP = {l = C 0, m = 10, y = 0, x < 10, x < 11, x < 12}. Since the atomic propositions for internal states and outputs explicitly state the current value, the following lemma follows. Lemma 2 Given RKS K with finite outputs and internal state as described above. Then This immediately implies another simple, but useful lemma. Lemma 3 Let s, r S Q with L(s) = L(r). Then s { x c} = r { x c} and s// c = r// c hold for arbitrary input vectors c. Definition 4 Given RKS K = (S, S 0, R, L, AP ) with finite outputs and internal state, and AP partitioned into AP = AP I AP M AP O as described above. If s 0, s 1 S : (L(s 0 ) = L(s 1 ) L(T (s 0 )) = L(T (s 1 ))) then AP is called an input equivalence class partitioning (IECP) of K, and its input classes are specified by I = { { c D I p L(s) AP I p[ c/ x] p AP I L(s) p[ c/ x] } s S } In presence of an IECP AP, all input changes { x c}, { x d} to a state s satisfying the same input-related atomic propositions, that is, L(s { x c}) AP I = L(s { x d}) AP I, lead to post states satisfying the same atomic propositions. In particular, these post states all have the same internal state and the same outputs. Recall that T maps quiescent states to themselves, so the IECP property is only non-trivial for the transient states of an RKS. Example 5. For the ceiling speed monitoring function, whose RKS K has been constructed in Example 2, atomic propositions AP = {V est = 0, V est > V MRSP, V MRSP > 110, V est > V MRSP + 7.5, V est > V MRSP + 15, l = NS, l = WS, l = IS, W, EB} 25

26 introduce an IECP for K. Consider, for example, the states s 0 labelled by L(s 0 ) = {V est > V MRSP, l = NS}. Each of these s 0 is transient and has a post state s 1 satisfying V est = V est V MRSP = V MRSP l = WS W. As a consequence, all of these post-states are labelled by L(s 1 ) = {V est > V MRSP, l = WS, W}. The following Lemma shows that input traces applied to the same state and passing through the same sequences of input equivalence classes produce identical outputs. Lemma 4 Given RKS K = (S, S 0, R, L, AP ) with finite outputs and internal state as described above, so that AP is an IECP for K with input classes I. Let ι = c 1... c k, τ = d 1... d k, c i, d i D I, i = 1,..., k, such that i = 1,..., k, X i I : { c i, d i } X i Then s S Q : (s/ι) M O = (s/τ) M O. Proof. Let X i I satisfying { c i, d i } X i, i = 1,..., k. Let s S Q. Denote s/ι = s 0.s 1... s k, s/τ = r 0.r 1... r k, where s = s 0 = r 0. We prove by induction over i = 0,..., k that s i M O = r i M O. For i = 0 it is trivial, since s = s 0 = r 0. Suppose that the induction hypothesis holds for i < k, s i M O = r i M O. Since s i { x c i+1 } M O = r i { x d i+1 } M O and, according to the assumptions of the lemma, { c i+1, d i+1 } X i+1, we conclude that L(s i { x c i+1 }) = L(r i { x d i+1 }).The IECP property of AP now implies that also L(T (s i { x c i+1 })) = L(T (r i { x d i+1 })), and by definition T (s i { x c i+1 }) = s i+1, T (r i { x d i+1 }) = r i+1, therefore s i+1 M O = r i+1 M O. This proves the lemma. Lemma 5 Given RKS K with finite outputs and internal state as described above, and AP an IECP. Let p 1,..., p n a set of fresh atomic propositions not contained in AP, with var(p i ) I, i = 1,..., n. Then AP 2 = AP {p 1,..., p n } is another IECP, called the refinement of AP. The input classes of AP 2, constructed according to Definition 4, are denoted by I 2. Observe that IECP refinement according to Lemma 5 introduces new propositions in AP I only, while AP M and AP O remain unchanged. 26

27 2.4 Test Hypotheses and Proof of Exhaustiveness The input equivalence class testing strategy to be introduced in this section yield exhaustive tests, provided that the following two test hypotheses are met. (TH1) Testability Hypothesis. There exists a RKS K = (S, S 0, R, L, AP ) with finite internal states and output as introduced in Section 2.3 describing the true behaviour of the SUT, and its state space S consists of valuation functions s : V D for variables from V as specified for the reference model K = (S, S 0, R, L, AP ). (TH2) Existence of Refined Equivalence Class Partitioning. For specification model K = (S, S 0, R, L, AP ) and SUT K = (S, S 0, R, L, AP ), both atomic proposition sets AP, AP are IECP of K and K with input classes I, I, respectively, and AP M = AP M, AP O = AP O. Moreover, there exists an input partition refinement AP 2 = AP 2I AP M AP O, in the sense of Lemma 5, such that X I, X I : X 2 I 2 : (X X X 2 X X ) Validity of (TH2) induces a finite input alphabet to K and K which will be shown below to suffice for uncovering any violation of I/O equivalence between K and K. Definition 5 Given RKS K, K with finite internal state and outputs, and input equivalence class partitionings AP, AP and AP 2 according to test hypothesis (TH2). Let A I denote a finite subset of input vectors c D I satisfying X I 2 : c A I : c X Then A I is called an input alphabet for equivalence class partition testing of K against K. For any nonnegative integer k, A k I is the set of all A I - sequences of length less than or equal to k (including the empty trace ε). Example 6. Let K be the RKS of the ceiling speed monitor model constructed in Example 2, with IECP AP as given in Example 5. Now suppose that the SUT implementing the monitor model has an error, as indicated in 27

28 Figure 2.2: it uses a faulty guard condition g ebi1 g ebi1 instead of g ebi1 g ebi1. Its IECP (which, of course, would be unknown in a black box test) is AP = {V est = 0, V est > V MRSP, V MRSP > 110, V est > V MRSP + 7.5, V est > V MRSP + 20, l = NS, l = WS, l = IS, W, EB} The IECP refinement of AP AP 2 = {V est = 0, V est > V MRSP, V MRSP > 110, V est > V MRSP + 7.5, V est > V MRSP + 15, V est > V MRSP , V est > V MRSP , l = NS, l = WS, l = IS, W, EB} fulfils test hypothesis (TH2). The input classes I, I and I 2 are I = { {(V est, V MRSP ) V est = 0 (V MRSP > 110)}, {(V est, V MRSP ) V est = 0 V MRSP > 110}, {(V est, V MRSP ) (V est = 0) (V est > V MRSP ) (V MRSP > 110)}, {(V est, V MRSP ) (V est = 0) (V est > V MRSP ) V MRSP > 110)}, {(V est, V MRSP ) V est > V MRSP (V MRSP > 110) (V est > V MRSP + 7.5)}, {(V est, V MRSP ) V est > V MRSP V MRSP > 110 (V est > V MRSP + 7.5)}, {(V est, V MRSP ) V est > V MRSP (V MRSP > 110) V est > V MRSP (V est > V MRSP + 15)}, {(V est, V MRSP ) V est > V MRSP V MRSP > 110 V est > V MRSP (V est > V MRSP + 15)}, {(V est, V MRSP ) V est > V MRSP (V MRSP > 110) V est > V MRSP + 15}, {(V est, V MRSP ) V est > V MRSP V MRSP > 110 V est > V MRSP + 15} } = { {(V est, V MRSP ) V est = 0 V MRSP 110)}, {(V est, V MRSP ) V est = 0 V MRSP > 110}, {(V est, V MRSP ) V MRSP V est > 0 V MRSP 110}, {(V est, V MRSP ) V MRSP V est > 0 V MRSP > 110}, {(V est, V MRSP ) V MRSP V est > V MRSP V MRSP 110}, {(V est, V MRSP ) V MRSP V est > V MRSP > 110}, {(V est, V MRSP ) V MRSP + 15 V est > V MRSP V MRSP 110}, {(V est, V MRSP ) V MRSP + 15 V est > V MRSP V MRSP > 110}, {(V est, V MRSP ) V est > V MRSP + 15 V MRSP 110}, {(V est, V MRSP ) V est > V MRSP + 15 V MRSP > 110} } 28

29 I = { {(V est, V MRSP ) V est = 0 V MRSP 110)}, {(V est, V MRSP ) V est = 0 V MRSP > 110}, {(V est, V MRSP ) V MRSP V est > 0 V MRSP 110}, {(V est, V MRSP ) V MRSP V est > 0 V MRSP > 110}, {(V est, V MRSP ) V MRSP V est > V MRSP V MRSP 110}, {(V est, V MRSP ) V MRSP V est > V MRSP > 110}, {(V est, V MRSP ) V MRSP + 20 V est > V MRSP V MRSP 110}, {(V est, V MRSP ) V MRSP + 20 V est > V MRSP V MRSP > 110}, {(V est, V MRSP ) V est > V MRSP + 20 V MRSP 110}, {(V est, V MRSP ) V est > V MRSP + 20 V MRSP > 110} } I 2 = { {(V est, V MRSP ) V est = 0 V MRSP 110}, {(V est, V MRSP ) V est = 0 V MRSP > 110}, {(V est, V MRSP ) V MRSP V est > 0 V MRSP 110}, {(V est, V MRSP ) V MRSP V est > 0 V MRSP > 110}, {(V est, V MRSP ) V MRSP V est > V MRSP V MRSP 110}, {(V est, V MRSP ) V MRSP V est > V MRSP > 110}, {(V est, V MRSP ) V MRSP + 15 V est > V MRSP V MRSP 110}, {(V est, V MRSP ) V MRSP + 15 V est > V MRSP V MRSP > 110}, {(V est, V MRSP ) V MRSP V est > V MRSP + 15 V MRSP 110}, {(V est, V MRSP ) V MRSP V est > V MRSP + 15 V MRSP > 110}, {(V est, V MRSP ) V MRSP V est > V MRSP V MRSP 110}, {(V est, V MRSP ) V MRSP V est > V MRSP V MRSP > 110}, {(V est, V MRSP ) V est > V MRSP V MRSP 110}, {(V est, V MRSP ) V est > V MRSP V MRSP > 110} } For each X 2 I 2, choose an arbitrary vector (V est, V MRSP ) X 2, in order to build an input Alphabet A I. For example, let A I = { c 1 = (0, 90), c 2 = (0, 135), c 3 = (60, 90), c 4 = (60, 135), c 5 = (95, 90), c 6 = (140, 135), c 7 = (100, 90), c 8 = (145, 135), c 9 = (106, 90), c 10 = (151, 135), c 11 = (110, 90), c 12 = (155, 135), c 13 = (115, 90), c 14 = (160, 135)} Consider, for example the intersection of K input class X = {(V est, V MRSP ) V est > V MRSP + 15 V MRSP > 110} 29

30 I 2 = {x 21,x 22,...,x 214 } V est X 214 X 212 X 213 X 210 X 211 X X X X X X 23 X 24 X 21 X V MRSP Figure 2.3: Input classes I 2 and the K input class X = {(V est, V MRSP ) V MRSP + 20 V est > V MRSP V MRSP > 110} Then the input class X 2 = {(V est, V MRSP ) V MRSP V est > V MRSP + 15 V MRSP > 110} of the refined IECP AP 2 is contained in the intersection X X. Indeed, any input from X 2 applied to the SUT in control state WS would reveal in most cases the erroneous guard condition, because K transits into IS, while K remains in WS. For example, (s 0, s 0) Q(S 0 ) Q(S 0), (c 0, c 1 ) = (c 0, c 1) X 2i I 2 (s.figure 2.3). Let s 1 = s 0 //(151, 135), s 1 = s 0//(151, 135). 1. i = 1,..., 6, 8, 10 : s 1 = (151, 135, IS, 1, 1), s 1 = (151, 135, WS, 1, 0) 2. i = 7, 9, 11, 13, 14 : s 1 = (151, 135, IS, 1, 1), s 1 = (151, 135, IS, 1, 1) 3. i = 12 c 0 c : s 1 = (151, 135, IS, 1, 1), s 1 = (151, 135, WS, 1, 0) 30

31 V est IS IS WS IS IS WS IS IS WS WS 20 IS IS WS 15 IS WS 7.5 WS NS NS NS NS 110 V MRSP Figure 2.4: Initial states in quiescent reduction Q(K) and Q(K ) 4. i = 12 c 0 > c : s 1 = (151, 135, IS, 1, 1), s 1 = (151, 135, IS, 1, 1) In the case (2) and (3), we observe that s 0 = s 0 = (c 0, c 1, IS, 1, 1). Let ι = (0, 90).(151, 135) A I, (s 0, s 0) Q(S 0 ) Q(S 0) s 0 = s 0 = (c 0, c 1, IS, 1, 1) s 0 //ι = (151, 135, IS, 1, 1), s 0//ι = (151, 135, WS, 1, 0) For practical application (since the IECP of the SUT is unknown), the input space D I is systematically partitioned by intersecting the input-related propositions from AP with interval vectors, partitioning D I into I -dimensional cubes. Theorem 1 Given RKS K = (S, S 0, R, L, AP ), K = (S, S 0, R, L, AP ), such that AP, AP are IECP of K and K with input classes I, I, respectively, and AP 2 is a refinement of AP according to test hypothesis (TH2). I 2 contains the input classes associated with AP 2. Let A I an input alphabet derived from I 2 according to Definition 5. Then for any quiescent states 31

32 s S Q, s S Q and any input trace ι, there exists an input trace τ A I with the same length, such that s/ι O = s/τ O and s /ι O = s /τ O. Hence, s ι s if and only if s τ s. Proof. If ι is empty, there is nothing to prove, since ε A I. Suppose therefore, that ι = c 1... c k with k 1 and let s/ι = s 0.s 1... s k, and s /ι = s 0.s 1... s k, where s 0 = s, s 0 = s. Consider the associated sequences of input classes X 1... X k I and X 1... X k I, where c i X i and c i X i, for all i = 1,..., k. Since c i X i X i, i = 1,..., k, (TH2) implies the existence of X 21,..., X 2k I 2 such that X 2i X i X i, i = 1,..., k ( ) According to Definition 5, we can select d 1,..., d k A I, such that d i X 2i for all i = 1,..., k. (*) implies d i X i X i i = 1,..., k. Therefore, setting τ = d 1... d k, Lemma 4 may be applied to conclude that (s/ι) O = (s/τ) O and (s /ι) O = (s /τ) O. Therefore s ι s s τ s, and this completes the proof. 2.5 Test Strategy Given specification model K = (S, S 0, R, L, AP ) and SUT K = (S, S 0, R, L, AP ), and the refined IECP AP 2 with input classes I 2 according to test hypothesis (TH2). Let A I the input alphabet constructed from I 2 as specified in Definition 5. Then AP, A I and each s 0 Q(S 0 ) induce a deterministic finite state machine (DFSM) abstraction M(K, s 0 ) = (Q, q 0, A I, D O, δ, ω) of K with state space Q = {[s] s S Q }, initial state q 0 = [s 0 ], and input alphabet A I, where [s] = {r S Q r s}. Let O the set of output variables of K. The output alphabet of M(K, s 0 ) is defined by D O = D y1... D y O. The state transition function δ : Q A I Q of M(K, s 0 ) is defined by δ(q, c) = q 1 if and only if s S Q : q = [s] q 1 = [s// c] The output function ω : Q A I D O of M(K, s 0 ) is defined by ω(q, c) = e if and only if s S Q : q = [s] (s// c) O = { y e} We extend the domain of the state transition function to input traces, δ : Q A I Q by setting recursively δ(q, ε) = q, δ(q, c.ι) = q.δ(δ(q, c), ι). The output function can be extended to ω : Q A I DO by setting 32

33 ω(q, ι) = e 0... e k, if and only if δ(q, ι) = [s 0 ]... [s k ] and s i O = { y e i }, i = 0,..., k. Lemma 6 The DFSMs M(K, s 0 ) = (Q, q 0, A I, D O, δ, ω) introduced above are well-defined. Proof. Let q = [s] and [r] = [s] for some s, r S Q. Then r s, and therefore s// c r// c, and this shows that δ(q, c) is well-defined. Since all members of [s// c] coincide on O, this also shows that ω is well-defined. By construction, the DFSMs are minimal, because each pair of different states [s 0 ] [s 1 ] can be distinguished by an input trace resulting in different outputs when applied to [s 0 ] or [s 1 ], respectively. Since AP is an IECP, all K-states s 0, s 1 carrying the same label L(s 0 ) = L(s 1 ) are I/O-equivalent, so {s 1 L(s 1 ) = L(s)} [s] for all quiescent states of K. It may be the case, however, that some states carrying different labels are still I/O-equivalent, that is, L(s 0 ) L(s 1 ), but {s L(s) = L(s 0 )} {s L(s) = L(s 1 )} [s 0 ] = [s 1 ]. In analogy to M(K, s 0 ), DFSMs M(K, s 0) can be constructed from K, AP, s 0 Q(S 0), and the same input alphabet A I as has been used for the DFSMs M(K, s 0 ). We write M(K, s 0 ) M(K, s 0) and q 0 q 0, if and only if ω(q 0, ι) = ω (q 0, ι) for every ι A I. Note that this differs from I/O equivalence between K and K, where s 0 s 0 if and only if (s 0 /ι) O = (s 0/ι) O for every ι D I. The following theorem states that I/O equivalence between specification model K and an implementation K can be established by investigating the equivalence of their associated DFSM, that is, using ι A I only. Theorem 2 Given RKS K = (S, S 0, R, L, AP ) (specification model) and K = (S, S 0, R, L, AP ) (SUT) with finite internal states and output, fulfilling test hypotheses (TH1) and (TH2). Let AP 2 with input classes I 2 denote the refined IECP according to (TH2), and A I a choice of an input alphabet according to Definition 5. Then the following statements are equivalent K and K are I/O equivalent, K K. s 0 Q(S 0 ), s 0 Q(S 0) : (s 0 I = s 0 I M(K, s 0 ) M(K, s 0)). Proof. Obviously, M(K, s 0 ) M(K, s 0) q 0 q 0 ( τ A I : s τ 0 s 0). By Theorem 1, we have ( ι DI ι : s 0 s 0) ( τ A I : s τ 0 s 0). Hence s 0 s 0 M(K, s 0 ) M(K, s 0). Now the assertion follows directly from the definition of K K (Definition 3). Definition 6 With the terms introduced above, a transition cover of M(K, s 0 ) 33

34 is a set of input traces ι A I satisfying the following condition: for any reachable state q Q and any c A I, there is an input trace ι T C such that δ(q 0, ι) = q and ι. c T C. Definition 7 With the terms introduced above and minimal M(K, s 0 ), define a characterisation set W of M(K, s 0 ) as a set of traces ι A I, such that for all q 1, q 2 Q, there exists an input trace ι W such that ω(q 1, ι) ω(q 2, ι). On DFSM M(K, s 0 ) we can apply Chow s W-method [Cho78] to conclude that the following finite test suite is exhaustive for testing I/O equivalence between K and K. Theorem 3 Let s 0 Q(S 0 ), s 0 Q(S 0) with s 0 I = s 0 I, and T C(s 0 ), W (s 0 ) the transition cover and characterisation set of M(K, s 0 ) as introduced above. Assume that M(K, s 0 ) has n states and that M(K, s 0) has at most m states and m 0 = max(n, m). Then W(K) = [s 0 ] Q(S 0 )/ ( T C(s0 ).A m 0 n I.W (s 0 ) ) is an exhaustive test suite for testing SUT K against specification model K. Proof. M(K, s 0 ) and M(K, s 0) are two minimal DFSMs with the same input alphabet A I. Applying Chow s W-method [Cho78] to M(K, s 0 ) and M(K, s 0), M(K, s 0 ) and M(K, s 0) are I/O equivalent if and only if they are T C(s 0 ).A m 0 n I.W (s 0 ) equivalent 7. Hence the assertion follows directly from Theorem 2. Example 7. Let K the RKS of the ceiling speed monitor model constructed in Example 2, with IECP AP as given in Example 5. And the SUT implementing the monitor model, the refinement AP 2 and the input alphabet A I are constructed in Example 6. Form the transition relation listed in Example 3, there are 3 equivalence classes of S Q, namely, [NS] := {s S Q s(l) = NS}, [WS] := {s S Q s(l) = WS}, and [IS] := {s S Q s(l) = IS}. For M(K, s 0), there are also 3 equivalence classes of S Q which are [NS] := {s S Q s (l) = NS}, [WS] := {s S Q s (l) = WS}, and [IS] := {s S Q s (l) = IS}. The transition covers are, for example, T C([NS]) = {ε} { c 1,..., c 14 } { c 5. c 1,..., c 5. c 14 } { c 10. c 1,..., c 10. c 14 } 7 Observe that in [Cho78], the author uses a slightly different notation, where A i I denotes the set of input traces with length i, while we use this term to denote the traces of length less or equal i. 34