Model Based Testing : principles and applications in the context of timed systems

Transcription

1 A. Rollet - ETR Brest (France) - August /63 Model Based Testing : principles and applications in the context of timed systems Antoine Rollet Université de Bordeaux - LaBRI (UMR CNRS 5800), France rollet@labri.fr

2 A. Rollet - ETR Brest (France) - August /63 Outline 1 Model Based Testing 2 Conformance Testing with IOLTS 3 Testing Timed Systems 4 Conclusion and further work

3 A. Rollet - ETR Brest (France) - August /63 Outline 1 Model Based Testing 2 Conformance Testing with IOLTS 3 Testing Timed Systems 4 Conclusion and further work

4 A. Rollet - ETR Brest (France) - August /63 Introduction on testing Why testing? Systems getting more and more complex potentially more bugs A failure may cost a lot (human and financial) earlier detection implies weaker consequences Limitations Testing can only be used to show the presence of bugs, but never to show their absence (Dijkstra) need to make some assumptions Objective : increase the confidence in the system

5 A. Rollet - ETR Brest (France) - August /63 Different kinds of testing black box / white box white box : most elements of the system are known, especially source code (structural testing) black box : implementation is considered as an unknown black box; only interfaces are known test generation based on the specification (functional testing) What do we intend to test User testing, performance testing, conformance testing, interoperability testing, robustness testing, etc... Testing that a black-box implementation (IUT) of a system behaves correctly wrt. its functional specification Spec.

6 A. Rollet - ETR Brest (France) - August /63 Different kinds of testing black box / white box white box : most elements of the system are known, especially source code (structural testing) black box : implementation is considered as an unknown black box; only interfaces are known test generation based on the specification (functional testing) What do we intend to test User testing, performance testing, conformance testing, interoperability testing, robustness testing, etc... Testing that a black-box implementation (IUT) of a system behaves correctly wrt. its functional specification Spec.

7 A. Rollet - ETR Brest (France) - August /63 Conformance testing of reactive systems Reactive system System which reacts to its environment through its interfaces. Environment: human, software, hardware Necessary to think about : Controllability : how the tester can lead the test Observability : how the tester can get information definition of Points of Control and Observation (PCO). definition of a test architecture

8 A. Rollet - ETR Brest (France) - August /63 Model Based Testing Industrial practice: manual design of test suites from informal specifications Model Based Testing Model Based Testing (MBT) testing with the ability to detect faults which do not conform to a model called specification. Specification specifies conforms Implementation Under Test (IUT) possible automation for test generation, test execution, test evaluation (verdict) Formal Methods

9 A. Rollet - ETR Brest (France) - August /63 Model Based Testing (2) Test cases are generated from the Model Problems : need to find a good model of the specification what does specify mean? what does conform mean? Implementation is supposed to be equivalent to a formal model (but Implementation is unknown) Need to define a conformance relation between the Specification and the Implementation

10 A. Rollet - ETR Brest (France) - August /63 Model Based Testing (2) Test cases are generated from the Model Problems : need to find a good model of the specification what does specify mean? what does conform mean? Implementation is supposed to be equivalent to a formal model (but Implementation is unknown) Need to define a conformance relation between the Specification and the Implementation At the beginning... Two main approaches of MBT : Finite State Machines Labeled Transition Systems

11 A. Rollet - ETR Brest (France) - August /63 General schema Property P S P? Specification S I conf S? Implementation I

12 A. Rollet - ETR Brest (France) - August /63 General schema Property P S P? VERIFICATION Specification S I conf S? Implementation I

13 A. Rollet - ETR Brest (France) - August /63 General schema Property P S P? Specification S I conf S? TEST Implementation I

14 A. Rollet - ETR Brest (France) - August /63 General schema Property P S P? Specification S I conf S? Implementation I?! observation control Test cases Verdict

15 A. Rollet - ETR Brest (France) - August /63 General schema Property P S P? Specification S Test Generation I conf S? Implementation I?! observation control Test cases Verdict

16 A. Rollet - ETR Brest (France) - August /63 General schema Property P Test Purpose S P? Specification S Test Generation I conf S? Implementation I?! observation control Test cases Verdict

17 A. Rollet - ETR Brest (France) - August /63 Main ingredients of a testing theory Specification, implementation and conformance Specification: model of requested behaviors, Implementations: model of observable real behavior (unknown) Conformance relation: formalizes IUT conforms to Spec Tests cases and their executions Test cases, test suites: model of tests (control/observation) Test execution: interaction test IUT, produced observations, associated verdicts (e.g. pass, fail) Test suite properties: IUT passes TS IUT conf S Test generation Algorithms : tests = testgen( Spec (+ TestPurpose))

18 A. Rollet - ETR Brest (France) - August /63 Outline 1 Model Based Testing 2 Conformance Testing with IOLTS 3 Testing Timed Systems 4 Conclusion and further work

19 A. Rollet - ETR Brest (France) - August /63 References Part essentially based on : [Tre96] J. Tretmans, Test generation with inputs, outputs, and repetitive quiescence, Software Concepts and Tools, vol. 17, pp , [JJ04] C. Jard and T. Jéron, Tgv: theory, principles and algorithms, a tool for the automatic synthesis of conformance test cases for non-deterministic reactive systems, Software Tools for Technology Transfer (STTT), [Jer04] T. Jéron, Contribution à la génération automatique de tests pour les systèmes réactifs, 2004, habilitation à Diriger des Recherches - Université de Rennes 1.

20 A. Rollet - ETR Brest (France) - August /63 Input Output Labelled Transition System (IOLTS) τ 2 s 0?reset?digit s 1?reset τ 1 s 2!beep?digit s 7!open s 3!alarm τ 3 s 4 s 5 s 6 τ 4 M = (Q M, A M, M, q M 0 ) with : Q M set of states q M 0 Q M initial state A M action alphabet, A M I input alphabet (with?) A M O output alphabet (with!) I M internal actions (τ k ) M Q M A M Q M transition relation A M V IS = AM I A M O set of visible actions

21 A. Rollet - ETR Brest (France) - August /63 Input Output Labelled Transition System (IOLTS) τ 2 s 0?r?d s 1?r τ 1 s 2!b?d s 7!o s 3!a τ 3 s 4 s 5 s 6 τ 4 M = (Q, A,, q 0 ) with : Q set of states q 0 Q initial state A action alphabet, A I input alphabet (with?) A O output alphabet (with!) I internal actions (τ k ) Q A Q transition relation A V IS = A I A O set of visible actions

22 A. Rollet - ETR Brest (France) - August /63 Runs / Traces τ 2 s 0?r?d s 1?r τ 1 s 2!b?d s 7!o s 3!a τ 3 s 4 s 5 s 6 τ 4 Runs: alternate sequences of states and actions fireable btw those states?d τ s 0 s 1?d!o 1 s2 s 3 s 4 Runs(M) Traces: projections of Runs on visible actions: Traces(M) = {ε,?d,?r,?d.?r,?r.?d,?d.!b,...} P after σ: set of states reachable from P after observation σ: {s 2 } after?d.!o = {s 0, s 4 } {s 0 } after?d,!a = M after σ {q 0 } after σ

23 A. Rollet - ETR Brest (France) - August /63 Non-determinism?x!a s 1 s 2!a s 3 Not to be confused with uncontrolled choice s τ 1 s 2!b!a s 3 s 4?x!a s 2 s 1!b s 3 M is deterministic if it has no internal action, and q, q, q a Q, a A V IS, (q q a q q ) q = q Determinization: det(m) = (2 Q, A V IS, det, q 0 after ɛ) with P a det P P, P 2 Q, a A V IS and P = P after a. Traces(M) = Traces(det(M))

24 A. Rollet - ETR Brest (France) - August /63 Non-determinism?x!a s 1 s 2!a s 3 Not to be confused with uncontrolled choice s τ 1 s 2!b!a s 3 s 4?x!a s 2 s 1!b s 3 M is deterministic if it has no internal action, and q, q, q a Q, a A V IS, (q q a q q ) q = q Determinization: det(m) = (2 Q, A V IS, det, q 0 after ɛ) with P a det P P, P 2 Q, a A V IS and P = P after a. Traces(M) = Traces(det(M))

25 A. Rollet - ETR Brest (France) - August /63 Observation of quiescence In testing practice, one can observe traces of the IUT, but also its quiescences with timers. Only quiescences of IUT unspecified in S should be rejected. τ 2?r s 0 s 1?d τ 1?r s 2!b?d s 7!o s 3!a τ 3 s 4 s 5 s 6 τ 4 Notation : Γ(q) {a A q } a deadlock : no possible evolution : Γ(q) =. outputlock : systems waiting for an action : Γ(q) A I. livelock : internal actions loop : τ 1,...τ n : q τ 1...τ n q. quiescent(m) = deadlock(m) livelock(m) outputlock(m)

26 A. Rollet - ETR Brest (France) - August /63 Observation of quiescence In testing practice, one can observe traces of the IUT, but also its quiescences with timers. Only quiescences of IUT unspecified in S should be rejected. τ 2?r s 0 s 1?d τ 1?r s 2!b?d s 7!o s 3!a τ 3 s 4 s 5 s 6 τ 4 Notation : Γ(q) {a A q } a deadlock : no possible evolution : Γ(q) =. outputlock : systems waiting for an action : Γ(q) A I. livelock : internal actions loop : τ 1,...τ n : q τ 1...τ n q. quiescent(m) = deadlock(m) livelock(m) outputlock(m)

27 A. Rollet - ETR Brest (France) - August /63 Observation of quiescence In testing practice, one can observe traces of the IUT, but also its quiescences with timers. Only quiescences of IUT unspecified in S should be rejected. τ 2?r s 0 s 1?d τ 1?r s 2!b?d s 7!o s 3!a τ 3 s 4 s 5 s 6 τ 4 Notation : Γ(q) {a A q } a deadlock : no possible evolution : Γ(q) =. outputlock : system waiting for an action : Γ(q) A I. livelock : internal actions loop : τ 1,...τ n : q τ 1...τ n q. quiescent(m) = deadlock(m) livelock(m) outputlock(m)

28 A. Rollet - ETR Brest (France) - August /63 Observation of quiescence In testing practice, one can observe traces of the IUT, but also its quiescences with timers. Only quiescences of IUT unspecified in S should be rejected. τ 2?r s 0 s 1?d τ 1?r s 2!b?d s 7!o s 3!a τ 3 s 4 s 5 s 6 τ 4 Notation : Γ(q) {a A q } a deadlock : no possible evolution : Γ(q) =. outputlock : system waiting for an action : Γ(q) A I. livelock : internal actions loop : τ 1,...τ n : q τ 1...τ n q. quiescent(m) = deadlock(m) livelock(m) outputlock(m)

29 A. Rollet - ETR Brest (France) - August /63 Observation of quiescence In testing practice, one can observe traces of the IUT, but also its quiescences with timers. Only quiescences of IUT unspecified in S should be rejected. τ 2?r s 0 s 1?d τ 1?r s 2!b?d s 7!o s 3!a τ 3 s 4 s 5 s 6 τ 4 Notation : Γ(q) {a A q } a deadlock : no possible evolution : Γ(q) =. outputlock : system waiting for an action : Γ(q) A I. livelock : internal actions loop : τ 1,...τ n : q τ 1...τ n q. quiescent(m) = deadlock(m) livelock(m) outputlock(m)

30 A. Rollet - ETR Brest (France) - August /63 Suspension automaton!δ Quiescence : special output δ s 0?d?r The suspension iolts of M = (Q, A,, q 0 ) is an iolts (M) = (Q, A {δ}, (M), q 0 ) where (M) = {q δ q q quiescent(m)}. τ 2 s 1?r τ 1 s 2!b?d s 7!δ s 3!o!a τ 3 s 4 s 5 s 6!δ τ 4!δ

31 Suspension traces!δ!δ s 0?r?d τ s 1 2?r τ 1 s!b 2 s 7!δ?d τ 3 s!o 4 s!a 3 s 5 s 6!o s 0, s 4?r?d?r s 1, s 2?d!b s 7!δ s 3!a s 5, s 6!δ τ 4!δ!δ Suspension traces (S) det( (S)) STraces(M) Traces( (M)) = Traces(det( (M))) STraces(S) and STraces(I) represent visible behaviors of S and I for testing a base for the definition of conformance. A. Rollet - ETR Brest (France) - August /63

32 A. Rollet - ETR Brest (France) - August /63 Testing framework Specification : iolts S = (Q S, A S, S, s S 0 ) Implementation : iolts IUT = (Q IUT, A IUT, IUT, s IUT 0 ) Unknown implementation, except for its interface, identical to S s Hyp.: IUT is input-complete : In any state, IUT accepts any input, possibly after internal actions.

33 A. Rollet - ETR Brest (France) - August /63 Conformance relation The conformance relation defines the set of implementations IU T conforming to S. Conformance IUT ioco S σ STraces(S), Out( (IU T ) after σ) Out( (S) after σ) with Out(P ) Γ(P ) A δ O a : set of outputs quiescences in P. a A δ O is equivalent notation for A O since δ is an output of (S) and (IUT ) Intuition : IUT conforms to S iff after any suspension trace of S and IUT, all outputs and quiescences of IUT are specified by S.

34 A. Rollet - ETR Brest (France) - August /63 ioco: example s 0!δ?a!z s 1!δ!x!y s 2 s 3 specification (S) s 0!δ?a!z s 1!x s 2 I 1 : Implem. choice s 0!δ?a!z s 1!δ!x!y!z s 2 s 3!δ s 4 I 3 : Unspec. output s 0!δ!z?b?a s 4!δ s 1!δ!z!x!y s 5 s 2 s 3 I 2 : Implem. of a partial spec. s 0!δ!z?a?a s 4 s 1!δ!x!y s 2 s 3!δ I 4 : Unspec. quiescence

35 A. Rollet - ETR Brest (France) - August /63 Canonical Tester From S (more precisely from det( (S)) = (Q d, A d, d, q0 d)), build an iolts Can(S) = (Q c, A c, c, q0 c ) the most general iolts permitting to detect non-conformance of implementation IUT. Q c = Q d {Fail} and q0 c = qd 0 A c = A c I Ac O where Ac I = Ad O and Ac O = Ad I inputs of the tester are outputs of S and reciprocally. a c = d {q c Fail q Q d, a A c I (q a d )}, all non-specified outputs lead to Fail.

36 A. Rollet - ETR Brest (France) - August /63 Canonical Tester From S (more precisely from det( (S)) = (Q d, A d, d, q0 d)), build an iolts Can(S) = (Q c, A c, c, q0 c ) the most general iolts permitting to detect non-conformance of implementation IUT. Q c = Q d {Fail} and q0 c = qd 0 A c = A c I Ac O where Ac I = Ad O and Ac O = Ad I input / output inversion a c = d {q c Fail q Q d, a A c I (q a d )}, all non-specified outputs lead to Fail.

37 A. Rollet - ETR Brest (France) - August /63 Canonical Tester From S (more precisely from det( (S)) = (Q d, A d, d, q0 d)), build an iolts Can(S) = (Q c, A c, c, q0 c ) the most general iolts permitting to detect non-conformance of implementation IUT.?δ q 0!r!r!d?o q 1?b q 4!d?δ q 2 q 3?a?δ

38 A. Rollet - ETR Brest (France) - August /63 Canonical Tester From S (more precisely from det( (S)) = (Q d, A d, d, q0 d)), build an iolts Can(S) = (Q c, A c, c, q0 c ) the most general iolts permitting to detect non-conformance of implementation IUT.?δ q 0!r!r!d?a,?o,?b?a,?o,?δ?o q 1?b q 4?a,?b,?o Fail!d?δ?b,?δ q 2?a q 3?a,?b,?o?δ

39 A. Rollet - ETR Brest (France) - August /63 Canonical Tester From S (more precisely from det( (S)) = (Q d, A d, d, q0 d)), build an iolts Can(S) = (Q c, A c, c, q0 c ) the most general iolts permitting to detect non-conformance of implementation IUT.?δ?o q 0!r?a,?o,?b!d!r?a,?o,?δ q 1!d?b q 4?a,?b,?o?δ Fail?b,?δ?a,?b,?o q 2?a q 3?δ IUT ioco S STraces(IUT ) Traces Fail (Can(S)) =

40 A. Rollet - ETR Brest (France) - August /63 Test cases A test case is a deterministic iolts (Q TC, A TC, TC, t TC 0 ), equipped with verdict states: Pass, Fail and Inconc s.t. A TC O = AS I and ATC I = A S O {δ} (input / output inversion) T C is controllable, i.e. never have to choose btw. several outputs or btw. inputs and outputs : q Q TC, ( a A TC O, q A TC b, (b a q TC )) a TC b All states permitting an input, are input-complete, except verdict states. t 0?δ?othw t 1!d t 2 Fail!d?othw t?a 3?othw?o Inconc t 4?δ Pass

41 A. Rollet - ETR Brest (France) - August /63 Test execution Modelled by the parallel composition T C (IU T ) synchronizing on common visible actions s 0, t 0 t 0?δ s 0, t 1?δ?othw!δ!d t 1 s 0?r s 1, t 2!d t?d!d 2 Fail τ s s!d?othw 2 1 2, t?b 3 Fail τ?r!b 1?o t?d?a 3 s Inconc s 6, t 3 5!δ?a s?othw s 4, t 4 2?o!o τ 1 τ Inconc 2 t!b 4 s 4 s!a 6 s 7!δ s 0, t 4 s 3?δ?δ Pass!δ Pass T C1 (IUT ) T C1 (IUT )

42 A. Rollet - ETR Brest (France) - August /63 Properties of test suites T C fails IUT iff an execution of T C (IUT ) reaches Fail Expresses a possibility for rejection. Due to non-controllable choices of IUT, a single test case applied on a single Implementation can produce all different verdicts! Soundness, Exhaustiveness, Completeness A set of test cases T S is Sound IUT : (IUT ioco S = T C T S : (T C fails IUT )), i.e. only non-conformant IUT may be rejected by a T C T S. Exhaustive IUT : ( (IUT ioco S) = T C T S : T C fails IUT ), i.e. any non-conformant IUT may be rejected by a T C T S. Complete = Sound and Exhaustive

43 A. Rollet - ETR Brest (France) - August /63 Properties of test suites T C fails IUT iff an execution of T C (IUT ) reaches Fail Expresses a possibility for rejection. Due to non-controllable choices of IUT, a single test case applied on a single Implementation can produce all different verdicts! Soundness, Exhaustiveness, Completeness A set of test cases T S is Sound IUT : (IUT ioco S = T C T S : (T C fails IUT )), i.e. only non-conformant IUT may be rejected by a T C T S. Exhaustive IUT : ( (IUT ioco S) = T C T S : T C fails IUT ), i.e. any non-conformant IUT may be rejected by a T C T S. Complete = Sound and Exhaustive

44 A. Rollet - ETR Brest (France) - August /63 Test selection Objective : Find an algorithm taking as input a finite state iolts S, and satisfying the following properties: produces only sound test suites is limit-exhaustive i.e. the infinite suite of test cases that can be produced is exhaustive Two techniques : 1 Non-deterministic selection (à la TorX) 2 Selection guided by a test purpose (à la TGV)

45 A. Rollet - ETR Brest (France) - August /63 Non-deterministic selection Algorithm: partial unfolding of Can(S) Start in q0 c. After any trace σ in Can(S) if Can(S) after σ Fail, emit a Fail verdict otherwise make a choice between produce a Pass verdict and stop, consider all inputs of Can(S) after σ and continue. choose one output in those of Can(S) after σ and continue. Properties T S = all possible Test cases generated with this algorithm : T S is sound and limit-exhaustive

46 A. Rollet - ETR Brest (France) - August /63 Non-deterministic selection Algorithm: partial unfolding of Can(S) Start in q0 c. After any trace σ in Can(S) if Can(S) after σ Fail, emit a Fail verdict otherwise make a choice between produce a Pass verdict and stop, consider all inputs of Can(S) after σ and continue. choose one output in those of Can(S) after σ and continue. Properties T S = all possible Test cases generated with this algorithm : T S is sound and limit-exhaustive

47 A. Rollet - ETR Brest (France) - August /63 Examples q 0?o?δ q 0!r?a,?o,?b!d!r?a,?o,?δ q 1?b q 4?a,?b,?o Fail!d?δ?b,?δ?a,?b,?o q 2?a q 3?δ Can(S)!r?a,?b,?o q 0 Fail?δ q 0!d q 1?b q 4?δ Pass?a,?o,?δ Fail?a,?o,?b Fail T C1

48 A. Rollet - ETR Brest (France) - August /63 Examples q 0?δ!d?o q 0!r?a,?o,?b!d!r?a,?o,?δ q 1!d?b q 4?a,?b,?o?δ Fail?b,?δ?a,?b,?o q 2?a q 3?δ Can(S)?a,?o,?b Fail q 3 q 1 q 2?a?o?b,?δ?δ q 0 Pass q 0?δ?a,?o,?b Pass T C2!d!r Fail Fail

49 A. Rollet - ETR Brest (France) - August /63 Test Purpose generation Previous algorithm : maybe quite long if we intend to focus on a specific behavior... Main characteristics of Test Purpose Generation: test selection by test purposes describing a set of behaviors to be tested, targeted by a test case, off-line selection, a posteriori execution.

50 Test Purpose definition Test Purpose Deterministic and complete iolts T P = (Q TP, A TP, TP, q TP 0 ) equipped with two sets Accept TP and Refuse TP of trap states, s.t. A T P = A S V IS {δ}?δ?o q 0!r?a,?o,?b!d!r?a,?o,?δ q 1!d?b q 4?a,?b,?o?δ Fail?b,?δ?a,?b,?o q 2?a q 3 p 3 Refuse?r p 0 p 1?r!o p 2 Accept?δ Can(S) T P A. Rollet - ETR Brest (France) - August /63

51 A. Rollet - ETR Brest (France) - August /63 Selection principle

52 A. Rollet - ETR Brest (France) - August /63 Synchronous Product : definition Definition of Synchronous Product The Synchronous Product of two iolts M 1 = (Q M1, A, M1, q0 M1), and M 2 = (Q M2, A, M2, q0 M2 ) is the iolts M 1 M 2 = (Q M1 Q M2, A,, q0 M1 q0 M2 ) where is defined by : (q M1, q M2 ) a (q M1, q M2) (q M1 a M1 q M1) (q M2 a M2 q M2)

53 The Synchronous Product Can(S) T P?δ?δ?o!r!d q 0 q 1!d!r?a,?o,?δ q 4?b?a,?b,?o?b,?δ?δ q 2 q 3?a p 3 Refuse?r?δ?a,?o,?b p 0 p 1?r!o p 2 Accept Fail?a,?b,?o q 0,p 0!d!r?o?δ q 0,p 1 q 1,p 0!r!d?b!d!r?δ,!r q 0,p 3 q 1,p 1 q 4,p 0 q 2,p 0!r!d?b!r!d?δ?a q 1,p 3?o q 2,p 1 q 4,p 1 q 3,p 0?δ?b!d?o?a?δ?δ q 4,p 3 q 2,p 3!r q 0,p 2 q 3,p 1?δ?a!d?δ q 3,p 3?δ q 1,p 2?othw F ail A. Rollet - ETR Brest (France) - August /63

54 A. Rollet - ETR Brest (France) - August /63 Complete Test Graph (CTG)?δ Keep the first Accept state in a path If q coreach(p ass) keep q If q {Fail} keep q If q coreach(p ass) input (tester point of view) successor of a state q coreach(p ass) then Inconc?δ q 0, p 0 q 0, p 1!d?o!r q 1, p 0!r!d?b!d q 1, p 1 Inconc q 2, p 0!d?b?a q 2, p 1 Inconc Inconc?o?a Pass Inconc?othw F ail

55 A. Rollet - ETR Brest (France) - August /63 Ensuring controlabillity of test cases q 0, p 0?δ?othw Fail?othw!r q 0, p 1?δ q 0, p 1!d q 1, p 1!d q 2, p 1?o?a Pass Inconc Example of Test Case?δ q 0, p 0!d!r?o q 0, p 1 q 1, p 0!r!d?b!d q 1, p 1 Inconc q 2, p 0!d?b?a q 2, p 1 Inconc Inconc?o?a Pass Inconc?othw F ail The test suite composed of the set of test cases that the algorithm can produce is sound and limit-exhaustive.

56 A. Rollet - ETR Brest (France) - August /63 Conclusion Testing theory for iolts Test generation for finite iolts Non-deterministic selection: unfolding of Can(S) Selection by test purpose: for finite iolts based on co-reachability analysis. Soundness and exhaustiveness.

57 A. Rollet - ETR Brest (France) - August /63 Outline 1 Model Based Testing 2 Conformance Testing with IOLTS 3 Testing Timed Systems 4 Conclusion and further work

58 A. Rollet - ETR Brest (France) - August /63 References Part essentially based on : [HLMNPS08] A. Hessel, K. Larsen, M. Mikucionis, B. Nielsen, P. Pettersson, and A. Skou, Testing real-time systems using uppaal, in Formal Methods and Testing, LNCS, vol Springer Berlin / Heidelberg, 2008, pp [MLN04] M. Mikucionis, K. G. Larsen, and B. Nielsen, T-uppaal: Online model-based testing of real-time systems, in 19th IEEE International Conference on Automated Software Engineering (ASE 2004), September 2004, Linz, Austria. IEEE Computer Society, 2004, pp [KT04] M. Krichen and S. Tripakis, Black-box conformance testing for real-time systems, in Model Checking Software, 11th International SPIN Workshop, Barcelona, Spain, April 1-3, 2004, LNCS vol Springer, 2004, pp

59 A. Rollet - ETR Brest (France) - August /63 Main lines Need a new model to describe real-time aspects : Timed Automata with Inputs and Outputs... and semantics. Need a new conformance relation : rtioco Non-deterministic online test generation Discussion about offline test generation

60 A. Rollet - ETR Brest (France) - August /63 Uppaal-like approach Explicit and separate model of the environment input Real Env. IUT i output E S o + test generation tool can synthesize only relevant scenario + designer can lead the test to specific situations

61 A. Rollet - ETR Brest (France) - August /63 Timed Automaton l 0 x 1!wCoffee?coin x 1!sCoffee x < 5 l 1 x 3 l 2 x 3 l 3 x 5 Semantics defined in terms of TIOTS. Possibly non-deterministic

62 A. Rollet - ETR Brest (France) - August /63 Timed Input Output Transition System (TIOTS) Given a set of actions A, divided in A out and A in, and τ A. (A τ A {τ}) if no precision is given, in the following a [k] is an action, d [k] is a delay TIOTS definition S = (S, s 0, A in, A out, ) where : S set of states, s 0 S the initial state S (A τ R 0 ) S transition relation with time determinism : (s d s s d s ) s = s time additivity : (s d1 s s d 2 s ) s d1+d2 s zero-delay : s, s 0 s Testing point of view : Timed Traces are considered, e.g. σ =?coin 1 2!wCoffee 9?coin

63 Notations / Definitions s a s iff s τ a τ s s d s iff s τ d 1 τ d 2 where d = n k=1 d k usually generalized to sequences Observable Timed Traces T T r(s) τ τ... d n T T r(s) = {σ (A R 0 ) s σ } Example : σ =?coin 1 2!wCoffee 9?coin τ s After s After σ = {s s σ s }, S Afterσ = s S s After σ Out Out(s) = {a A out R 0 s } a Out(S ) = s S Out(s) A. Rollet - ETR Brest (France) - August /63

64 Notations / Definitions s a s iff s τ a τ s s d s iff s τ d 1 τ d 2 where d = n k=1 d k usually generalized to sequences Observable Timed Traces T T r(s) τ τ... d n T T r(s) = {σ (A R 0 ) s σ } Example : σ =?coin 1 2!wCoffee 9?coin τ s After s After σ = {s s σ s }, S Afterσ = s S s After σ Out Out(s) = {a A out R 0 s } a Out(S ) = s S Out(s) A. Rollet - ETR Brest (France) - August /63

65 Notations / Definitions s a s iff s τ a τ s s d s iff s τ d 1 τ d 2 where d = n k=1 d k usually generalized to sequences Observable Timed Traces T T r(s) τ τ... d n T T r(s) = {σ (A R 0 ) s σ } Example : σ =?coin 1 2!wCoffee 9?coin τ s After s After σ = {s s σ s }, S Afterσ = s S s After σ Out Out(s) = {a A out R 0 s } a Out(S ) = s S Out(s) A. Rollet - ETR Brest (France) - August /63

66 Notations / Definitions s a s iff s τ a τ s s d s iff s τ d 1 τ d 2 where d = n k=1 d k usually generalized to sequences Observable Timed Traces T T r(s) τ τ... d n T T r(s) = {σ (A R 0 ) s σ } Example : σ =?coin 1 2!wCoffee 9?coin τ s After s After σ = {s s σ s }, S Afterσ = s S s After σ Out Out(s) = {a A out R 0 s } a Out(S ) = s S Out(s) A. Rollet - ETR Brest (France) - August /63

67 A. Rollet - ETR Brest (France) - August /63 Timed Automata (with Inputs and Outputs) : definition Given X set of clock variables, G(X) set of guards, U(X) set of updates. x 1!wCoffee l 2 x 3 x < 5 l 0 l 1?coin x 3 x 1!sCoffee l 3 x 5 Timed Automaton T A = (L, l 0, I, E) where L set of locations, l 0 initial location I : L G(X) assigns invariants to locations E L G(X) A τ U(X) L set of edges (written l g,α,u l ) Observable trace example : σ =?coin Out(?coin 6 3) = {scoffee} [0, 2]

68 most reasoning done on the semantics A. Rollet - ETR Brest (France) - August /63 Semantics of Timed Automata Semantics as a TIOTS defined by : States of the form s = (l, v), s.t. l is a location v R X 0 clock valuation satisfying invariant of l Delay transitions (l, v) d d.i l (d ) d (l, v + d) Discrete transitions l g(v) I l (v ), v = u(v) α (l, v) (l, v ) l g,α,u

69 A. Rollet - ETR Brest (France) - August /63 Relativized timed conformance rtioco e S = (S S, s S 0, A in, A out, S ) a weakly input enabled (i.e. s S S, i A in, s ) i TIOTS IUT = (S IUT, s IUT 0, A in, A out, IUT ) a weakly input enabled TIOTS E = (E E, e E 0, A out, A in, E ) (input / output inversion) weakly input enabled TIOTS. Let s S S, e E E and iut S IUT : iut rtioco e s iff σ T T r(e), Out((iut, e) After σ) Out((s, e) After σ) iff T T r(iut) T T r(e) T T r(s) T T r(e)

70 A. Rollet - ETR Brest (France) - August /63 Relativized timed conformance (2) rtioco ensures Implementation has only the behavior allowed by Specification : Implementation not allowed to produce an output at a time when not allowed by Specification Implementation not allowed to omit producing an output when required by the Specification

71 A. Rollet - ETR Brest (France) - August /63 rtioco examples!coin l 0 l 0?wCoffe?sCoffe x 1!wCoffee?coin x 1!sCoffee x 2!wCoffee?coin x 3!sCoffee!req x < 5 l 1 x 3 x 4 l 1 x > 4 l 2 x 3 l 3 x 5 l 2 x 2 l 3 x 3 Environment Specification s Implementation i 1 Trace σ Out(s After σ) Out(i 1 After σ) c 2 R 0 R 0 c 4 r 1 {wcoffee, scoffee} [0, 4] [0, 1] c 4 r 2 {wcoffee, scoffee} [0, 3] {wcoffee, 0} c 5 r 3 {scoffee} [0, 2] {scoffee, 0} c 5 r 5 {scoffee, 0}

72 A. Rollet - ETR Brest (France) - August /63 rtioco examples (2)!coin l 0 l 0?wCoffe?sCoffe x 1!wCoffee?coin x 1!sCoffee x 2!wCoffee?coin x 6!sCoffee!req x < 5 l 1 x 3 x 4 l 1 x > 4 l 2 x 3 l 3 x 5 l 2 x 3 l 3 x 7 Environment Specification s Implementation i 2 Trace σ Out(s After σ) Out(i 2 After σ) c 2 R 0 R 0 c 4 r 1 {wcoffee, scoffee} [0, 4] [0, 2] c 4 r 2 {wcoffee, scoffee} [0, 3] {wcoffee} [0, 1] c 5 r 3 {scoffee} [0, 2] [0, 4] c 5 r 5 {scoffee, 0} [0, 2]

73 A. Rollet - ETR Brest (France) - August /63 Online testing (à la TorX) On-the-fly testing : combines test generation and execution Non-deterministic generation Symbolic states Weakly input-enabled and non-blocking TIOTS Advantages : reduces state space explosion handles non-determinism Drawbacks : specification must be analyzed online, in real-time test runs may be long... coverage criteria can not be guaranteed

74 A. Rollet - ETR Brest (France) - August /63 Non-determinism Often used : as means of abstraction to model optional behavior, permitted but not required determinism definition An TIOTS (S) is deterministic if α (A τ R 0 ), s S, (s α s s α s ) s = s. l 1 l 0 x 7?a?a l 2 l 4 l 5 l 6?a l 3 (l 0, x = 3) After a = {(l 2, x = 3), (l 4, x = 3), (l3, x = 0)} (l 5, x = 0) After 4 = {(l 5, x = 4), (l 6, 0 x 4)}

75 Uppaal TRON algorithm T estgenexe(s, E, IUT, T ) while Z = iterations T do switch randomly choose btw action, delay and restart do case action /* offer an input */ if EnvOutput(Z) then randomly choose i EnvOutput(Z); send i to IUT ; Z := Z After i ; case delay /* wait for an output */ randomly choose d Delays(Z) ; sleep for d time units or wake up on output o at d d; if o occurs then Z := Z After d ; if o ImpOutput(Z) then return FAIL else Z := Z After o else Z := Z After d ; case restart Z := {(s 0, e 0)}, reset IUT /* reset and restart */ if Z = then return FAIL else return PASS A. Rollet - ETR Brest (France) - August /63

76 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe Symbolic state set: {(k 0 l 0, x = 0)} EnvOutput: coin ImpOutput: x 1!wCoffee x 5 l 2 x 3 l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!sCoffee x > 4 l 3 x 4 x = 0 Wait for ouput (delay) or offer input?

77 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe Symbolic state set: {(k 0 l 0, x = 0)} EnvOutput: coin ImpOutput: x 1!wCoffee x 5 l 2 x 3 l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!sCoffee x > 4 l 3 x 4 x = 0 Wait for ouput (delay) or offer input?

78 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe Symbolic state set: {(k 0 l 0, x = 0)} EnvOutput: coin ImpOutput: x 1!wCoffee x 5 l 2 x 3 l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 coin x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!sCoffee x > 4 l 3 x 4 x = 0 Let s offer an input. Choose (the only) coin

79 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe Symbolic state set: {(k 1 l 1, x = 0)} EnvOutput: req ImpOutput: x 1!wCoffee x 5 l 2 x 3 l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!sCoffee x > 4 l 3 x 4 x = 0 Update the state set and other variables

80 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe Symbolic state set: {(k 1 l 1, x = 0)} EnvOutput: req ImpOutput: x 1!wCoffee x 5 l 2 x 3 l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!sCoffee x > 4 l 3 x 4 x = 0 Wait or offer input? Let s wait for 5 units

81 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe Symbolic state set: {(k 1 l 1, x = 5)} EnvOutput: req ImpOutput: x 1!wCoffee x 5 l 2 x 3 l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!sCoffee x > 4 l 3 x 4 x = 5... no ouput so far... update the state set

82 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe Symbolic state set: {(k 1 l 1, x = 5)} EnvOutput: req ImpOutput: x 1!wCoffee x 5 l 2 x 3 l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 req x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!sCoffee x > 4 l 3 x 4 x = 5 Wait or offer input? Let s offer req

83 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe x 1!wCoffee x 5 l 2 x 3 Symbolic state set: {(k 2 l 2, x = 0), (k 2 l 3, x = 0)} EnvOutput: ImpOutput: l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!sCoffee x > 4 l 3 x 4 x = 0 Update the state set and other variables

84 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe x 1!wCoffee x 5 l 2 x 3 Symbolic state set: {(k 2 l 2, x = 0), (k 2 l 3, x = 0)} EnvOutput: ImpOutput: l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!sCoffee x > 4 l 3 x 4 x = 0 Wait or offer input? Let s wait for 4 units

85 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe x 1!wCoffee x 5 l 2 x 3 Symbolic state set: (k 2 l 3, x = 4)} EnvOutput: ImpOutput: {scof f ee} l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!sCoffee x > 4 l 3 x 4 x = 4... no output so far: update the state set

86 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe x 1!wCoffee x 5 l 2 x 3 Symbolic state set: {(k 2 l 3, x = 4)} EnvOutput: ImpOutput: {scof f ee} l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!sCoffee x > 4 l 3 x 4 x = 4 Wait or offer input? Let s wait for 2 units

87 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe x 1!wCoffee x 5 l 2 x 3 Symbolic state set: {(k 2 l 3, x = 4)} EnvOutput: ImpOutput: {scof f ee} l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 scoffee x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!sCoffee x > 4 l 3 x 4 x = 4 Got output after 0 delay: update the state set

88 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe x 1!wCoffee x 5 l 2 x 3 Symbolic state set: {(k 2 l 3, x = 4)} EnvOutput: ImpOutput: {scof f ee} l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!wCoffee x > 4 l 3 x 4 x = 4 What if there is a bug? Let s wait back for 2 units

89 A. Rollet - ETR Brest (France) - August /63 Example of test execution Tester k 0 Implementation l 0?wCoffe!coin k 1!req k 2?sCoffe x 1!wCoffee x 5 l 2 x 3 Symbolic state set: {(k 2 l 3, x = 4)} EnvOutput: ImpOutput: {scof f ee} l 0?coin l 1 x 3!sCoffee x 3 l 3 x 7 wcoffee x == 2!wCoffee x 4 l 2 x 2?coin l 1 x == 4!wCoffee x > 4 l 3 x 4 x = 6 output after 0 delay: wcof f ee {scoffee}

90 A. Rollet - ETR Brest (France) - August /63 Properties of test cases Let a S, E, and IUT three weakly input enabled TIOTS, with IUT deterministic. Soundness, Exhaustiveness Soundness : T estgenexe(s, E, IUT, T ) = Fail (IUT rtioco E S) Exhaustiveness : (IUT rtioco E S) Prob(T estgenexe(s, E, IUT, T ) = Fail T 1 If IUT is not deterministic, exhaustiveness is not guaranteed

91 A. Rollet - ETR Brest (France) - August /63 Offline test generation : main ideas Advantages : test cases are easier and faster to execute possibility to guarantee a coverage or a test objective Drawbacks : specification has to be analyzed entirely state explosion only deterministic (and impossible to determinize in general case) Test Generation with Test Purpose Synchronous Product btw Spec. and T.P. need a finite symbolic representation of TA (Region Graph, Zones,...) Test Case Generation with Uppaal Test Case Generation using Observers Still immature...

92 A. Rollet - ETR Brest (France) - August /63 Offline test generation : main ideas Advantages : test cases are easier and faster to execute possibility to guarantee a coverage or a test objective Drawbacks : specification has to be analyzed entirely state explosion only deterministic (and impossible to determinize in general case) Test Generation with Test Purpose Synchronous Product btw Spec. and T.P. need a finite symbolic representation of TA (Region Graph, Zones,...) Test Case Generation with Uppaal Test Case Generation using Observers Still immature...

93 A. Rollet - ETR Brest (France) - August /63 Offline test generation : main ideas Advantages : test cases are easier and faster to execute possibility to guarantee a coverage or a test objective Drawbacks : specification has to be analyzed entirely state explosion only deterministic (and impossible to determinize in general case) Test Generation with Test Purpose Synchronous Product btw Spec. and T.P. need a finite symbolic representation of TA (Region Graph, Zones,...) Test Case Generation with Uppaal Test Case Generation using Observers Still immature...

94 A. Rollet - ETR Brest (France) - August /63 Offline test generation : main ideas Advantages : test cases are easier and faster to execute possibility to guarantee a coverage or a test objective Drawbacks : specification has to be analyzed entirely state explosion only deterministic (and impossible to determinize in general case) Test Generation with Test Purpose Synchronous Product btw Spec. and T.P. need a finite symbolic representation of TA (Region Graph, Zones,...) Test Case Generation with Uppaal Test Case Generation using Observers Still immature...

95 A. Rollet - ETR Brest (France) - August /63 Offline test generation : main ideas Advantages : test cases are easier and faster to execute possibility to guarantee a coverage or a test objective Drawbacks : specification has to be analyzed entirely state explosion only deterministic (and impossible to determinize in general case) Test Generation with Test Purpose Synchronous Product btw Spec. and T.P. need a finite symbolic representation of TA (Region Graph, Zones,...) Test Case Generation with Uppaal Test Case Generation using Observers Still immature...

96 A. Rollet - ETR Brest (France) - August /63 Offline test generation : main ideas Advantages : test cases are easier and faster to execute possibility to guarantee a coverage or a test objective Drawbacks : specification has to be analyzed entirely state explosion only deterministic (and impossible to determinize in general case) Test Generation with Test Purpose Test Case Generation with Uppaal Still immature...

97 Test Case generation with Test Purpose using Uppaal Uppaal Tool : Model checker for temporal properties Symbolic efficient analysis (using DBM) Generates diagnostic traces (shortest or fastest) Assumptions : TIOTS are deterministic, weakly input enabled and output urgent Idea Formulate the problem as safety property (usually solved by a reachability analysis) obtain a trace of the form (s 0, e 0 ) γ 0 (s 1, e 1 )... γ n 1 (s n, e n ) Obtain a test sequence by projecting the trace to the E component (and summing delays) Add Verdicts to the test sequence to obtain a test case Test sequences are guaranteed to be included in the specification A. Rollet - ETR Brest (France) - August /63

98 Test Case generation with Test Purpose using Uppaal Uppaal Tool : Model checker for temporal properties Symbolic efficient analysis (using DBM) Generates diagnostic traces (shortest or fastest) Assumptions : TIOTS are deterministic, weakly input enabled and output urgent Idea Formulate the problem as safety property (usually solved by a reachability analysis) obtain a trace of the form (s 0, e 0 ) γ 0 (s 1, e 1 )... γ n 1 (s n, e n ) Obtain a test sequence by projecting the trace to the E component (and summing delays) Add Verdicts to the test sequence to obtain a test case Test sequences are guaranteed to be included in the specification A. Rollet - ETR Brest (France) - August /63

99 A. Rollet - ETR Brest (France) - August /63 Example of test case Fail z 0 Sequence :!in 0 delay?out 0 Fail z < delay?out 0 z := 0!in 0 z := 0 Fail z delay?out 1...?out n z == delay?out 0 z := 0 Pass

100 A. Rollet - ETR Brest (France) - August /63 Examples of Test Purposes (light controller) TP1 : Check that the light can become bright : Simple reachability property : eventually the system specification can enter location BRIGHT TP2 : Check the light switch off after 3 successive touches reachability property + specific environment :!touch z := 0 z T react!touch z := 0 z T react!touch z := 0?off goal?off?dim?bright?off?dim?bright?dim?bright

101 A. Rollet - ETR Brest (France) - August /63 Examples of Test Purposes (light controller) TP1 : Check that the light can become bright : Simple reachability property : eventually the system specification can enter location BRIGHT TP2 : Check the light switch off after 3 successive touches reachability property + specific environment :!touch z := 0 z T react!touch z := 0 z T react!touch z := 0?off goal?off?dim?bright?off?dim?bright?dim?bright

102 A. Rollet - ETR Brest (France) - August /63 Examples of Test Purposes (light controller) TP1 : Check that the light can become bright : Simple reachability property : eventually the system specification can enter location BRIGHT TP2 : Check the light switch off after 3 successive touches reachability property + specific environment :!touch z := 0 z T react!touch z := 0 z T react!touch z := 0?off goal?off?dim?bright?off?dim?bright?dim?bright

103 A. Rollet - ETR Brest (France) - August /63 Examples of Test Purposes (light controller) TP1 : Check that the light can become bright : Simple reachability property : eventually the system specification can enter location BRIGHT TP2 : Check the light switch off after 3 successive touches reachability property + specific environment :!touch z := 0 z T react!touch z := 0 z T react!touch z := 0?off goal?off?dim?bright?off?dim?bright?dim?bright

104 A. Rollet - ETR Brest (France) - August /63 Examples of coverage criteria Edge Coverage Reachability property : add a boolean variable e i for each edge to be covered, initially false add assignment e i := true for each edge to be covered property to reach : e i == true Location (l i ) Coverage add a boolean variable b i for each node, initially false (except initial) for every edge l g,a,u l i add assignment b i := true property to reach : b i == true Etc... but not always possible

105 A. Rollet - ETR Brest (France) - August /63 Examples of coverage criteria Edge Coverage Reachability property : add a boolean variable e i for each edge to be covered, initially false add assignment e i := true for each edge to be covered property to reach : e i == true Location (l i ) Coverage add a boolean variable b i for each node, initially false (except initial) for every edge l g,a,u l i add assignment b i := true property to reach : b i == true Etc... but not always possible

106 A. Rollet - ETR Brest (France) - August /63 Examples of coverage criteria Edge Coverage Reachability property : add a boolean variable e i for each edge to be covered, initially false add assignment e i := true for each edge to be covered property to reach : e i == true Location (l i ) Coverage add a boolean variable b i for each node, initially false (except initial) for every edge l g,a,u l i add assignment b i := true property to reach : b i == true Etc... but not always possible

107 A. Rollet - ETR Brest (France) - August /63 Examples of coverage criteria Edge Coverage Reachability property : add a boolean variable e i for each edge to be covered, initially false add assignment e i := true for each edge to be covered property to reach : e i == true Location (l i ) Coverage add a boolean variable b i for each node, initially false (except initial) for every edge l g,a,u l i add assignment b i := true property to reach : b i == true Etc... but not always possible

108 A. Rollet - ETR Brest (France) - August /63 Using observers Weakness of this offline approach : time-consuming to find the proper model annotation model-checking tools not adapted for test cases generation : may lead to performance problems Possibility to use a language of observers to describe coverage criteria Adaptation of model-checking algorithms for test generation based on observers

109 A. Rollet - ETR Brest (France) - August /63 Outline 1 Model Based Testing 2 Conformance Testing with IOLTS 3 Testing Timed Systems 4 Conclusion and further work

110 A. Rollet - ETR Brest (France) - August /63 Conclusion Testing theory and generation algorithms for finite iolts Extensions for Timed Automata with Inputs and Outputs Off-line and on-line algorithms Perspectives Mature tools (scaling) Real-time coverage criteria Testing seen as Game theory Add variables with complex assignments Run-time verification / enforcement dans le cadre temporisé

111 A. Rollet - ETR Brest (France) - August /63 Conclusion Testing theory and generation algorithms for finite iolts Extensions for Timed Automata with Inputs and Outputs Off-line and on-line algorithms Perspectives Mature tools (scaling) Real-time coverage criteria Testing seen as Game theory Add variables with complex assignments Run-time verification / enforcement dans le cadre temporisé

112 A. Rollet - ETR Brest (France) - August /63 Thank you for your attention rollet@labri.fr