Model-based testing. NATO Summer School Marktoberdorf, August, Ed Brinksma University of Twente

Transcription

1 Model-based testing NATO Summer School Marktoberdorf, August, 2012 Ed Brinksma University of Twente

2 CONTENTS 1. Introduction control-oriented testing 2. Input-output conformance testing 3. Real-time conformance testing 4. Test coverage measures NATO Summer School, Marktoberdorf August

3 CONTENTS 1. Introduction control-oriented testing 2. Input-output conformance testing 3. Real-time conformance testing 4. Test coverage measures NATO Summer School, Marktoberdorf August

4 PRACTICAL PROBLEMS OF TESTING Testing is: important much practiced 30% - 50% of project effort expensive time critical not constructive (but sadistic?) But also: ad-hoc, manual, error-prone limited theory / research little attention in curricula not cool : if you re a bad programmer you might be a tester Attitude is changing: more awareness more professional NATO Summer School, Marktoberdorf August

5 TYPES OF TESTING system Level integration unit Accessibility robustness performance usability glass box black box Aspect reliability functional behaviour NATO Summer School, Marktoberdorf August

6 TEST AUTOMATION Traditional test automation = tools to execute and manage test cases Why not generate test automatically?! specification test cases TTCN pass implementation under test test tool fail NATO Summer School, Marktoberdorf August 2012

7 VERIFICATION AND TESTING Verification : formal manipulation prove properties performed on model Testing : experimentation show error concrete system formal world concrete world Verification is only as good as the validity of the model on which it is based Testing can only show the presence of errors, not their absence NATO Summer School, Marktoberdorf August

8 TESTING WITH FORMAL METHODS Testing with respect to a formal specification Precise, formal definition of correctness : good and unambiguous basis for testing Formal validation of tests Algorithmic derivation of tests : tools for automatic test generation Allows to define measures expressing coverage and quality of testing NATO Summer School, Marktoberdorf August

9 CHALLENGES OF TESTING THEORY Infinity of testing: too many possible input combinations: infinite breadth too many possible input sequences: infinite depth too many invalid and unexpected inputs Exhaustive testing never possible: when to stop testing? how to invent effective and efficient test cases with high probability of detecting errors? Optimization problem of testing yield vs. effort usually stop when time is over... NATO Summer School, Marktoberdorf August

10 FORMAL TESTING correctness criterion implementation relation imp specification S implementation i test generation test suite T S exhaustive i imp s sound i passes T s test execution pass / fail NATO Summer School, Marktoberdorf August

11 FORMAL TESTING : CONFORMANCE specification S s SPECS IUT specification implementation under test IUT conforms-to s correctness criterion IUT is concrete, physical object Model the physical world implementation IUT But IUT is black box!? Assume that model i IUT exists NATO Summer School, Marktoberdorf August

12 FORMAL TESTING : CONFORMANCE specification S s SPECS Specification i IUT MODS model of IUT formal correctness criterion i IUT imp s Test assumption : each concrete IUT can be modelled by some i IUT MODS implementation i IUT Conformance : i IUT imp s i IUT is not known ; testing to learn about i IUT NATO Summer School, Marktoberdorf August

13 FORMAL TESTING : TEST DERIVATION specification S Test generation : test generation der : SPECS (TESTS) test suite T S Test suite - set of test cases : Test case : T TESTS t TESTS NATO Summer School, Marktoberdorf August

14 FORMAL TESTING : TEST EXECUTION Test execution leads to a set of observations : exec : TESTS IMPS (OBS ) Model of test execution : obs : TESTS MODS (OBS ) implementation IUT test suite T test execution OBS i IUT NATO Summer School, Marktoberdorf August

15 TEST HYPOTHESIS Observational framework : TESTS, OBS, exec, obs Test hypothesis : for all IUT in IMPS. i IUT MODS. t TESTS. exec (t, IUT) = obs (t, i IUT ) IUT test suite T i IUT exec obs test execution OBS NATO Summer School, Marktoberdorf August

16 FORMAL TESTING : VERDICTS Observations are interpreted : t : (OBS) { fail, pass} pass test execution OBS t fail NATO Summer School, Marktoberdorf August

17 TESTING FOR CONFORMANCE IUT passes T s? i conforms-to s IUT passes T s IUT t passes T s. IUT T s passes def t t T s. IUT passes t IUT t passes T s. t ( exec def ( t, IUT t ( exec ) ) ( = t, IUT pass) ) = pass t T s. t ( obs ( t, i IUT ) ) = pass Test hypothesis : iproof IUT imp obligation s : i MODS. t TESTS. exec ( t, IUT ) = obs ( t, i IUT ) IUT Definition conforms-to : i conforms-to s s ( t T s. t ( obs ( t, i ) ) = pass ) i imp s NATO Summer School, Marktoberdorf August

18 TESTING FOR CONFORMANCE Proof obligation : i MODS. ( t T s. t ( obs ( t, i ) ) = pass ) i imp s Proof of completeness on model leads to completeness for tested systems : IUT passes T s exhaustive sound i conforms-to s NATO Summer School, Marktoberdorf August

19 FORMAL TESTING s SPECS imp IUT i IUT IMPS MODS der : SPECS (TESTS) T s TESTS Test hypothesis : IUTIMPS. i IUT MODS. ttests. exec(t,iut) = obs(t,i IUT ) Soundness and exhaustivess: imods. ( tder(s). t (obs(t,i)) = pass ) i imp s pass exec obs : TESTS MODS IMPS (OBS) OBS t : (OBS) {fail,pass} fail NATO Summer School, Marktoberdorf August

20 TESTING PREORDERS implementation i specification s environment e environment e For all environments e all observations of an implementation i in e should be explained by observations of the specification s in e. i s e Env. obs ( e, i ) obs (e, s )??? NATO Summer School, Marktoberdorf August

21 LABELLED TRANSITION SYSTEMS An LTS is a tuple A S a set of states S 0 L a set of labels; L S L We write: s s s a S s' for s' for a s' for a CTraces { a... a 0 S, S 0 0 a nonempty set S s,a,s' and s and s, L, the transitionrelation; s n n S. s of initial states 0 s s 0 0 with L { } with the invisible action a 1 k1 s s s 1 a 1 a a 2... m1 s a 1 n s k2 n a m2 for any a L } 2... NATO Summer School, Marktoberdorf August kn a n mn s n

22 PARALLEL COMPOSITION August 2012 NATO Summer School, Marktoberdorf 22 }} { \,,,,,, { }} { \,,,,,, { },,,,,,,,,,,,,, L L a S s t,a,t t s a t s L L a S t s,a,s t s a t s a t a t s,a,s t s a s,t L L S S S S A B L S S B A { with Then be two LTSs. and Let

23 CLASSICAL TESTING PREORDER implementation i te specification s environment e Philosophical question: can we observe deadlocks? i te s e E. obs ( e, i ) obs (e, s ) LTS(L) CTraces(e s) environment In testing one may e ignore the difference between livelock and deadlock NATO Summer School, Marktoberdorf August

24 TESTING PREORDER implementation i te specification s environment p after refuses ea iff p p and a p for all a A environment e i te s e LTS(L). L*. e i { deadlocks Ctraces(e i)} after { e s deadlocks Ctraces(e s)} after FP(i) FP(s) FP(p) = {, A p after refuses A } NATO Summer School, Marktoberdorf August

25 Can we distinguish between these machines? QUIRKY COFFEE MACHINE [Langerak] coin coin coin coin tea bang bang coffee te tea bang bang coffee coffee tea tea coffee They are testing equivalent! NATO Summer School, Marktoberdorf August

26 REFUSAL PREORDER implementation i rf CTraces δ (e i) = specification s {(L{δ})* e i after refuses L} environment e environment e i rf s e E. obs ( e, i ) obs (e, s ) e observes with δ deadlock on all alternative actions LTS(L{δ}) CTraces δ (e i) NATO Summer School, Marktoberdorf August

27 QUIRKY COFFEE MACHINE REVISITED coin coin coin coin tea bang bang coffee te tea bang bang coffee coffee tea rf tea coffee coffee coffee coin bang tester δ only enabled if coffee is not NATO Summer School, Marktoberdorf August

28 CONTENTS 1. Introduction control-oriented testing 2. Input-output conformance testing 3. Real-time conformance testing 4. Test coverage measures NATO Summer School, Marktoberdorf August

29 CONTENTS 1. Introduction control-oriented testing 2. Input-output conformance testing 3. Real-time conformance testing 4. Test coverage measures NATO Summer School, Marktoberdorf August

30 I/O TRANSITION SYSTEMS testing actions are usually directed, i.e. there are inputs and outputs L=L in L out with L in L out = systems can always accept all inputs: input enabledness for all states s, for all al in s testers are I/O systems output (stimulus) is input for the SUT input (response) is output of the SUT a NATO Summer School, Marktoberdorf August

31 QUIESCENCE Because of input enabledness S T deadlocks iff T produces no stimuli and S no responses. This is known as quiescence Observing quiescence leads to two implementation relations for I/O systems I and S : 1. I iote S iff for all I/O testers T : CTraces(I T) CTraces(S T) (quiescence) 2. I iorf S iff for all I/O testers T : CTraces δ (I T) Ctraces δ (S T) (repetitive quiescence) NATO Summer School, Marktoberdorf August

32 INPUT-OUTPUT QCM coin! states must be saturated with input loops for input enabledness coffee? tea? coin? bang? coin? bang? tea? coffee? quiescent states coffee? coffee! tea! coffee! bang! coffee? coffee! iote iorf tea? tea! coffee! coffee? NATO Summer School, Marktoberdorf August

33 QUIESCENT LABELLED TRANSITION SYSTEMS A QLTS is an LTS A S, S with special (output)label such that if s s then s 0, L 1 L O, s and s is quiescent. This definition is closed under determinisation. Let A then its underlyingqlts ( A) is the QLTS S, S 0 S, S 0, L,, L { }, bean LTS with L with {( s,, s) s S, s is quiescent} L I L O and L, Moreover, traces ( A) traces A \{ } NATO Summer School, Marktoberdorf August

34 IMPLEMENTATION RELATION IOCO Let i and s bebeqtlss (possibly after over L L I L applying (.)) O, then we define i iorf s iff L * out i ( ) out s ( ) For implementations we will require input-enabledness, But not for specifications. In this setting it makes sense to restrict testing to the traces of the implementation: i ioco s iff traces s out ( ) out ( ) i s NATO Summer School, Marktoberdorf August

35 INTUITION BEHIND IOCO i ioco s iff traces s out i ( ) out ( ) s Intuition: i ioco-conforms to s, iff 1. if i produces output x after a specified trace, then s can produce x after 2. if i cannot produce any output after a specified trace, then s cannot produce any output after (quiescence ) NATO Summer School, Marktoberdorf August

36 ADDING QUIESCENCE i?quart δ(i)?quart?dime?dime?dime?quart!coffee?dime?quart!coffee?dime?quart?dime?quart NATO Summer School, Marktoberdorf August

37 CALCULATING OUT i?quart out i (ε) = out i (?dime) = { δ } {!coffee }?dime out i (?dime.?dime ) = {!coffee }?dime?quart?dime?quart!coffee out i (?dime.!coffee) = out i (?quart ) = out i (!coffee ) = out i (?dime.!tea ) = out i (δ) = { δ } { δ } {δ } NATO Summer School, Marktoberdorf August

38 IMPLEMENTATION RELATION IOCO i ioco s iff traces s out i ( ) out ( ) s i s?dime?dime?dime!coffee ioco!tea!coffee?dime out i (?dime) = {!coffee } out s (?dime) = {!coffee,!tea } NATO Summer School, Marktoberdorf August

39 IMPLEMENTATION RELATION IOCO i ioco s iff traces s out i ( ) out ( ) s i s?dime!tea?dime!coffee ioco?dime!coffee?dime?dime out i (?dime) = {!coffee,!tea } out s (?dime) = {!coffee} NATO Summer School, Marktoberdorf August

40 IMPLEMENTATION RELATION IOCO i ioco s iff traces s out i ( ) out ( ) s i?dime?quart s?dime?dime?quart!coffee!tea ioco!coffee out i (?dime) = {!coffee } out i (?quart) = {!tea } out s (?dime) = {!coffee } out s (?quart) = But?quart Traces δ ( s ) NATO Summer School, Marktoberdorf August

41 IMPLEMENTATION RELATION IOCO i i ioco s iff traces s out i ( ) outs( ) s?dime?dime?dime!tea?dime δ?dime?dime!coffee i s ioco ioco s i?dime?dime?dime!tea?dime δ?dime?dime!tea!coffee?dime out i (?dime.? dime) = out s (? dime.? dime) = {!tea,!coffee }?dime?dime out i (? dime.δ.? dime) = {!coffee } out s (? dime.δ.? dime) = {!tea,!coffee } NATO Summer School, Marktoberdorf August

42 FORMAL TESTING specification S correctness criterion implementation relation ioco test generation i ioco s? i passes T s implementation i test suite T S test execution pass / fail NATO Summer School, Marktoberdorf August

43 TEST CASES A test (case) t over L=L I L δ O is an LTS with t is deterministic t does not contain an infinite path t is acyclic and connected for all states s of t we have either Alternatively, a test case can be characterised by the prefix-closed set of its traces after(s)=, or (termination) after(s)= L δ O, or (response observation) after(s)= {a?} L O (stimulus) NATO Summer School, Marktoberdorf August

44 INPUT-OUTPUT QCM AGAIN coin! coffee? tea? coin? bang? coin? bang? tea? coffee? coffee? coffee! tea! coffee! bang! coffee? coffee! iote iorf tea? tea! coffee! coffee? NATO Summer School, Marktoberdorf August

45 TEST ANNOTATIONS Let t be a test case: an annotation of t is a function a: Ctraces t { pass, fail } the pair ṫ = (t,a) is an annotated test case When a is clear from the context, or irrelevant, we use t for both test case and its annotation. NATO Summer School, Marktoberdorf August

46 ANNOTATED TEST CASES VS TTCN coin? test case t!coin coin?!coin ; Start timer1?tea?timer1 fail fail coffee! tea!?coffee!coin ; Start timer2 coin? fail fail?tea?timer2?coffee pass pass fail tea! pass coffee! pass fail NATO Summer School, Marktoberdorf August

47 EXECUTION AND EVALUATION Let A be a QLTS over L and t a test over L. The executions of t with A are defined as exec t (A) = Ctraces(t A) in fact, exec t (A) = Ctraces(t) traces(a) Let ṫ = (t,a) be an annotated test case. The verdict of ṫ is the function v ṫ :QLTS(L) {pass,fail} with This can be lifted in the obvious way to sets of tests, i.e. test suites. v ṫ (A) = pass if for all exec t (A) a() = pass fail otherwise NATO Summer School, Marktoberdorf August

48 A SOUND AND COMPLETE TEST SUITE Given a specification s we define the annotation a ioco s () = fail if 1 traces s, a!l δ O 1 a! and 1 atraces s pass otherwise Tests(s) contains all tests over the same label set L as s Given s and any ttests(s), its annotated version (t, a s ioco ) is sound w.r.t. s under ioco. The test suite T={(t, a s ioco ) ttests(s)} is sound and complete w.r.t. s under ioco. NATO Summer School, Marktoberdorf August

49 DESIRABLE TEST CASE PROPERTIES Let s be specification over a label set L, then a test t is fail-fast w.r.t. s if traces s implies that al at a test t is input-minimal w.r.t. s if for all a?t with a?l I it holds that traces s implies a?traces s NATO Summer School, Marktoberdorf August

50 TEST GENERATION EXAMPLE Equation solver for y 2 =x specification test? x (x < 0)!9! x? x (x >= 0) otherwise?-3?3 FAIL! -x PASS!4 otherwise FAIL?-2?2 PASS PASS NATO Summer School, Marktoberdorf August

51 TEST EXECUTION EXAMPLE implementation test! n? n (n < 0)?n (n >= 0)! -n? k otherwise FAIL PASS otherwise!9?-3?3!4?-2?2 FAIL PASS PASS NATO Summer School, Marktoberdorf August

52 i? a specified input for some ts ANNOTATED TEST GENERATION ALGORITHM To generate a test case t(s) from a transition system specification with S set of states ( initially S = S 0 ) Apply the following steps recursively, non-deterministically 1 end test case PASS (in the limit) ioco-complete o! 3 supply input supply i? fail-fast, input-minimal, ioco-sound & observe output (no δ) t(s after i?) FAIL FAIL forbidden outputs 2 observe output allowed outputs o! t(s after o!) Allowed outputs: out s (S) = ts out s (t) NATO Summer School, Marktoberdorf August

53 FORMAL TESTING WITH TRANSITION SYSTEMS ioco s LTS der : LTS (TTS) i IUT IOTS T s TTS pass obs : TTS IOTS (traces) traces t : (traces) {fail,pass} fail NATO Summer School, Marktoberdorf August

54 SOME TEST GENERATION TOOLS FOR IOCO TVEDA (CNET - France Telecom) derives TTCN tests from single process SDL specification developed from practical experiences implementation relation R1 ioco TGV (IRISA - Rennes) derives tests in TTCN from LOTOS or SDL uses test purposes to guide test derivation implementation relation: unfair extension of ioco TestComposer (Verilog) Combination of TVEDA and TGV in ObjectGeode TestGen (Stirling) Test generation for hardware validation TorX (University of Twente, ESI) NATO Summer School, Marktoberdorf August

55 A TEST TOOL : TORX On-the-fly test generation and test execution Implementation relation: ioco Specification languages: LOTOS, Promela, FSP, Automata user: manual automatic specification next input check output TorX pass fail inconclusive offer input observe output IUT NATO Summer School, Marktoberdorf August

56 TORX TOOL ARCHITECTURE specification spec. TorX IUT explorer primer driver adapter IUT specification text states transitions abstract actions abstract actions concrete actions 56 NATO Aug Summer ust

57 ON-THE-FLY BATCH TESTING on the fly spec. explorer batch test generation primer driver adapter TTCN TTCN test rep. NATO Summer School, Marktoberdorf IUT batch test execution August

58 ON-THE-FLY TESTING New menu!! x! Check x (x (x < < 0) 0)!! x!? x (x (quiescence)?(x >= 3 >= 0) 0) Action Choice??(quiescence)! Abstract action action?! -1?! (quiescence) 93 Concrete action?!! (timeout) spec states abstract bits explorer primer driver adapter IUT transitions transition actions bytes specification implementation? n (n < 0)? n (n < 0)! n? n (n >= 0)! -n? n (n >= 0)! n? n NATO Summer School, Marktoberdorf August

59 TORX NATO Summer School, Marktoberdorf August

60 SOME TORX CASE STUDIES Conference Protocol EasyLink TV-VCR protocol Cell Broadcast Centre component Road Toll Payment Box protocol V5.1 Access Network protocol Easy Mail Notification FTP Client Oosterschelde storm surge barrier-control TANGRAM: testing VLSI lithography machine academic Philips CMG Interpay Lucent CMG academic CMG ASML NATO Summer School, Marktoberdorf August

61 THE CONFERENCE PROTOCOL EXPERIMENT Academic benchmarking experiment, initiated for test tool evaluation and comparison Based on really testing different implementations Simple, yet realistic protocol (chatbox service) Specifications in LOTOS, Promela, SDL, EFSM 28 different implementations in C one of them (assumed-to-be) correct others manually derived mutants NATO Summer School, Marktoberdorf August

62 THE CONFERENCE PROTOCOL join leave send receive CPE CPE CPE Conference Service UDP Layer NATO Summer School, Marktoberdorf August

63 CONFERENCE PROTOCOL TEST ARCHITECTURE A Tester TorX UT-PCO = C-SAP CPE = IUT B C U-SAP LT-PCO LT-PCO UDP Layer NATO Summer School, Marktoberdorf August

64 THE CONFERENCE PROTOCOL EXPERIMENTS TorX - LOTOS, Promela : on-the-fly ioco testing Axel Belinfante et al., Formal Test Automation: A Simple Experiment IWTCS 12, Budapest, Tau Autolink - SDL : semi-automatic batch testing TGV - LOTOS : automatic batch testing with test purposes Lydie Du Bousquet et al., Formal Test Automation: The Conference Protocol with TGV/TorX TestCom 2000, Ottawa. PHACT/Conformance KIT - EFSM : automatic batch testing Lex Heerink et al., Formal Test Automation: The Conference Protocol with PHACT TestCom 2000, Ottawa. NATO Summer School, Marktoberdorf August

65 CONFERENCE PROTOCOL RESULTS Results: fail pass core dump TorX LOTOS TorX Promela PHACT EFSM TGV LOTOS random TGV LOTOS purposes pass NATO Summer School, Marktoberdorf August

66 CONFERENCE PROTOCOL ANALYSIS Mutants 444 and 666 react to PDU s from non-existent partners: no explicit reaction is specified for such PDU s, so ioco-correct, and TorX does not test such behaviour So, for LOTOS/Promela with TGV/TorX: All ioco-erroneous implementations detected EFSM: two additional-state errors not detected one implicit-transition error not detected NATO Summer School, Marktoberdorf August

67 CONFERENCE PROTOCOL ANALYSIS TorX statistics all errors found after test events maximum length of tests : > 500,000 test events EFSM statistics 82 test cases with partitioned tour method ( = UIO ) length per test case : < 16 test events TGV with manual test purposes ~ 20 test cases of various length TGV with random test purposes ~ 200 test cases of 200 test events NATO Summer School, Marktoberdorf August

68 INTERPAY HIGHWAY TOLLING SYSTEM NATO Summer School, Marktoberdorf August

69 HIGHWAY TOLLING PROTOCOL Characteristics : simple protocol parallelism: many cars at the same time encryption system passed traditional testing phase NATO Summer School, Marktoberdorf August

70 HIGHWAY TOLLING SYSTEM Onboard Unit Road Side Equipment Payment Box (PB) Wireless UDP/IP NATO Summer School, Marktoberdorf August

71 HIGHWAY TOLLING: TEST ARCHITECTURE spec PB + ObuSim + TCP/IP + UDP/IP TorX PCO TCP/IP ObuSim Test Context UDP/IP IAP Payment Box SUT NATO Summer School, Marktoberdorf August

72 HIGHWAY TOLLING: RESULTS Test results : 1 serious error during validation (design error) 1 serious error during testing (coding error) Automated testing : beneficial: high volume and reliability many and long tests executed ( > 50,000 test events ) very flexible: adaptation and many configurations Real-time : interference computation time on-the-fly testing interference quiescence and time-outs NATO Summer School, Marktoberdorf August

73 STORM SURGE BARRIER CONTROL Oosterschelde Stormvloedkering (OSVK) NATO Summer School, Marktoberdorf August

74 SVKO EMERGENCY CLOSING SYSTEM Collect water level sensor readings (12x, 10Hz) Calculate mean outer-water level and mean innerwater level Determine closing conditions if (closing_condition) {notify officials start diesel engines block manual control control local computers} Failure rate: 10-4 /closing event NATO Summer School, Marktoberdorf August

75 TESTING SVKO water level sensor 12x collector controller diesel generator power control water level sensor user control barrier control test controller (Unix port) many timed observations shortest timed delay: 2 seconds longest timed delay: 85 minutes signal wire communication NATO Summer School, Marktoberdorf August

76 RESULTS real-time control systems can be tested with TorXtechnology addition of discrete real time time stamped actions quiescence action is not used time spectrum of 3 orders of magnitude deterministic system adhoc implementation relation NATO Summer School, Marktoberdorf August

77 RT TORX HACKS: APPROACH 1 Ignore RT functionality: 1. test pure functional behaviour 2. analyse timing requirements using TorX log files & assumed frequency of wire polling actions NATO Summer School, Marktoberdorf August

78 RT TORX HACKS: APPROACH 2 Add timestamps to observations 1. adapter adds timestamps to observations when they are made and passed on to the driver 2. timestamps are used to analyse TorX log files NATO Summer School, Marktoberdorf August

79 TIMING ERROR LOGGING int time,newtime;... input?value,time // input: stimulus... // variable time is set here output!param,&newtime // output: observation // & is extension to promela: if // variable newtime is set here :: (newtime==time+60) // expected delay of 60 is print( OK,...) // checked & logged :: (newtime!=time+60) print( NOK,...) fi; NATO Summer School, Marktoberdorf August

80 RT TORX HACKS: APPROACH 3 Add timestamps to stimuli & observations 1. adapter add timestamps to observations when they are made and passed on to the driver 2. adapter adds timestamps to stimuli when they are applied and returned to the driver 3. analysis: a. timing error logging: observed errors are written to TorX log file b. timing error failure: observed errors cause fail verdict of test case NATO Summer School, Marktoberdorf August

81 TIMING ERROR FAILURE int time;... // input: stimulus input?value,time // variable time is set here... if // output: observation :: output!param,(time+59) // wait 60 (-1) :: output!param,(time+60) // wait 60 :: output!param,(time+61) // wait 60 (+1) fi // if observation is not made // after approx. 60 units, // quiescence will be // observed NATO Summer School, Marktoberdorf August

82 CONTENTS 1. Introduction control-oriented testing 2. Input-output conformance testing 3. Real-time conformance testing 4. Test coverage measures NATO Summer School, Marktoberdorf August

83 CONTENTS 1. Introduction control-oriented testing 2. Input-output conformance testing 3. Real-time conformance testing 4. Test coverage measures NATO Summer School, Marktoberdorf August

84 REAL-TIME TESTING AND I/O SYSTEMS can the notion of repetitive quiescence be combined with real-time testing? is there a well-defined and useful conformance relation that allows sound and (relative) complete test derivation? can the TorX test tool be adapted to support Realtimed conformance testing? NATO Summer School, Marktoberdorf August

85 WITH REAL-TIME DO WE STILL NEED QUIESCENCE? Yes! coffee? coin? coin? tea? the example processes should also be distinct in a real-time context tea? bang? tea! coffee? coffee! coffee? bang? coffee! tea? tea! NATO Summer School, Marktoberdorf August

86 REAL-TIME AND QUIESCENCE s is quiescent iff: for no output action a and delay d: s special transitions: δ s s for every quiescent system state s testers observing quiescence take time: Test M : set of test processes having only δ(m)-actions to observe quiescence assume that implementations are M-quiescent: for all reachable states s and s : (M) if s s then s is quiescent a(d) NATO Summer School, Marktoberdorf August

87 REAL-TIME AND QUIESCENCE M i tiorf s T Test M : CTraces δ (i T) CTraces δ (s T) ( L { (M) } )*: out ì,m () out s,m () i tioco M s Traces δ(m) ( s ) : out ì,m () out s,m () NATO Summer School, Marktoberdorf August

88 PROPERTIES 1. for all M 1 M 2 : M 1 M 2 i tiorf s implies i tiorf s 2. for all time-independent i,s and M 1,M 2 >0 M 1 M 2 i tiorf s iff i tiorf s iff i iorf s NATO Summer School, Marktoberdorf August

89 A LIMITATION coin? coin? The first extention cannot be distinguished from the second x=m tea! tea? coffee? xm x<m bang? x xm bang? tea? x xm bang? coffee? xm x<m bang? x=m coffee! xm x=m coffee! tea! xm x=m coffee! tea! NATO Summer School, Marktoberdorf August

90 ANNOTATED REAL-TIME TEST CASES Test case t TTA TTA Timed Test Automata : tree-structured finite, deterministic final states pass and fail from each state pass, fail choose an input i? and a time k and wait for the time k accepting all outputs o! and after k time units provide input i? or wait for time M accepting all outputs o! and x=m pass off! fail off! x=5 x:=0 xm fail off! x:= 0 xk xm on? x:=0 off! x<5 fail NATO Summer School, Marktoberdorf August x=m fail

91 TIMED TEST GENERATION PROTO-ALGORITHM To generate a test case t(s) from a timed transition system specification with S set of states ( initially S = {s 0 } ) Apply the following steps recursively, non-deterministically 1. end test case PASS 2. choose k (0, M) and input μ x:=0 3. wait for observing possible output x:=0 forbidden outputs o i! after d time-units x k allowed outputs o j! after d time-units forbidden outputs o i! after d time-units x M allowed outputs o j! after d time-units o 1! o n! x=d 1 x=d n μ? x=k o 1! o n! x=d 1 x=d n x=d 1 o 1! x=d n o n! x=m o n! o 1! x=d 1 x=d n FAIL FAIL t μ t 1 t n FAIL FAIL t δ t 1 t n NATO Summer School, Marktoberdorf August

92 Example spec: δ impl: M=k t! c? m? m? t? c? b? b? c? c! t! t? t? c! m? c? m? t? t! c! t? c? b? b? x<k x<k t? x<k t! c! c? x<k test: c! fail c! fail c! pass c! fail c! fail c! pass x1 x1 xm m? x=1 x:=0 c? x=1 x:=0 δ x=m x:=0 x1 x1 b? x=1 x:=0 xm c? x=1 x:=0 fail δ x=m fail fail fail fail fail fail NATO Summer School, Marktoberdorf August t! t! t! t! t! t!

93 SOUNDNESS & COMPLETENESS the non-timed generation algorithm can be shown to generate only sound real-time test cases test generation is complete for every erroneous trace it can generate a test that exposes it test generation is not limit complete because of continuous time there are uncountably many timed error traces and only countably many test are generated by repeated runs test generation is almost limit complete non-spurious errors = errors with a positive probability of occurring repeated test geration runs will eventually generate a test case that will expose one of the non-spurious errors of a non-conforming implementation NATO Summer School, Marktoberdorf August

94 CONTENTS 1. Introduction control-oriented testing 2. Input-output conformance testing 3. Real-time conformance testing 4. Test coverage measures NATO Summer School, Marktoberdorf August

95 CONTENTS 1. Introduction control-oriented testing 2. Input-output conformance testing 3. Real-time conformance testing 4. Test coverage measures NATO Summer School, Marktoberdorf August

96 COVERAGE: MOTIVATION Testing is inherently incomplete Test selection is crucial Coverage metrics Quantitative evaluation of test suite Count how much of specification/implementation has been examined Examples: Transparent box (implementation coverage): Statement, path, condition coverage Black box (specification coverage) State, transition coverage NATO Summer School, Marktoberdorf August

97 TRADITIONAL COVERAGE MEASURES Traditional measures are: based on syntactic model features states, transitions, statements, tests uniform all system parts treated as bequally important Disadvantages: replacing the spec by an equivalent one yields different coverage needs a semantic approach some bugs are more important than others; test crucial behaviour first and better NATO Summer School, Marktoberdorf August

98 SEMANTIC APPROACH Considers black box coverage similar ideas could apply to white box coverage Semantically equivalent specs yield same coverage Risk-based more important bugs/system parts higher contribution to coverage Allows for optimization cheapest test suite with 90% coverage; or maximal coverage within cost budget NATO Summer School, Marktoberdorf August

99 FAULT MODELS f: Observation R 0 f(σ) = 0 : correct behaviour f(σ) > 0 : incorrect behaviour : f(σ) severity 0 < Σ σ f(σ) < Observations are traces Observations = L * L = (L I, L U ) How to obtain f? E.g. via fault automaton f: L* R 0 f(play? song!) = 0 correct f(play? silence!) = 10 incorrect f(song!) = 3 incorrect NATO Summer School, Marktoberdorf August

100 EXAMPLE TEST CASE Test t play? δ fail 10 fail δ song! play? song! 10 0 pass f: L* R f(play? song!) = 0 f(play? δ) = 10 f(play? song! play? δ ) = 10 f(song!) = 3 σ f(σ) = 100 (assumption) Absolute Coverage abscov(f,t) sum the error weights = 20 Relative Coverage abscov(f,t) totcov(f) should be 0, NATO Summer School, Marktoberdorf August =

101 EXAMPLE TEST SUITE Test t play? Test t play? δ song! δ song! 10 play? 10 δ song! δ song! Absolute Coverage count each trace once! = 22 Relative Coverage abscov(f,t) 22 = totcov(f) 100 = 22% NATO Summer School, Marktoberdorf August

102 FAULT SPECIFICATIONS fault model f(σ) = 0 if σ trace of automaton f(σ) = 3 α σ -1 if σ ends in 3-state f(σ) = 10 if σ ends in 10-state α σ -1 infinite total coverage!! σ f(σ) = = song! 3 play? song! δ 10 play? Solution 1: restrict to traces of lenght k Omit here, works as solution 2, less efficient, more boring Solution 2: discounting errors in short traces are worse Lower the weight proportional to length Use your favorite Formalism, e.g. UML state charts, LOTOS, etc NATO Summer School, Marktoberdorf August

103 FAULT SPECIFICATIONS fault model f(σ) = 0 if σ automaton trace f(σ) = 3 α σ -1 if σ end in 3-state f(σ) = 10 α σ -1 if σ ends in 10-state Example f(play?) = 0 f(play? δ ) = 10 α f(play? song! song! ) = 3 α 2... song! 3 play? song! 10 α < 1/out(spec) = 1/2 α can vary per transition tune α δ play? NATO Summer School, Marktoberdorf August

104 FAULT SPECIFICATIONS Total coverage becomes fcomputable: tc(s) = 3 + α tc(t) tc(t) = 10 + α tc(t) + α tc (s) tc(x) = wgt(x) + α y: succ(x) tc(y) Solve linear equations tc = wgt (I - α A) -1 with A adjacency matrix song! 3 s tc(s) = play? song! t α Relative Coverage abscov(f,t) totcov(f) δ 1 α - α 2 play? NATO Summer School, Marktoberdorf August

105 TEST SUITE COVERAGE play? play? δ song! δ song! 10 α 10 δ play? song! 10 α 10 δ 0 song! 3 3 α 2 10 α Relative test suite coverage Absolute test suite coverage count each trace once! merge test cases first abscov(f,t) = 10α+ 3α2 +10α 3 totcov(f) 10 +7α 1 α - α 2 NATO Summer School, Marktoberdorf August

106 OPTIMIZATION song! Find best test case of lenght n 3 play? song! v 1 (s) = 3 v 1 (t) = 10 v k+1 (s) = max(3, α v k (t)) v k+1 (t) = max(10 + α v k (s), α v k (t)) Complexity: O(n #transitions in spec) s t 10 δ play? More optimizations: Test suite of k tests & lenght n; Best test case in budget; Add costs... NATO Summer School, Marktoberdorf August

107 PROPERTIES Framework for black box coverage robustness small changes in weight yield small changes in coverage relcov(s) continuous tunable (calibration) change α : get as much total coverage as desired NATO Summer School, Marktoberdorf August

108 CALIBRATION play? song! 3 s song! t 10 δ play? α small tune α present is important, future unimportant few small test cases with high (>0.999) coverage make tests with length >k important, i.e make cov(t k, f) as small as desired. α(s) = 1/n(s) - lim 0 cov(t k,f ) = 0 n(s) =outinf(s) for all k NATO Summer School, Marktoberdorf August

109 CONCLUSIONS NATO Summer School, Marktoberdorf August

110 CONCLUSIONS model-based testing offers theory and tools for (real-time) conformance testing, in particular: test generation, execution & evaluation coverage analysis ioco-theory, TorX and related tools have been evaluated against many industrial cases on-the-fly application very productive good coverage with random test execution current theory is mostly control-oriented OK for classical embedded applications Is being extended to cope with data-intensive systems NATO Summer School, Marktoberdorf August

111 A major problem getting good models THE BIG PICTURE REVISITED specification S An effective theory of test selection and coverage implementation still needs workrelation test generation A fairly robust understanding of conformance relations and versions of the generic test generation algorithm Flexible Instrumentation for complex implementation systems; abstract test i action refinement test suite T S test execution pass / fail NATO Summer School, Marktoberdorf August

112 CURRENT DEVELOPMENTS Methods for model inference Incremental model learning Process mining Test-based modelling Stochastic approaches Importance of stochastic features Approach to coverage metrics Source of (highly) abstract models NATO Summer School, Marktoberdorf August

113 CURRENT DEVELOPMENTS Rigorous integration of control and data-oriented testing Symbolic approaches Deeper integration of methods for test data selection Leaving transparent data transfer paradigm (ubiquitous/wireless/smart dust networking) Engineering tool kit for instrumentation Mapping abstract action level to machine (inter)action level Use of compilation techniques Use of solutions for reconfigurable HW/SW platforms NATO Summer School, Marktoberdorf August

114 RESOURCES NATO Summer School, Marktoberdorf August