Software Specification 2IX20

Transcription

1 Software Specification 2IX20 Julien Schmaltz (with slides jointly with J. Tretmans, TNO&RUN) Lecture 11: Introduction to Model-Based Testing

2 Context & Motivation

3 Testing Testing: checking or measuring some quality characteristics of an executing object by performing experiments in a controlled way w.r.t. a specification tester specification IUT

4 Formal Methods F Use mathematics to model relevant parts of software F Precise, formal semantics: no room for ambiguity or misinterpretation F Allow formal validation and reasoning about systems F Amenable to tools: automation F Examples: Z Temporal Logic First order logic SDL (Specification and Description Language) LOTOS Promela Labelled Transition Systems Finite state machine Process algebra...

5 Software Testing Testing is: F important F much practiced F 30% - 50% of project effort F expensive F time critical F not constructive (but sadistic?) Improvements possible with formal methods!? But also: F ad-hoc, manual, error-prone F hardly theory / research F no attention in curricula F not cool : if you re a bad programmer you might be a tester Attitude is changing: F more awareness F more professional

6 Types of Testing Level of detail system integration module unit portability maintainability white box black box Accessibility efficiency usability reliability functionality Characteristics

7 informal ideas A Model-Based Development Process informal world validation specification formalizable world of models formal verification design code model- based testing real world realization

8 Verification and Testing Verification : F formal manipulation F prove properties F performed on model Testing : F experimentation F show error F concrete system formal world concrete world Verification is only as good as the validity of the model on which it is based Testing can only show the presence of errors, not their absence

9 Model-Based Testing FModel based testing has potential to combine practice theory - testing - formal methods FModel Based Testing : testing with respect to a (formal) model / specification state model, pre/post, CSP, Promela, UML, Spec#,.... promises better, faster, cheaper testing: algorithmic generation of tests and test oracles : tools formal and unambiguous basis for testing measuring the completeness of tests

10 Model Based Testing with Formal Methods Claims : F Formal specifications are more precise, more complete, unambiguous F Formal models can be formally verified F Formal models allow automatic generation of tests

11 Towards Model-Based Testing: Trends FIncrease in complexity, and quest for higher quality software FMore abstraction less detail model based development; OMG s UML, MDA FChecking quality Software bugs / errors cost US economy yearly: $ (~ 50 billion) ( $ 22 billion could be eliminated practice: testing - ad hoc, too late, expensive, lot of time research: formal verification - proofs, model checking,....

12 Towards Model-Based Testing: Trends Increase testing effort grows exponentially with system size testing cannot keep pace with development of complexity and size of systems x : [0..9] 10 ways that it can go wrong 10 combinations of inputs to check x : [0..9] y : [0..9] x : [0..9] y : [0..9] z : [0..9] 100 ways that it can go wrong 100 combinations of inputs to check 1000 ways that it can go wrong 1000 combinations of inputs to check

13 Testing & Formal Methods A Perfect Couple? Formal methods: F proving properties F research F sound theories F clean Testing : F trial & error F practice F heuristics F dirty hands Testing is not necessary after formal verification Testing can only detect the presence of errors, not their absence Formal methods are toys for boys Formal methods have extreme potential - but not for my project

14 Testing with Formal Methods FTesting with respect to a formal specification FPrecise, formal definition of correctness : good and unambiguous basis for testing FFormal validation of tests FAlgorithmic derivation of tests : tools for automatic test generation FAllows to define measures expressing coverage and quality of testing

15 Testing & Formal Methods Claims : FCombining the mathematically clean world of formal methods and the dirty hands world of testing FTesting and formal methods are both necessary in software development FFormal methods improve the testing process FFormal testing stimulates the use of formal methods

16 Automated Model-Based Testing IUT confto model exhaustive sound IUT passes tests TTCN test generation cases tool test tool model IUT IUT confto model pass fail

17 Approaches to Model-Based Testing Several modelling paradigms: F Finite State Machine F Pre/post-conditions Labelled Transition Systems F Labelled Transition Systems F Programs as Functions F Abstract Data Type testing

18 Model Based Testing with Transition Systems IUT confto i ioco model s exhaustive sound IUT i passes gen(s) tests pass gen test : LTS generation (TTS) tool test test execution t tool i tool s model LTS IUT i ioco confto s model IUT i IOTS pass fail

19 Formal Testing correctness criterion implementation relation specification S implementation i test generation test suite T S test execution i conforms-to s i passes T s pass / fail

20 Model-Based Testing for LTS Involves: model specification model confto = test generation implementation IUT + models of IUTs correctness test cases tests test generation IUT test execution test execution test result analysis pass / fail

21 Formal Testing with Transition Systems s LTS ioco iiut IMPS IUT IOTS = gen : LTS (TTS) T s TTS Test hypothesis : IUT IMP. i IUT IOTS. t TTS. IUT passes t i IUT passes t Proof soundness and exhaustiveness: i IOTS. ( t gen(s). i passes t ) i ioco s passes exec : : IOTS TESTS TTS {pass,fail} IMPS (OBS) pass / fail

22 LTS with inputs and outputs

23 Models: Labelled Transition Systems Labelled Transition System S, L, T, s 0 states actions initial state s 0 S transitions T S (L {τ}) S?coin!alarm?button?button

24 Labelled Transition Systems a a b a b a b a b a b a a a τ a b a b b a b c c

25 Labelled Transition Systems s dub S0 kwart S0 dub S1 transition S1 S3 coffee S2 τ S4 S0 S0 dub coffee kwart tea S3 transition composition executable sequence tea S5 S0 kwart soup non-executable sequence L = { dub,kwart, coffee,tea,soup} LTS(L) all transition systems over L

26 Labelled Transition Systems s Sequences of observable actions: dub S0 dub traces (s) = { σ L* s σ } S1 S2 traces(s) = { ε, dub, dub coffee, dub tea } S3 coffee tea S4 Reachable states: s after σ = { s s σ s } s after dub = { S1, S2 } s after dub tea = { S4 }

27 Input-Output Transition Systems dub, kwart coffee, tea S1? dub S0? kwart S2 from user to machine initiative with user machine cannot refuse from machine to user initiative with machine user cannot refuse! coffee! tea input output S3 S4 L I L U L I = {,?kwart } L U = {,!tea } L I L U = L I L U = L

28 Input-Output Transition Systems?kwart?kwart!tea?kwart Input-Output Transition Systems IOTS (L I,,L U ) LTS (L I L U ) IOTS is LTS with Input-Output?kwart?kwart and always enabled inputs: L I = {,?kwart } L U = {,!tea } for all states s, for all inputs?a L I : S?a

29 Input-Output Transition Systems?kwart?kwart?kwart!tea?kwart?kwart?kwart?kwart?kwart?kwart?kwart!tea?kwart?kwart

30 Formal Testing with Transition Systems s LTS ioco iiut IMPS IUT IOTS = gen : LTS (TTS) T s TTS Test hypothesis : IUT IMP. i IUT IOTS. t TTS. IUT passes t i IUT passes t Proof soundness and exhaustiveness: i IOTS. ( t gen(s). i passes t ) i ioco s passes exec : : IOTS TESTS TTS {pass,fail} IMPS (OBS) pass / fail

31 Labelled Transition System Testing FSPECS LTS ( L I L U ) LTS FMODS IOTS (L I, L U ) LTS FTESTS TTS ( L U, L I ) LTS FOBS traces Fobs t i Fder der : LTS ( LTS ) FWhich imp? (strong, weak, branching,... ) bisimulation trace-, testing-, refusal - preorder / equivalence conf, conf*, aconf, ioco

32 A conformance relation

33 Implementation Relation ioco Correctness expressed by implementation relation ioco: i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) Intuition: i ioco-conforms to s, iff if i produces output x after trace σ, then s can produce x after σ if i cannot produce any output after trace σ, then s cannot produce any output after σ ( quiescence δ )

34 Implementation Relation ioco Correctness expressed by implementation relation ioco: i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) p δ p =!x L U {τ}. p!x Straces ( s ) = { σ (L {δ})* s σ } p after σ = { p p σ p } out ( P ) = {!x L U p!x, p P } { δ p δ p, p P }

35 Implementation Relation ioco i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) out ( i after ε ) = { δ } i?kwart δ out ( i after ) = { } out ( i after. ) = { }?kwart out ( i after.) = out ( i after?kwart ) = { δ } { δ }?kwart δ out ( i after ) = out ( i after.!tea ) = out ( i after δ ) = { δ }

36 Implementation Relation ioco i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) i s ioco out (i after ε) = { δ } out (i after ) = { } out (i after.)= { δ } out (s after ε) = { δ } out (s after ) = { } out (s after.) = { δ }

37 Implementation Relation ioco i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) i s!tea ioco out (i after ) = { } out (s after ) = {,!tea }

38 Implementation Relation ioco i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) i s!tea ioco out (i after ) = {,!tea } out (s after ) = { }

39 Implementation Relation ioco i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) i s!tea!tea ioco out (i after ) = {,!tea } out (s after ) = {,!tea}

40 Implementation Relation ioco i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) i?kwart s?kwart!tea ioco out (i after ) = { } out (i after?kwart) = {!tea } out (s after ) = { } out (s after?kwart) = But?kwart Straces ( s )

41 Implementation Relation ioco i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) i?kwart s?kwart?kwart!tea ioco!tea out (i after ) = { } out (i after?kwart) = {!tea } out (s after ) = { } out (s after?kwart) = {!tea }

42 Implementation Relation ioco i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) i?kwart s?kwart?kwart ioco!tea?kwart out (i after?kwart) = { δ } out (s after?kwart) = {!tea }

43 Implementation Relation ioco i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) i s ioco out (i after ) = { δ, } out (s after ) = { }

44 Implementation Relation ioco i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) i s τ ioco out (i after ) = { δ, } out (s after ) = { δ, }

45 Implementation Relation ioco i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) i i ioco s s!tea s ioco i!tea!tea out (i after.) = out (s after.) = {!tea, } out (i after.δ.) = { } out (s after.δ.) = {!tea, }

46 Implementation Relation ioco?kwart?kwart ioco ioco!tea!choc?kwart!tea δ s?kwart ioco ioco!tea!choc

47 Test cases

48 Test Cases Test case t TTS TTS - Test Transition System :!dub!kwart labels in L { θ } tree-structured?coffee?tea θ finite, deterministic final states pass and fail!dub fail fail from each state pass, fail : either one input!a or all outputs?x and θ?tea θ?coffee pass pass fail

49 Test Cases!dub!kwart test case t!dub!kwart ; Start timer1?coffee?tea θ?tea?timer1 fail fail!dub fail fail?coffee!dub ; Start timer2?tea θ?coffee?tea?timer2 pass pass pass pass fail?coffee fail

50 Observations and Verdicts OBS ( L { θ } )* obs obs : TTS IOTS ( ( L { θ } )* ) obs (t,i) = { σ ( L { θ } )* t i σ pass i' or t i σ fail i' } ν t ν t : ( ( L { θ } )* ) { fail, pass } ν t (O) = pass if σ O. t σ pass = fail otherwise

51 Test Generation Look for test derivation algorithm der : LTS (TTS) such that der is sound and exhaustive: i IOTS. ( t der ( s ). ν t ( obs ( t, i ) ) = pass ) i ioco s i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) obs (t,i) = { σ (L {θ})* t i σ pass i' or t i σ fail i' } ν t (O) = pass if σ O. t σ pass = fail otherwise

52 Test Generation i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) s i test σ σ σ!x!y!x!z?x pass? y pass?z fail out (s after σ) = {!x,!y } out (i after σ) = {!x,!z } out (test after σ) = L U

53 Test Generation i ioco s = def σ Straces (s) : out (i after σ) out (s after σ) s i test σ σ σ!x!y δ!x!z δ θ?x?z?y pass pass fail pass out (s after σ) = {!x,!y, δ } out (i after σ) = {!x,!z, δ } out (test after σ) = L U { θ }

54 Test Generation Algorithm Algorithm To generate a test case t(s) from a transition system specification with S, with S set of states ( initially S = s 0 after ε ) Apply the following steps recursively, non-deterministically: 1 end test case 3 observe output pass 2 supply input forbidden outputs?y fail fail allowed outputs θ?x!a t ( S after!x ) t ( S after?a ) allowed outputs or δ:!x out ( S ) forbidden outputs or δ:!y out ( S )

55 Test Generation Example s test!dub!tea?coffee?tea θ?choc?coffee?tea θ pass?choc fail fail fail fail pass fail

56 Test Generation Example s δ test!dub δ?coffee?tea θ δ?coffee?tea θ fail pass fail fail pass

57 i Test Execution Example test!dub!tea i' Two test runs : dub tea t i i''!choc pass i'?coffee?tea?coffee θ?tea pass?choc fail fail pass fail θ?choc fail fail t i dub choc i fail i'' fails t

58 Test Execution Example i test!dub?coffee?tea θ Two test runs :?coffee?tea θ fail pass t i dub θ pass i' fail fail pass t i dub coffee θ pass i' i passes t

59 Test Generation Example Equation solver for y 2 =x specification test? x (x < 0)? x (x >= 0) otherwise!9?-3?3! x fail! - x pass!4 otherwise?2?-2 To cope with non-deterministic behaviour, tests are not linear traces, but trees fail pass pass

60 Test Execution Example implementation test? x (x < 0)!9? x (x >= 0) otherwise?-3?3! x! - x fail pass!4? y otherwise?2?-2 fail pass pass

61 Validity of Test Generation For every test t generated with algorithm we have: FSoundness : t will never fail with correct implementation i ioco s implies i passes t FExhaustiveness : each incorrect implementation can be detected with a generated test t i ioco s implies t : i fails t

62 Validity of Test Generation Theorem: FSoundness : i ioco s implies t der ( s ). ν t ( obs ( t, i ) ) = pass FExhaustiveness : t der ( s ). ν t ( obs ( t, i ) ) = pass implies i ioco s

63 Soundness and Exhaustiveness s δ test!dub δ?coffee?tea θ δ?coffee?tea θ /// fail pass pass /// fail fail fail pass Not sound Still anymore sound!!