Model Based Testing -- FSM based testing

Size: px

Start display at page:

Download "Model Based Testing -- FSM based testing"

Flora Horton
5 years ago
Views:

1 Model Based Testing -- FSM based testing Brian Nielsen

2 Automated Model Based Conformance Testing x>=2 Model DBLclick! click? x:=0 click? x<2 Test Test Generator tool Test Generator tool tool tool Selection & optimization Test suite Test Test execution tool tool Event mapping Driver pass fail Implementation Implementation Relation Under Test Does the behavior of the (blackbox) implementation comply to that of the specification?

3 Mealy FSM q 1 cof-but / cof coin / - q 2 coin / - q 3 Inputs = {cof-but, tea-but, coin} Outputs = {cof,tea} States: {q 1,q 2,q 3 } Initial state = q 1 Transitions= { (q 1, coin, -, q 2 ), (q 2, coin, -, q 3 ), (q 3, cof-but, cof, q 1 ), (q 3, tea-but, tea, q 1 ) } tea-but / tea condition current state Sample run: input q 1 coin q 2 q 2 coin - q 3 q 3 cof-but cof q 1 q 3 tea-but tea - output effect q 1 next state coin/ - coin/- cof-but / cof coin/ - q 1 q 2 q 3 q 1 q 2 coin/ - q 3 cof-but / cof q 1

4 State Machine : FSM Model FSM - Finite State Machine - or Mealy Machine is 5-tuple M = ( S, I, O, δ, λ ) S I O δ : S x I S λ : S x I O finite set of states finite set of inputs finite set of outputs transfer function output function Natural extension to sequences : δ : S x I* S λ : S x I* O*

5 Concepts Two states s and t are (language) equivalent iff s and t accepts same language has same traces: tr(s) = tr(t) Two Machines M0 and M1 are equivalent iff initial states are equivalent A minimized / reduced M is one that has no equivalent states for no two states s,t, s!=t, s equivalent t

6 Fundamental Results Every FSM may be determinized accepting the same language (potential explosion in size). For each FSM there exist a language-equivalent minimal deterministic FSM. FSM s are closed under and FSM s may be described as regular expressions (and vise versa)

7 Conformance Testing Reference FSM M s Tester test sequence SUT FSM M I Given a specification FSM M S a ( black box ) implementation FSM M I determine whether M I conforms to M S. i.e., M I behaves in accordance with M S i.e., whether outputs of M I are the same as of M S i.e., whether the reduced M I is equivalent to M S Today: Deterministic Specifications SUT is an (unknown) deterministic FSM (testing hypothesis)

8 Restrictions FSM restrictions: deterministic δ : S x I S and λ : S x I O are functions completely specified δ : S x I S and λ : S x I O are complete functions ( empty output is allowed; sometimes implicit completeness ) strongly connected from any state any other state can be reached reduced there are no equivalent states

9 Possible Errors q 1 q 1 q 1 cof-but / - coin / - coin / - tea-but / vodka cof-but / cof q 2 q 2 tea-but / tea coin / - tea-but / tea cof-but / cof q 2 coin / - coin / - coin / - q 3 q 3 q 3 output fault (wrong or missing) extra or missing states transition fault to other state to new state coin / - q 4

10 Desired Properties Nice, but rare / problematic status messages: Assume that tester can ask implementation for its current state (reliably!!) without changing state reset: reliably bring SUT to initial state set-state: reliably bring SUT to any given state status? currentstate=s10! reset? SUT Grey-box FSM M I set-state S10?

11 FSM Transition Testing Make test case for every transition in spec separately: S1 a? / x! S2 Test transition : 1. Go to state S1 2. Apply input a? 3. Check output x! 4. Verify state S2 ( optionally ) Test purpose: Test whether the system, when in state S1, produces output x! on input a? and goes to state S2

12 Coffee Machine FSM Model coin? / - coffee? / coffee? / - coin? / - token? / - coffee? / coffee! token? / coin! 10 token? / token! coin? / coin!

13 Transition Testing 1 To test token? / coin! : go to state 5 : set-state 5 give input token? check output coin! verify state: send status? check status=10 coin? / - coffee? / - 0 coin? / - 5 coffee? / - coffee? / coffee! token? / - token? / token! 10 coin? / coin! token? / coin! Test case : set-state 5/ * - token? / coin! - status? / 10! S * I test cases remaining

14 FSM Transition Tour Make Transition Tour that covers every transition (in spec) coin? / - coffee? / - 0 coin? / - 5 coffee? / - token? / - coffee? / coffee! 10 token? / coin! token? / token! coin? / coin! Test input sequence : reset? coffee? coin? coffee? coin? coin? token? coffee? token? coffee? coin? token? coffee? + check expected outputs and target state by status message

15 Transition Testing -1 Go to state S5 : No Set-state property??? use reset property if available go from S0 to S5 ( always possible because of determinism and completeness ) or: synchronizing sequence brings machine to particular known state, say S0, from any state ( but synchronizing sequence may not exist )

16 Transition Testing -1 synchronizing sequence : token? coffee? coin? / - coffee? / - coffee? coffee? / / coin? / - 5 coffee? / - token? / - coffee? / coffee! token? / token! 10 token? / coin! coin? / coin! To test token? / coin! : go to state 5 by : token? coffee? coin?

17 Transition Testing 2,3 To test token? / coin! : 1. go to state 5 by : token? coffee? coin? 2. give input token? 3. check output coin! 4. verify that machine is in state 10 coin? / - coffee? / - 0 coin? / - 5 coffee? / - coffee? / coffee! token? / - token? / token! 10 coin? / coin! token? / coin!

18 Transition Testing-4 No Status Messages?? State identification: What state am I in?? State verification : Am I in state s? Apply sequence of inputs in the current state of the FSM such that from the outputs we can identify that state where we started; or verify that we were in a particular start state Different kinds of sequences UIO sequences ( Unique Input Output sequence, SIOS) Distinguishing sequence ( DS ) W - set ( characterizing set of sequences ) UIOv SUIO MUIO Overlapping UIO

19 Transition Testing-4 State check : UIO sequences (verification) sequence x s that distinguishes state s from all other states : for all t s : λ ( s, x s ) λ ( t, x s ) each state has its own UIO sequence UIO sequences may not exist Distinguishing sequence (identification) sequence x that produces different output for every state : for all pairs t, s with t s : λ ( s, x ) λ ( t, x ) a distinguishing sequence may not exist W - set of sequences (identification) set of sequences W which can distinguish any pair of states : for all pairs t s there is x W : λ ( s, x ) λ ( t, x ) W - set always exists for reduced FSM

20 Transition Testing-4: UIO UIO sequences coin? / - coffee? / - 0 coin? / - 5 coffee? / - coffee? / coffee! token? / - token? / token! 10 coin? / coin! token? / coin! state 0 : coin? / - coffee? / - state 5 : token? / coin! state 10 : coffee? / coffee!

21 Transition Testing-4: DS DS sequence coin? / - coffee? / - 0 coin? / - 5 coffee? / - coffee? / coffee! token? / - token? / token! 10 coin? / coin! token? / coin! DS sequence : token? output state 0 : - output state 5 : coin! output state 10 : token!

22 Transition Testing 4 done To test token? / coin! : go to state 5 : token? coffee? coin? give input token? check output coin! Apply UIO of state 10 : coffee? / coffee! coin? / - coffee? / - 0 coin? / - 5 coffee? / - coffee? / coffee! token? / - token? / token! 10 coin? / coin! token? / coin! Test case : token? / * coffee? / * coin? / - token? / coin! coffee? / coffee!

23 Transition Testing - done coin? / - coffee? / - 0 coin? / - 5 coffee? / - coffee? / coffee! token? / - token? / token! 10 coin? / coin! token? / coin! - 9 transitions / test cases for coffee machine - if end-state of one corresponds with start-state of next then concatenate - different ways to optimize and remove overlapping / redundant parts - there are (academic) tools to support this

24 FSM Transition Testing Test transition : Go to state S1 Apply input a? Check output x! Verify state S2 Checks every output fault and transfer fault (to existing state) If we assume that the number of states of the implementation machine M I is less than or equal to the number of states of the specification machine to M S. then testing all transitions in this way leads to equivalence of reduced machines, i.e., complete conformance If not: exponential growth in test length in number of extra states.

25 Object Testing-1 Class PuckSupply{ int _count=10; Public: }; puck * get(){ if(_count>0) { _count--;return get_puck(); } else { return NULL; } A Test case s=new PuckSupply; Assert(s.get()== apuck); Assert(s.get()== apuck); Assert(s.get()== apuck); Assert(s.get()== apuck); Assert(s.get()== apuck); Assert(s.get()== apuck); Assert(s.get()== apuck); Assert(s.get()== apuck); Assert(s.get()== apuck); Assert(s.get()== apuck); Assert(s.get()==NULL); Assert(s.get()==NULL); s10 get() / puck s9 get() / puck s8 s1 get() / puck s0 get() / NULL

26 Object Testing-2 Class PuckSupply{ int _count=10; Public: //test helpers int getstate() {return _count;} void setstate(int count) { }; _count=count; alloc_puks(); } puck * get(){ if(_count>0) { _count--;return new puck(); } else { return NULL; } Assert(s.get()==aPuck); Assert(s.getState()==9); s.setstate(1); Assert(s.get()==aPuck); Assert(s.getState()==0); Assert(s.get()==NULL); Assert(s.getState()==0); Assert(s.get()==NULL); Assert(s.getState()==0); s10 get() / puck s9 get() / puck s8 s1 get() / puck s0 get() / NULL

27 Object Testing-2 Class PuckSupply{ int _count=10; Public: //test helpers int getstate() {return _count;} void setstate(int count) { }; _count=count; alloc_puks(); } puck * get(){ if(_count>0) { _count--;return new puck(); } else { return NULL; } Assert(s.get()==aPuck); Assert(s.getState()==9); s.setstate(1); Assert(s.get()==aPuck); Assert(s.getState()==0); Assert(s.get()==NULL); Assert(s.getState()==0); Assert(s.get()==NULL); Assert(s.getState()==0); s10 get() / puck s9 get() / puck s8 s1 get() / puck s0 get() / NULL How many states in corresponding FSM?

28 Object Testing 3: Partitioning Class PuckSupply{ int _count=10; Public: }; puck * get(){ if(_count>0) { _count--;return new puck(); } else { return NULL; } n=init n>1 get() / puck n-- n>1 getpuck() / puck n-- n 1 n==1 get() / puck n-- get() / NULL n=0 n==1 get() / puck n-- Generate tests systematically from more abstract descriptions, e.g., the quivalence classes to select reasonably number of tests

29 Test Generation using Class PuckSupply{ int _count=10; Public: Uppaal??!! }; puck * get(){ if(_count>0) { _count--;return new puck(); } else { return NULL; }

30 Test Generation using UppAal

31 Touch-sensitive Light- Controller Patient user: Wait= Impatient: Wait=15

32 Object Testing and Test Generation Using Uppaal

33 Timed Tests T_sw=4 T_idle=20 EXAMPLE test cases for Interface Epsilon=200ms Delta=500ms 0 grasp! 210 release! touch?.pass 0 grasp! 317 release! touch? 2½ grasp! 220 release! touch? PASS 1000 grasp! 517 starthold? 100 release! endhold? PASS INFINITELY MANY SEQUENCES!!!!!!

34 Test Purposes 1 A specific test objective (or observation) the tester wants to make on SUT Environment model System model Wait= mindelay=0 TP1: Check that the light can become bright: E<> L==10 Shortest (and fastest) Test: out(igrasp);silence(500);in(osetlevel,0);silence(1000); in(osetlevel,1);silence(1000);in(osetlevel,2); silence(1000); in(osetlevel,3);silence(1000);in(osetlevel,4);silence(1000); in(osetlevel,5);silence(1000);in(osetlevel,6);silence(1000); in(osetlevel,7);silence(1000);in(osetlevel,8);silence(1000); in(osetlevel,9);silence(1000);in(osetlevel,10); out(irelease);

35 Test Purposes 2 TP2: Check that controller can enter location DnPassive : E<> Dim.DnPassive DnPassive If delay=1000 Shortest (and fastest) Test: out(igrasp); silence(500); in(osetlevel,0); out(irelease); out(igrasp); silence(500);

36 Test Purposes 2 TP2: Check that controller can enter location DnPassive : E<> Dim.DnPassive DnPassive If delay=40? Shortest Test: Fastest Test: out(igrasp); silence(500); in(osetlevel,0); out(irelease); out(igrasp); silence(500); out(igrasp);silence(500);in(osetlevel,0);silence(40); in(osetlevel,1);silence(40);in(osetlevel,2); silence(40); in(osetlevel,3);silence(40);in(osetlevel,4); silence(40); in(osetlevel,5);silence(40);in(osetlevel,6); silence(40); in(osetlevel,7);silence(40);in(osetlevel,8); silence(40); in(osetlevel,9);silence(40);in(osetlevel,10);silence(40);

37 Test Purposes 2 TP2: Check that controller can enter location DnPassive : E<> Dim.DnPassive DnPassive If Wait=1500 and mindelay=400? Ask a tool

38 Test Purposes 3 TP3: Check that controller re-sets light level to previous value after switch-on. E<> Purpose1.goal out(igrasp); //set level to 5 silence(500); in(osetlevel,0); silence(1000); in(osetlevel,1); silence(1000); in(osetlevel,2); silence(1000); in(osetlevel,3); silence(1000); in(osetlevel,4); silence(1000); in(osetlevel,5); out(irelease); out(igrasp); //touch To Off silence(200); out(irelease); in(osetlevel,0); out(igrasp); //touch To On silence(200); out(irelease); in(osetlevel,5); silence(2000);

39 Other Environments Slow User: T_react=2 T_react=2 T_sw=4 T_pause=5 T_idle=20 TP1: Fastest Test: 0 touch! 0 dim? 2 touch! 0 bright?? PASS Pausing User: max 2 successive quick touches

40 Coverage Based Test Generation Multi purpose testing Cover measurement Examples: Location coverage, Edge coverage, Definition/use pair coverage l 1 a? x:=0 b! l 2 c! x 2 a? x<2 l 4 l 3

41 Coverage Based Test Generation Multi purpose testing Cover measurement Examples: Location coverage, Edge coverage, Definition/use pair coverage l 1 a? x:=0 b! l 2 c! x 2 a? x<2 l 4 l 3

42 Coverage Based Test Generation Multi purpose testing Cover measurement Examples: Location coverage, Edge coverage, Definition/use pair coverage l 1 a? x:=0 b! l 2 c! x 2 a? x<2 l 4 l 3

43 Coverage Based Test Generation Multi purpose testing Cover measurement Examples: Location Coverage, Edge Coverage, Definition/Use Pair Coverage c! l 1 a? x:=0 b! x 2 l 2 x<2 l 4 l 3

44 Location Coverage Test sequence traversing all locations Encoding: Enumerate locations l 0,,l n Add an auxiliary variable l i for each location Label each ingoing edge to location i l i :=true Mark initial visited l 0 :=true Check: E<>( l 0 =true l n =true ) l j :=true l j l j :=true

45 Edge Coverage Test sequence traversing all edges Encoding: Enumerate edges e 0,,e n Add auxiliary variable e i for each edge Label each edge e i :=true Check: E<>( e 0 =true e n =true ) a? x:=0 e0:=1 l 1 l 2 x 2 b! e1:=1 c! e4:=1 x<2 l 4 l e3:=1 3 a? e2:=1

46 Fastest Edge Coverage Cost=12600 ms out(igrasp); //touch:switch light on silence(200); out(irelease); in(osetlevel,0); out(igrasp); // touch: switch light off silence(200); out(irelease);//touch in(osetlevel,0); //9 out(igrasp); //Bring dimmer from ActiveUp silence(500); //hold //To Passive DN (level=0) in(osetlevel,0); out(irelease); Page 1 //13 out(igrasp); //@900 // Bring dimmer PassiveDn->ActiveDN-> silence(500);//hold // ActiveUP+increase to level 10 silence(1000); in(osetlevel,1); silence(1000); in(osetlevel,2); silence(1000); in(osetlevel,3); silence(1000); in(osetlevel,4); silence(1000); in(osetlevel,5); silence(1000); in(osetlevel,6); silence(1000); in(osetlevel,7); silence(1000); in(osetlevel,8); silence(1000); in(osetlevel,9); silence(1000); in(osetlevel,10 silence(1000); in(osetlevel,9); //bring dimm State to ActiveDN out(irelease); //check release->grasp is ignored out(igrasp); //@12400 out(irelease); silence(dftolerance); Page 2

47 Definition/Use Pair Coverage Dataflow coverage technique Def/use pair of variable x: x:=0 definition... no defs x 4 use Encoding:.. v d { false} { e 0,, e n }, initially false Boolean array du of size E x E At definition on edge i: v d :=e i At use on edge j: if( v d )thendu[v d,e j ]:=true

48 Definition/Use Pair Coverage Dataflow coverage technique Def/use pair of variable x: x:=0... x 4 du: 0 0 n-1 definition no defs use i Encoding: v d { false} { e 0,, e n }, initially false Boolean array du of size E x E n-1 j At definition on edge i: v d :=e i At use on edge j: if( v d )thendu[v d,e j ]:=true Check: E<>( all du[i,j] = true )

49 Test Suite Generation In general a set of test cases is needed to cover a test criteria Add global reset of SUT and environment model and associate a cost (of system reset) initial R x=c x C x:=0 reset? reset Same encodings and min-cost reachability Test sequence σ = ε 0,i 0,,ε 1, i 1, reset ε 2,i 2,,ε 0,i 0,reset,ε 1, i 1,ε 2,i 2, Test suite T = {σ 1,, σ n } with minimum cost σ i

50 Optimal Tests 1W 50W 1W 100W Shortest test for max light?? Fastest test for max light?? Fastest edge-covering test suite?? Least power consuming test??

Model Based Testing of Embedded Systems

Model Based Testing of Embedded Systems Brian Nielsen Arne Skou {bnielsen ask}@cs.auc.dk Automated Model Based Conformance Testing x>=2 Model DBLclick! click? x:=0 click? x