Acceptance Test. Mohamed Mussa, Ferhat Khendek

Transcription

1 Acceptance Optimization Mohamed Mussa, Ferhat Khendek SAM 2014

2 Outline Background Problem Statement Overall Approach Integration test cases selection Comparing test models Conclusion 2

3 Background process consists of several phases Unit ing Integration ing Acceptance ing Models Model Models Model Model Model 3

4 Background - Model Based ing Development Process Model CODE Execution Model CODE ing Process 4

5 Background -Model Based ing Different approaches for different test phases Unit, integration, acceptance Different notations/languages Different subsets of the same language Many test models are produced during different phases Redundancy generation/planning: no reusability Execution: no optimization 5

6 Background - Model Based Framework Goals Provide a systematic transition between the test phases Framework Strengthen the collaboration between the development and the testing teams Well know standards & reuse Improve the test process Enable reusability & optimization

7 Background - Model Based Framework Unit Unit test model cn Unit test model c1 Integration Integration test model int(n-1) Integration test model int1 Acceptance Optimized acceptance test model Acceptance test model 7

8 Problem Statement cases may be exercised several times across the testing phases Integration vs. Acceptance Goal: remove redundant acceptance test cases Reduce test execution time 8

9 Problem Statement Obvious solution Compare integration test cases and acceptance test cases Problem Some integration test cases may include stubs for subsequent system components Cannot be substituted to acceptance test cases Comp 1 Model 1 Comp 2 SbSys 1 Comp 3 Model 2 SbSys 2 Comp 4 Model n-1 SbSys n-1 Model n System Comp n 9

10 Overall Approach Integration test cases selection Compare integration test cases to acceptance test cases 10

11 Integration test cases selection cases of last integration round are applied on complete system Compare the behavior of test stubs of each test case to the behavior of CUTs of test cases of subsequent integration rounds No additional information beside the test models Comp 1 Model 1 Comp 2 SbSys 1 Comp 3 Model 2 SbSys 2 Comp 4 Model n-1 SbSys n-1 Model n System Comp n 11

12 Integration test cases selection stubs can be specified explicitly, or «Context» TC k SbSys k CUT k+1 m1 m2 m3 «Component» stub k m6 m5 m4 specified implicitly «Context» TC k m1 m3 SbSys k m2 CUT k+1 m4 m6 m5 12

13 Integration test cases selection Event based comparison Not instance based comparison Instances are different Not event name based but message event types: message, time, miscellaneous 13

14 Integration test cases selection Selection condition Let T k = {I k, E k, R k } be an integration test case at integration round k, T i = {I i, E i, R i } be an integration test case at integration round i, i> k T k does not use a test stub for the CUT of T i if and only if ( e, e ) e E, e E ( e e ) ( e = e ) ( e. owner. st SUT ) i k i i k k i k ( ). i k i 14

15 Integration test cases selection Comparing integration test cases Round k+1 Subsequent rounds «Context» TC k m1 m3 SbSys k m2 CUT k+1 «Context» TC m SbSys m CUT m+1 s1 s2 s4 s3 m4 m6 m5 «Context» TC n m1 SbSys n m3 CUT n+1 m6 m4 15

16 Integration test cases selection cases, which their stubs do not match with subsequent CUTs, are comparedtoacceptancetestcases 16

17 Comparing Models A lot of work has been done Compared models are evolved from the same source Two-Way vs. Three-Way Look up for differences (Add/Delete/Modify) Structure vs. Behavior Our case Models did not necessary evolve from the same source 17

18 Comparing Models Comparing MSCs or Sequence Diagrams is not straightforward Several researchers have tackled this issue But this is not difficult for test cases Finite behaviors 18

19 Comparing Models A test case T is a tuple(i, E, R), where I : a set of instances E : a set of events R (E x E): a partial order reflecting the transitive closure of the order relation between events on the same axis and the sending and reception events of the same message case inclusion T acc = {I a, E a, R a } and T int = {I i, E i, R i } T acc is included in T int iff E a E i R a R i 19

20 Comparing Models Comparing test cases Acceptance test case Integration test cases «Context» TC a m1 Sys «Context» TC m SbSys m CUT m+1 s1 s2 s4 s3 m6 «Context» TC n SbSys n CUT n+1 m1 m3 m6 m4 20

21 Conclusion We proposed an optimization approach that reduces the acceptance test suite length already done at integration phase Implemented and completed the framework What kind of systems would benefit? Requires evaluation of the gain 21