Integration Testing - Looking for a Solution to Testing Concurrent Components and Systems

Transcription

1 Integration Teting - Looking for a Solution to Teting Concurrent Component and Sytem Integration Teting 23. TAV 1 Andrej Pietchker, PhD Siemen AG, CT SE 1 andrej.pietchker@iemen.com ph: +49 (89) Preentation Overview Motivation Light-Weight Approach to Paive Teting Model-baed Approach to Paive Teting Concluion Integration Teting 23. TAV 2 1

2 Motivation Integration Teting 23. TAV 3 Fault Diagnoi in Ditributed Sytem Fault diagnoi conit of the tak Reproduce the failure oberved, e.g., during teting Examine the fault Hypotheize about the error Ditributed, concurrent ytem complicate matter Nondeterminitic behavior Reproducibility of failure difficult to achieve Error Error Fault Fault Integration Teting 23. TAV 4 Failure Failure 2

3 Paive Teting Tracing to upport fault diagnoi Well-proven mean for analyzing complex ytem Collect event from the running application Uually large amount of data collected Automatic offline analyi of trace Support of filtering, viualization, and detail-on-demand Flexible and eay uage of tool Paive Teting Active Teting: Verification by mean of interaction with SUT Integration Teting 23. TAV 5 Light-Weight Approach to Paive Teting Integration Teting 23. TAV 6 3

4 Information Seeking Mantra Paive teting approach applie concept of Viualization Trace data and analyi reult are made acceible to the uer Different output upported, e.g. in SVG or HTML Filtering Reduce the ize and information contained in a trace Filter define operation over trace, e.g. election, pruning Detail-on-Demand Not all information hould be preent at all time Diplay further information when cloer inpection i required Integration Teting 23. TAV 7 Structure of a Standardized Trace Format Trace contain the following event type Regitration and unregitration event of Device Procee and/or thread Object Communication event Send and receive of meage Local event Variable aignment Aertion Event tructure defined in XML chema Integration Teting 23. TAV 8 4

5 Trace Tranformation Trace preent a XML-file (converted from other format) Output of a filter i another trace in XML format Filter can be combined Support reue of filter Filter are not necearily commutative: f g g f Viualization component applied if needed Integration Teting 23. TAV 9 Example: Overview Automotive application Driver information and navigation ytem Embedded component-baed oftware development project Sytem wa deigned with tracing already in mind! Sytem failure Sytem lowed down unexpectedly when running on target hardware Tak: Analyze the caue of the low-down! Original trace contained more than 1 million event Analyi tep Identify thread with a large running time Viualize the communication of thee thread Analyi reult Slow-down caued by permanent polling of a device that wa not operational Integration Teting 23. TAV 10 5

6 Example: Performance Analyi in HTML Integration Teting 23. TAV 11 Example: Graphical Trace Viualization in SVG Integration Teting 23. TAV 12 6

7 More Example Integration Teting 23. TAV 13 Taking Concurrency into Account Integration Teting 23. TAV 14 7

8 Model-baed Approach to Paive Teting Integration Teting 23. TAV 15 Log, Trace, and Model pr1 n1 = 3 n1 = 4 n1 = 5 n2=2 n1=5 m1 m2 Zeit rm3 n1=5 n1=5 rm3 rm2 n2=2 rm2 n2=2 m3 pr2 n2 = 6 n2 = 4 Log pr1: n1=3 Trace pr2: n2=6 pr2: nd m1 to pr1 pr2: n2=4 pr1: rcv m1 from pr2 pr2: nd m3 to pr1 pr1: n1=4 pr1: nd m2 to pr2 pr1: rcv m3 from pr2 pr2: rcv m2 from pr1 pr1: n1=5 pr2: n2=2 Integration Teting 23. TAV 16 n2 = 2 m3 m2 rm2 m2 n2=4 m3 m3 n1=4 n2=4 m2 rm1 rm1 n2=4 m2 n1=3 n2=4 m2 n2=4 n1=4 n1=4 n1=3 rm1 m1 n2=6 n1=3 n1=3 m1 rm3 n2=6 Model 8

9 Model-baed Analyi Overview Trace Uer-defined Propertie (a Pattern) Recontruction of ytem behavior Simulation Reult OK? Integration Teting 23. TAV 17 Example: Sliding Window Protocol A connection-oriented protocol (ued e.g. in TCP) Allow data to be ent in one direction between a pair of protocol entitie, ubject to a maximum number of unacknowledged meage (window) If thi window become full, the protocol i blocked until an acknowledgement i received for the earliet outtanding meage. At thi point the tranmitter i clear to end more meage. Integration Teting 23. TAV 18 9

10 bl ock B lo c k 1 e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T Vo id, T W in d o wa c k, T W in d o w N a c k e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T Vo id, T W in d o wa c k, T W in d o w N a c k a 2 ( 1, 1 ) a 2 a 4 6 a 4 6 ( 1, 1 ) a 2 a 5 0 a 2 a 7 6 T C o n n e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T V o id, T W in d o w A c k a 5 0 ( 1, 1 ) a2 a 5 8 T C o n n e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T V o id, T W in d o w Ac k a 7 6 ( 1, 1 ) a 2 a 7 2 T C o n n e c t, T Co n n e c t A c k, T T e t, T T e t A c k, T V o id, T W in d o wa c k a 5 8 ( 1, 1 ) a 2 a 5 2 T C o n n e c t, T Co n n e c t A c k, T T e t, T T e t A c k, T V o id, T W in d o wa c k a 7 2 ( 1, 1 ) a 2 a 6 4 T C o n n e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T Vo id, T W in d o w A c k a 5 2 ( 1, 1 ) a 2 a 6 2 T C o n n e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T V o id, T W in d o w A c k a 6 4 ( 1, 1 ) a 2 a 6 8 T C o n n e c t, T C o n n e c t Ac k, T T e t, T T e t A c k, T V o id, T W in d o w A c k a 6 2 ( 1, 1 ) a 2 a 5 4 T C o n n e c t, T C o n n e c t Ac k, T T e t, T T e t A c k, T V o id, T W in d o w A c k a 6 8 ( 1, 1 ) a 2 a 6 0 T C o n n e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T V o id, T W in d o w A c k a5 4 ( 1, 1 ) a 2 a 6 6 a 6 0 ( 1, 1 ) a 2 a 4 4 a 6 6 ( 1, 1 ) a 2 a 2 8 a 4 4 ( 1, 1 ) a 2 a 5 6 T C o n n e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T V o id, T W in d o w A c k a 2 8 ( 1, 1 ) a 2 a 4 2 T C o n n e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T V o id, T W in d o w A c k a 5 6 ( 1, 1 ) a 2 a 2 6 T Co n n e c t, T Co n n e c t A c k, T T e t, T T e t A c k, T V o id, T W in d o w Ac k a 4 2 ( 1, 1 ) a 2 a 3 0 T Co n n e c t, T Co n n e c t A c k, T T e t, T T e t A c k, T V o id, T W in d o w Ac k a 2 6 ( 1, 1 ) a 2 a 4 0 T C o n n e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T Vo id, T W in d o wa c k a 3 0 ( 1, 1 ) a 2 a 7 4 T C o n n e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T Vo id, T W in d o w A c k a 4 0 ( 1, 1 ) a 2 a 3 2 a 2 a 3 8 T C o n n e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T V o id, T W in d o w A c k, T W in d o w N a c k a 7 4 ( 1, 1 ) a 3 2 ( 1, 1 ) a 2 a 3 6 T C o n n e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T V o id, T W in d o w A c k a 3 8 ( 1, 1 ) a 2 a 3 4 T C o n n e c t, T C o n n e c t A c k, T T e t, T T e t A c k, T V o id, T W in d o w A c k a 3 6 ( 1, 1 ) TConnect, TConnect Ack, TTet, TTet Ack, TVoid, TW indo a 3 4 ( 1, 1 ) T C o n n e c t, T C o n n e c t A c k, T T e t, T Architectural View of the Sytem 1 centralized component communicating with all other component (tar-form communication) Integration Teting 23. TAV 19 Paive Teting Workflow Exemplified Start: Formalize ytem property The liding window property i not violated. Rephrae property a pattern over trace Let SWS be the maximum ender window ize. Let LAR be the equence number of the lat acknowledgment received. Let LFS be the equence number of the lat frame ent. The ender maintain the invariant: LFS - LAR + 1 <= SWS. Simulation of trace and pattern uing a model checker Interpretation of reult Integration Teting 23. TAV 20 10

11 Main Part of the Analyi Pattern Integration Teting 23. TAV 21 Trace Analyi Execution Procee a2 and a40 are analyzed Integration Teting 23. TAV 22 1 error found during the verification proce 11

12 A Detected Error Scenario error-trace int_1_a2 PROCESS / ytem1/ block1/a2 (1) int_1_a40 PROCESS / ytem1/ block1/ a40(1) TConnect TConnect TWindowAck TWindowAck TVoid TVoid TVoid TVoid TVoid Integration Teting 23. TAV 23 Error trace report a 4th unacknowledged TVoid meage although window ize i et to 3 only. Analyi of Other Sytem Propertie Sytem architecture Analyi of initialization phae (thread or object view) Analyi of creation and termination of thread or object Deadlock analyi Race analyi Identification of object ued by more than one thread Identification of meage intended for a ingle recipient (race) Race are the main caue for nondeterminitic ytem behavior Performance analyi Analyi of communication, delay, uage time Integration Teting 23. TAV 24 12

13 Concluion Integration Teting 23. TAV 25 Summary (1) Input for Paive Teting A trace file with all the neceary information Deired or undeired ytem propertie Fully automated analyi Tool exit! Analyi reult Indicator for fault analyi Support during regreion teting Univeral analyi of ytem propertie Non-determinim Performance bottle-neck Hint for redeign Paive Teting uable for Ditributed ytem, epecially for ytem with many device Poorly documented ytem Sytem containing Component of the helve (COTS) Integration Teting 23. TAV 26 13

14 Summary (2) XML-baed approach to trace analyi Offer quick and cheap olution to upport fault diagnoi Enable an interactive way to extract and viualize information gathered during teting Ue tandardized XML technologie Benefit from the large variety of freely available XML tool Scale with the trace ize Model-baed approach to trace analyi Can be integrated into XML-baed approach (filter concept) Take concurrency into account Find non-reproducible fault Integration Teting 23. TAV 27 14