Integration and Higher Level Testing

Transcription

1 Integration and Higher Level Testing Software Testing and Verification Lecture 11 Prepared by Stephen M. Thebaut, Ph.D. University of Florida

2 Context Higher-level testing begins with the integration of (already unit-tested) modules to form higher-level program entities (e.g., components). The primary objective of integration testing is to discover interface and blatant higher-level design errors associated with the elements being integrated. (Somewhat in keeping with the spirit of smoke testing.)

3 Context (cont d) Once the elements have been successfully integrated (i.e., once they are able to function together), the functional and non-functional characteristics of the higher-level element can be tested thoroughly (via component, product, or system testing).

4 Integration Testing Integration testing is carried out when integrating (i.e., combining): Units or modules to form a component Components to form a product Products to form a system The strategy employed can significantly affect the time and effort required to yield a working, higher-level element.

5 Integration Testing (cont d) Note that integration testing is sometimes defined as the level of testing between unit and system. We use a more general model of the testing process.

6 Levels of Testing More General View Popular View System Test Integration Test Unit Test System Test Integration Test Product Test Integration Test Component Test Integration Test Unit Test

7 Integration Testing Strategies The first (and usually the easiest...) issue to address is the choice between instantaneous and incremental integration testing. The former is sometimes referred to as the big bang approach. (Guess why!) Locating subtle errors can be very difficult after the bang. Integration is a process, not an event.

8 Integration Testing Strategies (cont d) Incremental integration testing results in some additional overhead, but can significantly reduce error localization and correction time. (What is the overhead?) The optimum incremental approach is inherently dependent on the individual project and the pros and cons of the various alternatives available.

9 s an aside Here s a slide from an on-line lecture that I came across recently Principles of Integration Proper policy and plans dvocacy Manpower training Realistic tasks Coordination with other sectors Proper support ccess to drugs Integration of Substance buse Disorders in National Rural Health Mission (NRHM), Centre for Community Medicine, ll India Institute of Medical Sciences, New Delhi

10 s an aside Here s a slide from an on-line lecture that I came across recently Principles of Integration Proper policy and plans dvocacy Manpower training Realistic tasks Coordination with other sectors Proper support ccess to drugs Integration of Substance buse Disorders in National Rural Health Mission (NRHM), Centre for Community Medicine, ll India Institute of Medical Sciences, New Delhi

11 Incremental Strategies Suppose we are integrating a group of modules to form a component, the control structure of which will form a calling hierarchy as shown. B C D E F G H I J K L

12 Incremental Strategies (cont d) In what order should the modules be integrated? From the top ( root ) module toward the bottom? From bottom ( leaf ) modules toward the top? By function? Critical or high-risk modules first? By availability?

13 Incremental Strategies (cont d) How many should be combined at a time? What scaffolding (i.e., s and s to exercise the modules, and oracles to interpret/inspect test results) will be required? Is scaffolding ever required outside the context of integration testing?

14 Top-Down Strategy 1. Start with the root and one or more called modules. 2. Test this group using s to take the place of missing called modules, and one (if necessary) to call the root module. 3. dd one or more other called modules, replacing and providing new s as necessary. (cont d)

15 Top-Down Strategy (cont d) 4. Continue the process until all elements have been integrated and tested.

16 Top-Down Strategy (cont d) B C D E F G H I J K L

17 Top-Down Strategy (cont d) B C D E F G H I B J K L

18 Top-Down Strategy (cont d) B C D E F G H I B C J K L

19 Top-Down Strategy (cont d) B C D E F G H I B C D J K L

20 Top-Down Strategy (cont d) B C D E F G H I B C D J K L E

21 Top-Down Strategy (cont d) B C D E F G H I B C D J K L E F

22 Top-Down Strategy (cont d) B C D E F G H I B C D J K L E F G

23 Top-Down Strategy (cont d) B C D E F G H I B C D J K L E F G H

24 Top-Down Strategy (cont d) B C D E F G H I B C D J K L E F G H I

25 Top-Down Strategy (cont d) B C D E F G H I B C D J K L E F G H I J

26 Top-Down Strategy (cont d) B C D E F G H I B C D J K L E F G H I J K

27 Top-Down Strategy (cont d) B C D E F G H I J K L

28 Top-Down Strategy (cont d) Potential dvantages: llows early verification of high-level behavior. One (at most) is required. Modules can be added one at a time with each step if desired. Supports both breadth first and depth first approaches.

29 Top-Down Strategy (cont d) Potential Disadvantages: Delays verification of low-level behavior. Stubs are required for missing elements. Test case inputs may be difficult to formulate. Test case outputs may be difficult to interpret. (Oracles may be needed to interpret/inspect test results.)

30 Bottom-Up Strategy 1. Start at the bottom of the hierarchy with two or more sibling leaf modules, or an only-child leaf module with its parent. 2. Test this group using a. (No s are required.) 3. dd one or more additional siblings, replacing s with modules only when all modules they call have been integrated. (cont d)

31 Bottom-Up Strategy (cont d) 4. Continue the process until all elements of the subtree have been integrated and tested. 5. Repeat the steps above for the remaining subtrees in the hierarchy (or handle in parallel). 6. Incrementally combine the sub-trees and then add the root module.

32 Bottom-Up Strategy (cont d) B C D E F G H I J K L

33 Bottom-Up Strategy (cont d) B C D E F G H I J K L F J

34 Bottom-Up Strategy (cont d) B C D E F G H I J K L E F J

35 Bottom-Up Strategy (cont d) B C D E F G H I B J K L E F J

36 Bottom-Up Strategy (cont d) B C D E F G H I B J K L E F J K L

37 Bottom-Up Strategy (cont d) B C D E F G H I B J K L E F H J K L

38 Bottom-Up Strategy (cont d) B C D E F G H I B J K L E F H I J K L

39 Bottom-Up Strategy (cont d) B C D E F G H I B D J K L E F H I J K L

40 Bottom-Up Strategy (cont d) B C D E F G H I B D J K L E F G H I J K L

41 Bottom-Up Strategy (cont d) B C D E F G H I B C D J K L E F G H I J K L

42 Bottom-Up Strategy (cont d) B C D E F G H I B C D J K L E F G H I J K L

43 Bottom-Up Strategy (cont d) B C D E F G H I B C D J K L E F G H I J K L

44 Bottom-Up Strategy (cont d) B C D E F G H I B C D J K L E F G H I J K L

45 Bottom-Up Strategy (cont d) B C D E F G H I B C D J K L E F G H I J K L

46 Bottom-Up Strategy (cont d) B C D E F G H I J K L

47 Bottom-Up Strategy (cont d) Potential dvantages: llows early verification of low-level behavior. No s are required. Easier to formulate input data for some subtrees. Easier to interpret output data for others.

48 Bottom-Up Strategy (cont d) Potential Disadvantages: Delays verification of high-level behavior. Drivers are required for missing elements. s subtrees are combined, a large number of elements may be integrated at one time.

49 Hybrid Incremental Integration pproaches Risk Driven Start by integrating the most critical or complex modules together with modules they call or are called by. Schedule Driven To the extent possible, integrate modules as they become available. (cont d)

50 Hybrid Incremental Integration pproaches (cont d) Function or Thread Driven Integrate the modules associated with a key function (thread); continue the process by selecting another function, etc. B C D E F G H I J K L

51 Hybrid Incremental Integration pproaches (cont d) Function or Thread Driven Integrate the modules associated with a key function (thread); continue the process by selecting another function, etc. B C D E F G H I J K L

52 How about Object-Oriented Systems? Just as a calling hierarchy allows design of an integration strategy for imperative software, use/include relations serve this purpose for object-oriented software. Since there may be no single root class, testing usually proceeds cluster by cluster in a bottom-up fashion, starting with leaf classes that depend on no others. We will come back to this in Lecture 12.

53 Higher-Level Testing Higher-level tests focus on the core functionality specified for higher level elements, and on certain emergent properties that become more observable as testing progresses toward the system level. The black-box testing strategies already considered (e.g., partition and combinatorial approaches) apply to functional testing at any level.

54 Higher-Level Testing (cont d) Higher-level testing typically includes: Usability test Installability test Serviceability test Performance test Stress test Security test Software compatibility test Device and configuration test Recovery test Reliability test brief overview of each follows.

55 Usability Test Focus is on factors which influence the ease with which potential end-users are able to utilize the system to accomplish their goals. Specialized and sophisticated: HCI experts conduct experiments in simulated work environments. Protocol analysis is utilized to identify usability bottlenecks. pplication-specific metrics related to understandability, learnability, and operability may be employed.

56 Installability Test Focus is functional and non-functional requirements related to the installation of the product/system. Coverage includes: Media correctness and fidelity, Relevant documentation (including examples), and Installation processes and supporting system functions. (cont d)

57 Installability Test (cont d) Functions, procedures, documentation, etc., associated with product/system decommissioning must also be tested.

58 Serviceability Test Focus is requirements related to upgrading or fixing problems after installation. Coverage includes: Change procedures (adaptive, perfective, and corrective service scenarios), Supporting documentation, and System diagnostic tools.

59 Performance Test Focus is requirements related to throughput, response time, memory utilization, input/ output rates, etc. lso very specialized in some organizations; sophisticated test-beds and instrumentation are the norm. Statistical testing based on an operational profile is often employed. Requirements must be stated in quantifiable terms.

60 Stress Test Focus is system behavior at or near overload conditions (i.e., pushing the system to failure ). Often undertaken with performance testing. In general, products are required to exhibit graceful failures and non-abrupt performance degradation.

61 Security Test Focus is vulnerability of resources to unauthorized access or manipulation. Issues include: Physical security of computer resources, media, etc., Login and password procedures/policies, Levels of authorization for data or procedural access, etc.

62 Software Compatibility Test Focus is compatibility with other products/systems in the environment and/or with interoperability standards. May also concern source- or object-code compatibility with different operating environment versions. K compatibility/conversion testing when conversion procedures or processes are involved.

63 Device and Configuration Test Focus is configurability for, and/or compatibility with, all supported hardware configurations. Particularly taxing for client/serverbased applications Tests are usually limited to combinations of representative devices for each supported protocol.

64 Recovery Test Focus is ability to recover from exceptional conditions associated with hardware, software, or people. This can involve: detecting exceptional conditions, switch-overs to standby systems, recovery of data, maintaining audit trails, etc. May also involve external procedures such as storing backup tapes, etc.

65 Reliability Test Requirements may be expressed as: the probability of no failure in a specified time interval, or as the expected mean time to failure. ppropriate interpretations for failure and time are critical. Utilizes Statistical testing based on an operational profile.

66 Integration and Higher Level Testing Software Testing and Verification Lecture 11 Prepared by Stephen M. Thebaut, Ph.D. University of Florida