V&V of Complex Systems

Transcription

1 V&V of Complex Systems Bill Lanz and Hussein Youssef S5 Symposium- Beavercreek, OH June 14-16,

2 Introduction Current methods of V&V, which certify that the software is fit for use, require a significant amount of touch labor. Future complex software developments face cost hurdles so high that it may not be deployable. We will take the current V&V technology beyond formal methods (the current state of the art) We propose reducing the V&V problem to an optimization problem solvable by readily available processing methods. This effort examines an advanced method of V&V utilizing a heuristic approach to automatically determine system invariants as introduced by Hoare [Hoare, 1969]. We will illustrate our approach using optimization algorithms to automatically determine system invariants and hence system satisfiability. Violation of the satisfiability can be isolated as bugs or errors in the code. Solving the satisfiability problem is NP-Hard during back propagation, but we will show some possible methods of overcoming the time constraints. 2

3 Modern Systems are Highly Complex 787 Example nd Qtr Mar-10 Conceptual Des 1st Flight Schd 1st Flight Static Test Ultimate load Marketing Begins 1st Order/Program Launch 1st Deliveries Scheduled Months Delay Date Schedule Delays - Cause 1st Flight Deliveries 9/5/07 Software/Fasteners 3 10/10/07 Fasteners/Suppliers 3 6 1/16/08 Traveled Work 3 4/9/08 Other Delays from 1/16/ /4/08 Bad Fastener Installation/Strike 3 8/27/09 Reinforce Side-Body /27/10 Trent 1000 Engine Blowout 6 Roughly 65% of vehicle made outside Boeing 787 is about 8% overweight at last count, with a 10-15% range reduction 3

4 Modern Systems are Highly Complex A380 Example Ult Ld st Order/Program Launch 1st Flight 1st Delivery 1st Planned Delivery ( ) Months Delay Date Schedule Delays - Cause 1st Flight Deliveries 6/1/05 Wiring, Change Control, CATIA 4/5 6 2/14/06 Wing fails destructive 145% 6/13/06 Production Ramp-up Issues 6 10/3/06 Misrouted wire harnesses 12 Initial Op Wt Empty target Mid-program update Op Wt Empty actual

5 Modern Systems are Highly Complex Other Examples F-35 Current Program Next Generation Air Transportation Management System (NextGen) Future Program 2001 National Institute of Standard and Technology (NIST) study found that the cost of software errors to the U.S. economy is ~ approximately $60B/year1 1 RTI, The Economic Impacts of Inadequate Infrastructure for Software Testing, Planning Report 02-3, National Institute of Standard and Technology, May

6 V&V Labor Hours V&V Software Costs vs. Goals Software Costs Reflect the Complexities of Host Systems Cost Research and Cost Models indicate that Validation & Verification experience superlinear cost trends A linear V&V cost target offers large savings in time and money 100,000,000 90,000,000 80,000,000 70,000,000 60,000,000 Validation and Verification Costs V&V Manhours (for Military Ground Test) (1) = 0.049(SLOCs) Note: R 2 = ,000,000 40,000,000 All Software Hours (including V & V) (2) = 514(SLOCS/1000) ,000,000 20,000,000 10,000,000 Linear V&V Cost Target 0 0 2,000,000 4,000,000 6,000,000 8,000,000 10,000,000 12,000,000 Source lines of Code (SLOCs) References: 1) Gayek, et.al., Software Cost and Productivity Model, Journal of Parametrics, Summer 2006, pages 36-71, Figure A-5 2) Boehm, Barry, Revised Intermediate COCOMO (Constructive Cost Model), or REVIC, cost for embedded software (as described in 6

7 SLOC Growth Approaching Affordability Limits Nov 30, 2010, David Redman, Donald Ward, John Chilenski and Greg Pollari, Virtual Integration for Improved System Design, Proceedings of The First Analytic Virtual Integration of Cyber-Physical Systems Workshop in conjunction with RTSS

8 The Cost of V&V (Conventional Methods) Fisher describes a conventional Validation and Verification (V&V) methodology to show that a software development is (1) building the right software for the need and (2) building the software right. This top-down process is proven throughout the industry and supported by CMMI, IEEE-1012, ISO/IEC-25020, and other software quality standards. Conventional Methods represent Corporate Processes based on Industry Standards State of the Art at the Complex System Level 8

9 DO-178B Software Development Process Example Combined Flight and Mission Software Refine System or High Level Requirements CR SRS Requirements Derived Requirements Technical Reviews Baseline SPEs Task Assignments / Schedules Sub-Teams Individual Engineers CR Simulink Model Serena Dimensions SPEs CRs CM Software Development Team CR Model/Design SPEs VMS OFP Hand Code Real-Time Workshop Process Artifacts SPEs Autocode Merge AutoMet SDSRs Independent V&V Team Simulation Actuators, Aero, Air Data, CLAW, Sensors, Propulsion DO-178B Artifacts Plans Unit Test Static Analysis Reports SPE Reports Unit Test Results PSAC Requirements Coding Standards Source Code (Baselines) Algorithm Design Cycle Merges Model Standards/SDD Code Generation Tools Flight Control Utilities Construction Tools Code Design Phase CR 6-DOF Simulation SPEs CR Embedded Code OFP Target SPEs SIM Model Based Design Linear Model and Analysis Non-Linear Analysis Data Review Operational Flight Program Check Cases/Timing FCIL, PIL, AIL, HWIL Static Analysis Polyspace + LDRA CR SIL SPEs Flight Test SCRs Mission Control Laws Executive VMC Hardware OFP SGI OFP release to Flight Test 9

10 The Cost of V&V (Formal Methods) Formal methods involve a method of developing specifications, implementation, and verification of software utilizing a formal methodology. There are several formal method toolkits (Esterel, B-Method, Z) available and are often supplemented by an Automated Theorem Prover (ATP) that effectively proves the correctness of the theorem. Formal Methods have proven to be very successful at the component level and are the current state of the art 10

11 DARPA TTO Office Initiative On Complexity 11

12 New Directions Empirical Methods Conventional and formal methods have shown to be effective and are proven approaches to developing complex systems. Both approaches use top-down deductive methods that unfortunately have not significantly reduced the amount of touch labor needed to perform to demanding schedules and lower cost. Empirical methods may be used assuming a satisfactory representation of the entire system is modeled, and appropriate computing resources are available to (1) analyze the model to discover invariants and (2) discover anomalies automatically. The fundamental problem for this approach is to empirically discover all the likely invariants of large complex software design and discover deviations from the invariants. The deviations may be described as bugs or software defects. But how hard is this problem to solve in a general manner? 12

13 Natural Law Invariants Schmidt s Research Empirically extracting natural law invariants (Hamiltonians, Lagrangians) has been performed (Schmidt, 2009), but can this method be scaled for software with possibly tens of thousands of invariants? Michael Schmidt and Hod Lipson, Published 3 April 2009, Science 324, 81 (2009) DOI: /Science

14 Extraction of Software Invariants and Isolation of Invariant Violations is Computationally Expensive Garey and Johnson in Computers and Intractability have compiled the best resource available on the topic of polynomial and exponential time complexity problems. Basically, polynomial time problems denoted as P can be solved in reasonable time higher-performance computing resources generally solve this class of problems faster. Exponential time problems belong in a different class (denoted as Non- Deterministic Polynomial Time Complete or NP-Complete) because they cannot be solved in polynomial time and, for that matter, any computing resource in a practical time. For example, a P time-complexity function n 3 where n=60 may be solved in 216 msec. The exponential time complexity function 3 n would take 1.3 x centuries. (M.R. Garey, 1979, p. 7) Some tools have been developed to detect likely invariants in software (Daikon (Ernst, 2007) and DIDUCE (Hangel, May 2002)) that work in P time for limited problem sets. Limitations occur due to the size of the problem space (with v variables thus take O(v n ) time and space). 14

15 Invariants and Satisfiability Formulation For the purpose of a software model, the software invariants may be reduced to a combination of Boolean expressions S 1, S 2, S 3 S n of the form S i = S i (x 1, x 2, x 3 x n ) such that the combination of S 1 + S 2 + S S n = TRUE FALSE. Results are TRUE only if no anomalies are present, FALSE otherwise. This type of problem falls into a class of NP-Hard Logic problems called SAT (Satisfiability) (M.R. Garey, 1979, p. 259). Although this is in the class of NP-Complete and likely intractable (assuming the worst case), there may be a P time and space solution reasonably close using Quantum Annealing methods. It is important to note that a practical quantum computer can converge on a solution in 2 n/2 steps at best no quantum computer can solve an NP- Complete problem. This means that we can only hope for a quadratic speedup, for an exact solution to a problem (C. Bennett, 1996). Quadratic speedup isn t too bad, and this approach to Validation and Verification is searching for likely invariants. 15

16 Determining Invariants To illustrate the possibility of using Quantum Annealing methods (using Quantum hardware), we produced a set of simple requirements with a well-defined set of invariants, in the form of written requirements and implemented in C++ code. These requirements must be re-mapped into equivalent Boolean expressions though this is rather painstaking to do manually, future MBD tools can perform this reduction in an automated fashion. Note that invariants can be captured in any form (requirements, software code, or drawings) that are capable of remapping into this form. Back Propagation Model illustrates the much more difficult problem of determining, for a given output state, what are the available input states that determine it? 16

17 Demonstrated Example Considering S 1 (x 1, x 2 ) =O 1 as matching the logic of a simple AND gate, then the reverse model must match the behavior of the AND gate. Without prior knowledge of this behavior, how can the back propagation determine the truth table S 1? This can be accomplished for this simple example through the use of an algorithm used in constraint mathematics called the penalty method, which allows solving for the constraints (invariants) through unconstrained convergence. Invariants are isolated by detecting the minima using an objective function (H and (x1, x2;z). Solving for all possible values yields the truth table. Other related methods Quadratic Unconstrained Binary Optimization (QUBO), Quadratically Signed Weight Enumerators (QWGT) Objective Function 17

18 Simulated Annealing The penalty method illustrates what needs to be achieved to determine global minima. Simulated annealing exploits a link between statistical mechanics (thermodynamic behavior of large numbers of particles such as metallic crystals) and combinatorial optimization This is similar to the annealing process in metallurgy, and the property that physical systems tend to seek a low energy state through a heating and cooling process that causes crystals to escape local minima to seek lower energy. Despite the intensive amount of interest in simulated annealing within industry and academia, and the most powerful supercomputer resources, simulated annealing cannot determine the invariants of a moderate-sized software project approaching P time. Solving satisfiability through Quantum Annealing physics outperforms solving the problem by software means or using supercomputers. 18

19 Adiabatic Quantum Computers Adiabatic Quantum Computers (AQC) can be used to solve NP-Hard problems faster than classical computers when solving Adiabatic Quantum Optimization (AQO) problems (based on projected data) 19

20 Key Technology Adiabatic Quantum Computers Based on Ising Spin Model key is to determine the ground state (lowest energy) of an Ising Spin system Encode a Hamiltonian for each satisfiability (S) condition by building the physical system that yields ground state (E<0) when S=true (system is bug free) and E>0 when S=False (there are bugs in the system) Getting to the ground state is NP hard which gives the answer with 100% certainty The closer we get to the ground state the more certain we are of bug-free behavior The fewer levels of the ground state can be a measure for the probability of errors and perhaps quantifiable relationship between the amount of testing time and the probability of remaining bugs. Ising Model Bias Coupling Energy Objective Function Quadratic Terms Variables Tunable parameters Software (Classical Machine) Hardware (Quantum Machine) Reference: Nature, May 2011, Quantum annealing with manufactured spins, M.W. Johnson et al, doi: /nature

21 Adiabatic Quantum Computers An Emerging Technology AQC have moved from Basic Research to Applied Research very recently (February 2011) Simulated Annealing problem is a SAT problem all problems reduce to a Boolean algebra problems and automated tools do not exist yet. Modeling the problem is still painstaking Qubits must increase to solve practical problems current state of the art is 128 qubits 21

22 Conclusion A rapid, heuristic method for performing Validation and Verification of software using quantum simulated annealing can outperform classical computing methods. Experimental evidence has shown that the AQC is suitable to solve the quadratic unconstrained binary optimization (QUBO) problems to determine likely software invariants Deviations from the software invariants (lowest energy state) may be flagged as errors in the software or software requirements. Deducing invariance is a heuristic process. Quantum outperforms classical computing. Hardware implementation outperforms software. Quantum hardware for SAT exists today pioneering efforts are underway 22

23