On Example Models and Challenges Ahead for the Evaluation of Complex Cyber-Physical Systems with State of the Art Formal Methods V&V Research in Quantum Enabled V&V Technology June 9-11, 2015 Chris Elliott Flight Controls / Quantum Computing
Not Observable Late Defects Improved V&V Path Challenge to Quantify Savings wrt Non-Observable 2
Our Timeline to Optimize 3
Motivation for New Tools Consider 9 Phases of Development (SSS, SRS, SDD, OFP, FT, etc) P9 P1 P8 P2 P7 P3 P6 P4 P5 4
Engineering Time [Hrs] Motivation for New Tools Assume a Crude Model for Level of Effort for Design Team Engineering Attention: Exponential as a function of Phase, p Attention ~ 2 p 1 expert focuses for a couple hours on given design at phase 1 Releases Design for Team Review Attention On Problem Increases) Phase III ~ 4 Phase IV ~ 8 Phase IX ~ 512 hours 600 500 400 300 200 100 Designer Focuses on a Prelim Design for an Hour 0 0 2 4 6 8 10 # Design Phase Total Iterations Flight Test Team 25 experts validate ~2 days of sorties 5
Engineering Time [Hrs] Motivation for New Tools Assume Anomaly Detection at Any Phase Reinitiates Process Back to the Preliminary Drawing Board Phase 1 Engineering Time Continues to Accumulate Through Anomaly Burndown Model Anomaly Detections at Each Phase to Understand Impact 1200 1000 800 600 400 200 Nominal P1 SPAR P2 SPAR P3 SPAR P4 SPAR P5 SPAR P6 SPAR P7 SPAR P8 SPAR P9 SPAR 0 0 5 10 15 20 # Design Phase Total Iterations 6
Engineering Time [Hrs] Product Delayed [Weeks] Engineering Time [Hrs] Engineering Time [Hrs] Engineering Time [Hrs] Motivation for New Tools 1000 500 Perfect P1 P2 P3 0 0 5 10 15 20 # Design Phase Total Iterations 1100 1000 900 800 700 600 Finding SPARS in the final Design phases may nearly Double Costs! 1000 500 1000 P4 P5 P6 P7 0 0 5 10 15 20 # Design Phase Total Iterations 500 P8 P9 FT SPAR Discovered! 500 0 2 4 6 8 10 Anamoly Discovered at Phase # 15 10 5 Notional time delay assuming quadratic growth in phase durations 0 0 5 10 15 20 # Design Phase Total Iterations 0 0 2 4 6 8 10 Anamoly Discovered at Phase # We Must Avoid Doc Brown s Alternate 1985 7
Motivation for New Tools Advanced Systems: On Board Intelligence Real Time Adaptation New Sensor Technology Emerging Complexity Driving Need for Novel V&V Practice T-50 F-22 Prog-D JSF ISR Combat UAV Inter-System Communication & Dependencies UAV - Projected F-35 Est F-16 IDF YF-22 F-22 JSF CDA Block 60 System Complexity Exceeding Advancement of V&V Practice 8
Discussion Layout Quantum Enabled V&V Overview A Few Challenge Problems of Interest A Nonlinear Guidance Example Neural Networks Emergent Closed Loop Behavior Path Forward 9
Quantum Enabled V&V What is it? QVTrace*: This technology is a method for Software Verification & Validation using Quantum Computer Assisted Formal Methods. Requirements And Implementation (Software Code) Quantum V&V Classical Computation Defects (Bugs) Req/Code Inconsistency Report to Designer D-Wave Adiabatic Quantum Computer Who will use it? Target Users are System/Software Design Teams interested in: - Reducing development costs - Improving final product quality *Product Developed by Quantum Research Analytics 10
D-Wave Adiabatic Quantum Computer Current State-of-the-Art DW-2: 512-qubit Vesuvius Processor 11
Quantum Enabled V&V Overview Implementation Requirements SMT Instance False = Defect Space Satisfiability (All-SAT) Modulo Theory (Number Domain) Quantum Computer Ising (Binary Optimization) No Feasible? Consistent Yes Reqs/Implementation Defect Detected Inconsistency in Reqs/Implementation 12
Nonlinear Guidance Example Tangent Aim Point #1 Relative Position Vector to Tangent Aim Point #1 Standoff 1 UAV UAV Velocity Vector Relative Position Vector Relative Position Vector to Tangent Aim Point #2 Target Standoff 2 Target Velocity Vector Eta Plane Containing Relative Position and UAV Velocity Vector Tangent Aim Point #2 Derived from a 3D Collision Cone Approach used in a Reactive Obstacle Avoidance Paper, Reference [1] AIAA 2010-7729 Chawla and Padhi Goal: Compute Aim Points as a function of Observed Positions and Velocities 13
Nonlinear Guidance Example X ap1 r 1 d u 1 V X TARGET X V X R r 2 X ap2 d u 2 Eta Plane Containing X R and V Ref [1] 14
Nonlinear Guidance Example Note Substituting into (1) Expanding (1) Writing Unit Vector in components of UAV Velocity and Relative Position Vector (2) (3) Ref [1] 15
Nonlinear Guidance Example Using (1) and (2), find Beta (1) (2) Recalling (3) Ref [1] 16
Nonlinear Guidance Example Continue to Solve for Beta, Ref [1] 17
Nonlinear Guidance Example Finally, unit vector and relative position vectors are found which allows computation of Aim Points Recall Ref [1] 18
Nonlinear Guidance Example Two Aim Points were Computed Which one is selected? Chawla and Padhi paper resolves UAV Velocity Vector into relative position components to choose more efficient collision avoidance. e.g. We desire a Counter Clockwise Loiter Interception for Surveillance off the Port Wing (CCW when viewing Ground Plane from above) Sign of k th component in cross product inspected for CCW rotation Ref [1] 19
Nonlinear Guidance Example Estimate UAV Velocity Vector and Magnitude Target Velocity Vector and Magnitude Compute Relative Position Standoff 1 and Standoff 2 Relative Position Vectors from Target Location Relative Position Vector to Tangent Aim Point #1 and Tangent Aim Point #2 Aim Point Resulting in CCW Standoff Loiter Rotation One Factor at a Time Approach ODE Solver Scope A Dynamic Target Reveals Abhorrent Behavior 20
Nonlinear Guidance Example Theory Model Experiment Derive Build Understand (OFAT, Formal Methods) Iterate V Max DRIVE2 LAT DRIVE2 LON Target Inertial Position TARGET Inner Loop, 1 st Order Airframe Approx MATLAB FUNCTION UNDER TEST Nonlinear Geometric Guidance FLY2 LAT FLY2 LON V Constant Outer Loop Psi Inner Loop, 1 st Order Airframe Approx UAV Inertial Position Air Vehicle Winds (Disturbance) Formal Methods Powerful for Early Designer s Arsenal 21
Nonlinear Guidance Example Notional Requirement Theory Experiment Assess Mature Requirement Design Test Certify (Repeat) Counter- Example Formal Methods Property 22
Neural Network Example Inputs Exhaustive Input Sweep 1-x ODE Solver Network Topology 0.0-0.29313 0.29313-0.58626 0.58626-0.87938 0.87938-1.1725 1.1725-1.4656 1.4656-1.7588 1.7588-2.0519 Positive Negative Scope Inspecting Output for Forward Analysis Not Always Trivial, Especially for Higher Order Systems e.g. (2D case depicted) 2-y Counter- Example Inputs Layer 1 Layer2 Output Formal Methods Property 23
Neural Network Example Is the NN output correctly bounded? Counter- Example Formal Methods Property Property Proving Exceeds Time Limit (5 mins) 24
Neural Network Example QVTRACE 25
Neural Network Example QVTRACE 26
Emergent Closed Loop Behaviors Wing Leveler Controller Φ Can Formal Methods Identify this Preliminary Design Flaw? 27
Path Forward LM Aero Developing Set of Examples to Evaluate and Improve Formal Methods Toolsets for Early-Design Phase Algorithms of Interest Involve: - Nonlinear Arithmetic - Real Number Data Types and Transcendental Functions - Vector and Matrix Operations - Complex State Machines - Heterogeneous Discrete Systems Goal is to Transition Formal Methods to Early-Design Teams CHRIS ELLIOTT, christopher.m.elliott@lmco.com, 817-935-3054 28
29
State Machine Example 30