Cyber-Physical V&V Challenges for the Evaluation of State of the Art Model Checkers

Cyber-Physical V&V Challenges for the Evaluation of State of the Art Model Checkers Research in Quantum Enabled V&V Technology July 12-14, 2016 Chris Elliott Flight Controls / Quantum Computing

Overview I. Quantum Enabled V&V Overview II. Overview of 10 V&V Challenge Problems III. End to End Analysis Example IV. Summary 2

Quantum Enabled V&V What is it? QVTrace*: This technology is a method for Software Verification & Validation using Quantum Computer Assisted Formal Methods. Requirements And Implementation (Software Code) Quantum V&V Classical Computation Defects (Bugs) Req/Code Inconsistency Report to Designer D-Wave Adiabatic Quantum Computer Who will use it? Target Users are System/Software Design Teams interested in: - Reducing development costs - Improving final product quality *Product Developed by QRA Inc. 3

D-Wave Adiabatic Quantum Computer Current State-of-the-Art, DW-2X: 1152q Washington LM QA Solves a Quadratic Unconstrained Binary Optimization Problem 4

Quantum Optimization with Superconducting Qubits Quantum Superposition, Entanglement Enable Unique Optimization 5

QE-V&V Timeline Nov 2010 Early Access To QC Mar 2011 USC/ISI/LM Team for QC Center March 2016 QC Upgrade 1152 q DW2X Washington Early 2010 LM ID s Quantum as Key Tech Jan 2012 USC-LM QC Operational Mar 2013 QC Upgrade 128 q DW1 Rainier 512 q DW2 Vesuvius 6

Overview of Challenge Problems LM Aero Developed Set of 10 V&V Challenge Problems Goal: - Foster Collaboration in S5 Community (Ponder, Present, Publish) - Evaluate & Improve State-of-the-Art Formal Methods Toolsets Each Example in Package Includes: - Simulink Model Built in Matlab R2012B - Parameters, if any, for Simulating Model (.mat) - Documentation Containing Description and Requirements Difficult due to Transcendental Functions, Nonlinearities and Discontinuous Math, Vectors, Matrices, States Challenges Built with Commonly Used Blocks 7

Overview of Challenge Problems 1. Triplex Signal Monitor 2. Finite State Machine 3. Tustin Integrator 4. Control Loop Regulators 5. Nonlinear Guidance Algorithm 6. Feedforward Cascade Connectivity Neural Network 7. Abstraction of a Control Allocator (Effector Blender) 8. 6DOF with DeHavilland Beaver Autopilot* 9. System Safety Monitor 10. Euler Transformation Flight Control and Vehicle Management System Inspired Problems 8

1. Triplex Signal Monitor Description: this challenge problem involves the verification of a redundancy management system using quantum simulation techniques. The p Sensor A Sensor B Sensor C Online Monitoring Airborne Redundancy Management 9

2. Finite State Machine Description: this challenge problem involves the verification of a redundancy management system using quantum simulation techniques. The p Integrated Sensor Flight Control Embedded System Discrete Interwoven Modes in Integrated Cyber-Physical System 10

3. Tustin Integrator Description: this challenge problem involves the verification of a redundancy management system using quantum simulation techniques. The p Numerical Integration Fundamental Modeling and Simulation Component 11

4. Control Loop Regulators Description: this challenge problem involves the verification of a redundancy management system using quantum simulation techniques. The p PID Architecture Feedback Error Synthesis Kp Gain Ki Gain3 1 s Integrator Command Authority? Kd s s+1 Gain1 Transfer Fcn Attributes of Multi-Axis Control Law of Output Commands 12

5. Nonlinear Guidance Algorithm Description: this challenge problem involves the verification of a redundancy management system using quantum simulation techniques. The p Aim Point Validity? 3D Vector Mathematics for Outer Loop Intercept Guidance 13

5. Nonlinear Guidance Algorithm Block Types For NL Guidance Recent Focus on Import of Common Algorithmic Operators (Primitives) 14

6. Neural Network 1 Truth Model Description: this challenge problem involves the verification of a redundancy 0.6management system using quantum simulation techniques. The p 0.8 z 0.4 0.2 0-0.2 2 Network Topology 0.0-0.29313 0.29313-0.58626 0.58626-0.87938 0.87938-1.1725 1.1725-1.4656 1.4656-1.7588 1.7588-2.0519 1 0 y -1-2 -2-1.5-1 x -0.5 0 0.5 1 1.5 2 1-x Positive Negative Inputs Output Features? 2-y Inputs Layer 1 Layer2 Output 2x10x10x1 Feedforward Cascade Connectivity NN 15

1 trim_hdgref trim_turnknob trim_altref trim_pitchwheel boolean boolean boolean ID AD AD ID AP Eng HDG Mode ALT Mode HDG Ref Turn Knob ALT Ref Pitch Wheel Aileron Cmd Elevator Cmd Rudder Cmd AC Bus trim_flap trim_throttle trim_rudder Aileron Elevator Rudder Flap Throttle Rudder Trim EnvBus Controls DeHavilland Beaver model originally based on work created by Marc Rauw for Delft University of Technology, http://www.dutchroll.com and subsequently modified by the Mathworks http://www.mathworks.com/matlabcentral/fileexchange/ FLIGHT CONTROL Demonstration Author: elliocm Model Version: 1.80 Date: 21-Sep-2015 15:23:12 and More Description: this challenge problem involves the verification of a redundancy management system using quantum simulation techniques. The p Cyber-Physical V&V Challenge Problems LM Aeronautics Quantum Information Science Research Team 2015 Copyright 2015 Lockheed Martin Corporation Sensors Environment APeng Constant7 Data Type Conversion3 Data Type Conversion1 HDGmode Constant DeHavilland Beaver Airframe Constant1 Data Type Conversion2 Constant2 Constant3 HDGref Signal Conditioning Constant4 TurnKnob Constant5 Constant6 Autopilot 16

End to End Analysis (Tustin) Cyber-Physical V&V Challenge Problems LM Aeronautics Quantum Information Science Research Team 2015 Copyright 2015 Lockheed Martin Corporation Description: this challenge problem involves 1 the verification of xin Input Signal to Be Integrated a redundancy management system using quantum simulation techniques. The p cmd xin Discrete Time Step Top Limit 5 TL Bottom Limit Boolean Reset Initial Condition Upon Reset 2 reset 4 ic 3 T 6 BL T TL BL reset ic Tustin yout Integrator (Limited, Resettable, States) Definitions: Normal operation: the integrator is not in reset mode, and the output is within the specified limits (TL and BL). ypv: prior yout value xinpv: prior xin input value SP: Saturation Point 1 yout Output Signal Documentation Provides ICD, Definitions, and Requirements 17

End to End Analysis (Tustin) 3 TL TLc [TL] TL Description: this challenge problem involves Goto the verification of a redundancy management system using quantum simulation techniques. The p 4 BL BL BLc [BL] Goto1 bounds 1 z TUSTIN INTEGRATOR (LIMITED, RESETTABLE, STATES) Unit Delay1 [TL] From 6 ic up 5 ~= u y 1 reset lo Saturation yout 1 xin 1.5 Gain Switch [BL] Dynamic z Unit Delay From1 2 T Product 18

End to End Analysis (Tustin) Description: this challenge problem involves the verification of a redundancy management system using quantum simulation techniques. The p 2 BL 1 TL < TLc Relational Operator Switch1 ~= 1 ~= 2 BLc Switch2 19

End to End Analysis (Tustin) Requirements: 1. When Reset is True and the Initial Condition (ic) is bounded by the provided Top and Bottom Limits (BL <= ic <= TL), the Output (yout) shall equal the Initial Condition (ic). 2. The Output Detailed (yout) Formal shall be Property bounded Derivation: by the provided Top and Bottom limits (TL and BL) 3. When in normal # 1. When operation, Reset is the True output and the shall Initial be the Condition result of the (ic) equation, is yout = T/2*(xin + xinpv)+ # ypv bounded by the provided Top and Bottom Limits (BL<=ic<=TL), # the Output (yout) shall equal the Initial Condition (ic). 4. The Output of this function shall approximate the integration of the value of the input # If the Initial Condition is not bound by the Limits signal over # during time within a Reset, a specified the Output tolerance, shall equal defined the in saturation subtests below: a. After # 10 point seconds (nominally of Computation with TL>=BL, at an ic>=tl execution impl frequency SP==TL of and 10 ic<=bl hz, the impl Output SP==BL. should equal 10 within a +/- 0.1 tolerance, for a Constant Input (xin = 1.0), and the # Off-nominally with TL<BL, ic, ic>=bl impl SP==BL and ic<=tl impl sample SP==TL. delta time T = 0.1 seconds when in normal mode of operation. ((reset and ic<=tl and ic>=bl) impl yout == ic); #1a b. Over a 10 second computational duration at an execution frequency of 10 hz, ((reset and ic>=tl and ic>=bl and TL>=BL) impl yout == TL); #1b the Output ((reset and should ic<=bl equal and the ic>=bl sine and of time TL>=BL) t, sin(t), impl where yout == time BL); #1c is defined as a vector ((reset from and 0 to ic>=bl 10 by and increments ic<=tl and of TL<BL) 0.1 seconds impl yout within == BL); a +/- #1d 0.1 tolerance for ((reset and ic<=tl and ic>=bl and TL<BL) impl yout == TL); #1e an input equal to the cosine of time t, cos(t), with the sample delta time T = 0.1 seconds when in normal mode of operation. 21

End to End Analysis (Tustin) Requirements: Detailed Formal Property Derivation: 1. When Reset # Over is a True 10 second and the computational Initial Condition duration (ic) is at bounded an execution by the provided Analytic Top and vs Bottom frequency Limits (BL of <= 10 ic hz, <= the TL), Output the Output should (yout) equal shall the sine equal of the time Initial t, Condition Numerical (ic). sin(t), where time is defined as a vector from 0 to 10 by 2. The Output (yout) shall be bounded by the provided Top and Bottom limits (TL and BL) increments of 0.1 seconds within a +/- 0.1 tolerance for an 3. When input normal equal operation, to the cosine the of output time t, shall cos(t), be with the result the sample of the delta equation, yout = T/2*(xin 10 s + xinpv)+ time ypv T = 0.1 seconds when in normal mode of operation 4. The Output (xin{0}==1 of this and function xin{1}==0.995 shall approximate and xin{100}==-0.83907 the integration of and Tustin Error <.05 the value of the input T{all}==0.1 and reset{never} and (TL{all}>=BL{all}) and signal over (yout{all}>bl{all}) time within a and specified (yout{all}<tl{all})) tolerance, defined impl (abs(yout{0}- in subtests below: a. After 0)<=0.1 10 seconds and abs(yout{1}-0.099833)<=0.1 of Computation at an execution and frequency abs(yout{98}-- of 10 hz, the Output should 0.36648)<=0.1 equal 10 within and abs(yout{99}--0.45754)<=0.1 a +/- 0.1 tolerance, for a Constant and Input (xin = 1.0), and the 10 s abs(yout{100}--0.54402)<=0.1); sample delta time T = 0.1 seconds when in normal mode of operation. b. Over a 10 second computational duration at an execution frequency of 10 hz, the Output should equal the sine of time t, sin(t), where time is defined as a vector from 0 to 10 by increments of 0.1 seconds within a +/- 0.1 tolerance for an input equal to the cosine of time t, cos(t), with the sample delta time T = 0.1 seconds when in normal mode of operation. Requirements Properties (Tests) is At Least Half the Challenge 22

End to End Analysis (Triplex) Signal A Description: this challenge problem involves the verification of a redundancy management system using quantum simulation techniques. The p Signal B Signal C Threshold Level Fault Code Persistence Limit (Duration Trigger) FC: 0-nofail, 1-branchC, 2-branchB, 4-branchA # detailed formal property (abs(ia{all}-ib{all})>tlevel{all} or abs(ia{all}- ic{all})>tlevel{all} and PC>PClimit and PClimit{all}==1 and Tlevel{all}==1) impl (FC{3}==4); Given These Conditions, Prove the Correct Fault Report 23

10000 inputs 5000 ia ib ic 0 0 0.5 1 1.5 2 2.5 3 t [sec] 1 0.5 FC 0-0.5-1 0 0.5 1 1.5 2 2.5 3 t [sec] Counter Example Data As a Test Harness to Model 26

inputs 10000 5000 ia ib ic FC 1 0.5 0-0.5 0 0 1 2 3 t [sec] -1 0 1 2 3 t [sec] 4 inputs 3 2 1 0 0 1 2 3 t [sec] ia-ib ia-ic ib-ic This is a Valid Defect Discovered By QVTrace v0.9.1 Closer Inspection Yields a Problem 27

2 inputs 0-2 ia ib ic -4 0 1 2 3 4 5 6 7 8 9 10 t [sec] 2 1.5 Nominal Behavior FC 1 0.5 0 0 1 2 3 4 5 6 7 8 9 10 t [sec] FC: 0-nofail, 1-branchC, 2-branchB, 4-branchA 28

2 inputs 0-2 ia ib ic -4 0 1 2 3 4 5 6 7 8 9 10 t [sec] 1 0.5 Faulty Behavior FC 0-0.5-1 0 1 2 3 4 5 6 7 8 9 10 t [sec] FC: 0-nofail, 1-branchC, 2-branchB, 4-branchA 29

Summary and Path Forward Round 1 V&V Challenge Problems In Use to Develop Novel QE-V&V Requirements Formalization is Difficult Alone and Reduces Defects - Requirements Properties (Tests) is At Least Half the Challenge - Beneficial to Front Load Design Process with Formalization - Need Near if Not Equivalent Primitives Capability in Properties - Interested in Deploying Challenges Requirements to SPeAR Goals: - Publish Results on Current Round of Challenges - Round 2 V&V Challenge Problems To Increase Complexity Further - Transition Formal Methods Analysis Process/Tools to Programs - Interested? Contact: Chris Elliott, christopher.m.elliott@lmco.com, 817-935-3054 Thank You 30

Biography slide Mr. Greg Tallant Program Manager and LM Fellow Lockheed Martin Aeronautics Skunk Works Dr. Edward H. Ned Allen Chief Scientist and LM Senior Fellow Lockheed Martin Corporation Mr. Peter Stanfill Quantum Apps Team Lockheed Martin Aeronautics Skunk Works Dr. Kristen Pudenz Quantum Apps Team Lockheed Martin Aeronautics Skunk Works Chris Elliott Quantum Apps Team Lockheed Martin Aeronautics Skunk Works 32