Scalable Underapproximative Verification of Stochastic LTI Systems using Convexity and Compactness. HSCC 2018 Porto, Portugal

Transcription

1 Scalable Underapproximative Verification of Stochastic LTI Systems using ity and Compactness Abraham Vinod and Meeko Oishi Electrical and Computer Engineering, University of New Mexico April 11, 2018 HSCC 2018 Porto, Portugal Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 1 / 16

2 Motivation Real world problems stochastic and high-dimensional Motion planning in stochastic environments transportation Human-automation collaboration systems biomedical systems Need for probabilistic guarantees of safety and performance Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 2 / 16

3 Motivation Target Safe set Stochastic reach-avoid problem viability and reachability maximize subject to P{stay safe and reach target at the time horizon} dynamics, initial state, policy constraints Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 2 / 16

4 Motivation Target P θ Safe set Stochastic reach-avoid problem viability and reachability maximize subject to P{stay safe and reach target at the time horizon} dynamics, initial state, policy constraints Stochastic reach-avoid set acceptable initial states P θ High computation costs and lacks scalability Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 2 / 16

5 Motivation Target P θ Safe set Stochastic reach-avoid problem viability and reachability maximize subject to P{stay safe and reach target at the time horizon} dynamics, initial state, policy constraints Stochastic reach-avoid set acceptable initial states P θ High computation costs and lacks scalability Objective: Compute a polytopic underapproximation of P θ Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 2 / 16

6 Main contributions Target P θ Safe set Polytopic underapproximation of stochastic reach-avoid set Open-loop underapproximation compact sets Sufficient conditions for closed, compact, and convex Stochastic reach-avoid set Open-loop underapproximation Admittance of optimal bang-bang Markov policies Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 3 / 16

7 Related work Backward stochastic reachability Abate, Amin, Prandini, Lygeros, & Sastry (2007); Abate, Prandini, Lygeros, & Sastry (2008); Summers, & Lygeros (2010) Approximation techniques Lesser, Oishi, & Erwin (2013); Manganini, Pirotta, Restelli, Piroddi, & Prandini (2015); Kariotoglou, Kamgarpour, Summers, & Lygeros (2017); Gleason, Vinod, & Oishi (2017); Vinod & Oishi (2017) Forward stochastic reachability Lasota & Mackey (1985); Althoff, Stursberg, & Buss (2009); Vinod, HomChaudhuri, & Oishi (2017); HomChaudhuri, Vinod, & Oishi (2017) Uncertain system reachability (discrete time) Bertsekas and Rhodes (1971); Girard (2005); Kurzhanskiy & Varaiya (2006); Kvasnica, Takács, Holaza, & Ingole (2015); Althoff (2015); Bak & Duggirala (2017) Uncertain system reachability (continuous time) Tomlin, Mitchell, Bayen, & Oishi (2003); Bokanowski, Forcadel, & Zidani (2010); Huang, Ding, Zhang, & Tomlin (2015); Chen, Herbert, & Tomlin (2017) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 4 / 16

8 System formulation Discrete-time LTI system (time horizon N and initial state x 0 X ) x k+1 = Ax k + Bu k + w k x k X = R n, u k U R m w k W R p, w k ψ w ( ) Continuous state Markov decision process (when IID noise) X = [x 1 P π,x X 0 P ρ,x X 0 Q(dy x, u) = ψ w (y Ax Bu)dy... x N ] X N random vector with probability measure Markov (closed-loop) policy π : N [0,N 1] X U, π M Open-loop policy ρ : X U N, ρ(x 0 ) M Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 5 / 16

9 Stochastic reach-avoid problem and its solution S, T X Terminal stochastic reach-avoid problem P π,x { ( V X 0 X S N 1 T )} 0 (x 0) = max. π s. t. π M Solution via dynamic programming VN(x) = 1 T (x) Vk (x) = sup 1 S (x) u U X V k+1(y)q(dy x, u) Discretization approach For compact U, S, T and Lipschitz Q Curse of dimensionality n 3 S π 2 x 0 x 0 S x 1 S x 2 S T π. x N 1 S x N T X S N 1 T X Abate, Amin, Prandini, Lygeros, and Sastry, HSCC 2007 Summers and Lygeros, Automatica 2010 Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 6 / 16

10 Stochastic reach-avoid problem and its solution S, T X Terminal stochastic reach-avoid problem [( N 1 ) ] V0 (x 0) = max. E π x 0 k=0 1 S(x k ) 1 T (x N ) π s. t. π M Solution via dynamic programming VN(x) = 1 T (x) Vk (x) = sup 1 S (x) u U X V k+1(y)q(dy x, u) Discretization approach For compact U, S, T and Lipschitz Q Curse of dimensionality n 3 S π 2 x 0 x 0 S x 1 S x 2 S T π. x N 1 S x N T X S N 1 T X Abate, Amin, Prandini, Lygeros, and Sastry, HSCC 2007 Summers and Lygeros, Automatica 2010 Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 6 / 16

11 Stochastic reach-avoid problem and its solution S, T X Terminal stochastic reach-avoid problem [( N 1 ) ] V0 (x 0) = max. E π x 0 k=0 1 S(x k ) 1 T (x N ) π s. t. π M Solution via dynamic programming VN(x) = 1 T (x) Vk (x) = sup 1 S (x) u U X V k+1(y)q(dy x, u) Discretization approach For compact U, S, T and Lipschitz Q Curse of dimensionality n 3 S π 2 x 0 T π X Abate, Amin, Prandini, Lygeros, and Sastry, HSCC 2007 Summers and Lygeros, Automatica 2010 Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 6 / 16

12 Underapproximation of the stochastic reach-avoid problem Stochastic reach-avoid problem { } max. P π,x 0 X X (S N 1 T ) s.t. π M Underapproximation max. P ρ,x 0 X s.t. { X (S N 1 T ) ρ(x 0 ) U N Optimal value function V0 (x 0) Optimal value function W0 (x 0) Search over Markov policies Search over open-loop policies } Vinod and Oishi, LCSS 2017 Vinod, HomChaudhuri, and Oishi, HSCC 2017 Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 7 / 16

13 Underapproximation of the stochastic reach-avoid problem Stochastic reach-avoid problem { } max. P π,x 0 X X (S N 1 T ) s.t. π M Underapproximation max. S N 1 T ψ X(X; ρ, x 0 )dx s.t. ρ(x 0 ) U N Optimal value function V0 (x 0) Optimal value function W0 (x 0) Search over Markov policies Search over open-loop policies Vinod and Oishi, LCSS 2017 Vinod, HomChaudhuri, and Oishi, HSCC 2017 Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 7 / 16

14 Underapproximation of the stochastic reach-avoid problem Stochastic reach-avoid problem { } max. P π,x 0 X X (S N 1 T ) s.t. π M Underapproximation max. S N 1 T ψ X(X; ρ, x 0 )dx s.t. ρ(x 0 ) U N Optimal value function V0 (x 0) Optimal value function W0 (x 0) Search over Markov policies Search over open-loop policies Fourier transform IID noise Ψ W = Ψ w ψ W Ψ W Ψ w X = A x 0 + H ρ(x 0 ) + G W Fourier transform ψ X ( ; ρ, x 0 ) Inverse Fourier transform Ψ X ( ; ρ, x 0 ) ψ w W 0 (x 0) V 0 (x 0) a.s. Quadrature ψ(x; ρ, x 0 )dx + patternsearch Vinod and Oishi, LCSS 2017 Vinod, HomChaudhuri, and Oishi, HSCC 2017 Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 7 / 16

15 Underapproximation of stochastic reach-avoid sets Stochastic reach-avoid set and its underapproximation, θ [0, 1] L π (θ, S, T ) = {x 0 X : V0 (x 0 ) θ} K ρ (θ, S, T ) = {x 0 X : W0 (x 0 ) θ} L π (θ, S, T ) compact set tight polytopic underapproximation W 0 (x 0) V 0 (x 0) a.s. Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 8 / 16

16 Underapproximation of stochastic reach-avoid sets Stochastic reach-avoid set and its underapproximation, θ [0, 1] L π (θ, S, T ) = {x 0 X : V0 (x 0 ) θ} K ρ (θ, S, T ) = {x 0 X : W0 (x 0 ) θ} L π (θ, S, T ) compact set tight polytopic underapproximation W 0 (x 0) V 0 (x 0) a.s. Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 8 / 16

17 Underapproximation of stochastic reach-avoid sets Stochastic reach-avoid set and its underapproximation, θ [0, 1] L π (θ, S, T ) = {x 0 X : V0 (x 0 ) θ} K ρ (θ, S, T ) = {x 0 X : W0 (x 0 ) θ} L π (θ, S, T ) compact set tight polytopic underapproximation W 0 (x 0) V 0 (x 0) a.s. Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 8 / 16

18 Underapproximation of stochastic reach-avoid sets Stochastic reach-avoid set and its underapproximation, θ [0, 1] L π (θ, S, T ) = {x 0 X : V0 (x 0 ) θ} K ρ (θ, S, T ) = {x 0 X : W0 (x 0 ) θ} L π (θ, S, T ) compact set tight polytopic underapproximation W 0 (x 0) V 0 (x 0) a.s. Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 8 / 16

19 Underapproximation of stochastic reach-avoid sets Stochastic reach-avoid set and its underapproximation, θ [0, 1] L π (θ, S, T ) = {x 0 X : V0 (x 0 ) θ} K ρ (θ, S, T ) = {x 0 X : W0 (x 0 ) θ} L π (θ, S, T ) compact set tight polytopic underapproximation W 0 (x 0) V 0 (x 0) a.s. Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 8 / 16

20 Underapproximation of stochastic reach-avoid sets Stochastic reach-avoid set and its underapproximation, θ [0, 1] L π (θ, S, T ) = {x 0 X : V0 (x 0 ) θ} K ρ (θ, S, T ) = {x 0 X : W0 (x 0 ) θ} L π (θ, S, T ) compact set tight polytopic underapproximation W 0 (x 0) V 0 (x 0) a.s. Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 8 / 16

21 Underapproximation of stochastic reach-avoid sets Stochastic reach-avoid set and its underapproximation, θ [0, 1] L π (θ, S, T ) = {x 0 X : V0 (x 0 ) θ} K ρ (θ, S, T ) = {x 0 X : W0 (x 0 ) θ} L π (θ, S, T ) compact set tight polytopic underapproximation W 0 (x 0) V 0 (x 0) a.s. Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 8 / 16

22 Underapproximation of stochastic reach-avoid sets Stochastic reach-avoid set and its underapproximation, θ [0, 1] L π (θ, S, T ) = {x 0 X : V0 (x 0 ) θ} K ρ (θ, S, T ) = {x 0 X : W0 (x 0 ) θ} L π (θ, S, T ) compact set tight polytopic underapproximation W 0 (x 0) V 0 (x 0) a.s. Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 8 / 16

23 Underapproximation of stochastic reach-avoid sets Stochastic reach-avoid set and its underapproximation, θ [0, 1] L π (θ, S, T ) = {x 0 X : V0 (x 0 ) θ} K ρ (θ, S, T ) = {x 0 X : W0 (x 0 ) θ} L π (θ, S, T ) compact set tight polytopic underapproximation W 0 (x 0) V 0 (x 0) a.s. Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 8 / 16

24 Problem statements Q1 Compute a tight polytopic underapproximation of K ρ (θ, S, T ) in a grid-free manner Q2 When are L π (θ, S, T ) and K ρ (θ, S, T ) convex and compact? Log-concavity and upper semicontinuity of V 0 (x 0) and W 0 (x 0) Admittance of bang-bang optimal Markov policies Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 9 / 16

25 Tight polytopic underapproximation of K ρ (θ, S, T ) V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) upper semi-continuity Closed Closed Compact θ (0, 1] Compact S T Q( x, u) U Continuous Compact Closed Continuous Compact β > W 0 (x max) > α > 0 K ρ (β, S, T ) = D bisections to ɛ accuracy Runtime: O( D log( ɛ )) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 10 / 16

26 Tight polytopic underapproximation of K ρ (θ, S, T ) V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) upper semi-continuity Closed Closed Compact θ (0, 1] Compact S T Q( x, u) U Continuous Compact Closed Continuous Compact β > W 0 (x max) > α > 0 K ρ (β, S, T ) = D bisections to ɛ accuracy Runtime: O( D log( ɛ )) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 10 / 16

27 Tight polytopic underapproximation of K ρ (θ, S, T ) V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) upper semi-continuity Closed Closed Compact θ (0, 1] Compact S T Q( x, u) U Continuous Compact Closed Continuous Compact β > W 0 (x max) > α > 0 K ρ (β, S, T ) = D bisections to ɛ accuracy Runtime: O( D log( ɛ )) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 10 / 16

28 Tight polytopic underapproximation of K ρ (θ, S, T ) V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) upper semi-continuity Closed Closed Compact θ (0, 1] Compact S T Q( x, u) U Continuous Compact Closed Continuous Compact β > W 0 (x max) > α > 0 K ρ (β, S, T ) = D bisections to ɛ accuracy Runtime: O( D log( ɛ )) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 10 / 16

29 Tight polytopic underapproximation of K ρ (θ, S, T ) V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) upper semi-continuity Closed Closed Compact θ (0, 1] Compact S T Q( x, u) U Continuous Compact Closed Continuous Compact β > W 0 (x max) > α > 0 K ρ (β, S, T ) = D bisections to ɛ accuracy Runtime: O( D log( ɛ )) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 10 / 16

30 Tight polytopic underapproximation of K ρ (θ, S, T ) V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) upper semi-continuity Closed Closed Compact θ (0, 1] Compact S T Q( x, u) U Continuous Compact Closed Continuous Compact β > W 0 (x max) > α > 0 K ρ (β, S, T ) = D bisections to ɛ accuracy Runtime: O( D log( ɛ )) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 10 / 16

31 Tight polytopic underapproximation of K ρ (θ, S, T ) V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) upper semi-continuity Closed Closed Compact θ (0, 1] Compact S T Q( x, u) U Continuous Compact Closed Continuous Compact β > W 0 (x max) > α > 0 K ρ (β, S, T ) = D bisections to ɛ accuracy Runtime: O( D log( ɛ )) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 10 / 16

32 Tight polytopic underapproximation of K ρ (θ, S, T ) V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) upper semi-continuity Closed Closed Compact θ (0, 1] Compact S T Q( x, u) U Continuous Compact Closed Continuous Compact β > W 0 (x max) > α > 0 K ρ (β, S, T ) = Grid-free, parallelizable, tight polytopic underapproximation! Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 10 / 16

33 Special case: Bang-bang optimal Markov policies V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) Avoid T at N: Reach X \ T at N, S X, and U = convexhull(u vertices ) S T Q( x, u) U Continuous Compact u 1 T u 1 T x 0 u 2 X x 0 u 2 X Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 11 / 16

34 Special case: Bang-bang optimal Markov policies V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) Avoid T at N: Reach X \ T at N, S X, and U = convexhull(u vertices ) S T Q( x, u) U Continuous Compact V0 (x 0 ) = sup P π,x X 0{x N X \ T } π M = 1 inf π M Pπ,x 0 X {x N T } 1 V 0,bang(x) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 11 / 16

35 Special case: Bang-bang optimal Markov policies V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) Avoid T at N: Reach X \ T at N, S X, and U = convexhull(u vertices ) S T Q( x, u) U Continuous Compact V0 (x 0 ) = sup P π,x X 0{x N X \ T } π M = 1 inf π M Pπ,x 0 X {x N T } 1 V 0,bang(x) V N,bang(x) = 1 T (x) V k,bang(x) = inf u U X V k+1(y)q(dy x, u) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 11 / 16

36 Special case: Bang-bang optimal Markov policies V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) Avoid T at N: Reach X \ T at N, S X, and U = convexhull(u vertices ) S T Q( x, u) U Continuous Compact V0 (x 0 ) = sup P π,x X 0{x N X \ T } π M = 1 inf π M Pπ,x 0 X {x N T } 1 V 0,bang(x) V N,bang(x) = 1 T (x) V k,bang(x) = inf u U vertices X V k+1(y)q(dy x, u) exp ( x ) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 11 / 16

37 Special case: Bang-bang optimal Markov policies V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) Avoid T at N: Reach X \ T at N, S X, and U = convexhull(u vertices ) S T Q( x, u) U Continuous Compact V0 (x 0 ) = sup P π,x X 0{x N X \ T } π M = 1 inf π M Pπ,x 0 X {x N T } 1 V 0,bang(x) V N,bang(x) = 1 T (x) V k,bang(x) = inf u U vertices X V k+1(y)q(dy x, u) ( log exp ( x )) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 11 / 16

38 Special case: Bang-bang optimal Markov policies V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) Avoid T at N: Reach X \ T at N, S X, and U = convexhull(u vertices ) S T Q( x, u) U Continuous Compact V0 (x 0 ) = sup P π,x X 0{x N X \ T } π M = 1 inf π M Pπ,x 0 X {x N T } 1 V 0,bang(x) V N,bang(x) = 1 T (x) V k,bang(x) = inf u U vertices X V k+1(y)q(dy x, u) ( log exp minimize exp ( )) x ( x s.t x [2.5, 6] ) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 11 / 16

39 Special case: Bang-bang optimal Markov policies V 0 (x) and W 0 (x) Property Lπ (θ) and K ρ (θ) Avoid T at N: Reach X \ T at N, S X, and U = convexhull(u vertices ) S T Q( x, u) U Continuous Compact V0 (x 0 ) = sup P π,x X 0{x N X \ T } π M = 1 inf π M Pπ,x 0 X {x N T } 1 V 0,bang(x) V N,bang(x) = 1 T (x) V k,bang(x) = inf u U vertices X V k+1(y)q(dy x, u) ( log exp minimize exp ( )) x ( x s.t x [2.5, 6] ) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 11 / 16

40 Chain of integrators n = 2 (4 6 min. vs 34 min.) n = 40 (16 31 min.) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 12 / 16

41 Satellite docking problem Clohessy-Wiltshire-Hill dynamics } x t+1 = Ax t + Bu t + w t ẍ 3ωx 2ωẏ = u 1 x ÿ + 2ωẋ = u t = [x t y t ẋ t ẏ t] 2 w t N (0, Σ w) Stay within line-of-sight cone and reach target ẋ 0 = 0 ẏ 0 = 0 ẋ 0 = 0.1 ẏ 0 = 0.1 Compute time (min.) Proposed method Chance const. (CDC 13) Lagrangian (CDC 17) Dyn. prog. (HSCC 07) Value of v (ẋ 0 = ẏ 0 = v) Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 13 / 16

42 Conclusion Summary Polytopic underapproximation for the stochastic reach-avoid set Scalable, grid-free, parallelizable Sufficient conditions for Closed, compact, convex sets Optimal bang-bang policies Future work Mitigate noisy optimization Feedback-based underapproximation Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 14 / 16

43 Acknowledgement This work was supported by the following grants: NSF CMMI (CAREER, Oishi) CNS Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 15 / 16

44 Conclusion Summary Polytopic underapproximation for the stochastic reach-avoid set Scalable, grid-free, parallelizable Sufficient conditions for Closed, compact, convex sets Optimal bang-bang policies Future work Mitigate noisy optimization Feedback-based underapproximation Vinod and Oishi Scalable Underapproximative Verification of Stochastic LTI Systems 16 / 16