Controlling probabilistic systems under partial observation an automata and verification perspective

Transcription

1 Controlling probabilistic systems under partial observation an automata and verification perspective Nathalie Bertrand, Inria Rennes, France Uncertainty in Computation Workshop October 4th 2016, Simons Institute, Berkeley October 4th 2016 Simons Institute, Berkeley

2 Partially observable probabilistic systems Why probabilities? randomized algorithms, unpredictable behaviours, abstraction of non-determinism Why partial observation? abstraction of large systems, security concerns Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 2

3 Partially observable probabilistic systems Why probabilities? randomized algorithms, unpredictable behaviours, abstraction of non-determinism Why partial observation? abstraction of large systems, security concerns this talk: known automaton-like model b,1/2 c a q0 f,1/2 u,1/2 f1 q1 a,1/2 c,1/2 f2 q2 a,1/3 1 a,1/3 a,1/3 2 3 c c b b W L b,1/2 c a Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 2

4 Partially observable probabilistic systems Why probabilities? randomized algorithms, unpredictable behaviours, abstraction of non-determinism Why partial observation? abstraction of large systems, security concerns this talk: known automaton-like model b,1/2 c a q0 f,1/2 u,1/2 f1 q1 a,1/2 c,1/2 f2 q2 a,1/3 1 a,1/3 a,1/3 2 3 c c b b W L b,1/2 c a language-theoretic questions: languages defined by prob. automata monitoring issues: fault diagnosis, supervision, etc. control problems: optimization for a given objective Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 2

5 Outline Probabilistic automata Partially observable MDP Discussion Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 3

6 Motivating example for probabilistic automata (PA) Planning holidays in advance: 1. choose an airline type (lowcost/highcost); 2. book accommodation (internet/phone); 3. choose tour (seeall/missnothing). each action fails with some probability 1 start 3 4 lowcost+1highcost 4 lowcost airport 3 4 internet+ 1 2 phone fail 1 4 internet+ 1 2 phone hotel seeall+ 7 4 seeall+ 1 8 missnothing 8 missnothing success success probability of plan lowcost internet seeall is Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 4

7 Control strategies in PA Strategies are words what is the probability to reach a final state after word w? The acceptance probability of w = a 1... a n by A is: Pr A (w) = π 0 [q] ( n ) P ai [q, q ] = π 0 P w 1 T F q Q q F i=1 Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 5

8 Control strategies in PA Strategies are words what is the probability to reach a final state after word w? The acceptance probability of w = a 1... a n by A is: Pr A (w) = π 0 [q] ( n ) P ai [q, q ] = π 0 P w 1 T F q Q q F i=1 Optimal strategies may not exist b 1a + 1 q 0 q 2 b a + 1b 1 2 a Pr A (a 1... a n ) = n 2 i n 1 1 ai =b i=1 Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 5

9 Control strategies in PA Strategies are words what is the probability to reach a final state after word w? The acceptance probability of w = a 1... a n by A is: Pr A (w) = π 0 [q] ( n ) P ai [q, q ] = π 0 P w 1 T F q Q q F i=1 Optimal strategies may not exist b 1a + 1 q 0 q 2 b a + 1b 1 2 a n Pr A (a 1... a n ) = 2 i n 1 1 ai =b Find good enough strategies, i.e. that guarantee a given probability i=1 Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 5

10 Existence of good-enough strategies L θ (A) = {w A Pr A (w) θ} The problem, given a PA A of telling whether L 1 (A) is 2 undecidable. Paz 71 Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 6

11 Existence of good-enough strategies L θ (A) = {w A Pr A (w) θ} The problem, given a PA A of telling whether L 1 (A) is 2 undecidable. Paz 71 Undecidability is robust refined emptiness assuming that for ɛ > 0 either w Pr A (w) 1 ε or w Pr A (w) < ε, decide which is the case Condon et al. 03 value one problem does there exist (w n ) n N such that lim sup n Pr A (w n ) = 1? Gimbert and Oualhadj 10 parametric probability values does there exist a valuation of probabilities such that A has value one? Fijalkow et al. 14 Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 6

12 Anything decidable? Almost-sure language: L =1 (A) Emptiness of almost-sure language is PSPACE-complete. equivalent to universality problem for NFA Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 7

13 Anything decidable? Almost-sure language: L =1 (A) Emptiness of almost-sure language is PSPACE-complete. Quantitative language equivalence Input: A and A PA Output: yes iff w A Pr A (w) = Pr A (w) equivalent to universality problem for NFA Quantitative language equivalence is decidable in PTIME. Schützenberger 61, Tzeng 92 linear algebra argument polynomial bound on length of counterexample to equivalence Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 7

14 Recap on Probabilistic Automata Partial observation: the plan must be decided in advance! Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 8

15 Recap on Probabilistic Automata Partial observation: the plan must be decided in advance! model of system is known the effect of a plan can be computed: after word w, probability distribution over states yet most optimization problems are undecidable Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 8

16 Recap on Probabilistic Automata Partial observation: the plan must be decided in advance! model of system is known the effect of a plan can be computed: after word w, probability distribution over states yet most optimization problems are undecidable What if we execute a plan, but have feedback, and can modify the plan? partially observable Markov decision processes Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 8

17 Outline Probabilistic automata Partially observable MDP Discussion Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 9

18 Example of partially observable MDP (POMDP) McCallum maze: robot with limited sensor abilities, and imperfect moves Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 10

19 Example of partially observable MDP (POMDP) McCallum maze: robot with limited sensor abilities, and imperfect moves robot only sees walls surrounding it, not the precise cell observations Ω = {{L, U}, {U, D}, {U, R}, {L, D, R} } actions A = {N, W, S, E} are not implemented accurately action N leads to north with probability 2 3 and others with 1 3 Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 10

20 Example of partially observable MDP (POMDP) McCallum maze: robot with limited sensor abilities, and imperfect moves robot only sees walls surrounding it, not the precise cell observations Ω = {{L, U}, {U, D}, {U, R}, {L, D, R} } actions A = {N, W, S, E} are not implemented accurately action N leads to north with probability 2 3 and others with 1 3 Reachability objective: move to target cell Optimization: minimum expected time Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 10

21 Strategies Strategy: maps history ρ (AΩ) with distribution over actions; ν : (AΩ) Dist(A) ν(ρ, a): probability that a is chosen given history ρ pure strategy: all distributions are Dirac belief-based strategy: based on set of current possible states word in PA pure strategy in POMDP with Ω = 1 Consequence: all hardness results lift from PA to POMDP Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 11

22 Infinite horizon objectives Objectives Reachability F visited at least once: F = {q 0 q 1 q 2 S ω n, q n F } Safety always stay in F : F = {q 0 q 1 q 2 S ω n, q n F } Büchi F visited an infinite number of times: F = {q 0 q 1 q 2 S ω m n m, q n F } Goal: For ϕ an objective, evaluate sup ν P ν (M = ϕ). Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 12

23 Infinite horizon objectives Objectives Reachability F visited at least once: F = {q 0 q 1 q 2 S ω n, q n F } Safety always stay in F : F = {q 0 q 1 q 2 S ω n, q n F } Büchi F visited an infinite number of times: F = {q 0 q 1 q 2 S ω m n m, q n F } Goal: For ϕ an objective, evaluate sup ν P ν (M = ϕ). Pure strategies suffice! For every strategy ν, there exists a pure strategy ν such that P ν (M = ϕ) P ν (M = ϕ). Chatterjee et al. 15 Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 12

24 Undecidability results Undecidability of qualitative objectives beyond the ones already mentioned for PA Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 13

25 Undecidability results Undecidability of qualitative objectives beyond the ones already mentioned for PA positive repeated reachability does there exist ν such that P ν (M = F ) > 0? Baier et al. 08 combined objectives does there exist ν such that P ν (M = F 1 ) = 1 and P ν (M = F 2 ) > 0? Bertrand et al. 14 Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 13

26 Undecidability results Undecidability of qualitative objectives beyond the ones already mentioned for PA positive repeated reachability does there exist ν such that P ν (M = F ) > 0? Baier et al. 08 combined objectives does there exist ν such that P ν (M = F 1 ) = 1 and P ν (M = F 2 ) > 0? Bertrand et al. 14 Proof of first statement: reduction from the value one problem for PA q f f pure strategies in M: ν w = w 1 w 2 w 3 val(a) = 1 (w i) i N P A(w i) > 0 ν w P νw (M = f ) > 0 i Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 13

27 Decidability results Good news: decidable problems for PA remain decidable Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 14

28 Decidability results Good news: decidable problems for PA remain decidable almost-sure safety existence of ν such that P ν (M = F ) = 1 positive safety existence of ν such that P ν (M = F ) > 0 almost-sure repeated reachability existence of ν such that P ν (M = F ) = 1 are all EXPTIME-complete. Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 14

29 Decidability results Good news: decidable problems for PA remain decidable almost-sure safety existence of ν such that P ν (M = F ) = 1 positive safety existence of ν such that P ν (M = F ) > 0 almost-sure repeated reachability existence of ν such that P ν (M = F ) = 1 are all EXPTIME-complete. fixpoint algorithms on a powerset construction belief-based strategies suffice except for positive safety Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 14

30 Decidability results Good news: decidable problems for PA remain decidable almost-sure safety existence of ν such that P ν (M = F ) = 1 positive safety existence of ν such that P ν (M = F ) > 0 almost-sure repeated reachability existence of ν such that P ν (M = F ) = 1 are all EXPTIME-complete. fixpoint algorithms on a powerset construction belief-based strategies suffice except for positive safety q a 1 2 a + b 1 2 a q 1 q 2 a b 1 2 b 1 2 a q 3 a no belief-based strategy can achieve P ν (M = {q 0, q 1, q 2}) > 0 alternate a and b forever, guarantees a probability 1 2 Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 14

31 Decidability results Good news: decidable problems for PA remain decidable almost-sure safety existence of ν such that P ν (M = F ) = 1 positive safety existence of ν such that P ν (M = F ) > 0 almost-sure repeated reachability existence of ν such that P ν (M = F ) = 1 are all EXPTIME-complete. fixpoint algorithms on a powerset construction belief-based strategies suffice except for positive safety q a 1 2 a + b 1 2 a q 1 q 2 a b 1 2 b 1 2 a q 3 a no belief-based strategy can achieve P ν (M = {q 0, q 1, q 2}) > 0 alternate a and b forever, guarantees a probability 1 2 Open: decidability of non-null proportion with positive probability ν, P ν (M = lim sup n #visits to F in n first steps n > 0) > 0? Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 14

32 Outline Probabilistic automata Partially observable MDP Discussion Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 15

33 Life is hard = Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 16

34 Life is hard = most optimization problems are undecidable notably quantitative questions but also some qualitative questions and undecidability is robust Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 16

35 ... but there is still hope usual way arounds decidable subclasses Fijalkow et al. 12 restricted classes of strategies approximations, although with no termination guarantees Yu 06 Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 17

36 ... but there is still hope usual way arounds decidable subclasses Fijalkow et al. 12 restricted classes of strategies approximations, although with no termination guarantees Yu 06 promising alternative: discretization continuous distributions approximated by large discrete population Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 17

37 ... but there is still hope usual way arounds decidable subclasses Fijalkow et al. 12 restricted classes of strategies approximations, although with no termination guarantees Yu 06 promising alternative: discretization continuous distributions approximated by large discrete population a,1/2 a,1/2 a,1/2 a a,1/2 a b b Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 17

38 ... but there is still hope usual way arounds decidable subclasses Fijalkow et al. 12 restricted classes of strategies approximations, although with no termination guarantees Yu 06 promising alternative: discretization continuous distributions approximated by large discrete population a,1/2 a,1/2 a,1/2 a a,1/2 a b limit for large populations differs from continuous semantics possible alternative semantics to PA/POMDP models, with more decidability results b Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 17

39 ... but there is still hope usual way arounds decidable subclasses Fijalkow et al. 12 restricted classes of strategies approximations, although with no termination guarantees Yu 06 promising alternative: discretization continuous distributions approximated by large discrete population a,1/2 a,1/2 a,1/2 a a,1/2 a b limit for large populations differs from continuous semantics possible alternative semantics to PA/POMDP models, with more decidability results b Thank you! Controlling probabilistic systems under partial observation October 4th 2016 Simons Institute, Berkeley 17