On Model Checking Techniques for Randomized Distributed Systems. Christel Baier Technische Universität Dresden

Transcription

1 On Model Checking Techniques for Randomized Distributed Systems Christel Baier Technische Universität Dresden joint work with Nathalie Bertrand Frank Ciesinski Marcus Größer / 6

2 biological systems, resilient systems, security protocols... / 6 Probability elsewhere int-0 randomized algorithms [Rabin 960] breaking symmetry, fingerprints, input sampling,... stochastic control theory [Bellman 957] operations research performance modeling [Markov, Erlang, Kolm., 900] emphasis on steady-state and transient measures

3 Probability elsewhere int-0 randomized algorithms [Rabin 960] breaking symmetry, fingerprints, input sampling,... models: discrete-time Markov chains Markov decision processes stochastic control theory [Bellman 957] operations research models: Markov decision processes performance modeling [Markov, Erlang, Kolm., 900] emphasis on steady-state and transient measures models: continuous-time Markov chains biological systems, resilient systems, security protocols... 3 / 6

4 Model checking [Clarke/Emerson, Queille/Sifakis] mc reactive system abstract model M requirements (safety, liveness) specification, e.g., temporal formula Φ model checking does M = Φ hold? no yes 4 / 6

5 Probabilistic model checking probabilistic reactive system probabilistic model M int-03 quantitative requirements specification, e.g., temporal formula Φ probabilistic model checking does M = Φ hold? probability for bad behaviors is < 0 6 probability for good behaviors is expected costs for... 5 / 6

6 Probabilistic model checking probabilistic reactive system Markov decision process M int-03 quantitative requirements linear temporal formula Φ (path event) probabilistic model checking quantitative analysis of M against Φ probability for bad behaviors is < 0 6 probability for good behaviors is 6 / 6

7 Outline overview Markov decision processes (MDP) and quantitative analysis against path events partial order reduction for MDP partially-oberservable MDP conclusions 7 / 6

8 Markov decision process (MDP) mdp-0 operational model with nondeterminism and probabilism 8 / 6

9 Markov decision process (MDP) mdp-0 operational model with nondeterminism and probabilism modeling randomized distributed systems by interleaving process tosses a coin s process tosses a coin process tosses a coin process tosses a coin 9 / 6

10 Markov decision process (MDP) mdp-0 operational model with nondeterminism and probabilism modeling randomized distributed systems by interleaving nondeterminism useful for abstraction, underspec., modeling interactions with an unkown environment process tosses a coin s process tosses a coin process tosses a coin process tosses a coin 0 / 6

11 Markov decision process (MDP) mdp-0-r M =(S, Act, P,...) / 6

12 Markov decision process (MDP) mdp-0-r M =(S, Act, P,...) finite state space S / 6

13 Markov decision process (MDP) mdp-0-r M =(S, Act, P,...) finite state space S Act finite set of actions P : S Act S [0, ] 3 / 6

14 Markov decision process (MDP) mdp-0-r M =(S, Act, P,...) finite state space S Act finite set of actions P : S Act S [0, ] 4 α 3 4 s 6 β 3 nondeterministic choice probabilistic choice 4 / 6

15 Markov decision process (MDP) mdp-0-r M =(S, Act, P,...) Act(s) = set of actions that are enabled finite state space S in state s Act finite set of actions P : S Act S [0, ] s.t. s S α Act. P(s, α, s ) {0, } s S α / Act(s) α Act(s) 4 α 3 4 s 6 β 3 nondeterministic choice probabilistic choice 5 / 6

16 Markov decision process (MDP) mdp-0-r M =(S, Act, P, s 0, AP, L, rew,...) finite state space S Act finite set of actions P : S Act S [0, ] s.t. s S α Act. P(s, α, s ) {0, } s S α / Act(s) α Act(s) s 0 initial state AP set of atomic propositions labeling L : S AP reward function rew : S Act R 6 / 6

17 Randomized mutual exclusion protocol mdp-05 concurrent processes P, P with 3 phases: n i noncritical actions of process P i w i waiting phase of process P i c i critical section of process P i competition of both processes are waiting 7 / 6

18 Randomized mutual exclusion protocol mdp-05 concurrent processes P, P with 3 phases: n i noncritical actions of process P i w i waiting phase of process P i c i critical section of process P i competition of both processes are waiting resolved by a randomized arbiter who tosses a coin 8 / 6

19 Randomized mutual exclusion protocol mdp-05 interleaving of the request operations competition if both processes are waiting randomized arbiter tosses a coin if both are waiting MDP n n w n n w w w c n n c release enter request release release release request request coin request request release release release c w w c enter request release release release 9 / 6

20 Randomized mutual exclusion protocol mdp-05 interleaving of the request operations competition if both processes are waiting randomized arbiter tosses a coin if both are waiting MDP n n w n n w w c n w n c release enter request release release release request request coin request request release release release c w w c enter request release release release 0 / 6

21 Randomized mutual exclusion protocol mdp-05 interleaving of the request operations competition if both processes are waiting randomized arbiter tosses a coin if both are waiting MDP n n w n n w w c n w n c release enter request release release release request request coin request request release release release c w w c enter request release release release / 6

22 Randomized mutual exclusion protocol mdp-05 interleaving of the request operations competition if both processes are waiting randomized arbiter tosses a coin if both are waiting MDP n n w n n w w c n w n c release enter request release release release request request toss a coin request request release release release c w w c enter request release release release / 6

23 Reasoning about probabilities in MDP mdp-0 requires resolving the nondeterminism by schedulers 3 / 6

24 Reasoning about probabilities in MDP mdp-0 requires resolving the nondeterminism by schedulers a scheduler is a function D : S Act s.t. action D (s 0...s n ) is enabled in state s n 4 / 6

25 Reasoning about probabilities in MDP mdp-0 requires resolving the nondeterminism by schedulers a scheduler is a function D : S Act s.t. action D (s 0...s n ) is enabled in state s n each scheduler induces an infinite Markov chain α 3 3 MDP β γ δ σ α 3 δ γ 3 3 α β σ 3 σ / 6

26 Reasoning about probabilities in MDP mdp-0 requires resolving the nondeterminism by schedulers a scheduler is a function D : S Act s.t. action D (s 0...s n ) is enabled in state s n each scheduler induces an infinite Markov chain yields a notion of probability measure Pr D on measurable sets of infinite paths 6 / 6

27 Reasoning about probabilities in MDP mdp-0 requires resolving the nondeterminism by schedulers a scheduler is a function D : S Act s.t. action D (s 0...s n ) is enabled in state s n each scheduler induces an infinite Markov chain yields a notion of probability measure Pr D on measurable sets of infinite paths typical task: given a measurable path event E, check whether E holds almost surely, i.e., Pr D (E) == for all schedulers D 7 / 6

28 Reasoning about probabilities in MDP mdp-0 requires resolving the nondeterminism by schedulers a scheduler is a function D : S Act s.t. action D (s 0...s n ) is enabled in state s n each scheduler induces an infinite Markov chain yields a notion of probability measure Pr D on measurable sets of infinite paths typical task: given a measurable path event E, check whether E holds almost surely compute the worst-case probability for E, i.e., sup Pr D (E) or inf PrD (E) D D 8 / 6

29 Quantitative analysis of MDP mdp-5 given: task: MDP M =(S, Act, P,...) with initial state s 0 ω-regular path event E, e.g., given by an LTL formula compute Pr M max(s 0, E) =suppr D (s 0, E) D 9 / 6

30 Quantitative analysis of MDP mdp-5 given: task: MDP M =(S, Act, P,...) with initial state s 0 ω-regular path event E, e.g., given by an LTL formula compute Pr M max(s 0, E) =suppr D (s 0, E) D method: compute x s = Pr M max (s, E) for all s S via graph analysis and linear program [Vardi/Wolper 86] [Courcoubetis/Yannakakis 88] [Bianco/de Alfaro 95] [Baier/Kwiatkowska 98] 30 / 6

31 probabilistic system bad behaviors 3 / 6

32 probabilistic system bad behaviors MDP M 3 / 6

33 probabilistic system MDP M bad behaviors LTL formula ϕ deterministic automaton A 33 / 6

34 probabilistic system MDP M bad behaviors LTL formula ϕ deterministic automaton A quantitative analysis in the product-mdp M A Pr M max(s, ϕ) = Pr M A( ( max s, inits, acceptance cond. of A ) 34 / 6

35 probabilistic system MDP M bad behaviors LTL formula ϕ deterministic automaton A quantitative analysis in the product-mdp M A maximal probabilility for reaching an accepting end component Pr M max(s, ϕ) = Pr M A( ( s, inits, accec ) max 35 / 6

36 probabilistic system MDP M bad behaviors LTL formula ϕ deterministic automaton A probabilistic reachability analysis in the product-mdp M A linear program maximal probabilility for reaching an accepting end component Pr M max(s, ϕ) = Pr M A( ( s, inits, accec ) max 36 / 6

37 probabilistic system MDP M bad behaviors LTL formula ϕ deterministic automaton A probabilistic reachability analysis in the product-mdp M A linear program polynomial in M A Pr M max(s, ϕ) = Pr M A( ( s, inits, accec ) max 37 / 6

38 probabilistic system MDP M bad behaviors LTL formula ϕ deterministic automaton A exp probabilistic reachability analysis in the product-mdp M A linear program polynomial in M A Pr M max(s, ϕ) = Pr M A( ( s, inits, accec ) max 38 / 6

39 probabilistic system MDP M state explosion problem bad behaviors LTL formula ϕ deterministic automaton A probabilistic reachability analysis in the product-mdp M A linear program polynomial in M A Pr M max(s, ϕ) = Pr M A( ( s, inits, accec ) max 39 / 6

40 Advanced techniques for PMC por-0-cmu symbolic model checking with variants of BDDs e.g., in PRISM [Kwiatkowska/Norman/Parker] ProbVerus [Hartonas-Garmhausen, Campos, Clarke] state aggregation with bisimulation e.g., in MRMC [Katoen et al] abstraction-refinement e.g., in RAPTURE [d Argenio/Jeannet/Jensen/Larsen] PASS [Hermanns/Wachter/Zhang] partial order reduction e.g., in LiQuor [Baier/Ciesinski/Größer] 40 / 6

41 Advanced techniques for PMC por-0-cmu symbolic model checking with variants of BDDs e.g., in PRISM [Kwiatkowska/Norman/Parker] ProbVerus [Hartonas-Garmhausen, Campos, Clarke] randomized distributed algorithms, communication and multimedia protocols, power management, security,... state aggregation with bisimulation e.g., in MRMC [Katoen et al] abstraction-refinement e.g., in RAPTURE [d Argenio/Jeannet/Jensen/Larsen] PASS [Hermanns/Wachter/Zhang] partial order reduction e.g., in LiQuor [Baier/Ciesinski/Größer] 4 / 6

42 Advanced techniques for PMC por-0-cmu symbolic model checking with variants of BDDs e.g., in PRISM [Kwiatkowska/Norman/Parker] ProbVerus [Hartonas-Garmhausen, Campos, Clarke] randomized distributed algorithms, communication and multimedia protocols, power management, security,... state aggregation with bisimulation e.g., in MRMC [Katoen et al] abstraction-refinement e.g., in RAPTURE [d Argenio/Jeannet/Jensen/Larsen] PASS [Hermanns/Wachter/Zhang] partial order reduction e.g., in LiQuor [Baier/Ciesinski/Größer] 4 / 6

43 Partial order reduction por-0 technique for reducing the state space of concurrent systems [Godefroid,Peled,Valmari, ca. 990] attempts to analyze a sub-system by identifying redundant interleavings explores representatives of paths that agree up to the order of independent actions 43 / 6

44 Partial order reduction por-0 technique for reducing the state space of concurrent systems [Godefroid,Peled,Valmari, ca. 990] attempts to analyze a sub-system by identifying redundant interleavings explores representatives of paths that agree up to the order of independent actions e.g., x := x+y }{{} action α } z := {{ z+3 } action β has the same effect as α; β or β; α 44 / 6

45 Partial order reduction por-0 technique for reducing the state space of concurrent systems [Godefroid,Peled,Valmari, ca. 990] attempts to analyze a sub-system by identifying redundant interleavings explores representatives of paths that agree up to the order of independent actions DFS-based on-the-fly generation of a reduced system foreachexpandedstates choose an appropriate subset Ample(s) of Act(s) expand only the α-successors of s for α Ample(s) (but ignore the actions in Act(s) \ Ample(s)) 45 / 6

46 Partial order reduction por-0a concurrent execution of processes P, P no communication no competition α λ β λ α µ γ λ β µ α ν transition system for P P where P = α; β; γ P = λ; µ; ν λ γ µ β ν γ ν µ β ν γ α 46 / 6

47 Partial order reduction por-0a concurrent execution of processes P, P no communication no competition β λ α transition system for P P where P = α; β; γ P = λ; µ; ν µ ν γ idea: explore just path as representative for all paths 47 / 6

48 Ample-set method [Peled 993] por-03 given: processes P i of a parallel system P... P n with transition system T =(S, Act,,...) task: on-the-fly generation of a sub-system T r s.t. (A) stutter condition... (A) dependency condition... (A3) cycle condition / 6

49 Ample-set method [Peled 993] por-03 given: task: (A) (A) (A3) processes P i of a parallel system P... P n with transition system T =(S, Act,,...) on-the-fly generation of a sub-system T r s.t. } stutter condition π πr π r dependency condition by permutations of cycle condition independent actions Each path π in T is represented by an equivalent path π r in T r 49 / 6

50 Ample-set method [Peled 993] por-03 given: task: (A) (A) (A3) processes P i of a parallel system P... P n with transition system T =(S, Act,,...) on-the-fly generation of a sub-system T r s.t. } stutter condition π πr π r dependency condition by permutations of cycle condition independent actions Each path π in T is represented by an equivalent path π r in T r T and T r satisfy the same stutter-invariant events, e.g., next-free LTL formulas 50 / 6

51 Ample-set method for MDP por-04 given: task: processes P i of a probabilistic system P... P n with MDP-semantics M =(S, Act, P,...) on-the-fly generation of a sub-mdp M r s.t. M r and M have the same extremal probabilities for stutter-invariant events 5 / 6

52 Ample-set method for MDP por-04 given: task: processes P i of a probabilistic system P... P n with MDP-semantics M =(S, Act, P,...) on-the-fly generation of a sub-mdp M r s.t. For all schedulers D for M there is a scheduler D r for M r s.t. for all measurable, stutter-invariant events E: M(E) D Pr D M = PrD r M r (E) M r and M have the same extremal probabilities for stutter-invariant events D r 5 / 6

53 Independence of actions por / 6

54 Independence of non-probabilistic actions por-06 Actions α and β are called independent in a transition system T iff: α whenever s t and s u then β () α is enabled in u () β is enabled in t (3) if u α β v and t w then v = w t α β s β α v u 54 / 6

55 Independence of actions in an MDP por-06 Let M =(S, Act, P,...)beaMDPandα, β Act. α and β are independent in M if for each state s s.t. α, β Act(s): () if P(s, α, t) > 0thenβ β Act(t) () if P(s, β, u) > 0thenα α Act(u) (3) / 6

56 Independence of actions in an MDP por-06 Let M =(S, Act, P,...)beaMDPandα, β Act. α and β are independent in M if for each state s s.t. α, β Act(s): () if P(s, α, t) > 0thenβ β Act(t) () if P(s, β, u) > 0thenα α Act(u) (3) for all states w: P(s, αβ, w) = P(s, βα, w) P(s, α, t) P(t, β, w) P(s, β, u) P(u, α, w) t S u S 56 / 6

57 Example: ample set method s β γ por-08 α α β γ α δ δ original system T α independent from β and γ 57 / 6

58 Example: ample set method por-08 β s γ β s γ α α β γ α α α δ δ δ δ original system T α independent from β and γ T r reduced system T r (A)-(A3) are fulfilled 58 / 6

59 Example: ample set method fails for MDP por-08 s α β γ β γ γ β α α δ δ s β γ α α δ δ original MDP M reduced MDP M r M r M r (A)-(A3) are fulfilled α independent from β and γ 59 / 6

60 Example: ample set method fails for MDP por-08 s α β γ β γ γ β α α δ δ s β γ α α δ δ original MDP M reduced MDP M r M r M r Pr M max(s, green) = Pr M max(s, green) = Pr M max(s, green) = eventually 60 / 6

61 Example: ample set method fails for MDP por-08 s α β γ β γ γ β α α δ δ s β γ α α δ δ original MDP M reduced MDP M r M r M r Pr M max(s, green) = Pr M max(s, green) = Pr M max(s, green) = > = PrM r max(s, green) > = PrM r max(s, green) > = PrM r max(s, green) 6 / 6

62 Partial order reduction for MDP por-09 extend Peled s conditions (A)-(A3) for the ample-sets (A) stutter condition... (A) dependency condition... (A3) cycle condition... (A4) probabilistic condition β β If there is a path β s n α... in M s.t. β,...,β n, α / Ample(s) and α is probabilistic then Ample(s) = =. 6 / 6

63 Partial order reduction for MDP por-09 extend Peled s conditions (A)-(A3) for the ample-sets (A) stutter condition... (A) dependency condition... (A3) cycle condition... (A4) probabilistic condition β β If there is a path β s n α... in M s.t. β,...,β n, α / Ample(s) and α is probabilistic then Ample(s) = =. M r If (A)-(A4) hold then M and M r have the same extremal probabilities for all stutter-invariant properties. 63 / 6

64 Probabilistic model checking probabilistic system por-ifm-3 quantitative requirements Markov decision process M LTL \ formula ϕ (path event) quantitative analysis of M against ϕ maximal/minimal probability for ϕ 64 / 6

65 Probabilistic model checking, e.g., LiQuor por-ifm-3 modeling language P... P n partial order reduction reduced MDP M r quantitative requirements LTL \ formula ϕ (path event) quantitative analysis of M r against ϕ maximal/minimal probability for ϕ 65 / 6

66 Probabilistic model checking, e.g., LiQuor por-ifm-3a modeling language P... P n partial order reduction reduced MDP M r quantitative requirements LTL \ formula ϕ (path event) quantitative analysis of M r against ϕ maximal/minimal probability for ϕ } worst-case analysis 66 / 6

67 Outline overview-pomdp Markov decision processes (MDP) and quantitative analysis against path events partial order reduction for MDP partially-oberservable MDP conclusions 67 / 6

68 Monty-Hall problem pomdp-0 3doors initially closed candidate show master 68 / 6

69 Monty-Hall problem no prize prize no prize pomdp-0 3doors initially closed candidate show master 69 / 6

70 Monty-Hall problem no prize prize no prize pomdp-0 3doors initially closed candidate show master. candidate chooses one of the doors 70 / 6

71 Monty-Hall problem pomdp-0 no prize prize no prize 3doors initially closed candidate show master. candidate chooses one of the doors. show master opens a non-chosen, non-winning door 7 / 6

72 Monty-Hall problem pomdp-0 no prize prize no prize 3doors initially closed candidate show master. candidate chooses one of the doors. show master opens a non-chosen, non-winning door 3. candidate has the choice: keep the choice or switch to the other (still closed) door 7 / 6

73 Monty-Hall problem pomdp-0 no prize Euro no prize 3doors initially closed candidate show master. candidate chooses one of the doors. show master opens a non-chosen, non-winning door 3. candidate has the choice: keep the choice or switch to the other (still closed) door 4. show master opens all doors 73 / 6

74 Monty-Hall problem pomdp-0 no prize Euro no prize 3doors initially closed candidate show master optimal strategy for the candidate: initial choice of the door: arbitrary revision of the initial choice (switch) probability for getting the prize: 3 74 / 6

75 MDP for the Monty-Hall problem pomdp-0 75 / 6

76 MDP for the Monty-Hall problem no prize prize no prize pomdp-0 3doors initially closed candidate s actions. choose one door 3. keep or switch? show master s actions. opens a non-chosen, non-winning door 4. opens all doors 76 / 6

77 MDP for the Monty-Hall problem no prize prize no prize pomdp-0 3doors initially closed candidate s actions. choose one door 3. keep or switch? start show master s actions. opens a non-chosen, non-winning door 4. opens all doors door door door 3 keep switch switch lost keep switchkeep won 77 / 6

78 MDP for the Monty-Hall problem no prize prize no prize pomdp-0 3doors initially closed candidate s actions. choose one door 3. keep or switch? start 3 3 Pr max (start,, won) = optimal scheduler requires complete information on the states 3 door door door 3 keep switch lost switch won 78 / 6

79 MDP for the Monty-Hall problem no prize prize no prize pomdp-0 3doors initially closed candidate s actions. choose one door 3. keep or switch? start cannot be distinguished by the candidate door door door 3 keep switch keep keep switch lost switch won 79 / 6

80 MDP for the Monty-Hall problem no prize prize no prize pomdp-0 3doors initially closed candidate s actions. choose one door 3. keep or switch? start 3 3 observation-based strategy: choose action switch in state door i 3 door door door 3 switch switch lost switch won 80 / 6

81 MDP for the Monty-Hall problem no prize prize no prize pomdp-0 3doors initially closed candidate s actions. choose one door 3. keep or switch? start 3 3 observation-based strategy: choose action switch in state door i probability for won: 3 3 door door door 3 switch switch lost switch won 8 / 6

82 Partially-observable Markov decision process pomdp-05 A partially-observable MDP (POMDP for short) is an MDP M =(S, Act, P,...) together with an equivalence relation on S 8 / 6

83 Partially-observable Markov decision process pomdp-05 A partially-observable MDP (POMDP for short) is an MDP M =(S, Act, P,...) together with an equivalence relation on S if s s then s, s cannot be distinguished from outside (or by the scheduler) observables: equivalence classes of states 83 / 6

84 Partially-observable Markov decision process pomdp-05 A partially-observable MDP (POMDP for short) is an MDP M =(S, Act, P,...) together with an equivalence relation on S if s s then s, s cannot be distinguished from outside (or by the scheduler) observables: equivalence classes of states observation-based scheduler: scheduler D : S Act such that for all π, π S : D(π ) = D(π ) if obs(π ) = obs(π ) where obs(s 0 s...s n ) =[s 0 ][s ]...[s n ] 84 / 6

85 Extreme cases of POMDP pomdp- extreme cases of an POMDP: s s iff s = s s s for all s, s 85 / 6

86 Extreme cases of POMDP pomdp- extreme cases of an POMDP: s s iff s = s s s for all s, s standard MDP 86 / 6

87 Probabilistic automata are special POMDP pomdp- extreme cases of an POMDP: s s iff s = s s s for all s, s standard MDP probabilistic automata note that for totally non-observable POMDP: observation-based scheduler = = = function D : N Act = = = infinite word over Act 87 / 6

88 Undecidability results for POMDP pomdp- extreme cases of an POMDP: s s iff s = s s s for all s, s standard MDP probabilistic automata note that for totally non-observable POMDP: observation-based scheduler = = = function D : N Act = = = infinite word over Act undecidability results for PFA carry over to POMDP maximum probabilistic reachability problem = = = does Probs max( F obs ( F) ) > p hold? non-emptiness problem for PFA 88 / 6

89 Undecidability results for POMDP pomdp-30-new The model checking problem for POMDP and quantitative properties is undecidable, e.g., probabilistic reachability properties. 89 / 6

90 Undecidability results for POMDP pomdp-30-new The model checking problem for POMDP and quantitative properties is undecidable, e.g., probabilistic reachability properties. There is no even no approximation algorithm for reachability objectives. [Paz 7], [Madani/Hanks/Condon 99], [Giro/d Argenio 07] 90 / 6

91 Undecidability results for POMDP pomdp-30-new The model checking problem for POMDP and quantitative properties is undecidable, e.g., probabilistic reachability properties. There is no even no approximation algorithm for reachability objectives. The model checking problem for POMDP and several qualitative properties is undecidable, e.g., repeated reachability with positive probability does Probs max( F obs ( F) ) > 0 hold? = = = infinitely often 9 / 6

92 Undecidability results for POMDP pomdp-30-new The model checking problem for POMDP and quantitative properties is undecidable, e.g., probabilistic reachability properties. There is no even no approximation algorithm for reachability objectives. The model checking problem for POMDP and several qualitative properties is undecidable, e.g., repeated reachability with positive probability does Probs max( F obs ( F) ) > 0 hold? Many interesting verification problems for distributed probabilistic multi-agent systems are undecidable. 9 / 6

93 Undecidability results for POMDP pomdp-30-new The model checking problem for POMDP and quantitative properties is undecidable, e.g., probabilistic reachability properties. There is no even no approximation algorithm for reachability objectives. The model checking problem for POMDP and several qualitative properties is undecidable, e.g., repeated reachability with positive probability does Probs max( F obs ( F) ) > 0 hold?... already holds for totally non-observable POMDP }{{} probabilistic Büchi automata 93 / 6

94 Remind: LTL model checking for MDP pomdp-50 MDP M requirements LTL formula ϕ deterministic automaton A exp probabilistic reachability analysis in the product-mdp M A 94 / 6

95 PA rather than DA? MDP M pomdp-50 requirements LTL formula ϕ probabilistic automaton A? probabilistic reachability analysis in the product-mdp M A 95 / 6

96 PA rather than DA? MDP M pomdp-50 requirements LTL formula ϕ probabilistic automaton A? probabilistic reachability analysis in the product-mdp M A impossible, due to undecidability results 96 / 6

97 Decidability results for POMDP pomdp-5-fm 97 / 6

98 Decidability results for POMDP pomdp-5-ifm The model checking problem for POMDP and several qualitative properties is decidable, e.g., invariance with positive probability does Probs obs max ( F ) > 0 hold? almost-sure reachability does Pr max( F) obs ) = hold? almost-sure repeated reachability does Pr max( F) obs ) = hold? persistence with positive probability does Pr max( F) obs ) > 0 hold? 98 / 6

99 Decidability results for POMDP pomdp-5-ifm The model checking problem for POMDP and several qualitative properties is decidable, e.g., invariance with positive probability does Probs obs max ( F ) > 0 hold? almost-sure reachability does Pr max( F) obs ) = hold? almost-sure repeated reachability does Pr max( F) obs ) = hold? persistence with positive probability does Pr max( F) obs ) > 0 hold? algorithms use a certain powerset construction 99 / 6

100 Probabilistic Automata and Verification overview-conc Markov decision processes (MDP) and quantitative analysis against path events partial order reduction for MDP partially-oberservable MDP conclusions 00 / 6

101 Conclusion conc worst/best-case analysis of MDP solvable by numerical methods for solving linear programs known techniques for non-probabilistic systems graph algorithms, LTL--AUT translators,... techniques to combat the state explosion problem (such as partial order reduction) 0 / 6

102 Conclusion conc worst/best-case analysis of MDP solvable by numerical methods for solving linear programs known techniques for non-probabilistic systems graph algorithms, LTL--AUT translators,... techniques to combat the state explosion problem (such as partial order reduction) but: strongly simplified definition of schedulers assumption full knowledge of the history is inadequate, e.g., for agents of distributed systems 0 / 6

103 Conclusion conc worst/best-case analysis of MDP solvable by numerical methods for solving linear programs known techniques for non-probabilistic systems more realistic model: partially-observable MDP and multi-agents variants with distributed schedulers 03 / 6

104 Conclusion conc worst/best-case analysis of MDP solvable by numerical methods for solving linear programs known techniques for non-probabilistic systems more realistic model: partially-observable MDP and multi-agents variants with distributed schedulers many algorithms for finite-horizon properties few decidability results for qualitative properties undecidability for quantitative properties and, e.g., repeated reachability with positive probability 04 / 6

105 Conclusion conc worst/best-case analysis of MDP solvable by numerical methods for solving linear programs known techniques for non-probabilistic systems more realistic model: partially-observable MDP and multi-agents variants with distributed schedulers many algorithms for finite-horizon properties few decidability results for qualitative properties undecidability for quantitative properties and, e.g., repeated reachability with positive probability proof via probabilistic language acceptors (PFA/PBA) 05 / 6

106 Conclusion conc worst/best-case analysis of MDP solvable by numerical methods for solving linear programs known techniques for non-probabilistic systems more realistic model: partially-observable MDP and multi-agents variants with distributed schedulers many algorithms for finite-horizon properties few decidability results for qualitative properties undecidability for quantitative properties and, e.g., repeated reachability with positive probability probabilistic Büchi automata interesting in their own / 6

107 Probabilistic Büchi automaton (PBA) pba-0 P = (Q, Σ, δ, µ, F ) Q finite state space Σ alphabet δ : Q Σ Q [0, ] s.t. for all q Q, a Σ: δ(q, a, p) {0, } initial distribution µ p Q set of final states F Q 07 / 6

108 Probabilistic Büchi automaton (PBA) pba-0 P = (Q, Σ, δ, µ, F ) Q finite state space Σ alphabet POMDP where Σ = Act and = = = Q Q δ : Q Σ Q [0, ] s.t. for all q Q, a Σ: δ(q, a, p) {0, } initial distribution µ p Q set of final states F Q 08 / 6

109 Probabilistic Büchi automaton (PBA) pba-0 P = (Q, Σ, δ, µ, F ) Q finite state space Σ alphabet POMDP where Σ = Act and = = = Q Q δ : Q Σ Q [0, ] s.t. for all q Q, a Σ: δ(q, a, p) {0, } initial distribution µ p Q set of final states F Q For each infinite word x Σ ω : Pr(x) = probability for the accepting runs for x accepting run: visits F infinitely often 09 / 6

110 Probabilistic Büchi automaton (PBA) pba-0 P = (Q, Σ, δ, µ, F ) Q finite state space Σ alphabet POMDP where Σ = Act and = = = Q Q δ : Q Σ Q [0, ] s.t. for all q Q, a Σ: δ(q, a, p) {0, } initial distribution µ p Q set of final states F Q For each infinite word x Σ ω : Pr(x) = probability for the accepting runs for x probability measure in the infinite Markov chain induced by x viewed as a scheduler 0 / 6

111 Accepted language of a PBA pba-03 P = (Q, Σ, δ, µ, F ) Q finite state space, Σ alphabet δ : Q Σ Q [0, ] s.t.... initial distribution µ set of final states F Q three types of accepted language: L >0 (P) = { x Σ ω : Pr(x) > 0 } probable semantics L = (P) = { x Σ ω : Pr(x) = } almost-sure sem. L >λ (P) = { x Σ ω : Pr(x) > λ } threshold semantics where 0 < λ < / 6

112 Example for PBA pba-5 b a, a, a / 6

113 Example for PBA pba-5 b a, a, a accepted language: L >0 (P) = (a + b) a ω 3 / 6

114 Example for PBA pba-5 b a, a, a accepted language: L >0 (P) = (a + b) a ω L = (P) = b a ω 4 / 6

115 Example for PBA pba-5 b a, a, a accepted language: L >0 (P) = (a + b) a ω L = (P) = b a ω Thus: PBA >0 are strictly more expressive than DBA 5 / 6

116 Example for PBA pba-5 b a, a, a accepted language: L >0 (P) = (a + b) a ω L = (P) = b a ω Thus: PBA >0 are strictly more expressive than DBA b b c a, a, 3 6 / 6

117 Example for PBA pba-5 b a, a, a accepted language: L >0 (P) = (a + b) a ω L = (P) = b a ω Thus: PBA >0 are strictly more expressive than DBA b b c a, a, 3 NBA accepts ((ac) ab) ω 7 / 6

118 Example for PBA pba-5 b a, a, a accepted language: L >0 (P) = (a + b) a ω L = (P) = b a ω Thus: PBA >0 are strictly more expressive than DBA b a, a, b c 3 accepted language: L >0 (P) =(ab + ac) (ab) ω but NBA accepts ((ac) ab) ω 8 / 6

119 Example for PBA pba-5 b a, a, a accepted language: L >0 (P) = (a + b) a ω L = (P) = b a ω Thus: PBA >0 are strictly more expressive than DBA b a, a, b c 3 accepted language: L >0 (P) =(ab + ac) (ab) ω L = (P) =(ab) ω but NBA accepts ((ac) ab) ω 9 / 6

120 Expressiveness of PBA with probable semantics pba-0 0 / 6

121 Expressiveness of PBA with probable semantics pba-0 PBA >0 are strictly more expressive than NBA / 6

122 Expressiveness of PBA with probable semantics pba-0 PBA >0 are strictly more expressive than NBA from NBA to PBA: NBA Courcoubetis/ Yannakakis NBA deterministic in limit = = = PBA >0 / 6

123 Expressiveness of PBA with probable semantics pba-0 PBA >0 are strictly more expressive than NBA from NBA to PBA: NBA Courcoubetis/ Yannakakis NBA deterministic in limit = = = PBA >0 d d a a b c b 3 / 6

124 Expressiveness of PBA with probable semantics pba-0 PBA >0 are strictly more expressive than NBA from NBA to PBA: NBA Courcoubetis/ Yannakakis NBA deterministic in limit = = = PBA >0 d d deterministic a a b c b 4 / 6

125 Expressiveness of PBA with probable semantics pba-0 PBA >0 are strictly more expressive than NBA from NBA to PBA: NBA Courcoubetis/ Yannakakis NBA deterministic in limit = = = PBA >0 d d deterministic a a b c b = = = d, d, a, b a, c b 5 / 6

126 PBA >0 are strictly more expressive than NBA pba- from NBA to PBA: via NBA that are deterministic in limit PBA can accept non-ω-regular languages a, a a, b 6 / 6

127 PBA >0 are strictly more expressive than NBA pba- from NBA to PBA: via NBA that are deterministic in limit PBA can accept non-ω-regular languages a, a a, b accepted language (probable semantics): { } L L>0 >0 (P) = a k bak bak 3 b / 6

128 PBA >0 are strictly more expressive than NBA pba- from NBA to PBA: via NBA that are deterministic in limit PBA can accept non-ω-regular languages a, a a, b accepted language (probable semantics): { L L>0 >0 (P) = a k bak bak 3 b... ( ( ) ) ki > 0 i= } 8 / 6

129 Expressiveness of PBA pba-5 PBA >0 DBA 9 / 6

130 Expressiveness of PBA pba-5 PBA >0 NBA DBA 30 / 6

131 Expressiveness of PBA pba-5 PBA with thresholds PBA >0 NBA DBA 3 / 6

132 Expressiveness of PBA pba-5 PBA with thresholds PBA >0 NBA DBA PBA = almost-sure semantics 3 / 6

133 Expressiveness of PBA pba-5 (a + b) a ω PBA with thresholds PBA >0 NBA DBA PBA = almost-sure semantics 33 / 6

134 Expressiveness of PBA pba-5 (a + b) a ω PBA with thresholds PBA >0 NBA DBA PBA = almost-sure semantics { k a k bak b...: ( ( )k i ) > 0} i= 34 / 6

135 Expressiveness of PBA pba-5 (a + b) a ω PBA with thresholds PBA >0 NBA DBA PBA = almost-sure semantics { k a k bak b...: ( ( )k i ) > 0} i= { a k ba k b...: ( ( i=( )k i )=0 } 35 / 6

136 Expressiveness of PBA pba-5 (a + b) a ω PBA with thresholds PBA >0 NBA DBA PBA = almost-sure semantics emptiness problem: undecidable for PBA >0 decidable for PBA = 36 / 6

137 Decidability results for POMDP pomdp-6 The model checking problem for POMDP and several qualitative properties is decidable: almost-sure reachability does Pr max( F) obs ) = hold? invariance with positive probability does Probs max( F obs ) > 0 hold? almost-sure repeated reachability does Pr max( F) obs ) = hold? persistence with positive probability does Probs obs max ( F) ) > 0 hold? algorithms use a certain powerset construction 37 / 6

138 Decidability results for POMDP pomdp-6 The model checking problem for POMDP and several qualitative properties is decidable: almost-sure reachability does Pr max( F) obs ) = hold? invariance with positive probability does Probs max( F obs ) > 0 hold? almost-sure repeated reachability does Pr max( F) obs ) = hold? persistence with positive probability does Probs obs max ( F) ) > 0 hold? algorithms use a certain powerset construction 38 / 6

139 Almost-sure reachability/repeated reachability pomdp-7 The almost-sure repeated reachability problem does Probs obs ( F) ) == hold? max is polynomially reducible to the almost-sure reachability problem does Probs max( F obs ) == hold? 39 / 6

140 Almost-sure reachability/repeated reachability pomdp-7 The almost-sure repeated reachability problem does Probs obs ( F) ) == hold? max is polynomially reducible to the almost-sure reachability problem does Probs max( F obs ) == hold? F POMDP M objective: repeated reachability F 40 / 6

141 Almost-sure reachability/repeated reachability pomdp-7 The almost-sure repeated reachability problem does Probs obs ( F) ) == hold? max is polynomially reducible to the almost-sure reachability problem does Probs max( f obs ) == hold? F f POMDP M objective: repeated reachability F M POMDP M objective: reachability f 4 / 6

142 Almost-sure reachability/repeated reachability pomdp-7 The almost-sure repeated reachability problem does Probs obs ( F) ) == hold? max is polynomially reducible to the almost-sure reachability problem does Probs max( f obs ) == hold? s F 3 3 POMDP M objective: repeated reachability F s F 6 3 M POMDP M objective: reachability f f 4 / 6

143 Almost-sure reachability pomdp-8 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? 43 / 6

144 Almost-sure reachability pomdp-8 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? POMDP M with equivalence MDP Pow (M) fully observable 44 / 6

145 Almost-sure reachability pomdp-8 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? POMDP M with equivalence MDP Pow (M) fully observable max( F) ) =inm iff Pr max ( F ) = inpow (M) Pr obs 45 / 6

146 Almost-sure reachability pomdp-8 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? POMDP M with equivalence MDP Pow (M) fully observable max( F) ) =inm iff Pr max ( F ) = inpow (M) state s in M states s, R where s R [s] Pr obs [s] = equivalence class of s w.r.t. 46 / 6

147 Almost-sure reachability pomdp-8 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? POMDP M with equivalence MDP Pow (M) fully observable max( F) ) =inm iff Pr max ( f ) = state s in M states s, R where s R [s] Pr obs fresh goal state f [s] = equivalence class of s w.r.t. inpow (M) 47 / 6

148 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? state s in M action α 48 / 6

149 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? state s in M action α if Post(s, α) F = 49 / 6

150 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? state s in M state s, R in Pow (M) action α action α if Post(s, α) F = where s R [s] 50 / 6

151 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? state s in M state s, R in Pow (M) action α action α t if Post(s, α) F = where s R [s] state t,... t Post(s) 5 / 6

152 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? state s in M state s, R in Pow (M) action α action α t if Post(s, α) F = where s R [s] U = Post(R, α) state t, U [t] t Post(s) 5 / 6

153 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? state s in M state s, R in Pow (M) action α action α t if Post(s, α) F = state t, U [t] P(s, α, t) = P ( s, R,α, t, U [t] ) 53 / 6

154 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? state s in M action α v F if Post(s, α) F where s R [s] U = Post(R, α) 54 / 6

155 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? action α s s v F if Post(s, α) F where s, s R [s] U = Post(R, α) 55 / 6

156 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? action α s s u F state s, R v F if Post(s, α) F where s, s R [s] U = Post(R, α) 56 / 6

157 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? action α s s u F state s, R v F t if Post(s, α) F where s, s R [s] U = Post(R, α) state t, U [t] where t U \ F 57 / 6

158 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? action α s s u F state s, R v F t / F if Post(s, α) F where s, s R [s] U = Post(R, α) state t, U [t] where t U \ F 58 / 6

159 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? action α s s u F state s, R v F if Post(s, α) F where s, s R [s] U = Post(R, α) objective: f f 59 / 6

160 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? action α s s if Post(s, α) F u F state s, R K K f P ( s, R,α, t, U [t] ) = K where K = Post(R, α) \ F 60 / 6

161 Almost-sure reachability pomdp-9 powerset construction for almost-sure reachability does Pr max( F) obs ) == hold? action α Probs max obs s s u F state s, R K max( F iff Prmax( f ) = original fully-observable POMDP M MDP Pow (M) K f 6 / 6