Probabilistic model checking with PRISM

Size: px
Start display at page:

Download "Probabilistic model checking with PRISM"

Transcription

1 Probabilistic model checking with PRISM Marta Kwiatkowska Department of Computer Science, University of Oxford 4th SSFT, Menlo College, May 204

2 Part 2 Markov decision processes

3 Overview (Part 2) Introduction Model checking for Markov decision processes (MDPs) MDPs: definition Paths, strategies & probability spaces PCTL model checking Costs and rewards Case study: Firewire root contention Strategy synthesis for MDPs Properties and objectives Verification vs synthesis Case study: Dynamic power management Summary 3

4 Recap: Discrete-time Markov chains Discrete-time Markov chains (DTMCs) state-transition systems augmented with probabilities Formally: DTMC D = (S, s init, P, L) where: S is a set of states and s init S is the initial state P : S S [0,] is the transition probability matrix L : S 2 AP labels states with atomic propositions define a probability space Pr s over paths Path s Properties of DTMCs can be captured by the logic PCTL e.g. send P 0.95 [ F deliver ] key question: what is the probability of reaching states T S from state s? reduces to graph analysis + linear equation system s {try} s s 0.0 s 3 {succ} {fail} 4

5 Nondeterminism Some aspects of a system may not be probabilistic and should not be modelled probabilistically; for example: Concurrency - scheduling of parallel components e.g. randomised distributed algorithms - multiple probabilistic processes operating asynchronously Underspecification - unknown model parameters e.g. a probabilistic communication protocol designed for message propagation delays of between d min and d max Unknown environments e.g. probabilistic security protocols - unknown adversary 5

6 Markov decision processes Markov decision processes (MDPs) extension of DTMCs which allow nondeterministic choice Like DTMCs: discrete set of states representing possible configurations of the system being modelled transitions between states occur in discrete time-steps Probabilities and nondeterminism in each state, a nondeterministic choice between several discrete probability distributions over successor states {init} s 0 c a s 2 0.7b 0.3 s {heads} s 3 {tails} a a 6

7 Markov decision processes Formally, an MDP M is a tuple (S,s init,α,δ,l) where: S is a set of states ( state space ) s init S is the initial state α is an alphabet of action labels {init} δ S α Dist(S) is the transition probability relation, where Dist(S) is the set of all discrete probability distributions over S L : S 2 AP is a labelling with atomic propositions s 0 c a s 2 0.7b 0.3 s {heads} s 3 {tails} a a Notes: we also abuse notation and use δ as a function i.e. δ : S 2 α Dist(S) where δ(s) = { (a,µ) (s,a,µ) δ} we assume δ(s) is always non-empty, i.e. no deadlocks MDPs, here, are identical to probabilistic automata [Segala] usually, MDPs take the form: δ : S α Dist(S) 7

8 Simple MDP example A simple communication protocol after one step, process starts trying to send a message then, a nondeterministic choice between: (a) waiting a step because the channel is unready; (b) sending the message if the latter, with probability 0.99 send successfully and stop and with probability 0.0, message sending fails, restart restart {fail} {try} 0.0 s 2 s 0 start s send 0.99 s 3 stop wait {succ} 8

9 Example - Parallel composition Asynchronous parallel composition of two 3-state DTMCs t 0 t t 2 Action labels omitted here s 0 s 0 t 0 s 0 t s 0 t 2 s s t 0 s t s t 2 s 2 s 2 t 0 s 2 t s 2 t 2 9

10 Paths and strategies A (finite or infinite) path through an MDP is a sequence (s 0...s n ) of (connected) states represents an execution of the system resolves both the probabilistic and nondeterministic choices {init} s a 2 s 0 0.7b 0.3 s {heads} c s 3 {tails} a a A strategy σ (aka. adversary or policy ) of an MDP is a resolution of nondeterminism only is (formally) a mapping from finite paths to distributions induces a fully probabilistic model i.e. an (infinite-state) Markov chain over finite paths on which we can define a probability space over infinite paths 0

11 Classification of strategies Strategies are classified according to randomisation: σ is deterministic (pure) if σ(s 0...s n ) is a point distribution, and randomised otherwise memory: σ is memoryless (simple) if σ(s 0...s n ) = σ(s n ) for all s 0...s n σ is finite memory if there are finitely many modes such as σ(s 0...s n ) depends only on s n and the current mode, which is updated each time an action is performed otherwise, σ is infinite memory A strategy σ induces, for each state s in the MDP: a set of infinite paths Path σ (s) a probability space Pr σ s over Path σ (s)

12 Example strategy Fragment of induced Markov chain for strategy which picks b then c in s s s 0 s 0.7 s 0 s s 0 s 0 s s s 0 s s 0 s s 0 s s s 2 s 0 s s 0 s s 2 s 0 s s 0 s s 3 s 0 s s s 2 s 2 {heads} s 0 s s s 3 s 0 s s s 3 s 3 {init} s a 2 s 0 0.7b 0.3 s c s 3 {tails} a a finite-memory, deterministic 2

13 PCTL Temporal logic for properties of MDPs (and DTMCs) extension of (non-probabilistic) temporal logic CTL key addition is probabilistic operator P quantitative extension of CTL s A and E operators PCTL syntax: φ ::= true a φ φ φ P ~p [ ψ ] ψ ::= X φ φ U k φ φ U φ (state formulas) (path formulas) where a is an atomic proposition, used to identify states of interest, p [0,] is a probability, ~ {<,>,, }, k N Example: send P 0.95 [ true U 0 deliver ] 3

14 PCTL semantics for MDPs PCTL formulas interpreted over states of an MDP s φ denotes φ is true in state s or satisfied in state s Semantics of (non-probabilistic) state formulas: for a state s of the MDP (S,s init,α,δ,l): s a a L(s) s φ φ 2 s φ and s φ 2 s φ s φ is false Semantics of path formulas: for a path ω = s 0 (a 0,µ 0 )s (a,µ )s 2 in the MDP: ω X φ s φ ω φ U k φ 2 i k such that s i φ 2 and j<i, s j φ ω φ U φ 2 k 0 such that ω φ U k φ 2 4

15 PCTL semantics for MDPs Semantics of the probabilistic operator P can only define probabilities for a specific strategy σ s P ~p [ ψ ] means the probability, from state s, that ψ is true for an outgoing path satisfies ~p for all strategies σ formally s P ~p [ ψ ] Pr sσ (ψ) ~ p for all strategies σ where we use Pr sσ (ψ) to denote Pr sσ { ω Path sσ ω ψ } s ψ ψ Pr sσ (ψ) ~ p Some equivalences: F φ φ true U φ G φ φ (F φ) (eventually, future ) (always, globally ) 5

16 Minimum and maximum probabilities Letting: Pr s max (ψ) = sup σ Pr sσ (ψ) Pr s min (ψ) = inf σ Pr sσ (ψ) We have: if ~ {,>}, then s P ~p [ ψ ] Pr s min (ψ) ~ p if ~ {<, }, then s P ~p [ ψ ] Pr s max (ψ) ~ p Model checking P ~p [ ψ ] reduces to the computation over all strategies of either: the minimum probability of ψ holding the maximum probability of ψ holding Crucial result for model checking PCTL on MDPs memoryless strategies suffice, i.e. there are always memoryless strategies σ min and σ max for which: Pr s σmin(ψ) = Pr s min (ψ) and Pr s σmax(ψ) = Pr s min (ψ) 6

17 Quantitative properties For PCTL properties with P as the outermost operator quantitative form (two types): P min=? [ ψ ] and P max=? [ ψ ] i.e. what is the minimum/maximum probability (over all adversaries) that path formula ψ is true? corresponds to an analysis of best-case or worst-case behaviour of the system model checking is no harder since compute the values of Pr s min (ψ) or Pr s max (ψ) anyway useful to spot patterns/trends Example: CSMA/CD protocol min/max probability that a message is sent within the deadline 7

18 Some real PCTL examples Byzantine agreement protocol P min=? [ F (agreement rounds 2) ] what is the minimum probability that agreement is reached within two rounds? CSMA/CD communication protocol P max=? [ F collisions=k ] what is the maximum probability of k collisions? Self-stabilisation protocols P min=? [ F t stable ] what is the minimum probability of reaching a stable state within k steps? 8

19 PCTL model checking for MDPs Algorithm for PCTL model checking [BdA95] inputs: MDP M=(S,s init,α,δ,l), PCTL formula φ output: Sat(φ) = { s S s φ } = set of states satisfying φ Basic algorithm same as PCTL model checking for DTMCs proceeds by induction on parse tree of φ non-probabilistic operators (true, a,, ) straightforward Only need to consider P ~p [ ψ ] formulas reduces to computation of Pr s min (ψ) or Pr s max (ψ) for all s S dependent on whether ~ {,>} or ~ {<, } these slides cover the case Pr s min (φ U φ 2 ), i.e. ~ {,>} case for maximum probabilities is very similar next (X φ) and bounded until (φ U k φ 2 ) are straightforward extensions of the DTMC case 9

20 PCTL until for MDPs Computation of probabilities Pr min s (φ U φ 2 ) for all s S First identify all states where the probability is or 0 precomputation algorithms, yielding sets S yes, S no Then compute (min) probabilities for remaining states (S? ) either: solve linear programming problem or: approximate with an iterative solution method or: use policy iteration Example: P p [ F a ] P p [ true U a ] s s 2 s 0 {a} s

21 PCTL until - Precomputation Identify all states where Pr s min (φ U φ 2 ) is or 0 S yes = Sat(P [ φ U φ 2 ]), S no = Sat( P >0 [ φ U φ 2 ]) Two graph-based precomputation algorithms: algorithm ProbA computes S yes for all strategies the probability of satisfying φ U φ 2 is algorithm Prob0E computes S no there exists a strategy for which the probability is 0 {a} S yes = Sat(P [ F a ]) Example: P p [ F a ] 0.4 s s 2 0. s s 3 S no = Sat( P >0 [ F a ]) 2

22 Method - Linear programming Probabilities Pr s min (φ U φ 2 ) for remaining states in the set S? = S \ (S yes S no ) can be obtained as the unique solution of the following linear programming (LP) problem: s S? maximize x s subject to the constraints : x s µ(s') x s' + s' S? s' S yes µ(s') for all s S? and for all (a, µ) δ(s) Simple case of a more general problem known as the stochastic shortest path problem [BT9] This can be solved with standard techniques e.g. Simplex, ellipsoid method, branch-and-cut 22

23 Example - PCTL until (LP) {a} 0.4 s s 2 0. S yes Let x i = Pr s (F a) imin S yes : x 2 =, S no : x 3 =0 For S? = {x 0, x } : Maximise x 0 +x subject to constraints: x 0 x s s 3 S no x x 0 + x 0. x 0 + x

24 Example - PCTL until (LP) {a} 0.4 s s 2 0. S yes Let x i = Pr s (F a) imin S yes : x 2 =, S no : x 3 =0 For S? = {x 0, x } : Maximise x 0 +x subject to constraints: x 0 x s s 3 S no x 0 2/3 x 0.2 x x x x x 0 x 0.8 x 0 2/3 x 0.2 x x /3 x x 0 24

25 Example - PCTL until (LP) {a} 0.4 s s 2 0. S yes Let x i = Pr s (F a) imin S yes : x 2 =, S no : x 3 =0 For S? = {x 0, x } : Maximise x 0 +x subject to constraints: x 0 x s s 3 S no x 0 2/3 x 0.2 x x 0.8 max Solution: (x 0, x ) = (2/3, 4/5) x 0 0 x 0 0 2/3 25

26 Example - PCTL until (LP) {a} 0.4 s s 2 0. S yes Let x i = Pr s (F a) imin S yes : x 2 =, S no : x 3 =0 For S? = {x 0, x } : Maximise x 0 +x subject to constraints: x 0 x s s 3 S no x 0 2/3 x 0.2 x x x 0.2 x max Two memoryless adversaries x 0 x x 0 2/3 x 0 0 x 0 0 2/3 26

27 Method 2 Value iteration For probabilities Pr s min (φ U φ 2 ) it can be shown that: Pr s min (φ U φ 2 ) = lim n x s (n) where: x s (n) = min (a,µ) Steps(s) ifs S yes 0 ifs S no 0 ifs S? andn = 0 (n ) µ(s') x s' ifs S? andn > 0 s' S This forms the basis for an (approximate) iterative solution iterations terminated when solution converges sufficiently 27

28 Example - PCTL until (value iteration) {a} 0.4 s s 2 0. S yes Compute: Pr s (F a) imin S yes = {x 2 }, S no ={x 3 }, S? = {x 0, x } [ x (n) 0,x (n),x (n) 2,x (n) 3 ] n=0: [ 0, 0,, 0 ] s s 3 S no n=: [ min(0, ), ,, 0 ] = [ 0, 0.4,, 0 ] n=2: [ min(0.4, ), ,, 0 ] = [ 0.4, 0.6,, 0 ] n=3: 28

29 Example - PCTL until (value iteration) s 0 {a} 0.4 s s s 3 S yes S no [ x (n) 0,x (n),x (n) 2,x (n) 3 ] n=0: [ , ,, 0 ] n=: [ , ,, 0 ] n=2: [ , ,, 0 ] n=3: [ , ,, 0 ] n=4: [ , ,, 0 ] n=5: [ , ,, 0 ] n=6: [ , ,, 0 ] n=7: [ , ,, 0 ] n=8: [ , ,, 0 ] n=9: [ , ,, 0 ] n=20: [ , ,, 0 ] n=2: [ , ,, 0 ] [ 2/3, 4/5,, 0 ] 29

30 Example - Value iteration + LP [ x 0 (n),x (n),x 2 (n),x 3 (n) ] x n=0: [ , ,, 0 ] n=: [ , ,, 0 ] n=2: [ , ,, 0 ] n=3: [ , ,, 0 ] n=4: [ , ,, 0 ] n=5: [ , ,, 0 ] n=6: [ , ,, 0 ] n=7: [ , ,, 0 ] n=8: [ , ,, 0 ] n=9: [ , ,, 0 ] 0 0 2/3 x 0 n=20: [ , ,, 0 ] n=2: [ , ,, 0 ] [ 2/3, 4/5,, 0 ] 30

31 Method 3 - Policy iteration Value iteration: iterates over (vectors of) probabilities Policy iteration: iterates over strategies ( policies ). Start with an arbitrary (memoryless) strategy σ 2. Compute the reachability probabilities Pr σ (F a) for σ 3. Improve the strategy in each state 4. Repeat 2/3 until no change in strategy Termination: finite number of memoryless strategies improvement in (minimum) probabilities each time 3

32 Method 3 - Policy iteration. Start with an arbitrary (memoryless) strategy σ pick an element of δ(s) for each state s S 2. Compute the reachability probabilities Pr σ (F a) for σ probabilistic reachability on a DTMC i.e. solve linear equation system 3. Improve the adversary in each state σ'(s) = argmin s' S µ(s') Pr σ s' (Fa) (a,µ) δ(s) 4. Repeat 2/3 until no change in strategy 32

33 Example - Policy iteration {a} S yes Arbitrary strategy σ: 0.4 s s 2 0. Compute: Pr σ (F a) Let x i = Pr s (F a) iσ x 2 =, x 3 =0 and: s s 3 S no x 0 = x x = 0. x 0 + x Solution: Pr σ (F a) = [,,, 0 ] Refine σ in state s 0 : min{(), ()+0.25(0)+0.25()} = min{, 0.75} =

34 Example - Policy iteration {a} S yes Refined strategy σ : s s 2 s s 3 S no Compute: Pr σ (F a) Let x i = Pr s (F a) iσ x 2 =, x 3 =0 and: x 0 = 0.25 x 0 + x = 0. x 0 + x Solution: Pr σ (F a) = [ 2/3, 4/5,, 0 ] This is optimal 34

35 Example - Policy iteration {a} S yes 0.4 s s 2 0. s s 3 S no x x = 0.2 x σ σ x 0 = x x 0 = 2/3 x 0 0 x 0 0 2/3 35

36 PCTL model checking - Summary Computation of set Sat(Φ) for MDP M and PCTL formula Φ recursive descent of parse tree combination of graph algorithms, numerical computation Probabilistic operator P: X Φ : one matrix-vector multiplication, O( S 2 ) Φ U k Φ 2 : k matrix-vector multiplications, O(k S 2 ) Φ U Φ 2 : linear programming problem, polynomial in S (assuming use of linear programming) Complexity: linear in Φ and polynomial in S S is states in MDP, assume δ(s) is constant 36

37 Costs and rewards for MDPs We can augment MDPs with rewards (or, conversely, costs) real-valued quantities assigned to states and/or transitions these can have a wide range of possible interpretations Some examples: elapsed time, power consumption, size of message queue, number of messages successfully delivered, net profit Extend logic PCTL with R operator, for expected reward as for PCTL, either R ~r [ ], R min=? [ ] or R max=? [ ] Some examples: R min=? [ I =90 ], R max=? [ C 60 ], R max=? [ F end ] the minimum expected queue size after exactly 90 seconds the maximum expected power consumption over one hour the maximum expected time for the algorithm to terminate 37

38 Case study: FireWire root contention FireWire (IEEE 394) high-performance serial bus for networking multimedia devices; originally by Apple "hot-pluggable" - add/remove devices at any time no requirement for a single PC (but need acyclic topology) Root contention protocol leader election algorithm, when nodes join/leave symmetric, distributed protocol uses randomisation (electronic coin tossing) and timing delays nodes send messages: "be my parent" root contention: when nodes contend leadership random choice: "fast"/"slow" delay before retry 38

39 Case study: FireWire root contention Detailed probabilistic model: probabilistic timed automaton (PTA), including: concurrency: messages between nodes and wires timing delays taken from official standard underspecification of delays (upper/lower bounds) maximum model size: 70 million states Probabilistic model checking (with PRISM) verified that root contention always resolved with probability P [ F (end elected) ] investigated worst-case expected time taken for protocol to complete R max=? [ F (end elected) ] investigated the effect of using biased coin 39

40 Case study: FireWire root contention minimum probability of electing leader by time T maximum expected time to elect a leader (using a biased coin) (using a biased coin) 40

41 Verification vs synthesis Majority of research to date has focused on verification scalability and performance of algorithms extending expressiveness of models and logics Some work to date on counterexamples but difficult to represent them compactly In this lecture, focus on simpler problem of strategy synthesis can we find a strategy to guarantee that a given quantitative property is satisfied? advantage: correct-by-construction Not a well known fact incidentally, can reuse the verification algorithms 4

42 Quantitative (probabilistic) verification Automatic verification and strategy synthesis from quantitative properties for probabilistic models System Probabilistic model e.g. Markov chain Result Quantitative results Probabilistic model checker e.g. PRISM Strategy System requirements P <0.0 [ F t fail] Probabilistic temporal logic specification e.g. PCTL, CSL, LTL 42

43 Running example Example MDP robot moving through terrain divided into 3 x 2 grid States: 0.4 {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } s 0, s, s 2, s 3, s 4, s 5 Actions: north, east, south, west, stuck Labels (atomic propositions): hazard, goal, goal 2 43

44 Properties and objectives The syntax: φ ::= P ~p [ ψ ] R ~r [ ρ ] ψ is true with probability ~p expected reward is ~r ψ ::= true a ψ ψ ψ X ψ ψ U k ψ ψ U ψ ρ ::= F b C C k next bounded until until reachability cumulative where b is an atomic proposition, used to identify states of interest, p [0,] is a probability, ~ {<,>,, }, k N, and r R 0 F b true U b We refer to φ as property, ψ and ρ as objectives (branching time more challenging for synthesis) 44

45 Properties and objectives Semantics of the probabilistic operator P can only define probabilities for a specific strategy σ s P ~p [ ψ ] means the probability, from state s, that ψ is true for an outgoing path satisfies ~p for all strategies σ formally s P ~p [ ψ ] Pr sσ (ψ) ~ p for all strategies σ where we use Pr sσ (ψ) to denote Pr sσ { ω Path sσ ω ψ } R ~r [ ] means the expected value of satisfies ~r Some examples: P 0.4 [ F goal ] probability of reaching goal is at least 0.4 R <5 [ C 60 ] expected power consumption over one hour is below 5 R 0 [ F end ] expected time to termination is at most 0 45

46 Verification and strategy synthesis The verification problem is: Given an MDP M and a property φ, does M satisfy φ under any possible strategy σ? The synthesis problem is dual: Given an MDP M and a property φ, find, if it exists, a strategy σ such that M satisfies φ under σ Verification and strategy synthesis is achieved using the same techniques, namely computing optimal values for probability objectives: Pr s min (ψ) = inf σ Pr s σ (ψ) Pr s max (ψ) = sup σ Pr s σ (ψ) and similarly for expectations 46

47 Computing reachability for MDPs Computation of probabilities Pr s max (F b) for all s S Step : pre-compute all states where probability is or 0 graph-based algorithms, yielding sets S yes, S no Step 2: compute probabilities for remaining states (S? ) (i) solve linear programming problem (i) approximate with value iteration (iii) solve with policy (strategy) iteration. Precomputation: algorithm ProbE computes S yes there exists a strategy for which the probability of "F b" is algorithm Prob0A computes S no for all strategies, the probability of satisfying "F b" is 0 47

48 Example - Reachability 0.4 {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } Example: P 0.4 [ F goal ] So compute: Pr s max (F goal ) 48

49 Example - Precomputation 0.4 {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } S yes S no Example: P 0.4 [ F goal ] So compute: Pr s max (F goal ) 49

50 Reachability for MDPs 2. Numerical computation compute probabilities Pr s max (F b) for remaining states in S? = S \ (S yes S no ) obtained as the unique solution of the linear programming (LP) problem: minimize s S? x s subject to the constraints: x s δ(s,a)(s ) x s + s S? s S yes δ(s,a)(s ) for all s S? and for all a A(s) This can be solved with standard techniques e.g. Simplex, ellipsoid method, branch-and-cut 50

51 Example Reachability (LP) 0.4 {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } Example: P 0.4 [ F goal ] Let x i = Pr s (F goal imax ) S yes : x 4 =x 5 = S no : x 2 =x 3 =0 For S? = {x 0, x } : Minimise x 0 +x subject to: x x x (east) x 0 0. x + 0. (south) x (south) x 0 (east) So compute: Pr s max (F goal ) 5

52 Example - Reachability (LP) 0.4 {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } Let x i = Pr s (F goal imax ) S yes : x 4 =x 5 = S no : x 2 =x 3 =0 For S? = {x 0, x } : Minimise x 0 +x subject to: x 0 x (east) x 0 0. x + 0. (south) x (south) x x x x 0 x x 0 0. x + 0. x 0 0 x /3 x x 0 52

53 Example - Reachability (LP) 0.4 {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } Let x i = Pr s (F goal imax ) S yes : x 4 =x 5 = S no : x 2 =x 3 =0 For S? = {x 0, x } : Minimise x 0 +x subject to: x 0 x x 0 0. x + 0. x x min Solution: (x 0, x ) = (, ) i.e. Pr s (F goal 0max ) = 0 0 2/3 x 0 53

54 Reachability for MDPs 2. Numerical computation (alternative method) value iteration it can be shown that: Pr s max (F b) = lim n x s (n) where: x (n) s = ifs S yes 0 ifs S no 0 ifs S? andn =0 (n ) max δ(s,a)(s ) x s a A(s) ifs S? andn >0 s S Approximate iterative solution technique iterations terminated when solution converges sufficiently 54

55 Example Reachability (val. iter.) 0.4 {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } Compute: Pr max s (F goal ) S yes : x 4 =x 5 = S no : x 2 =x 3 =0 S? = {x 0, x } [ x (n) 0,x (n),x (n) 2,x (n) 3,x (n) 4,x (n) 5 ] n=0: [ 0, 0, 0, 0,, ] n=: [ max( , ), max(0, ), 0, 0,, ] = [ 0.,, 0, 0,, ] n=2: [ max( , ), max(0, ), 0, 0,, ] = [ 0.34,, 0, 0,, ] 55

56 Example Reachability (val. iter.) 0.4 {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } x min 0 0 x 0 [x (n) 0,x (n),x (n) 2,x (n) 3,x (n) 4,x (n 5 ] n=0: [0, 0, 0, 0,, ] n=: [0.,, 0, 0,, ] n=2: [0.34,, 0, 0,, ] n=3: [0.436,, 0, 0,, ] n=4: [0.4744,, 0, 0,, ] n=5: [ ,, 0, 0,, ] n=6: [ ,, 0, 0,, ] n=7: [ ,, 0, 0,, ] n=8: [ ,, 0, 0,, ] n=6: [ ,, 0, 0,, ] n=7: [ ,, 0, 0,, ] [, 0, 0,, ] 56

57 Memoryless strategies Memoryless strategies suffice for probabilistic reachability i.e. there exist memoryless strategies σ min & σ max such that: Prob σ min(s, F a) = p min (s, F a) for all states s S Prob σ max(s, F a) = p max (s, F a) for all states s S Construct strategies from optimal solution: σ min (s) = argmin σ max (s) = argmax s' S s' S µ(s') p min (s',fa) (a,µ) Steps(s) µ(s') p max (s',fa) (a,µ) Steps(s) 57

58 Strategy synthesis Compute optimal probabilities Pr s max (F b) for all s S To compute the optimal strategy σ*, choose the locally optimal action in each state in general depends on the method used to compute the optimal probabilities For reachability memoryless strategies suffice For step-bounded reachability need finite-memory strategies typically requires backward computation for a fixed number of steps 58

59 Example - Strategy 0.4 {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } Optimal strategy: s 0 : east s : south s 2 : - s 3 : - s 4 : east s 5 : - x x 0 x (east) min x (south) 0 0 2/3 x 0 59

60 Example Bounded reachability 0.4 {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } Example: P max=? [ F 3 goal 2 ] So compute: Pr max s (F 3 goal 2 ) = 0.99 Optimal strategy is finite-memory: s 4 (after step): east s 4 (after2 steps): west 60

61 Strategy synthesis for LTL objectives Reduce to the problem of reachability on the product of MDP M and an omega-automaton representing ψ for example, deterministic Rabin automaton (DRA) Need only consider computation of maximum probabilities Pr s max (ψ) since Pr s min (ψ) = - Pr s max ( ψ) To compute the optimal strategy σ* find memoryless deterministic strategy on the product convert to finite-memory strategy with one mode for each state of the DRA for ψ 6

62 Example - LTL P 0.05 [ (G hazard) (GF goal ) ] avoid hazard and visit goal infinitely often Pr s 0max ((G hazard) (GF goal )) = {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } Optimal strategy: (in this instance, memoryless) s 0 : south s : - s 2 : - s 3 : - s 4 : east s 5 : west 62

63 Multi-objective strategy synthesis Consider conjunctions of probabilistic LTL formulas P ~p [ψ] require all conjuncts to be satisfied Reduce to a multi-objective reachability problem on the product of MDP M and the omega-automata representing the conjuncts convert (by negation) to formulas with lower probability bounds (, <), then to DRA need to consider all combinations of objectives The problem can be solved using LP methods [TACAS07] or via approximations to Pareto curve [ATVA2] strategies may be finite memory and randomised Continue as for single-objectives to compute the strategy σ* find memoryless deterministic strategy on the product convert to finite-memory strategy 63

64 Example Multi-objective 0.4 {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } ψ ψ = G hazard ψ 2 = GF goal ψ Multi-objective formula P 0.7 [ G hazard ] P 0.2 [ GF goal ]? True (achievable) Numerical query P max=? [ GF goal ] such that P 0.7 [ G hazard ]? ~ Pareto query for P max=? [ G hazard ] P max=? [ GF goal ]? 64

65 Example Multi-objective strategies 0.4 {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } Strategy (deterministic) s 0 : east s : south s 2 : - s 3 : - s 4 : east s 5 : west ψ ψ = G hazard ψ 2 = GF goal ψ 65

66 Example Multi-objective strategies 0.4 {hazard} {goal 2 } 0.6 east s 0 east s s 2 0. south stuck south north 0. stuck east 0.6 west s 3 s 4 s 5 west {goal 2 } 0.4 {goal } Strategy 2 (deterministic) s 0 : south s : south s 2 : - s 3 : - s 4 : east s 5 : west ψ ψ = G hazard ψ 2 = GF goal ψ 66

67 Example Multi-objective strategies stuck 0.4 {hazard} 0.6 east s 0 east s 0. south south s 3 {goal 2 } ψ west s 4 west {goal 2 } 0.9 s 2 north east s 5 {goal } ψ = G hazard ψ 2 = GF goal stuck 0. ψ Optimal strategy: (randomised) s 0 : : east : south s :.0 : south s 2 : - s 3 : - s 4 :.0 : east s 5 :.0 : west 67

68 Case study: Dynamic power management Synthesis of dynamic power management schemes for an IBM TravelStar VP disk drive 5 different power modes: active, idle, idlelp, stby, sleep power manager controller bases decisions on current power mode, disk request queue, etc. Build controllers that minimise energy consumption, subject to constraints on e.g. probability that a request waits more than K steps expected number of lost disk requests See: 68

69 Summary (Part 2) Markov decision processes (MDPs) extend DTMCs with nondeterminism to model concurrency, underspecification, Property specifications PCTL: exactly same syntax as for DTMCs but quantify over all strategies Model checking algorithms covered three basic techniques for MDPs: linear programming, value iteration, or policy iteration Strategy synthesis can reuse model checking algorithms 69

70 PRISM: Recent & new developments Major new features:. multi-objective model checking 2. parametric model checking 3. real-time: probabilistic timed automata (PTAs) 4. games: stochastic multi-player games (SMGs) Further new additions: strategy (adversary) synthesis CTL model checking & counterexample generation enhanced statistical model checking (approximations + confidence intervals, acceptance sampling) efficient CTMC model checking (fast adaptive uniformisation) [Mateescu et al., CMSB'3] benchmark suite & testing functionality [QEST'2] 70

71 Acknowledgements My group and collaborators in this work Project funding ERC, EPSRC, Microsoft Research Oxford Martin School, Institute for the Future of Computing See also PRISM 7

72 PhD Comics and Oxford You are welcome to visit Oxford! PhD scholarships, postdocs in verification and synthesis, and more 72 72

SFM-11:CONNECT Summer School, Bertinoro, June 2011

SFM-11:CONNECT Summer School, Bertinoro, June 2011 SFM-:CONNECT Summer School, Bertinoro, June 20 EU-FP7: CONNECT LSCITS/PSS VERIWARE Part 3 Markov decision processes Overview Lectures and 2: Introduction 2 Discrete-time Markov chains 3 Markov decision

More information

Probabilistic Model Checking Michaelmas Term Dr. Dave Parker. Department of Computer Science University of Oxford

Probabilistic Model Checking Michaelmas Term Dr. Dave Parker. Department of Computer Science University of Oxford Probabilistic Model Checking Michaelmas Term 20 Dr. Dave Parker Department of Computer Science University of Oxford Overview PCTL for MDPs syntax, semantics, examples PCTL model checking next, bounded

More information

Probabilistic Model Checking and Strategy Synthesis for Robot Navigation

Probabilistic Model Checking and Strategy Synthesis for Robot Navigation Probabilistic Model Checking and Strategy Synthesis for Robot Navigation Dave Parker University of Birmingham (joint work with Bruno Lacerda, Nick Hawes) AIMS CDT, Oxford, May 2015 Overview Probabilistic

More information

Probabilistic Model Checking Michaelmas Term Dr. Dave Parker. Department of Computer Science University of Oxford

Probabilistic Model Checking Michaelmas Term Dr. Dave Parker. Department of Computer Science University of Oxford Probabilistic Model Checking Michaelmas Term 2011 Dr. Dave Parker Department of Computer Science University of Oxford Probabilistic model checking System Probabilistic model e.g. Markov chain Result 0.5

More information

Probabilistic Model Checking Michaelmas Term Dr. Dave Parker. Department of Computer Science University of Oxford

Probabilistic Model Checking Michaelmas Term Dr. Dave Parker. Department of Computer Science University of Oxford Probabilistic Model Checking Michaelmas Term 2011 Dr. Dave Parker Department of Computer Science University of Oxford Overview Temporal logic Non-probabilistic temporal logic CTL Probabilistic temporal

More information

PRISM An overview. automatic verification of systems with stochastic behaviour e.g. due to unreliability, uncertainty, randomisation,

PRISM An overview. automatic verification of systems with stochastic behaviour e.g. due to unreliability, uncertainty, randomisation, PRISM An overview PRISM is a probabilistic model checker automatic verification of systems with stochastic behaviour e.g. due to unreliability, uncertainty, randomisation, Construction/analysis of probabilistic

More information

Modelling and Analysis (and towards Synthesis)

Modelling and Analysis (and towards Synthesis) Modelling and Analysis (and towards Synthesis) Marta Kwiatkowska Department of Computer Science, University of Oxford Dagstuhl meeting 15041, 19-23 rd Jan 2015 Outline Introduction The context: role of

More information

Probabilistic model checking with PRISM

Probabilistic model checking with PRISM Probabilistic model checking with PRISM Marta Kwiatkowska Department of Computer Science, University of Oxford IMT, Lucca, May 206 Lecture plan Course slides and lab session http://www.prismmodelchecker.org/courses/imt6/

More information

Probabilistic Model Checking: Advances and Applications

Probabilistic Model Checking: Advances and Applications Probabilistic Model Checking: Advances and Applications Dave Parker University of Birmingham Highlights 18, Berlin, September 2018 Overview Probabilistic model checking & PRISM Markov decision processes

More information

Probabilistic Model Checking Michaelmas Term Dr. Dave Parker. Department of Computer Science University of Oxford

Probabilistic Model Checking Michaelmas Term Dr. Dave Parker. Department of Computer Science University of Oxford Probabilistic Model Checking Michaelmas Term 2011 Dr. Dave Parker Department of Computer Science University of Oxford Overview CSL model checking basic algorithm untimed properties time-bounded until the

More information

Probabilistic model checking with PRISM

Probabilistic model checking with PRISM Probabilistic model checking with PRISM Marta Kwiatkowska Department of Computer Science, University of Oxford 4th SSFT, Menlo College, May 2014 What is probabilistic model checking? Probabilistic model

More information

Stochastic Model Checking

Stochastic Model Checking Stochastic Model Checking Marta Kwiatkowska, Gethin Norman, and David Parker School of Computer Science, University of Birmingham Edgbaston, Birmingham B15 2TT, United Kingdom Abstract. This tutorial presents

More information

Verification and Control of Partially Observable Probabilistic Systems

Verification and Control of Partially Observable Probabilistic Systems Verification and Control of Partially Observable Probabilistic Systems Gethin Norman 1, David Parker 2, and Xueyi Zou 3 1 School of Computing Science, University of Glasgow, UK 2 School of Computer Science,

More information

Variations on Itai-Rodeh Leader Election for Anonymous Rings and their Analysis in PRISM

Variations on Itai-Rodeh Leader Election for Anonymous Rings and their Analysis in PRISM Variations on Itai-Rodeh Leader Election for Anonymous Rings and their Analysis in PRISM Wan Fokkink (Vrije Universiteit, Section Theoretical Computer Science CWI, Embedded Systems Group Amsterdam, The

More information

Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties and its Application to Cyber-Physical Systems

Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties and its Application to Cyber-Physical Systems Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties and its Application to Cyber-Physical Systems Alberto Puggelli DREAM Seminar - November 26, 2013 Collaborators and PIs:

More information

Probabilistic Model Checking (1)

Probabilistic Model Checking (1) Probabilistic Model Checking () Lecture # of GLOBAN Summerschool Joost-Pieter Katoen Software Modeling and Verification Group affiliated to University of Twente, Formal Methods and Tools Warsaw University,

More information

Quantitative analysis with the probabilistic model checker PRISM 1

Quantitative analysis with the probabilistic model checker PRISM 1 QAPL 2005 Preliminary Version Quantitative analysis with the probabilistic model checker PRISM 1 Marta Kwiatkowska Gethin Norman David Parker 2 School of Computer Science, University of Birmingham Edgbaston,

More information

Probabilistic Model Checking of Randomised Distributed Protocols using PRISM

Probabilistic Model Checking of Randomised Distributed Protocols using PRISM Probabilistic Model Checking of Randomised Distributed Protocols using PRISM Marta Kwiatkowska University of Birmingham VPSM PhD School, Copenhagen, October 2006 Tutorial overview Part I - Probabilistic

More information

Probabilistic Model Checking of Deadline Properties in the IEEE 1394 FireWire Root Contention Protocol 1

Probabilistic Model Checking of Deadline Properties in the IEEE 1394 FireWire Root Contention Protocol 1 Under consideration for publication in Formal Aspects of Computing Probabilistic Model Checking of Deadline Properties in the IEEE 1394 FireWire Root Contention Protocol 1 Marta Kwiatkowska a, Gethin Norman

More information

Probabilistic Model Checking Michaelmas Term Dr. Dave Parker. Department of Computer Science University of Oxford

Probabilistic Model Checking Michaelmas Term Dr. Dave Parker. Department of Computer Science University of Oxford Probabilistic Model Checking Michaelmas Term 20 Dr. Dave Parker Department of Computer Science University of Oxford Next few lectures Today: Discrete-time Markov chains (continued) Mon 2pm: Probabilistic

More information

Quantitative Verification and Synthesis of Attack-Defence Scenarios

Quantitative Verification and Synthesis of Attack-Defence Scenarios Quantitative Verification and Synthesis of Attack-Defence Scenarios Zaruhi Aslanyan, Flemming Nielson DTU Compute Technical University of Denmark Denmark Email: {zaas,fnie}@dtu.dk David Parker School of

More information

Temporal logics and explicit-state model checking. Pierre Wolper Université de Liège

Temporal logics and explicit-state model checking. Pierre Wolper Université de Liège Temporal logics and explicit-state model checking Pierre Wolper Université de Liège 1 Topics to be covered Introducing explicit-state model checking Finite automata on infinite words Temporal Logics and

More information

Alan Bundy. Automated Reasoning LTL Model Checking

Alan Bundy. Automated Reasoning LTL Model Checking Automated Reasoning LTL Model Checking Alan Bundy Lecture 9, page 1 Introduction So far we have looked at theorem proving Powerful, especially where good sets of rewrite rules or decision procedures have

More information

Chapter 13: Model Checking Linear-Time Properties of Probabilistic Systems

Chapter 13: Model Checking Linear-Time Properties of Probabilistic Systems Chapter 13: Model Checking Linear-Time Properties of Probabilistic Systems Christel Baier, Marcus Größer, and Frank Ciesinski Technische Universität Dresden, Fakultät Informatik, Institut für Theoretische

More information

A Brief Introduction to Model Checking

A Brief Introduction to Model Checking A Brief Introduction to Model Checking Jan. 18, LIX Page 1 Model Checking A technique for verifying finite state concurrent systems; a benefit on this restriction: largely automatic; a problem to fight:

More information

Automated Verification and Strategy Synthesis for Probabilistic Systems

Automated Verification and Strategy Synthesis for Probabilistic Systems Automated Verification and Strategy Synthesis for Probabilistic Systems Marta Kwiatkowska 1 and David Parker 2 1 Department of Computer Science, University of Oxford, UK 2 School of Computer Science, University

More information

State Explosion in Almost-Sure Probabilistic Reachability

State Explosion in Almost-Sure Probabilistic Reachability State Explosion in Almost-Sure Probabilistic Reachability François Laroussinie Lab. Spécification & Vérification, ENS de Cachan & CNRS UMR 8643, 61, av. Pdt. Wilson, 94235 Cachan Cedex France Jeremy Sproston

More information

Probabilistic verification and approximation schemes

Probabilistic verification and approximation schemes Probabilistic verification and approximation schemes Richard Lassaigne Equipe de Logique mathématique, CNRS-Université Paris 7 Joint work with Sylvain Peyronnet (LRDE/EPITA & Equipe de Logique) Plan 1

More information

Timo Latvala. March 7, 2004

Timo Latvala. March 7, 2004 Reactive Systems: Safety, Liveness, and Fairness Timo Latvala March 7, 2004 Reactive Systems: Safety, Liveness, and Fairness 14-1 Safety Safety properties are a very useful subclass of specifications.

More information

Quantitative Verification: Models, Techniques and Tools

Quantitative Verification: Models, Techniques and Tools Quantitative Verification: Models, Techniques and Tools Marta Kwiatkowska Oxford University Computing Laboratory Parks Road, Oxford OX 3QD, UK mzk@comlab.ox.ac.uk ABSTRACT Automated verification is a technique

More information

Verifying Randomized Distributed Algorithms with PRISM

Verifying Randomized Distributed Algorithms with PRISM Verifying Randomized Distributed Algorithms with PRISM Marta Kwiatkowska, Gethin Norman, and David Parker University of Birmingham, Birmingham B15 2TT, United Kingdom {M.Z.Kwiatkowska,G.Norman,D.A.Parker}@cs.bham.ac.uk

More information

On Model Checking Techniques for Randomized Distributed Systems. Christel Baier Technische Universität Dresden

On Model Checking Techniques for Randomized Distributed Systems. Christel Baier Technische Universität Dresden On Model Checking Techniques for Randomized Distributed Systems Christel Baier Technische Universität Dresden joint work with Nathalie Bertrand Frank Ciesinski Marcus Größer / 6 biological systems, resilient

More information

From Liveness to Promptness

From Liveness to Promptness From Liveness to Promptness Orna Kupferman Hebrew University Nir Piterman EPFL Moshe Y. Vardi Rice University Abstract Liveness temporal properties state that something good eventually happens, e.g., every

More information

Overview. overview / 357

Overview. overview / 357 Overview overview6.1 Introduction Modelling parallel systems Linear Time Properties Regular Properties Linear Temporal Logic (LTL) Computation Tree Logic syntax and semantics of CTL expressiveness of CTL

More information

Quantitative Verification

Quantitative Verification Quantitative Verification Chapter 3: Markov chains Jan Křetínský Technical University of Munich Winter 207/8 / 84 Motivation 2 / 84 Example: Simulation of a die by coins Knuth & Yao die Simulating a Fair

More information

Leveraging Weighted Automata in Compositional Reasoning about Concurrent Probabilistic Systems

Leveraging Weighted Automata in Compositional Reasoning about Concurrent Probabilistic Systems Leveraging Weighted Automata in Compositional Reasoning about Concurrent Probabilistic Systems Fei He,2,3 Xiaowei Gao,2,3 Bow-Yaw Wang Lijun Zhang 5 Tsinghua National Laboratory for Information Science

More information

The State Explosion Problem

The State Explosion Problem The State Explosion Problem Martin Kot August 16, 2003 1 Introduction One from main approaches to checking correctness of a concurrent system are state space methods. They are suitable for automatic analysis

More information

Automata-based Verification - III

Automata-based Verification - III COMP30172: Advanced Algorithms Automata-based Verification - III Howard Barringer Room KB2.20: email: howard.barringer@manchester.ac.uk March 2009 Third Topic Infinite Word Automata Motivation Büchi Automata

More information

Modeling and Analysis of Probabilistic Timed Systems

Modeling and Analysis of Probabilistic Timed Systems Modeling and Analysis of Probabilistic Timed Systems Abhishek Dubey Derek Riley Sherif Abdelwahed Ted Bapty Institute for Software Integrated Systems, Vanderbilt University, Nashville, TN, USA Electrical

More information

Automatic Verification of Real-time Systems with Discrete Probability Distributions

Automatic Verification of Real-time Systems with Discrete Probability Distributions Automatic Verification of Real-time Systems with Discrete Probability Distributions Marta Kwiatkowska a, Gethin Norman a, Roberto Segala b and Jeremy Sproston a a University of Birmingham, Birmingham B15

More information

Outline. The Leader Election Protocol (IEEE 1394) IEEE 1394 High Performance Serial Bus (FireWire) Motivation. Background. Protocol Descriptions

Outline. The Leader Election Protocol (IEEE 1394) IEEE 1394 High Performance Serial Bus (FireWire) Motivation. Background. Protocol Descriptions Outline The Leader Election Protocol (IEEE 1394) Thai Son Hoang (inspired by the slides of Jean-Raymond Abrial) Department of Computer Science Swiss Federal Institute of Technology Zürich (ETH Zürich)

More information

Infinite Games. Sumit Nain. 28 January Slides Credit: Barbara Jobstmann (CNRS/Verimag) Department of Computer Science Rice University

Infinite Games. Sumit Nain. 28 January Slides Credit: Barbara Jobstmann (CNRS/Verimag) Department of Computer Science Rice University Infinite Games Sumit Nain Department of Computer Science Rice University 28 January 2013 Slides Credit: Barbara Jobstmann (CNRS/Verimag) Motivation Abstract games are of fundamental importance in mathematics

More information

Course 16:198:520: Introduction To Artificial Intelligence Lecture 13. Decision Making. Abdeslam Boularias. Wednesday, December 7, 2016

Course 16:198:520: Introduction To Artificial Intelligence Lecture 13. Decision Making. Abdeslam Boularias. Wednesday, December 7, 2016 Course 16:198:520: Introduction To Artificial Intelligence Lecture 13 Decision Making Abdeslam Boularias Wednesday, December 7, 2016 1 / 45 Overview We consider probabilistic temporal models where the

More information

Chapter 4: Computation tree logic

Chapter 4: Computation tree logic INFOF412 Formal verification of computer systems Chapter 4: Computation tree logic Mickael Randour Formal Methods and Verification group Computer Science Department, ULB March 2017 1 CTL: a specification

More information

Computing Laboratory STOCHASTIC GAMES FOR VERIFICATION OF PROBABILISTIC TIMED AUTOMATA. Marta Kwiatkowska Gethin Norman David Parker CL-RR-09-05

Computing Laboratory STOCHASTIC GAMES FOR VERIFICATION OF PROBABILISTIC TIMED AUTOMATA. Marta Kwiatkowska Gethin Norman David Parker CL-RR-09-05 Computing Laboratory STOCHASTIC AMES FOR VERIFICATION OF PROBABILISTIC TIMED AUTOMATA Marta Kwiatkowska ethin Norman David Parker CL-RR-09-05 Oxford University Computing Laboratory Wolfson Building, Parks

More information

Compositional Verification of Probabilistic Systems using Learning

Compositional Verification of Probabilistic Systems using Learning Compositional Verification of Probabilistic Systems using Learning Lu Feng, Marta Kwiatkowska, David Parker Oxford University Computing Laboratory, Parks Road, Oxford, OX1 3QD Email: {lu.feng, marta.kwiatkowska,

More information

A Counterexample Guided Abstraction-Refinement Framework for Markov Decision Processes

A Counterexample Guided Abstraction-Refinement Framework for Markov Decision Processes A Counterexample Guided Abstraction-Refinement Framework for Markov Decision Processes ROHIT CHADHA and MAHESH VISWANATHAN Dept. of Computer Science, University of Illinois at Urbana-Champaign The main

More information

Quantitative Verification: Models, Techniques and Tools

Quantitative Verification: Models, Techniques and Tools Quantitative Verification: Models, Techniques and Tools Marta Kwiatkowska Computing Laboratory, University of Oxford http://web.comlab comlab.ox.ac..ox.ac.uk/oucl/people/marta.kwiatkowska.html Overview

More information

LTL Control in Uncertain Environments with Probabilistic Satisfaction Guarantees

LTL Control in Uncertain Environments with Probabilistic Satisfaction Guarantees LTL Control in Uncertain Environments with Probabilistic Satisfaction Guarantees Xu Chu (Dennis) Ding Stephen L. Smith Calin Belta Daniela Rus Department of Mechanical Engineering, Boston University, Boston,

More information

Planning Under Uncertainty II

Planning Under Uncertainty II Planning Under Uncertainty II Intelligent Robotics 2014/15 Bruno Lacerda Announcement No class next Monday - 17/11/2014 2 Previous Lecture Approach to cope with uncertainty on outcome of actions Markov

More information

On Verification and Controller Synthesis for Probabilistic Systems at Runtime

On Verification and Controller Synthesis for Probabilistic Systems at Runtime On Verification and Controller Synthesis for Probabilistic Systems at Runtime by Mateusz Ujma A thesis submitted for the degree of Doctor of Philosophy in Computer Science Wolfson College, Oxford Hilary

More information

Automatic Synthesis of Distributed Protocols

Automatic Synthesis of Distributed Protocols Automatic Synthesis of Distributed Protocols Rajeev Alur Stavros Tripakis 1 Introduction Protocols for coordination among concurrent processes are an essential component of modern multiprocessor and distributed

More information

Optimal Control of Markov Decision Processes with Temporal Logic Constraints

Optimal Control of Markov Decision Processes with Temporal Logic Constraints Optimal Control of Markov Decision Processes with Temporal Logic Constraints Xuchu (Dennis) Ding Stephen L. Smith Calin Belta Daniela Rus Abstract In this paper, we develop a method to automatically generate

More information

Automata-based Verification - III

Automata-based Verification - III CS3172: Advanced Algorithms Automata-based Verification - III Howard Barringer Room KB2.20/22: email: howard.barringer@manchester.ac.uk March 2005 Third Topic Infinite Word Automata Motivation Büchi Automata

More information

A Survey of Stochastic ω-regular Games

A Survey of Stochastic ω-regular Games A Survey of Stochastic ω-regular Games Krishnendu Chatterjee Thomas A. Henzinger EECS, University of California, Berkeley, USA Computer and Communication Sciences, EPFL, Switzerland {c krish,tah}@eecs.berkeley.edu

More information

Formal Verification via MCMAS & PRISM

Formal Verification via MCMAS & PRISM Formal Verification via MCMAS & PRISM Hongyang Qu University of Sheffield 1 December 2015 Outline Motivation for Formal Verification Overview of MCMAS Overview of PRISM Formal verification It is a systematic

More information

Reasoning about Time and Reliability

Reasoning about Time and Reliability Reasoning about Time and Reliability Probabilistic CTL model checking Daniel Bruns Institut für theoretische Informatik Universität Karlsruhe 13. Juli 2007 Seminar Theorie und Anwendung von Model Checking

More information

Model checking and strategy synthesis for mobile autonomy: from theory to practice

Model checking and strategy synthesis for mobile autonomy: from theory to practice Model checking and strategy synthesis for mobile autonomy: from theory to practice Marta Kwiatkowska Department of Computer Science, University of Oxford University of Maryland, 24 th October 2016 Mobile

More information

Chapter 6: Computation Tree Logic

Chapter 6: Computation Tree Logic Chapter 6: Computation Tree Logic Prof. Ali Movaghar Verification of Reactive Systems Outline We introduce Computation Tree Logic (CTL), a branching temporal logic for specifying system properties. A comparison

More information

Stochastic Games with Time The value Min strategies Max strategies Determinacy Finite-state games Cont.-time Markov chains

Stochastic Games with Time The value Min strategies Max strategies Determinacy Finite-state games Cont.-time Markov chains Games with Time Finite-state Masaryk University Brno GASICS 00 /39 Outline Finite-state stochastic processes. Games over event-driven stochastic processes. Strategies,, determinacy. Existing results for

More information

Chapter 5: Linear Temporal Logic

Chapter 5: Linear Temporal Logic Chapter 5: Linear Temporal Logic Prof. Ali Movaghar Verification of Reactive Systems Spring 94 Outline We introduce linear temporal logic (LTL), a logical formalism that is suited for specifying LT properties.

More information

PRISM: Probabilistic Model Checking for Performance and Reliability Analysis

PRISM: Probabilistic Model Checking for Performance and Reliability Analysis PRISM: Probabilistic Model Checking for Performance and Reliability Analysis Marta Kwiatkowska, Gethin Norman and David Parker Oxford University Computing Laboratory, Wolfson Building, Parks Road, Oxford,

More information

CONTROLLER DEPENDABILITY ANALYSIS BY PROBABILISTIC MODEL CHECKING. Marta Kwiatkowska, Gethin Norman and David Parker 1

CONTROLLER DEPENDABILITY ANALYSIS BY PROBABILISTIC MODEL CHECKING. Marta Kwiatkowska, Gethin Norman and David Parker 1 CONTROLLER DEPENDABILITY ANALYSIS BY PROBABILISTIC MODEL CHECKING Marta Kwiatkowska, Gethin Norman and David Parker 1 School of Computer Science, University of Birmingham, Birmingham, B15 2TT, United Kingdom

More information

Analysing Randomized Distributed Algorithms

Analysing Randomized Distributed Algorithms Analysing Randomized Distributed Algorithms Gethin Norman School of Computer Science, University of Birmingham, Birmingham B15 2TT, United Kingdom G.Norman@cs.bham.ac.uk Abstract. Randomization is of paramount

More information

Playing Stochastic Games Precisely

Playing Stochastic Games Precisely Playing Stochastic Games Precisely Taolue Chen 1, Vojtěch Forejt 1, Marta Kwiatkowska 1, Aistis Simaitis 1, Ashutosh Trivedi 2, and Michael Ummels 3 1 Department of Computer Science, University of Oxford,

More information

T Reactive Systems: Temporal Logic LTL

T Reactive Systems: Temporal Logic LTL Tik-79.186 Reactive Systems 1 T-79.186 Reactive Systems: Temporal Logic LTL Spring 2005, Lecture 4 January 31, 2005 Tik-79.186 Reactive Systems 2 Temporal Logics Temporal logics are currently the most

More information

Using first-order logic, formalize the following knowledge:

Using first-order logic, formalize the following knowledge: Probabilistic Artificial Intelligence Final Exam Feb 2, 2016 Time limit: 120 minutes Number of pages: 19 Total points: 100 You can use the back of the pages if you run out of space. Collaboration on the

More information

Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties

Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties Alberto Alessandro Angelo Puggelli Wenchao Li Alberto L. Sangiovanni-Vincentelli Sanjit A. Seshia Electrical Engineering

More information

Model checking and strategy synthesis for mobile autonomy: from theory to practice

Model checking and strategy synthesis for mobile autonomy: from theory to practice Model checking and strategy synthesis for mobile autonomy: from theory to practice Marta Kwiatkowska Department of Computer Science, University of Oxford Uncertainty in Computation, Simons Institute, 4

More information

Counterexamples for Robotic Planning Explained in Structured Language

Counterexamples for Robotic Planning Explained in Structured Language Counterexamples for Robotic Planning Explained in Structured Language Lu Feng 1, Mahsa Ghasemi 2, Kai-Wei Chang 3, and Ufuk Topcu 4 Abstract Automated techniques such as model checking have been used to

More information

Kwiatkowska, M., Norman, G., Parker, D. and Qu, H. (2010) Assume-guarantee verification for probabilistic systems. In: Esparza, J. and Majumdar, R. (eds.) Tools and Algorithms for the Construction and

More information

Computation Tree Logic

Computation Tree Logic Computation Tree Logic Hao Zheng Department of Computer Science and Engineering University of South Florida Tampa, FL 33620 Email: zheng@cse.usf.edu Phone: (813)974-4757 Fax: (813)974-5456 Hao Zheng (CSE,

More information

Linear Temporal Logic and Büchi Automata

Linear Temporal Logic and Büchi Automata Linear Temporal Logic and Büchi Automata Yih-Kuen Tsay Department of Information Management National Taiwan University FLOLAC 2009 Yih-Kuen Tsay (SVVRL @ IM.NTU) Linear Temporal Logic and Büchi Automata

More information

ONR MURI AIRFOILS: Animal Inspired Robust Flight with Outer and Inner Loop Strategies. Calin Belta

ONR MURI AIRFOILS: Animal Inspired Robust Flight with Outer and Inner Loop Strategies. Calin Belta ONR MURI AIRFOILS: Animal Inspired Robust Flight with Outer and Inner Loop Strategies Provable safety for animal inspired agile flight Calin Belta Hybrid and Networked Systems (HyNeSs) Lab Department of

More information

The Leader Election Protocol (IEEE 1394)

The Leader Election Protocol (IEEE 1394) The Leader Election Protocol (IEEE 1394) J.R. Abrial, D. Cansell, D. Méry July 2002 This Session - Background :-) - An informal presentation of the protocol :-) - Step by step formal design :- - Short

More information

Optimal Control of MDPs with Temporal Logic Constraints

Optimal Control of MDPs with Temporal Logic Constraints 52nd IEEE Conference on Decision and Control December 10-13, 2013. Florence, Italy Optimal Control of MDPs with Temporal Logic Constraints Mária Svoreňová, Ivana Černá and Calin Belta Abstract In this

More information

Automata on Infinite words and LTL Model Checking

Automata on Infinite words and LTL Model Checking Automata on Infinite words and LTL Model Checking Rodica Condurache Lecture 4 Lecture 4 Automata on Infinite words and LTL Model Checking 1 / 35 Labeled Transition Systems Let AP be the (finite) set of

More information

Computation Tree Logic (CTL) & Basic Model Checking Algorithms

Computation Tree Logic (CTL) & Basic Model Checking Algorithms Computation Tree Logic (CTL) & Basic Model Checking Algorithms Martin Fränzle Carl von Ossietzky Universität Dpt. of Computing Science Res. Grp. Hybride Systeme Oldenburg, Germany 02917: CTL & Model Checking

More information

A note on the attractor-property of infinite-state Markov chains

A note on the attractor-property of infinite-state Markov chains A note on the attractor-property of infinite-state Markov chains Christel Baier a, Nathalie Bertrand b, Philippe Schnoebelen b a Universität Bonn, Institut für Informatik I, Germany b Lab. Specification

More information

Model Checking: An Introduction

Model Checking: An Introduction Model Checking: An Introduction Meeting 3, CSCI 5535, Spring 2013 Announcements Homework 0 ( Preliminaries ) out, due Friday Saturday This Week Dive into research motivating CSCI 5535 Next Week Begin foundations

More information

Lecture notes for Analysis of Algorithms : Markov decision processes

Lecture notes for Analysis of Algorithms : Markov decision processes Lecture notes for Analysis of Algorithms : Markov decision processes Lecturer: Thomas Dueholm Hansen June 6, 013 Abstract We give an introduction to infinite-horizon Markov decision processes (MDPs) with

More information

Simulation and verification for computational modelling of signalling pathways

Simulation and verification for computational modelling of signalling pathways Simulation and verification for computational modelling of signalling pathways Marta Kwiatkowska Computing Laboratory, University of Oxford http://www.comlab comlab.ox.ac..ox.ac.uk/people/marta.kwiatkowska.html

More information

Helsinki University of Technology Laboratory for Theoretical Computer Science Research Reports 66

Helsinki University of Technology Laboratory for Theoretical Computer Science Research Reports 66 Helsinki University of Technology Laboratory for Theoretical Computer Science Research Reports 66 Teknillisen korkeakoulun tietojenkäsittelyteorian laboratorion tutkimusraportti 66 Espoo 2000 HUT-TCS-A66

More information

Theoretical Foundations of the UML

Theoretical Foundations of the UML Theoretical Foundations of the UML Lecture 17+18: A Logic for MSCs Joost-Pieter Katoen Lehrstuhl für Informatik 2 Software Modeling and Verification Group moves.rwth-aachen.de/teaching/ws-1718/fuml/ 5.

More information

MODEL CHECKING. Arie Gurfinkel

MODEL CHECKING. Arie Gurfinkel 1 MODEL CHECKING Arie Gurfinkel 2 Overview Kripke structures as models of computation CTL, LTL and property patterns CTL model-checking and counterexample generation State of the Art Model-Checkers 3 SW/HW

More information

arxiv: v2 [cs.lo] 22 Jul 2017

arxiv: v2 [cs.lo] 22 Jul 2017 Tableaux for Policy Synthesis for MDPs with PCTL* Constraints Peter Baumgartner, Sylvie Thiébaux, and Felipe Trevizan Data61/CSIRO and Research School of Computer Science, ANU, Australia Email: first.last@anu.edu.au

More information

Department of Computer Science CS-RR-16-03

Department of Computer Science CS-RR-16-03 Department of Computer Science Compositional Strategy Synthesis for Stochastic ames with Multiple Objectives Nicolas Basset, Marta Kwiatkowska and Clemens Wiltsche CS-RR-6-03 Department of Computer Science,

More information

Timo Latvala. February 4, 2004

Timo Latvala. February 4, 2004 Reactive Systems: Temporal Logic LT L Timo Latvala February 4, 2004 Reactive Systems: Temporal Logic LT L 8-1 Temporal Logics Temporal logics are currently the most widely used specification formalism

More information

Model Checking & Program Analysis

Model Checking & Program Analysis Model Checking & Program Analysis Markus Müller-Olm Dortmund University Overview Introduction Model Checking Flow Analysis Some Links between MC and FA Conclusion Apology for not giving proper credit to

More information

Introduction to Temporal Logic. The purpose of temporal logics is to specify properties of dynamic systems. These can be either

Introduction to Temporal Logic. The purpose of temporal logics is to specify properties of dynamic systems. These can be either Introduction to Temporal Logic The purpose of temporal logics is to specify properties of dynamic systems. These can be either Desired properites. Often liveness properties like In every infinite run action

More information

Symmetry Reduction for Probabilistic Model Checking

Symmetry Reduction for Probabilistic Model Checking Symmetry Reduction for Probabilistic Model Checking Marta Kwiatkowska, Gethin Norman, and David Parker School of Computer Science, University of Birmingham, Birmingham B15 2TT, United Kingdom {mzk,gxn,dxp}@cs.bham.ac.uk

More information

Logic Model Checking

Logic Model Checking Logic Model Checking Lecture Notes 10:18 Caltech 101b.2 January-March 2004 Course Text: The Spin Model Checker: Primer and Reference Manual Addison-Wesley 2003, ISBN 0-321-22862-6, 608 pgs. the assignment

More information

Synthesis weakness of standard approach. Rational Synthesis

Synthesis weakness of standard approach. Rational Synthesis 1 Synthesis weakness of standard approach Rational Synthesis 3 Overview Introduction to formal verification Reactive systems Verification Synthesis Introduction to Formal Verification of Reactive Systems

More information

Verification and control of partially observable probabilistic systems

Verification and control of partially observable probabilistic systems Real-Time Syst (2017) 53:354 402 DOI 10.1007/s11241-017-9269-4 Verification and control of partially observable probabilistic systems Gethin Norman 1 David Parker 2 Xueyi Zou 3 Published online: 8 March

More information

Advances and Challenges of Probabilistic Model Checking

Advances and Challenges of Probabilistic Model Checking Author manuscript, published in "48th Annual Allerton Conference on Communication, Control and Computing (2010) 1691-1698" Advances and Challenges of Probabilistic Model Checking Marta Kwiatkowska Computing

More information

Complexity Issues in Automated Addition of Time-Bounded Liveness Properties 1

Complexity Issues in Automated Addition of Time-Bounded Liveness Properties 1 Complexity Issues in Automated Addition of Time-Bounded Liveness Properties 1 Borzoo Bonakdarpour and Sandeep S. Kulkarni Software Engineering and Network Systems Laboratory, Department of Computer Science

More information

Lecture 7 Synthesis of Reactive Control Protocols

Lecture 7 Synthesis of Reactive Control Protocols Lecture 7 Synthesis of Reactive Control Protocols Richard M. Murray Nok Wongpiromsarn Ufuk Topcu California Institute of Technology AFRL, 25 April 2012 Outline Review: networked control systems and cooperative

More information

Lecture 9 Synthesis of Reactive Control Protocols

Lecture 9 Synthesis of Reactive Control Protocols Lecture 9 Synthesis of Reactive Control Protocols Nok Wongpiromsarn Singapore-MIT Alliance for Research and Technology Richard M. Murray and Ufuk Topcu California Institute of Technology EECI, 16 May 2012

More information

ω-automata Automata that accept (or reject) words of infinite length. Languages of infinite words appear:

ω-automata Automata that accept (or reject) words of infinite length. Languages of infinite words appear: ω-automata ω-automata Automata that accept (or reject) words of infinite length. Languages of infinite words appear: in verification, as encodings of non-terminating executions of a program. in arithmetic,

More information

Probabilistic verification and synthesis

Probabilistic verification and synthesis Probabilistic verification and synthesis Marta Kwiatkowska Department of Computer Science, University of Oxford KTH, Stockholm, August 2015 What is probabilistic verification? Probabilistic verification

More information