Stochastic Games with Time The value Min strategies Max strategies Determinacy Finite-state games Cont.-time Markov chains

Transcription

1 Games with Time Finite-state Masaryk University Brno GASICS 00 /39

2 Outline Finite-state stochastic processes. Games over event-driven stochastic processes. Strategies,, determinacy. Existing results for with reachability. Games over discrete-time. Games over continuous-time. Games over event-driven stochastic processes.. GASICS 00 /39

3 in formal verification Finite-state In general: the arena (game board) corresponds to the state-space of a given system; a state reacts to some events whose impact in uncertain; there are special control states where two players, controller and environment can choose some action whose impact may also be uncertain; Is there a strategy for the controller such that the system satisfies a certain property no matter what the environment does? GASICS 00 3/39

4 stochastic processes Finite-state e e 3 3 state-space can be (countably) infinite; each event is either discrete-time or continuous-time; this model is closely related to real-time probabilistic processes [Alur, Courcoubetis, Dill] and generalized semi-markov processes. GASICS 00 4/39

5 stochastic processes () Finite-state u e e l r restart e e d Suppose that e and e have densities f and f and restart is a discrete-time event which takes zero time. The event e awaited in l has actually been fired in u. How do we capture this formally? What is the semantics of a given event-driven stochastic process G? GASICS 00 5/39

6 stochastic processes (3) u e e l r restart e e d Finite-state A fully rigorous approach: define the associated Markov process M G (with uncountable state-space). Usually, the state-space of M G is formed by tuples of the form (s, t,..., t n ). This is not appropriate in our setting. Alternatively, the state-space of M may consist of computational histories of G. A lightweight approach: define a suitable probability space over the runs in G. GASICS 00 6/39

7 stochastic processes (4) Finite-state A run of a given event-driven stochastic process is an infinite sequence (s 0, t 0, e 0 ), (s, t, e ), (s, t, e ),... where t i is the time spent in s i and e i is the triggering event. A basic cylinder determined by a finite sequence (s 0, I 0, e 0 ),... (s n, I n, e n ) consists of all runs of the form (s 0, t 0, e 0 ),... (s n, t n, e n ),... where t i I i for all 0 i n. We define the probability of basic cylinders in the natural way. Thus, we obtain the probability space (Run, F, P). GASICS 00 7/39

8 Games over event-driven stochastic processes Finite-state We add special control states V V where player and player can choose successor states. The impact of this choice may be uncertain in general. Players decisions are timeless. 3 3 GASICS 00 8/39

9 Games over event-driven stochastic processes () e 0.3 Finite-state e e e GASICS 00 9/39

10 Games over event-driven stochastic processes (3) A history of a game G is a finite sequence of the form (s 0, t 0, e 0, v 0 ),..., (s n, t n, e n, v n ) Finite-state A strategy of player, where {, }, is a measurable function which to every history (s 0, t 0, e 0, v 0 ),..., (s n, t n, e n, v n ) such that v n V assigns a probability distribution over the set of actions enabled in v n. Let µ 0 be an initial probability distribution over the set of states of G. Then each pair σ, π of strategies for player and player determines a unique play of G, denoted by G σ,π, which is an event-driven stochastic process. GASICS 00 0/39

11 Games over event-driven stochastic processes (4) A run in a game G is an infinite sequence of the form (s 0, t 0, e 0, v 0 ),..., (s n, t n, e n, v n ),... Finite-state One can define (Borel) σ-algebra over the runs of G which is the least σ-algebra containing all basic cylinders. A Borel objective is Borel set of runs in G. Various Borel are definable by timed automata and linear-time logics. GASICS 00 /39

12 Finite-state Games over event-driven stochastic processes (5) Theorem Games over event-driven stochastic processes with Borel have a value. That is, sup inf σ π Pσ,π (R) = inf sup P σ,π (R) π σ where R Run is Borel. Thm. follows directly from the result of Maitra & Sudderth [998] (which relies on Martin s determinacy result for Blackwell ). Thm. implies the existence of ε-optimal strategies for both players, but not the existence of optimal strategies. One can use various formalisms (e.g., timed automata) to construct finite representations of time-dependant strategies. GASICS 00 /39

13 Finite-state Let G be a game and T a set of target nodes. reach(t) consists of all runs that visit a target node. reach t (T) consists of all runs that visit a target node before time t. The goal of player / is to maximize/minimize the probability of reach t (T) (or reach(t)). The problems of our interest. Do the players have optimal strategies? And of what type? Can we compute the value and ε-optimal strategies? How about win-lose of the form P ϱ (reach(t))? Are such determined? If so, what is the type of winning strategies? GASICS 00 3/39

14 over Also known as simple stochastic. There is only one discrete-time event e with delay. Recall that the state-space can be infinite. Finite-state GASICS 00 4/39

15 have a value Finite-state Theorem Let G = (V, E, (V, V, V ), Prob) be a game, T V target vertices. For every v V we have that sup σ inf π Pσ,π v (reach(t)) = inf sup π σ P σ,π v (reach(t)) GASICS 00 5/39

16 Finite-state have a value () Proof sketch. Let Γ : [0, ] V [0, ] V be a (monotonic) function defined by if v T; sup {α(v ) (v, v ) E} if v T and v V ; Γ(α)(v) = inf {α(v ) (v, v ) E} if v T and v V ; (v,v ) E Prob(v, v ) α(v ) if v T and v V. µγ(v) sup inf σ π Pσ,π v (reach(t)) inf sup π σ P σ,π v (reach(t)) the second inequality holds for all Borel ; the tuple of all sup inf σ π Pσ,π v (reach(t)) is a fixed-point of Γ. It cannot be that µγ(v) < inf π sup σ P σ,π v (reach(t)) For all ε > 0 and v V, there is a strategy ˆπ such that sup σ P σ,ˆπ v (reach(t)) µγ(v) + ε. GASICS 00 6/39

17 Minimizing strategies () Finite-state Definition 3 (Locally optimal minimizing strategy) Let G = (V, E, (V, V, V ), Prob) be a game. An edge (v, v ) E is value minimizing if val(v ) = min { val(ˆv) V (v, ˆv) E } A locally optimal minimizing strategy is a strategy which in every play selects only value minimizing edges. GASICS 00 7/39

18 Finite-state Minimizing strategies () Theorem 4 Every locally optimal min. strategy is an optimal min. strategy. Proof. Let v V be an initial vertex, and u V a target vertex. () After playing k rounds according to a locally optimal minimizing strategy, player can switch to ε-optimal minimizing strategies in the current vertices of the play. Thus, we always (for every k and ε > 0) obtain an ε-optimal minimizing strategy for v. () Let π be a locally optimal min. strategy which is not optimal. Then there is a strategy σ of player such that (reach(t)) = val(v) + δ, where δ > 0. P σ,π v This means that there is k N such that P σ,π v (reachk (T)) > val(v) + δ. Hence, if player switches to δ 4-optimal minimizing strategy after playing k rounds according to π, we do not obtain a δ 4-optimal minimizing strategy for v. GASICS 00 8/39

19 Minimizing strategies (3) Finite-state Corollary 5 (Properties of minimizing strategies.) In every finitely-branching game, there is an optimal minimizing MD strategy. Theorem 6 Every optimal min. strategy is a locally optimal min. strategy. Hence, if player has some optimal minimizing strategy, then she also has an MD optimal minimizing strategy. GASICS 00 9/39

20 Minimizing strategies (4) Theorem 7 Optimal minimizing strategies do not necessarily exist, and (ε-) optimal minimizing strategies may require infinite memory. Proof. Finite-state r v i s s s 3 s i 4 8 i GASICS 00 0/39

21 Maximizing strategies () Observation 8 A locally optimal maximizing strategy is not necessarily an optimal maximizing strategy. This holds even for finite-state MDPs. Finite-state Proof. v t GASICS 00 /39

22 Finite-state Maximizing strategies () Theorem 9 Let v V be a vertex with finitely many successors t,..., t n. Then there is i n such that val(v) does not change if all edges (v, t j ), where i j, are deleted from the game. Proof. v V (σ,π) t k = P(u) P(u)+P( ) if P(u) + P( ) > 0; 0 otherwise; t k v u V σ t k = inf π V (σ,π) t k V tk = sup σ V σ t k There must be some k such that V tk = val(v). We put i = k. GASICS 00 /39

23 Maximizing strategies (3) Theorem 0 Optimal maximizing strategies may not exist, even in finitely-branching MDPs. Finite-state v GASICS 00 3/39

24 Maximizing strategies (4) Theorem Optimal maximizing strategies may require infinite memory, even in finitely-branching. Finite-state ˆv d d d 3 d 4 d 5 v e e e 3 e 4 e 5 s s s 3 s 4 s 5 GASICS 00 4/39

25 Summary Finite-state Minimizing strategies: Optimal minimizing strategies may not exist. Optimal and ε-optimal minimizing strategies may require infinite memory. In finitely-branching, there are MD optimal minimizing strategies. Maximizing strategies: Optimal maximizing strategies may not exist, even in finitely-branching. Optimal maximizing strategies may require infinite memory. In finite-state, there are MD optimal maximizing strategies. GASICS 00 5/39

26 as a win-lose objective () Finite-state Let ϱ [0, ]. A strategy σ Σ is ( ϱ)-winning in v if for every π Π we have that P (σ,π) v (reach(t) ϱ). A strategy π Π is (<ϱ)-winning if for every σ Σ we have that P (σ,π) v (reach(t) < ϱ). Is there a winning strategy for one of the two players? GASICS 00 6/39

27 as a win-lose objective () Finite-state Theorem Turn-based stochastic with reachability are not necessarily determined. However, finitely-branching are determined. GASICS 00 7/39

28 as a win-lose objective (3) u s v Finite-state u v GASICS 00 8/39

29 Algorithms for finite-state MDP and Finite-state We show how to compute the values and optimal strategies for reachability in finite-state and MDPs. For finite-state MDPs we have that the values and optimal strategies are computable in polynomial time by linear programming; For finite-state we have that the values and optimal strategies are computable in polynomial space (for a fixed number of randomized vertices, the problem is in P [Gimbert, Horn, 008]); There are also algorithms for certain classes of infinite-state. GASICS 00 9/39

30 References Finite-state D.A. Martin. The of Blackwell Games. The Journal of Symbolic Logic, Vol. 63, No. 4 (Dec., 998), pp A. Maitra and W. Sudderth. Finitely Additive Games with Borel Measurable Payoffs. International Journal of Game Theory, Vol. 7 (998), pp M.L. Puterman. Markov Decision Processes, Wiley, 994. T. Brázdil, V. Brožek, V. Forejt, A. Kučera. in recursive Markov decision processes. A. Kučera. Turn-based Games. In Lectures in Game Theory for Computer Scientists, Cambridge. To appear. A. Condon. The Complexity of Games. Information and Computation, 96():03 4, 99. L.S. Shapley.. Proceedings of the National Academy of Sciences USA, 39:095 00, 953. H. Gimbert, F. Horn. Simple Games with Few Random Vertices Are Easy to Solve. Proc. FoSSaCS 008, pp. 5 9, LNCS 496, Springer, 008. GASICS 00 30/39

31 Games over cont.-time Finite-state All events are continuous and exponentially distributed, i.e., P(d e t) = e λt where λ > 0 is a rate of the event e. For simplicity, we assume that the impact of players choice is determined (i.e., the distributions associated to the available actions are Dirac). Some facts about exponential distribution: Let X E(λ). Then P(X t + t X t ) = P(X t). Let X E(λ) and Y E(κ). Then min(x, Y) E(λ + κ) and P(X < Y) = λ/(λ + κ). GASICS 00 3/39

32 Games over cont.-time () Finite-state e E(λ) e E(κ) 3 3 λ κ+λ e E(κ + λ) κ κ+λ 3 3 GASICS 00 3/39

33 Games over cont.-time (3) Finite-state GASICS 00 33/39

34 Games over cont.-time (4) Finite-state -bounded reachability have been so far studied mainly for time abstract strategies. Theorem 3 Let G = (V, E, (V, V, V ), Prob) be a game, T V target vertices. For every v V we have that sup σ inf π Pσ,π v (reach t (T)) = inf sup π σ P σ,π v (reach t (T)) where σ and π range over time abstract strategies. GASICS 00 34/39

35 Games over cont.-time (5) Finite-state Proof sketch. Let H be the set of all histories i : R(G) N 0 where a R(G) i(a) <. Let Γ : (H V [0, ]) (H V [0, ]) be a (monotonic) function defined by F i (t) v T Γ(H)(i, v) = sup a E(v) u V P(a)(u) H(i + Rate(a), u) v V \ T inf a E(v) u V P(a)(u) H(i + Rate(e), u) v V \ T Let µγ be the least fixed-point of Γ. of a given vertex v is equal to µγ(0, v). Observation: as a R(G) i(a) increases, F i (t) approaches zero (assuming the rates are bounded). Hence, Γ allows to compute ε-optimal strategies. GASICS 00 35/39

36 Games over cont.-time (6) Finite-state In general, optimal strategies do not exist. In finitely-branching, player is guaranteed to have an optimal CD strategy. In finitely-branching with bounded rates, player is guaranteed to have an optimal CD strategy. In finitely-branching uniform, both players have BCD optimal strategies that are effectively computable. GASICS 00 36/39

37 Games over cont.-time (7) Finite-state References: C. Baier, H. Hermanns, J.-P. Katoen, and B.R. Haverkort. Efficient computation of time-bounded reachability probabilities in uniform continuous-time Markov decision processes. Theoretical Computer Science, 345: 6, 005. M. Neuhäußer, M. Stoelinga, and J.-P. Katoen. Delayed nondeterminism in continuous-time Markov decision processes. In Proceedings of FoSSaCS 009, volume 5504 of LNCS, pages Springer, 009. T. Brázdil, V. Forejt, J. Krčál, J. Křetínský, and A. Kučera. Continuous-time stochastic -bounded reachability. In Proceedings of FST&TCS 009, pages 6 7, 009. M. Rabe and S. Schewe. Optimal time-abstract schedulers for CTMDPs and Markov. In Eighth Workshop on Quantitative Aspects of Programming Languages, 00. GASICS 00 37/39

38 Games over event-driven stochastic processes Finite-state Assuming that all events are continuous and the objective is encoded as a deterministic timed automaton, one can decide if player has an almost-sure winning strategy and compute a finite description of this strategy [Brázdil et al., Concur 00]. It is not easy to extend this result to the general model with both continuous and discrete events. GASICS 00 38/39

39 , open problems Finite-state Games over event-driven stochastic processes can model concurrent systems with stochastic delays that are not necessarily exponentially distributed. One can rely on rich theory of discrete-time stochastic and Markov processes with general state-space. Almost everything is open. New theoretical results can also bring efficient algorithms for solving the considered problems. GASICS 00 39/39