Game Relations and Metrics

Transcription

1 Game Relation and Metric Luca de Alfaro Computer Engineering Department Univerity of California, Santa Cruz, USA Vihwanath Raman Computer Science Department Univerity of California, Santa Cruz, USA and Synopy Inc, Mountain View, USA Rupak Majumdar Department of Computer Science Univerity of California, Lo Angele, USA Mariëlle Stoelinga FMT Group Univerity of Twente, the Netherland Abtract We conider two-player game played over finite tate pace for an inite number of round. At each tate, the player imultaneouly chooe move; the move determine a ucceor tate. It i often advantageou for player to chooe probability ditribution over move, rather than ingle move. Given a goal e.g., reach a target tate ), the quetion of winning i thu a probabilitic one: what i the maximal probability of winning from a given tate?. On thee game tructure, two fundamental notion are thoe of equivalence and metric. Given a et of winning condition, two tate are equivalent if the player can win the ame game with the ame probability from both tate. Metric provide a bound on the difference in the probabilitie of winning acro tate, capturing a quantitative notion of tate imilarity. We introduce equivalence and metric for two-player game tructure, and we how that they characterize the difference in probability of winning game whoe goal are expreed in the quantitative µ-calculu. The quantitative µ- calculu can expre a large et of goal, including reachability, afety, and ω-regular propertie. Thu, we claim that our relation and metric provide the canonical extenion to game, of the claical notion of biimulation for tranition ytem. We develop our reult both for equivalence and metric, which generalize biimulation, and for aymmetrical verion, which generalize imulation. Thi reearch wa ponored in part by the grant NSF-CCF , NSF-CCF , and NSF-CCR Introduction We conider two-player game played for an inite number of round over finite tate pace. At each round, the player imultaneouly and independently elect move; the move then determine a probability ditribution over ucceor tate. Thee game, known variouly a tochatic game [24] or concurrent game [3, 1, 5], generalize many common tructure in computer cience, from tranition ytem, to Markov chain [12] and Markov deciion procee [6]. The game are turn-baed if, at each tate, at mot one of the player ha a choice of move, and determinitic if the ucceor tate i uniquely determined by the current tate, and by the move choen by the player. It i well-known that in uch game with imultaneou move it i often advantageou for the player to randomize their move, o that at each round, they play not a ingle pure move, but rather, a probability ditribution over the available move. Thee probability ditribution over move, called mixed move [20], lead to variou notion of equilibria [29, 20], uch a the equilibrium reult expreed by the minimax theorem [29]. Intuitively, the benefit of playing mixed, rather than pure, move lie in preventing the adverary from tailoring a repone to the individual move played. Even for imple reachability game, the ue of mixed move may allow player to win, with probability 1, game that they would loe i.e., win with probability 0) if retricted to playing pure move [3]. With mixed move, the quetion of winning a game with repect to a goal i thu a probabilitic one: what i the maximal probability a player can be guaranteed of winning, regardle of how the other player play? Thi probability i known, in brief, a the winning probability. In LICS 07: Proceeding of the 22nd Annual IEEE Sympoium on Logic in Computer Science, 2007

2 In tructure ranging from tranition ytem to Markov deciion procee and game, a fundamental quetion i the one of equivalence of tate. Given a uitably large cla Φ of propertie, containing all propertie of interet to the modeler, two tate are equivalent if the ame propertie hold in both tate. For a property φ, denote the value of φ at by φ): in the cae of game, thi might repreent the maximal probability of a player winning with repect to a goal expreed by φ. Two tate and t are equivalent if φ) = φt) for all φ Φ. For finite-branching) tranition ytem, and for the cla of propertie Φ expreible in the µ-calculu [14], tate equivalence i captured by biimulation [19]; for Markov deciion procee, it i captured by probabilitic biimulation [22]. For quantitative propertie, a notion related to equivalence i that of a metric: a metric provide a tight bound for how much the value of a property can differ at tate of the ytem, and provide thu a quantitative notion of imilarity between tate. Given a et Φ of propertie, the metric ditance of two tate and t can be defined a φ Φ φ) φt). Metric for Markov deciion procee have been tudied in [7, 27, 28, 8, 9]. Obviouly, the metric and relation are connected, in the ene that the relation are the kernel of the metric the pair of tate having metric ditance 0). The metric and relation are at the heart of many verification technique, from approximate reaoning one can ubtitute tate that are cloe in the metric) to ytem reduction one can collape equivalent tate) to compoitional reaoning and refinement providing a notion of ubtitutivity of equivalent). We introduce metric and equivalence relation for concurrent game, with repect to the cla of propertie Φ expreible in the quantitative µ-calculu [5, 18]. We claim that thee metric and relation repreent the canonical extenion of biimulation to game. We alo introduce aymmetrical verion of thee metric and equivalence, which contitute the canonical extenion of imulation. An equivalence relation for determinitic game that are either turn-baed, or where the player are contrained to playing pure move, ha been introduced in [2] and called alternating biimulation. Relation and metric for the general cae of concurrent game have o far proved eluive, with ome previou attempt at their definition by a ubet of the author following a ubtly flawed approach [4, 16]. The caue of the difficulty goe to the heart of the definition of biimulation. In the definition of biimulation for tranition ytem, for every pair, t of biimilar tate, we require that if can go to a tate, then t hould be able to go to t, uch that and t are again biimilar we alo ak that, t have an equivalent predicate valuation). Thi definition ha been extended to Markov deciion procee by requiring that for every mixed move from, there i a mixed move from t, uch that the move induce probability ditribution over ucceor tate that are equivalent modulo the underlying biimulation [22, 21]. Unfortunately, the generalization of thi appealing definition to game fail. It turn out, a we prove in thi paper, that requiring player to be able to replicate probability ditribution over ucceor modulo the underlying equivalence) lead to an equivalence that i too fine, and that may fail to relate tate at which the ame quantitative µ-calculu formula hold. We how that phraing the definition in term of ditribution over ucceor tate i the wrong approach for game; rather, the definition hould be phraed in term of expectation of certain metric-bounded quantitie. Our tarting point i a cloer look at the definition of metric for Markov deciion procee. We oberve that we can manipulate the definition of metric given in [28], obtaining an alternative form, which we call the a priori form, in contrat with the original form of [28], which we call the a poteriori form. Informally, the a poteriori form i the traditional definition, phraed in term of imilarity of probability ditribution; the a priori form i intead phraed in term of expectation. We how that, while on Markov deciion procee thee two form coincide, thi i not the cae for game; moreover, we how that it i the a priori form that provide the canonical metric for game. We prove that the a priori metric ditance between two tate and t of a concurrent game i equal to φ Φ φ) φt), where Φ i the et of propertie expreible via the quantitative µ-calculu. Thi reult can be ummarized by aying that the quantitative µ-calculu provide a logical characterization for the a priori metric, imilar to the way the ordinary µ-calculu provide a logical characterization of biimulation. Furthermore, we prove that a priori metric and their kernel, the a priori relation atify a reciprocity property, tating that propertie expreed in term of player-1 and player-2 winning condition have the ame ditinguihing power. Thi property i intimately connected to the fact that concurrent game, played with mixed move, are determined for ω-regular goal [17, 5]: the probability that player 1 achieve a goal ψ i one minu the probability that player 2 achieve the goal ψ. Reciprocity enure that there i one, canonical, notion of game equivalence. Thi i in contrat to the cae of alternating biimulation of [2], in which there are ditinct player-1 and player-2 verion, a a conequence of the fact that concurrent game, when played with pure move, are not determined. The logical characterization and reciprocity reult jutify our claim that a priori metric and relation are the canonical notion of metric, and equivalence, for concurrent game. Neither the logical characterization nor the reciprocity reult hold for the a poteriori metric and relation. While thi introduction focued motly on metric and equivalence relation, we alo develop reult for the aymmetrical verion of thee notion, related to imulation. 2

3 2. Game and Goal We will develop metric for game tructure over a et S of tate. We tart with ome preliminary definition. For a finite et A, let DitA) denote the et of probability ditribution over A. We ay that p DitA) i determinitic if there i a A uch that pa) = 1. For a et S, a valuation over S i a function f : S [0, 1] aociating with every element S a value 0 f) 1; we let F be the et of all valuation. For c [0, 1], we denote by c the contant valuation uch that c) = c at all S. We order valuation pointwie: for f, g F, we write f g iff f) g) at all S; we remark that F, under, form a lattice. Given a, b IR, we write a b = max{a, b}, and a b = min{a, b}; we alo let a b = min{1, max{0, a + b}} and a b = max{0, min{1, a b}}. We extend,, +,,, to valuation by interpreting them in pointwie fahion. A directed metric i a function d : S 2 IR 0 which atifie d, ) = 0 and d, t) d, u) + du, t) for all, t, u S. We denote by M S 2 IR the pace of all metric; thi pace, ordered pointwie, form a lattice which we indicate with M, ). Given a metric d M, we denote by d it oppoite verion, defined by d, t) = dt, ) for all, t S; we ay that d i ymmetrical if d = d Game Structure We aume a fixed, finite et V of obervation variable. A two-player, concurrent) game tructure G = S, [ ], Move, Γ 1, Γ 2, δ conit of the following component [1, 3]: A finite et S of tate. A variable interpretation [ ] : V S [0, 1], which aociate with each variable v V a valuation [v]. A finite et Move of move. Two move aignment Γ 1, Γ 2 : S 2 Move \. For i {1, 2}, the aignment Γ i aociate with each tate S the nonempty et Γ i ) Move of move available to player-i at tate. A probabilitic tranition function δ: S Move Move DitS), that give the probability δ, a 1, a 2 )t) of a tranition from to t when player-1 play move a 1 and player-2 play move a 2. At every tate S, player 1 chooe a move a 1 Γ 1 ), and imultaneouly and independently player 2 chooe a move a 2 Γ 2 ). The game then proceed to the ucceor tate t S with probability δ, a 1, a 2 )t). We denote by Det, a 1, a 2 ) = {t S δ, a 1, a 2 )t) > 0} the et of detination tate when action a 1, a 2 are choen at. The variable in V naturally induce an equivalence on tate: for tate, t, define t if for all v V we have [v]) = [v]t). In the following, unle otherwie noted, the definition refer to a game tructure with component G = S, [ ], Move, Γ 1, Γ 2, δ. For player i {1, 2}, we write i = 3 i for the opponent. We alo conider the following ubclae of game tructure. Turn-baed game tructure. A game tructure G i turn-baed if we can write S a the dijoint union of two et: the et S 1 of player-1 tate, and the et S 2 of player-2 tate, uch that S 1 implie Γ 2 ) = 1, and S 2 implie Γ 1 ) = 1, and further, there i a pecial variable turn V, uch that [turn] = 1 iff S 1, and [turn] = 0 iff S 2 : thu, the variable turn indicate whoe turn it i to play at a tate. Turn-baed game are often called perfect ormation game [20]. Markov deciion procee. A game tructure G i a Markov deciion proce MDP) [6] if only one of the two player ha a choice of move. For i {1, 2}, we ay that a tructure i an i-mdp if S, Γ i ) = 1. For MDP, we omit the ingle) move of the player without a choice of move, and write δ, a) for the tranition function. Determinitic game tructure. A game tructure G i determinitic if, for all S, a 1 Move, and a 2 Move, there exit a t S uch that δ, a 1, a 2 )t) = 1; we denote uch t by τ, a 1, a 2 ). We ometime call probabilitic a general game tructure, to emphaize the fact that it i not necearily determinitic. Pure and mixed move. A mixed move i a probability ditribution over the move available to a player at a tate. We denote by D i ) = DitΓ i )) the et of mixed move available to player i {1, 2} at S. The move in Move are called pure move, in contrat to mixed move. We extend the tranition function to mixed move. For S and x 1 D 1 ), x 2 D 2 ), we write δ, x 1, x 2 ) for the next-tate probability ditribution induced by the mixed move x 1 and x 2, defined for all t S by δ, x 1, x 2 )t) = a 1 Γ 1) a 2 Γ 2) δ, a 1, a 2 )t) x 1 a 1 ) x 2 a 2 ). In the following, we ometime retrict the move of the player to pure move. We identify a pure move a Γ i ) available to player i {1, 2} at a tate with a determinitic ditribution that play a with probability 1. The determinitic etting. The determinitic etting conit in conidering determinitic game tructure, and player retricted to play pure move. 3

4 Predeceor operator. Given a valuation f F, a tate S, and two mixed move x 1 D 1 ) and x 2 D 2 ), we define the expectation of f from under x 1, x 2 : E x1,x2 f) = δ, x 1, x 2 )t) ft) t S For a game tructure G, for i {1, 2} we define the valuation tranformer Pre i : F F by, for all f F and S, Pre i f)) = x i D i) x i D i) E x1,x2 f). Intuitively, Pre i f)) i the maximal expectation player i can achieve of f after one tep from : thi i the claical one-day or next-tage operator of the theory of repeated game [10]. We alo define a determinitic verion of thi operator, in which player are forced to play pure move: Pre Γ i f)) = max x i Γ i) 2.2. Quantitative µ-calculu min x i Γ i) Ex1,x2 f). We conider the et of propertie expreed by the quantitative µ-calculu qµ). A dicued in [13, 5, 18], a large et of propertie can be encoded in qµ, panning from baic propertie uch a maximal reachability and afety probability, to the maximal probability of atifying a general ω- regular pecification. Syntax. The yntax of quantitative µ-calculu i defined with repect to the et of obervation variable V a well a a et MVar of calculu variable, which are ditinct from the obervation variable in V. The yntax i given a follow: φ ::= c v Z φ φ φ φ φ φ c φ c pre 1 φ) pre 2 φ) µz. φ νz. φ for contant c [0, 1], obervation variable v V, and calculu variable Z MVar. In the formula µz. φ and νz. φ, we furthermore require that all occurrence of the bound variable Z in φ occur in the cope of an even number of occurrence of the complement operator. A formula φ i cloed if every calculu variable Z in φ occur in the cope of a quantifier µz or νz. From now on, with abue of notation, we denote by qµ the et of cloed formula of qµ. A formula i a player-i formula, for i {1, 2}, if φ doe not contain the pre i operator; we denote with qµ i the yntactic ubet of qµ coniting only of cloed player-i formula. A formula i in poitive form if the negation appear only in front of game variable, i.e., in the context v; we denote with qµ + and qµ + i the ubet of qµ and qµ i coniting only of poitive formula. We remark that the fixpoint operator µ and ν will not be needed to achieve our reult on the logical characterization of game relation. They have been included in the calculu becaue they allow the expreion of many intereting propertie, uch a afety, reachability, and in general, ω-regular propertie. Semantic. A variable valuation ξ: MVar F i a function that map every variable Z MVar to a valuation in F. We write ξ[z f] for the valuation that agree with ξ on all variable, except that Z i mapped to f. Given a game tructure G and a variable valuation ξ, every formula φ of the quantitative µ-calculu define a valuation [φ] G ξ F the ercript G i omitted if the game tructure i clear from the context): [c] ξ = c [v ] ξ = [v] [Z ] ξ = ξz) [ φ] ξ = 1 [φ] ξ [φ { } c]ξ = [φ] ξ { } c { [φ { 1 } φ2 ] ξ = [φ 1 ] ξ } [φ2 ] ξ [pre i φ)] ξ = Pre i [φ] ξ ) [ { } µ ν Z. φ]ξ = { } {f F f = [φ]ξ[z f] } where i {1, 2}. The exitence of the fixpoint i guaranteed by the monotonicity and continuity of all operator and can be computed by Picard iteration [5]. If φ i cloed, [φ] ξ i independent of ξ, and we write imply [φ]. We alo define a determinitic emantic [ ] Γ for qµ, in which player can elect only pure move in the operator pre 1, pre 2. [ ] Γ i defined a [ ], except for the claue [pre i φ)] Γ ξ = Pre Γ i [φ] Γ ξ ). Example 1 Given a et T S, the characteritic valuation T of T i defined by T) = 1 if T, and T) = 0 otherwie. With thi notation, the maximal probability with which player i {1, 2} can enure eventually reaching T S i given by [µz.t pre i Z))], and the maximal probability with which player i can guarantee taying in T forever i given by [νz.t pre i Z))] [5]. 3. Metric We are intereted in developing a metric on tate of a game tructure that capture an approximate notion of equivalence: tate cloe in the metric hould yield imilar value to the player for any winning objective. Specifically, we are intereted in defining a biimulation metric [ g ] M uch that for any game tructure G and tate, t of G, the following continuity property hold: [ g ], t) = [φ]) [φ]t). 1) φ qµ In particular, the kernel of the metric, that i, tate at ditance 0, are equivalent: each player can get exactly the ame value from either tate for any objective. Notice that 4

5 in defining the metric independent of a player, we are expecting our metric to be reciprocal, that i, invariant under a change of player. Reciprocity i expected to hold ince the underlying game we conider are determined for any game, the value obtained by player 2 i 1 minu the value obtained by player 1 and yield canonical metric on game. Thu, our metric will generalize equivalence and refinement relation that have been tudied on MDP and in the determinitic etting. To underline the connection between claical equivalence and the metric we develop, we write [ g t] for [ g ], t), o that the deired property of the biimulation metric can be tated a [ g t] = φ qµ [φ]) [φ]t). Metric of thi type have already been developed for Markov deciion procee MDP) [27, 8]. Our contruction of metric for game tart from an analyi of thee contruction Metric for MDP We conider the cae of 1-MDP; the cae for 2-MDP i ymmetrical. Throughout thi ubection, we fix a 1-MDP S, [ ], Move, Γ 1, Γ 2, δ. Before we preent the metric correpondent of probabilitic imulation, we firt rephrae claical probabilitic bi)imulation on MDP [15, 11, 22, 23] a a fixpoint of a relation tranformer F : 2 S S 2 S S. For all relation R S S and, t S, let, t) F R) iff t x 1 M 1 ). y 1 M 1 t). δ, x 1 ) R δt, y 1 ), 2) for all tate, t S. In 2), i the predicate equivalence relation: t if the predicate have the ame value at and t. The relation R lift the relation R on tate to a relation on ditribution. Preciely, for a relation R S S and two ditribution p, q DitS), we let p R q if there i a function : S S [0, 1] uch that i), ) > 0 implie, ) R, ii) p) = S, ) for any S, and iii) q ) = S, ) for any S. Probabilitic imulation i the greatet fixpoint of 2); probabilitic biimulation i the greatet ymmetrical fixpoint of 2). To obtain a metric equivalent of probabilitic imulation, we lift the above fixpoint from relation ubet of S 2 ) to metric map S 2 IR). Firt, we define [ ] M for all, t S by [ t] = max v V [v]) [v]t). Second, we lift 2) to metric, defining a metric tranformer Hpot 1MDP : M M. For all d M, let Dδ, x 1 ), δt, y 1 ))d) be the ditribution ditance between δ, x 1 ) and δt, y 1 ) with repect to the metric d. We will how later how to define uch a ditribution ditance. For, t S, we let Hpot 1MDP d), t) 3) = [ t] Dδ, x 1 ), δt, y 1 ))d). x 1 Γ 1) y 1 Γ 1t) In thi definition, the and of 2) have been replaced by and, repectively. Since equivalent tate hould have ditance 0, the imulation metric in MDP i defined a the leat rather than greatet) fixpoint of 3) [27, 8]. Similarly, the biimulation metric i defined a the leat ymmetrical fixpoint of 3). For a ditance d M and two ditribution p, q DitS), the ditribution ditance Dp, q)d) between p and q with repect to d, i a meaure of how much work we have to do to make p look like q, given that moving a unit of probability ma from S to t S ha cot d, t). More preciely, Dp, q)d) i defined via the tran-hipping problem, a the minimum cot of hipping the ditribution p into q, with edge cot d. Thu, Dp, q)d) i the olution of the following linear programming LP) problem over the et of variable {λ,t },t S [27]: Minimize d, t)λ,t ubject to λ,t = p), S,t S λ,t = qt), λ,t 0. t S Equivalently, we can define Dp, q)d) via the dual of the above LP problem. Given a metric d M, let Cd) F be the ubet of valuation k F uch that k) kt) d, t) for all, t S. Then the dual formulation i: Maximize q)k) 4) S p) k) S ubject to k Cd). The contraint Cd) on the valuation k, tate the value of k acro tate cannot differ by more than d. Thi mean, intuitively, that k behave like the valuation of a qµ formula: a we will ee, the logical characterization implie that d i a bound for the difference in valuation of qµ formula acro tate. Indeed, the logical characterization of the metric i proved by contructing formula whoe valuation approximate that of the optimal k. Plugging 4) into 3), we obtain: H 1MDP pot d), t) = [ t] 5) E x 1 k) E y1 t k) ) x 1 Γ 1) y 1 Γ 1t) k Cd) We can interpret thi definition a follow. State t i trying to imulate tate thi i a definition of a imulation metric). Firt, tate chooe a mixed move x 1, attempting to make imulation a hard a poible; then, tate t chooe a mixed move y 1, trying to match the effect of x 1. Once x 1 and y 1 have been choen, the reulting ditance between and t i equal to the maximal difference in expectation, for move x 1 and y 1, of a valuation k Cd). We call the metric tranformer Hpot 1MDP the a poteriori metric tranformer: the valuation k in 5) i choen after the move x 1 and y 1 are choen. We can define an a priori metric tranformer, 5

6 where k i choen before x 1 and y 1 : H 1MDP prio d), t) = [ t] 6) E x 1 k) Ey1 t k) ) k Cd) x 1 Γ 1) y 1 Γ 1t) Intuitively, in the a priori tranformer, firt a valuation k Cd) i choen. Then, tate t mut imulate tate with repect to the expectation of k. State chooe a move x 1, trying to maximize the difference in expectation, and tate t chooe a move y 1, trying to minimize it. The ditance between and t i then equal to the difference in the reulting expectation of k. Theorem 1 below tate that for MDP, a priori and a poteriori imulation metric coincide. In the next ection, we will ee that thi i not the cae for game. Theorem 1 For all MDP, H 1MDP pot = H 1MDP prio. Proof Conider two tate, t S, and a metric d M. We have to prove that k x1 y1 E x1 k) Ey1 t k) = x1 y1 k E x1 k) Ey1 t k). In the left-hand ide, we can exchange the two outer. Then, noticing that the difference in expectation i bilinear in k and y 1 for a fixed x 1, that y 1 i a probability ditribution, and that k i choen from a compact convex ubet, we apply the generalized minimax theorem [25] to exchange k y1 into y1 k, thu obtaining the right-hand ide. The metric defined above are logically characterized by qµ. Preciely, let [ ] M be the leat ymmetrical fixpoint of Hprio 1MDP = Hpot 1MDP. Then, the reult of [8] originally tated for Hpot 1MDP ) tate that for all tate, t of a 1-MDP, we have [ t] = φ qµ [φ]) [φ]t) Metric for Concurrent Game We now extend the imulation and biimulation metric from MDP to general game tructure. A we hall ee, unlike for MDP, the a priori and the a poteriori metric do not coincide over game. In particular, we how that the a priori formulation atifie both a tight logical characterization a well a reciprocity while, perhap urpriingly, the more natural a poteriori verion doe not. A poteriori metric are defined via the metric tranformer H 1 : M M a follow, for all d M and, t S: H 1 d), t) = [ t] = [ t] x 1 D 1) y 1 D 1t) y 2 D 2t) x 2 D 2) Dδ, x 1, x 2 ), δt, y 1, y 2 ), d) x 1 D 1) y 1 D 1t) y 2 D 2t) x 2 D 2) k Cd) E x 1,x 2 k) E y1,y2 t k) ). 7) We define a poteriori game imulation metric [ 1 ] a the leat fixpoint of H 1, and we define a poteriori game biimulation metric [ = 1 ] a the leat ymmetrical fixpoint of H 1. The a poteriori imulation metric [ 1 ] ha been introduced in [4, 16]. A priori metric are defined by bringing the k outide. Preciely, we define a metric tranformer H 1 : M M a follow, for all d M and, t S: H 1 d), t) = [ t] k Cd) x 1 D 1) y 1 D 1t) y 2 D 2t) x 2 D 2) E x 1,x 2 = [ t] k Cd) k) E y1,y2 t k) ) x 1 D 1) x 2 D 2) y 1 D 1t) y 2 D 2t) E x1,x2 k) E y1,y2 t k) = [ t] Pre1 k)) Pre 1 k)t) ). 8) k Cd) The a priori imulation metric [ 1 ] i the leat fixpoint of H 1, and the a priori biimulation metric [ 1 ] i the leat ymmetrical fixpoint of H 1. We now how ome baic propertie of thee metric. We how that a priori fixpoint give a directed) metric. We how that a priori and a poteriori metric are ditinct. We then focu on the a priori metric, and how through our reult that they are the natural metric for concurrent game. Theorem 2 For all game tructure G, and all tate, t, u of G, we have 1) [ 1 t] 0, and [ 1 t] 0, and 2) [ 1 u] [ 1 t] + [t 1 u]. Proof The firt part follow by conidering k) = 0 for all S. The econd part i proved by induction on the iteration ued to compute the fixpoint of H 1, a given in 8). The crucial tep conit in oberving that, for d M, we have Pre1 k)) Pre 1 k)t) ) k Cd) + k Cd) Pre1 k)t) Pre 1 k)u) ) Pre1 k)) Pre 1 k)u) ). k Cd) A priori and a poteriori metric are ditinct. Firt, we how that a priori and a poteriori metric are ditinct in general: the a priori metric never exceed the a poteriori one, and there are concurrent game where it i trictly maller. Intuitively, thi can be explained a follow. Simulation entail trying to imulate the expectation of a valuation k, a we ee from 7), 8). It i eaier to imulate a tate ) 6

7 δ *, *, *)w) 0 1/9 1/3 4/9 5/9 2/3 8/9 b,f 1 8/9 2/3 5/9 4/9 1/3 1/9 0 δ *, *, *)u) a,f c,f b,g c,g Figure 1. Tranition probabilitie from tate, t to tate u, w. from a tate t if the valuation i known in advance, a in a priori metric 8), than if the valuation k i choen after all the move have been choen, a in a poteriori metric 7). A a pecial cae, we hall ee that equality hold for turn-baed game tructure, in addition to MDP a we have een in the previou ubection. Theorem 3 a,g The following aertion hold. 1. For all game tructure G, and for all tate, t of G, we have [ 1 t] [ 1 t]. 2. There i a game tructure G, and tate, t of G, uch that [ 1 t] = 0 and [ 1 t] > For all turn-baed game tructure, we have 1 = 1. Proof The firt aertion i a conequence of the fact that, for all function f : IR 2 IR, we have x y fx, y) y x fx, y) intuitively, it i eaier to maximize if we can chooe x after y). Repeated application of thi allow u to how that, for all d M, we have H d) H d) with pointwie ordering). The reult then follow from the monotonicity of H and H. For the econd aertion, we give an example where a priori ditance are trictly le than a poteriori ditance. Conider a game with tate S = {, t, u, w}. State u and w are ink tate with [u w] = 1; tate and t are uch that [ t] = 0. At tate and t, player-2 ha move {f, g}. Player-1 ha a ingle move {a} at tate, and move {b, c} at tate t. The move from and t lead to u and w with tranition probabilitie indicated in Figure 1. In the figure, the point b, f indicate the probability of going to u and w when the move pair b, f) i played, with δ, b, f)u) + δ, b, f)w) = 1; imilarly for the other move pair. The thick line egment between the point a, f and a, g repreent the tranition probabilitie ariing when player 1 play move nd player 2 play a mixed move a mix of f and g). We how that, in thi game, we have [ 1 t] > 0. Conider the metric d where du, w) = 1 recall that [u w] = 1, and note the other ditance do not matter, 1 ince u, w are the only two detination). We need to how y 1 D 1 t). y 2 D 2 t). x 2 D 2 ). k Cd). E a,x 2 k) E y1,y2 t k) ) > 0 9) Conider any mixed move y 1 = αb + 1 α)c, where b, c are the move available to player 1 at t, and 0 α 1. If α < 1 2, chooe move f from t a y 2, and chooe kw) = 1, ku) = 0. Otherwie, chooe move g from t a y 2, and chooe kw) = 0, ku) = 1. With thee choice, the tranition probability δt, y 1, y 2 ) will fall outide of the egment [a, f), a, g)] in Figure 1. Thu, with the choice of k above, we enure that the difference in 9) i alway poitive. To how that in the game we have [ 1 t] = 0, it uffice to how given that [ 1 t] 0) that k Cd). y 1 D 1 t). y 2 D 2 t). x 2 D 2 ). E a,x 2 k) E y1,y2 t k) ) 0. If ku) = kw), the reult i immediate. Aume otherwie, without lo of generality, that ku) < kw), and chooe y 1 = c. For every y 2, the ditribution of ucceor tate and of k-expectation) will be in the interval [c, f), c, g)] in Figure 1. By chooing x 2 = f, we have that E a,f k) < E c,y2 t k) for all y 2 D 2 t), leading to the reult. The lat aertion of the theorem i proven in the ame way a Theorem 1. Reciprocity of a priori metric. The previou theorem etablihe that the a priori and a poteriori metric are in general ditinct. We now prove that it i the a priori metric, rather than the a poteriori one, that enjoy reciprocity, and that provide a quantitative) logical characterization of [qµ]. We begin by conidering reciprocity. Theorem 4 The following aertion hold. 1. For all game tructure G, we have [ 1 ] = [ 2 ], and [ 1 ] = [ 2 ]. 2. There i a concurrent game tructure G, with tate and t, where [ 1 ] [ 2 ]. Proof For the firt aertion, it uffice to how that, for all d M, and tate, t S, we have H 1 d), t) = H 2 d)t, ). We proceed a follow: Pre1 k)) Pre 1 k)t) ) 10) k Cd) = k Cd) Pre2 1 k)) + Pre 2 1 k)t) ) 11) = Pre2 k)t) Pre 2 k)) ) 12) k C d) The tep from 10) to 11) ue Pre 1 k)) = 1 Pre 2 1 k)) [5], and the tep from 11) to 12) ue the change of variable k 1 k. 7

8 To how that reciprocity fail for a poteriori imulation metric, conider the game from Figure 1. We add two new move to player 2 at tate t, namely {e, h}, uch that for any α [0, 1], δt,, αe + 1 α)h)u) = δ, a, αf + 1 α)g)u): uing move {e, h} player 2 can enure that tranition probabilitie are on the olid line in Figure 1. Reaoning a in Theorem 3, it till i the cae that [ 1 t] > 0. Since the player-2 move {e, h} from tate t are identical in induced tranition probabilitie to player-2 move {f, g} from tate, player 2 can retrict herelf to move {e, h} at tate t and enure that [ 2 t] = 0. A a conequence of thi theorem, we write [ g ] in place of [ 1 ] = [ 2 ], to emphaize that the player-1 and player-2 verion of game equivalence metric coincide. Logical characterization of a priori metric. The following theorem expree the fact that [qµ] provide a logical characterization for the a priori metric. The proof of the theorem ue idea from [16] and [8]. Theorem 5 The following aertion hold for all game tructure G, and for all tate, t of G: [ 1 t] = [φ]) [φ]t)) φ qµ + 1 [ g t] = [φ]) [φ]t) φ qµ We note that, due to Theorem 3, an analogou reult doe not hold for the a poteriori metric. Together with the lack of reciprocity of the a poteriori metric, thi i a trong indication that the a priori metric, and not the a poteriori one, are the natural metric on concurrent game. The Kernel. The kernel of the metric [ g ] define an equivalence relation g on the tate of a game tructure: g t iff [ g t] = 0. We call thi the game biimulation relation. Notice that by the reciprocity property of g, the game biimulation relation i canonical: 1 = 2 = g. Similarly, we define the game imulation preorder 1 t a the kernel of the directed metric [ 1 ], that i, 1 t iff [ 1 t] = 0. Alternatively, it i poible to define 1 and g directly. Given a relation R S S, let BR) F conit of all valuation k F uch that, for all, t S, if Rt then k) kt). We have the following reult. Theorem 6 Given a game tructure G, 1 rep. 1 ) can be characterized a the larget rep. larget ymmetrical) relation R uch that, for all tate, t with Rt, we have t and k BR). x 1 D 1 ). y 1 D 1 t). y 2 D 2 t). x 2 D 2 ). E y1,y2 t k) E x1,x2 k) ). 13) We note that the above theorem allow the computation of 1 via a partition-refinement cheme. From the logical characterization theorem, we obtain the following corollary. Corollary 1 For any game tructure G and tate, t of G, we have g t iff [φ]) = [φ]t) hold for every φ qµ and 1 t iff [φ]) [φ]t) hold for every φ qµ + 1. Computation. The next theorem tate that the metric are computable to any degree of preciion. Thi follow, ince the definition of the ditance between two tate of a given game, a the leat fixpoint of the metric tranformer 8), can be written a a formula in the theory of real, which i decidable [26]. Since the ditance between two tate may not be rational, we can only guarantee an approximate computation in general. Theorem 7 For any game tructure G, and tate, t of G, the following aertion hold. 1. For all rational v, and all ɛ > 0, it i decidable if [ 1 t] v < ɛ and if [ g t] v < ɛ. 2. It i decidable if 1 t and if g t. Game Metric and Bi-)imulation Metric. The a priori metric aume an adverarial relationhip between the player. We how that, on turn-baed game, the a priori biimulation metric coincide with the claical biimulation metric where the player cooperate. Moreover, on 1-MDP, the player-1 a priori imulation metric coincide with the cooperative imulation metric. The metric analog of claical bi)imulation [19, 22] i obtained through the metric tranformer H 12 : M M and H 12 : M M given by H 12 d), t) = [ t] k Cd) x 1 D 1) x 2 D 1) y 1 D 1t) y 2 D 1t) {E x1,x2 k) E y1,y2 t k)} H 12 d), t) = H 12 d), t) H 12 d)t, ) The metric [ 12 ] and [ 12 ] are defined a the leat fixed point of H 12 and H 12 repectively. The kernel of thee metric define the claical probabilitic imulation and biimulation relation. Theorem 8 The following aertion hold. 1. On turn-baed game tructure, [ g ] = [ 12 ]. 2. There i a determinitic game tructure G and tate, t in G uch that [ g t] > [ 12 t]. 3. There i a determinitic game tructure G and tate, t in G uch that [ g t] < [ 12 t]. The firt part follow eaily from the definition. The econd part i proven by the game in Figure 2, where it hold that [ g t] = 1 2 and [ 12 t] = 0. In thi figure, a in all ubequent one, different tate color denote that obervation variable have different value at the tate, o that the 8

9 *, * *, * a, b a, b u b, b t a, b b, a v u t v Figure 2. [ g t] = 1 2 and [ 12 t] = 0 a, b b, a b, b u t Figure 3. [ g t] = 0 but [ 12 t] = 1. tate are ditinguiheable in qµ. The third part i proven by the game in Figure 3 where [ t] = 0 and [u v] = 1. Theorem 9 The following aertion hold. 1. For i-mdp we have [ i ] = [ 12 ]. 2. There i a determinitic 2-MDP G with tate, t uch that [ 1 t] < [ 12 t]. 3. There i a determinitic 2-MDP G with tate, t uch that [t 1 ] > [t 12 ]. Again, the firt tatement follow eaily from the definition. The econd and third are proven by the determinitic 2- MDP in Figure 4, where again [ t] = 0 and [u v] = Dicuion Our derivation of i and g, for i {1, 2}, a kernel of metric, eem omewhat abtrue: mot equivalence or imilarity relation have been defined, after all, without reorting to metric. We now point out how a generalization of the uual definition [22, 2, 7, 8], uggeted in [4, 16], fail to produce the right relation. Furthermore, the flawed relation obtained a a generalization of [22, 2, 7, 8] are no impler than our definition, baed on kernel metric. Thu, our tudy of game relation a kernel of metric carrie no drawback in term of leading to more complicated definition. Indeed, we believe that the metric approach i the erior one for the tudy of game relation. We outline the flawed generalization of [22, 2, 7, 8] a propoed in [4, 16], explaining why it would eem a natural generalization. The alternating imulation of [2] i defined over determinitic game tructure. Player-i alternating imulation, for i {1, 2}, i the larget relation R atifying the following condition, for all tate, t S: Rt implie t and a i Γ i ). y i Γ i t). y i Γ i t). x i Γ i ). τ, x 1, x 2 )Rτt, y 1, y 2 ). v Figure 4. [ 1 t] = 0 and [ 12 t] = 1. Alo, [t 1 ] = 1 and [t 12 ] = 0. The MDP relation of [22], later extended to metric by [7, 8], rely on the fixpoint 2), where play the role of, play the role of, and R i replaced by ditribution equality modulo R, or R. Thi trongly ugget incorrectly that equivalence for general game probabilitic, concurrent game) can be obtained by taking the double alternating of in the definition of alternating imulation, changing all into, all into, and replacing R by R. The definition that would reult i a follow. We parameterize the new relation by a player i {1, 2}, a well a by whether mixed move or only pure move are allowed. For a relation R S S, for M {Γ, D}, for all, t S and i {1, 2} conider the following condition: loc) R t implie t. M-i-altim) R t implie x i M i ). y i M i t). y i M i t). x i M i ). δ, x 1, x 2 ) R δt, y 1, y 2 ); We then define the following relation: For i {1, 2} and M {Γ, D}, player-i M- alternating imulation M i i the larget relation that atifie loc) and M-i-altim). For i {1, 2} and M {Γ, D}, player-i M- alternating biimulation = M i i the larget ymmetrical relation that atifie loc) and M-i-altim). Over determinitic game tructure, the definition of Γ i and = Γ i coincide with the alternating imulation and biimulation relation of [2]. In fact, Γ i and = Γ i capture the determinitic emantic of qµ, and thu in ome ene generalize the reult of [2] to probabilitic game tructure. Theorem 10 For any game tructure G and tate, t of G, the following aertion hold: 1. = Γ i t iff [φ] Γ ) = [φ] Γ t) hold for every φ qµ i. 2. Γ i t iff [φ]γ ) [φ] Γ t) hold for every φ qµ + i. The following lemma tate that D i and = D i are the kernel of [ i ] and [ = i ], connecting thu the reult of combining the definition of [22] and [2] with a poteriori metric. Lemma 1 For all game tructure G, all player i {1, 2}, and all tate, t of G, we have D i t iff [ i t] = 0, and = D i t iff [ = i t] = 0. 9

10 We are now in a poition to prove that neither the Γ- relation not the D-relation are the correct relation on general concurrent game, ince neither characterize [qµ]. In particular, the D-relation are too fine, and the Γ-relation are incomparable with the relation i and g, for i {1, 2}. We prove thee negative reult firt for the D-relation. They follow from Theorem 3 and 5. Theorem 11 The following aertion hold: 1. For all game tructure G, all tate, t of G, and all i {1, 2}, we have that D i t implie i t, and = D i t implie i t. 2. There i a game tructure G, and tate, t of G, uch that i t but D i t. 3. There i a game tructure G, and tate, t of G, uch that [φ]) = [φ]t) for all φ qµ, but = D i t for ome i {1, 2}. We now turn our attention to the Γ-relation, howing that they are incomparable with i and g, for i {1, 2}. Theorem 12 The following aertion hold: 1. There exit a determinitic game tructure G and tate, t of G uch that Γ 1 t but 1 t, and = Γ 1 t but g t. 2. There exit a turn-baed game tructure G and tate, t of G uch that 1 t but Γ 1 t. and g t but = Γ 1 t. Finally, we remark that, in view of Theorem 6, the definition of the relation i and g for i {1, 2} are no more complex than the definition of D 1, Γ 1, = D 1, and = Γ 1. Reference [1] R. Alur, T.A. Henzinger, and O. Kupferman. Alternating time temporal logic. J. ACM, 49: , [2] R. Alur, T.A. Henzinger, O. Kupferman, and M.Y. Vardi. Alternating refinement relation. In CONCUR 98, LNCS 1466, pp Springer, [3] L. de Alfaro, T.A. Henzinger, and O. Kupferman. Concurrent reachability game. In FOCS 98, pp IEEE, [4] L. de Alfaro, T.A. Henzinger, and R. Majumdar. Dicounting the future in ytem theory. In ICALP 03, LNCS 2719, pp Springer, [5] L. de Alfaro and R. Majumdar. Quantitative olution of omega-regular game. Journal of Computer and Sytem Science, 68: , [6] C. Derman. Finite State Markovian Deciion Procee. Academic Pre, [7] J. Deharnai, V. Gupta, R. Jagadeean, and P. Panangaden. Metric for labelled markov ytem. In CONCUR 99, LNCS 1664, pp Springer, [8] J. Deharnai, V. Gupta, R. Jagadeean, and P. Panangaden. Approximating labelled markov procee. Information and Computation, 1841): , [9] J. Deharnai, V. Gupta, R. Jagadeean, and P. Panangaden. The metric analogue of weak biimulation for probabilitic procee. In LICS 02, pp , [10] J. Filar and K. Vrieze. Competitive Markov Deciion Procee. Springer-Verlag, [11] C.-C. Jou and S.A. Smolka. Equivalence, congruence and complete axiomatization for probabilitic procee. In CONCUR 90, LNCS 458, pp Springer, [12] J.G. Kemeny, J.L. Snell, and A.W. Knapp. Denumerable Markov Chain. D. Van Notrand Company, [13] D. Kozen. A probabilitic PDL. In STOC 83, pp , [14] D. Kozen. Reult on the propoitional µ-calculu. Theoretical Computer Science, 273): , [15] K.G. Laren and A. Skou. Compoitional Verification of Probabilitic Procee. In CONCUR 92, LNCS 630. Springer, [16] R. Majumdar. Symbolic algorithm for verification and control. PhD thei, Univerity of California, Berkeley, [17] D.A. Martin. The determinacy of Blackwell game. J. of Symbolic Logic, 634): , [18] A. McIver and C. Morgan. Abtraction, Refinement, and Proof for Probabilitic Sytem. Springer, [19] R. Milner. Operational and algebraic emantic of concurrent procee. In J. van Leeuwen, editor, Handbook of Theoretical Computer Science, vol B, pp Elevier, [20] M.J. Oborne and A. Rubintein. A Coure in Game Theory. MIT Pre, [21] R. Segala. Modeling and Verification of Randomized Ditributed Real-Time Sytem. PhD thei, MIT, Technical Report MIT/LCS/TR-676. [22] R. Segala and N.A. Lynch. Probabilitic imulation for probabilitic procee. In CONCUR 94, LNCS 836, pp Springer, [23] R. Segala and N.A. Lynch. Probabilitic imulation for probabilitic procee. Nordic Journal of Computing, 22): , [24] L.S. Shapley. Stochatic game. Proceeding of the National Academy of Science, USA, 39: , [25] M. Sion. On general minimax theorem. Pacific J. Math., 8: , [26] A. Tarki. A Deciion Method for Elementary Algebra and Geometry. Univerity of California Pre, [27] F. van Breugel and J. Worrel. An algorithm for quantitative verification of probabilitic tranition ytem. In CONCUR 01, LNCS 2154, pp Springer, [28] F. van Breugel and J. Worrel. Toward quantitative verification of probabilitic ytem. In ICALP 01, LNCS 2976, pp Springer, [29] J. von Neumann and O. Morgentern. Theory of Game and Economic Behavior. New York: John Wiley and Son,