Approximative Methods for Monotone Systems of min-max-polynomial Equations

Transcription

1 Approximative Methods or Monotone Systems o min-max-polynomial Equations Javier Esparza, Thomas Gawlitza, Stean Kieer, and Helmut Seidl Institut ür Inormatik Technische Universität München, Germany {esparza,gawlitza,kieer,seidl}@in.tum.de Abstract. A monotone system o min-max-polynomial equations (min-max- MSPE) over the variables X 1,..., X n has or every i exactly one equation o the orm X i = i(x 1,..., X n) where each i(x 1,..., X n) is an expression built up rom polynomials with non-negative coeicients, minimum- and maximum-operators. The question o computing least solutions o min-max- MSPEs arises naturally in the analysis o recursive stochastic games [4, 5, 12]. Min-max-MSPEs generalize MSPEs or which convergence speed results o Newton s method are established in [9, 2]. We present the irst methods or approximatively computing least solutions o min-max-mspes which converge at least linearly. Whereas the irst one converges aster, a single step o the second method is cheaper. Furthermore, we compute ɛ-optimal positional strategies or the player who wants to maximize the outcome in a recursive stochastic game. 1 Introduction In this paper we study monotone systems o min-max polynomial equations (min-max- MSPEs). A min-max-mspe over the variables X 1,..., X n contains or every 1 i n exactly one equation o the orm X i = i (X 1,..., X n ) where every i (X 1,..., X n ) is an expression built up rom polynomials with non-negative coeicients, minimumand maximum-operators. An example o such an equation is X 1 = 3X 1 X 2 +5X 2 1 4X 2 (where is the minimum-operator). The variables range over non-negative reals. Minmax-MSPEs are called monotone because i is a monotone unction in all arguments. Min-max-MSPEs naturally appear in the study o two-player stochastic games and competitive Markov decision processes, in which, broadly speaking, the next move is decided by one o the two players or by tossing a coin, depending on the game s position (see e.g. [10, 6]). The min and max operators model the competition between the players. The product operator, which leads to non-linear equations, allows to deal with recursive stochastic games [4, 5], a class o games with an ininite number o positions, and having as special case extinction games, games in which players inluence with their actions the development o a population whose members reproduce and die, and the player s goals are to extinguish the population or keep it alive (see Section 4). Min-max-MSPEs generalize several other classes o equation systems. I product is disallowed, we obtain systems o min-max linear equations, which appear in classical This work was in part supported by the DFG project Algorithms or Sotware Model Checking.

2 two-person stochastic games with a inite number o game positions. The problem o solving these systems has been thoroughly studied [1, 7, 8]. I both min and max are disallowed, we obtain monotone systems o polynomial equations, which are central to the study o recursive Markov chains and probabilistic pushdown systems, and have been recently studied in [3, 9, 2]. I only one o min or max is disallowed, we obtain a class o systems corresponding to recursive Markov decision processes [4]. All these models have applications in the analysis o probabilistic programs with procedures [12]. In vector orm we denote a min-max-mspe by X = (X) where X denotes the vector (X 1,..., X n ) and denotes the vector ( 1,..., n ). By Kleene s theorem, i a min-max-mspe has a solution then it also has a least one, denoted by µ, which is also the relevant solution or the applications mentioned above. Kleene s theorem also ensures that the iterative process κ (0) = 0, κ (k+1) = (κ (k) ), k N, the so-called Kleene sequence, converges to µ. However, this procedure can converge very slowly: in the worst case, the number o accurate bits o the approximation grows with the logarithm o the number o iterations (c. [3]). Thus, the goal is to replace the unction by an operator G : R n R n such that the respective iterative process also converges to µ but aster. In [3, 9, 2] this problem was studied or min-max-mspes without the min and max operator. There, G was chosen as one step o the well-known Newton s method (c. or instance [11]). This means that, or a given approximate x (k), the next approximate x (k+1) = G(x (k) ) is determined by the unique solution o a linear equation system which is obtained rom the irst order Taylor approximation o at x (k). It was shown that this choice guarantees linear convergence, i.e., the number o accurate bits grows linearly in the number o iterations. Notice that when characterizing the convergence behavior the term linear does not reer to the size o. However, this technique no longer works or arbitrary min-max-mspes. I we approximate at x (k) through its irst order Taylor approximation at x (k) there is no guarantee that the next approximate still lies below the least solution, and the sequence o approximants may even diverge. For this reason, the PReMo tool [12] uses roundrobin iteration or min-max-mspes, an optimization o Kleene iteration. Unortunately, this technique also exhibits logarithmic convergence behavior in the worst case. In this paper we overcome the problem o Newton s method. Instead o approximating (at the current approximate x (k) ) by a linear unction, both o our methods approximate by a piecewise linear unction. In contrast to the applications o Newton s method in [3, 9, 2], this approximation may not have a unique ixpoint, but it has a least ixpoint which we use as the next approximate x (k+1) = G(x (k) ). Our irst method uses an approximation o at x (k) whose least ixpoint can be determined using the algorithm or systems o rational equations rom [8]. The approximation o at x (k) used by our second method allows to use linear programming to compute x (k+1). Our methods are the irst algorithms or approximatively computing µ which converge at least linearly, provided that is quadratic, an easily achievable normal orm. The rest o the paper is organized as ollows. In Section 2 we introduce basic concepts and state some important acts about min-max-mspes. A class o games which can be analyzed using our techniques is presented in Section 4. Our main contribution, the two approximation methods, is presented and analyzed in Sections 5 and 6. In Sec- 2

3 tion 7 we study the relation between our two approaches and compare them to previous work. We conclude in Section 8. Missing proos can be ound in a technical report [?]. 2 Notations, Basic Concepts and a Fundamental Theorem As usual, R and N denote the set o real and natural numbers. We assume 0 N. We write R 0 or the set o non-negative real numbers. We use bold letters or vectors, e.g. x R n. In particular 0 denotes the vector (0,..., 0). The transpose o a matrix or a vector is indicated by the superscript. We assume that the vector x R n has the components x 1,..., x n. Similarly, the i-th component o a unction : R n R m is denoted by i. As in [2], we say that x R n has i N valid bits o y R n i x j y j 2 i y j or j = 1,..., n. We identiy a linear unction rom R n to R m with its representation as a matrix rom R m n. The identity matrix is denoted by I. The Jacobian o a unction : R n R m at x R n is the matrix o all irst-order partial derivatives o at x, i.e., the m n-matrix with the entry i X j (x) in the i-th row and the j-th column. We denote it by (x). The partial order on R n is deined by setting x y i x i y i or all i = 1,..., n. We write x < y i x y and x y. The operators and are deined by x y := min{x, y} and x y := max{x, y} or x, y R. These operators are also extended component-wise to R n and point-wise to R n -valued unctions. A unction : D R n R m it called monotone on M D i (x) (y) or every x, y M with x y. Let X R n and : X X. A vector x X is called ixpoint o i x = (x). It is the least ixpoint o i y x or every ixpoint y X o. I it exists we denote the least ixpoint o by µ. We call easible i has some ixpoint x X. Let us ix a set X = {X 1,..., X n } o variables. We call a vector = ( 1,..., m ) o polynomials 1,..., m in the variables X 1,..., X n a system o polynomials. is called linear (resp. quadratic) i the degree o each i is at most 1 (resp. 2), i.e., every monomial contains at most one variable (resp. two variables). As usual, we identiy with its interpretation as a unction rom R n to R m. As in [9, 2] we call a monotone system o polynomials (MSP or short) i all coeicients are non-negative. Min-max-MSPs. Given polynomials 1,..., k we call 1 k a min-polynomial and 1 k a max-polynomial. A unction that is either a min- or a maxpolynomial is also called min-max-polynomial. We call = ( 1,..., n ) a system o min-polynomials i every component i is a min-polynomial. The deinition o systems o max-polynomials and systems o min-max-polynomials is analogous. A system o min-max-polynomials is called linear (resp. quadratic) i all occurring polynomials are linear (resp. quadratic). By introducing auxiliary variables every system o minmax-polymials can be transormed into a quadratic one in time linear in the size o the system (c. [9]). A system o min-max-polynomials where all coeicients are rom R n 0 is called a monotone system o min-max-polynomials (min-max-msp) or short. The terms min-msp and max-msp are deined analogously. Example 1. (x 1, x 2 ) = ( 1 2 x , x 1 2) is a quadratic min-max-msp. 3

4 A min-max-msp = ( 1,..., n ) can be considered as a mapping rom R n 0 to R n 0. The Kleene sequence (κ(k) ) k N is deined by κ (k) := k (0), k N. We have: Lemma 1. Let : R n 0 Rn 0 be a min-max-msp. Then: (1) is monotone and continuous on R n 0 ; and (2) I is easible (i.e., has some ixpoint), then has a least ixpoint µ and µ = lim k κ (k). Strategies. Assume that denotes a system o min-max-polynomials. A -strategy σ or is a unction that maps every max-polynomial i = i,1 i,ki occurring in to one o the i,j s and every min-polynomial i to i. We also write i σ or σ( i ). Accordingly, a -strategy π or is a unction that maps every min-polynomial i = i,1 i,k occurring in to one o the i,j s and every max-polynomial i to i. We denote the set o -strategies or by Σ and the set o -strategies or by Π. For s Σ Π, we write s or ( s 1,..., s n). We deine Π := {π Π π is easible}. We drop the subscript whenever it is clear rom the context. Example 2. Consider rom Example 1. Then π : 1 2 x , x 1 2 x 1 2 is a -strategy. The max-msp π is given by π (x 1, x 2 ) = (3, x 1 2). We collect some elementary acts concerning strategies. Lemma 2. Let be a easible min-max-msp. Then (1) µ σ µ or every σ Σ; (2) µ π µ or every π Π ; (3) µ π = µ or some π Π. In [4] the authors consider a subclass o recursive stochastic games or which they prove that a positional optimal strategy exists or the player who wants to maximize the outcome (Theorem 2). The outcome o such a game is the least ixpoint o some min-max-msp. In our setting, Theorem 2 o [4] implies that there exists a -strategy σ such that µ σ = µ provided that is derived rom such a recursive stochastic game. Example 4 shows that this property does not hold or arbitrary min-max-msps. Example 3. Consider rom Example 1. Let σ 1, σ 2 Σ be deined by σ 1 (x 1 2) = x 1 and σ 2 (x 1 2) = 2. Then µ σ1 = (1, 1), µ σ2 = ( 5 2, 2) and µ = (3, 3). The proo o the ollowing undamental result is inspired by the proo o Theorem 2 in [4]. Although the result looks very natural it is non-trivial to prove. Theorem 1. Let be a easible max-msp. Then µ σ = µ or some σ Σ..mine 3 An Example o Application: Extinction Games ======= 4

5 4 A Class o Applications: Extinction Games.r1222 In order to illustrate the interest o min-max-msps we consider extinction games, which are special stochastic games. Consider a world o n dierent species s 1,..., s n. Each species s i is controlled by one o two adversarial players. For each s i there is a non-empty set A i o actions. An action a A i replaces a single individual o species s i by other individuals speciied by the action a. The actions can be probabilistic. E.g., an action could transorm an adult rabbit to zero individuals with probability 0.2, to an adult rabbit with probability 0.3 and to an adult and a baby rabbit with probability 0.5. Another action could transorm an adult rabbit to a at rabbit. The max-player (min-player) wants to maximize (minimize) the probability that some initial population is extinguished. During the game each player continuously chooses an individual o a species s i controlled by her/him and applies an action rom A i to it. Note that actions on dierent species are never in conlict and the execution order is irrelevant. What is the probability that the population is extinguished i the players ollow optimal strategies? To answer those questions we set up a min-max-msp with one min-maxpolynomial or each species, thereby ollowing [?,4]. The variables X i represent the probability that a population with only a single individual o species s i is extinguished. In the rabbit example we have X adult = X adult + 0.5X adult X baby X at, assuming that the adult rabbits are controlled by the max-player. The probability that an initial population with p i individuals o species s i is extinguished is given by n i=1 ((µ) i) pi. The stochastic termination games o [4, 5, 12] can be considered as extinction games. In the ollowing we present another instance. The primaries game. In order to illustrate the interest o min-max-msps we consider the ollowing stochastic game. Hillary Clinton has to decide her strategy in the primaries. Her team estimates that undecided voters have not yet decided to vote or her or three possible reasons: they consider her (a) cold and calculating, (b) too much part o Washington s establishment, or (c) they listen to Obama s campaign. So the team decides to model those problems as species in an extinction game. The larger the population o a species, the more inluenced is an undecided voter by the problem. The goal o Clinton s team is to maximize the extinction probabilities..mine So the team decides to model the state o an undecided voter by a triple (n a, n b, n c ) o natural numbers indicating the degree to which the voter is inluenced by these three problems (the higher the number, the more inluenced he or she is). The goal o Clinton s team is to bring as many voters as possible to the state (0, 0, 0). I n a > 1, Clinton can tackle problem (a) by either showing emotions or concentrating on her program. I she shows emotions, her team estimates that the voter s state changes by ( 1, 0, 0) with probability 0.3, but with probability 0.7 the action may backire and change it by (+1, 0, 0). I she concentrates on her program, the state changes by ( 1, 0, 0) with probability 0.1, and by ( 1, 0, +1) with probability 0.9. Let X 1 (resp. X 2, X 3 ) denote the probability that the voters s state changes eventually rom (1, 0, 0) (resp. (0, 1, 0), (0, 0, 1)) to (0, 0, 0), assuming Clinton and Obama ollow perect strategies. Then X 1 satisies equation (1) below. I n b > 1, Clinton can choose between ======= Clinton s possible actions or problem (a) are showing emotions or concentrating on her program. I she shows emotions, her team estimates that the individual o problem (a) is removed with probability 0.3, but with probability 0.7 the 5

6 (*I ve added some lines here!*) action backires and produces yet another individual o (a). This and the eect o concentrating on her program can be read o rom Equation (1) below. For problem (b), Clinton can choose between.r1222 concentrating on her voting record or her statement I ll be ready rom day 1. Her team estimates the eect as given in Equation (2). Problem (c) is controlled by Obama, who has the choice between his change message, or attacking Clinton or her position on Iraq, see Equation (3). X a = X 2 a X c (1) X b = X c 0.4X b + 0.3X c (2) X c = 0.5X b + 0.3X 2 b X a + 0.4X a X b + 0.1X b (3).mine What is the probability, assuming perect strategies, that a voter in state (1, 0, 0) is eventually moved to (0, 0, 0)? It is easy to see that it is given by the irst component o the least solution o this mn-max-msp. We are also interested in the strategies by Clinton and Obama that lead to this probability. In the next two sections we show how to eiciently solve these problems. This is an example o an extinction game. We have three populations o problem instances (a state (n a, n b, n c ) indicates that the populations have n a, n b, and n c individuals). Individuals can reproduce or die. The goal o one o the players (Clinton in our case) is to get all populations extinct, while the other player wants to keep them alive. Another example or an extinction game in appendix??. ======= What should Clinton and Obama do? What are the extinction probabilities, assuming perect strategies? In the next sections we show how to eiciently solve these problems..r The τ -Method Assume that denotes a easible min-max-msp. In this section we present our irst method or computing µ approximatively. We call it τ -method. This method computes, or each approximate x (i), the next approximate x (i+1) as the least ixpoint o a piecewise linear approximation L(, x (i) ) x (i) (see below) o at x (i). This approximation is a system o linear min-max-polynomials where all coeicients o monomials o degree 1 are non-negative. Here, we call such a system a monotone linear min-maxsystem (min-max-mls or short). Note that a min-max-mls is not necessarily a minmax-msp, since negative coeicients o monomials o degree 0 are allowed, e.g. the min-max-mls (x 1 ) = x 1 1 is not a min-max-msp. In [8] a min-max-mls is considered as a system o equations (called system o rational equations in [8]) which we denote by X = (X) in vector orm. We identiy a min-max-mls with its interpretation as a unction rom R n to R n (R denotes the complete lattice R {, }). Since is monotone on R n, it has a least ixpoint µ R n which can be computed using the strategy improvement algorithm rom [8]. We now deine the min-max-mls L(, y), a piecewise linear approximation o at y. As a irst step, let us consider a monotone polynomial : R n 0 R 0. Given some approximate y R n 0, a linear approximation L(, y) : Rn R o at y is given by the irst order Taylor approximation at y, i.e., L(, y)(x) := (y) + (y)(x y), x R n. 6

7 This is precisely the linear approximation which is used or Newton s method. Now consider a max-polynomial = 1 k : R n R. We deine the approximation L(, y) : R n R o at y by L(, y) := L( 1, y) L( k, y). We emphasize that in this case, L(, y) is in general not a linear unction but a linear max-polynomial. Accordingly, or a min-msp = 1 k : R n R, we deine L(, y) := L( 1, y) L( k, y). In this case L(, y) is a linear min-polynomial. Finally, or a min-max-msp : R n R n, we deine the approximation L(, y) : R n R n o at y by L(, y) := (L( 1, y),..., L( n, y)) which is a min-max-mls. Example 4. Consider the min-max-msp rom Example 1. The approximation L(, ( 1 2, 1 2 )) is given by L(, ( 1 2, 1 2 ))(x 1, x 2 ) = ( 1 2 x , x 1 2 ). Using the approximation L(, x (i) ) we deine the operator N : R n 0 Rn 0 which gives us, or an approximate x (i), the next approximate x (i+1) by N (x) := µ(l(, x) x), x R n 0. Observe that L(, x) x is still a min-max-mls (at least ater introducing auxiliary variables in order to eliminate components which contain - and -operators). Example 5. In Example 5 we have: N ( 1 2, 1 2 )=µ(l(, ( 1 2, 1 2 )) ( 1 2, 1 2 ) )=( 11 8, 2). We collect basic properties o N in the ollowing lemma: Lemma 3. Let be a easible min-max-msp and x, y R n 0. Then: 1. x, (x) N (x); 2. x = N (x) whenever x = (x); 3. (Monotonicity o N ) N (x) N (y) whenever x y; 4. N (x) (N (x)) whenever x (x); 5. N (x) N σ(x) or every -strategy σ Σ; 6. N (x) N π(x) or every -strategy π Π; 7. N (x) = N π(x) or some -strategy π Π. In particular Lemma 3 implies that the least ixpoint o N is equal to the least ixpoint o. Moreover, iteration based on N is at least as ast as Kleene iteration. We thereore use this operator or computing approximates to the least ixpoint. Formally, we deine: Deinition 1. We call the sequence (τ (k) (k) ) o approximates deined by τ := N k(0) or k N the τ -sequence or. We drop the subscript i it is clear rom the context. Proposition 1. Let be a easible min-max-msp. The τ -sequence (τ (k) ) or (see deinition 1) is monotonically increasing, bounded rom above by µ, and converges to µ. Moreover, κ (k) τ (k) or all k N. We now show that the new approximation method converges at least linearly to the least ixpoint. Theorem 6.2 o [2] implies the ollowing lemma about the convergence o Newton s method or MSPs, i.e., systems without maxima and minima. Lemma 4. Let be a easible quadratic MSP. The sequence (τ (k) ) k N converges linearly to µ. More precisely, there is a k N such that τ (k +i (n+1) 2 n) has at least i valid bits o µ or every i N. 7

8 We emphasize that linear convergence is the worst case. In many practical examples, in particular i the matrix I (µ) is invertible, Newton s method converges exponentially. We mean by this that the number o accurate bits o the approximation grows exponentially in the number o iterations. As a irst step towards our main result or this section, we use Lemma 4 to show that our approximation method converges linearly whenever is a max-msps. In this case we obtain the same convergence speed as or MSPs. Lemma 5. Let be a easible max-msp. Let M := {σ Σ µ σ = µ}. The set M is non-empty and τ (i) τ (i) σ or all σ M and i N. Proo. Theorem 1 implies that there exists a -strategy σ Σ such that µ σ = µ. Thus M is non-empty. Let σ M. By induction on k Lemma 3 implies τ (k) N k (0) N k = (k) σ(0) = τ σ or every k N. Combining Lemma 4 and Lemma?? we get linear convergence or max-msps: Theorem 2. Let be a easible quadratic max-msp. The τ -sequence (τ (k) ) or (see deinition 1) converges linearly to µ. More precisely, there is a k N such that τ (k +i (n+1) 2 n) has at least i valid bits o µ or every i N. A direct consequence o Lemma?? is that the τ -sequence (τ (i) ) converges exponentially i (τ (i) σ) converges exponentially or some σ Σ with µ σ = µ. This is in particular the case i the matrix I ( σ ) (µ) is invertible. In order to extend this result to min-max-msps we state the ollowing lemma which enables us to relate the sequence (τ (i) (i) ) to the sequences (τ π) where µ π = µ. Lemma 6. Let be a easible min-max-msp and m denote the number o strategies π Π with µ = µ π. There is a constant k N such that or all i N there exists some strategy π Π with µ = µ π and τ (i) τ (k+m i) π. We now present the main result o this section which states that our approximation method converges at least linearly also in the general case, i.e., or min-max-msps. Theorem 3. Let be a easible quadratic min-max-msp and m denote the number o strategies π Π with µ = µ π. The τ -sequence (τ (k) ) or (see deinition 1) converges linearly to µ. More precisely, there is a k N such that τ (k +i m (n+1) 2 n ) has at least i valid bits o µ or every i N. The upper bound on the convergence rate provided by Theorem 2 is by the actor m worse than the upper bound obtained or MSPs. Since m is the number o strategies π Π with µ π = µ, m is trivially bounded by Π but is usually much smaller. The τ -sequence (τ (i) (i) ) converges exponentially whenever (τ π) converges exponentially or every π with µ π = µ (see [?]). The latter condition is typically satisied (see the discussion ater Theorem 2). In order to determine the approximate τ (i+1) = N (τ (i) ) rom τ (i) we must compute the least ixpoint o the min-max-mls L(, τ (i) ) τ (i). This can be done by using the strategy improvement algorithm rom [8]. The algorithm iterates over -strategies. 8

9 For each strategy it solves a linear program or alternatively iterates over -strategies. The number o -strategies used by this algorithm is trivially bounded by the number o -strategies or L(, τ (i) ) τ (i) which is exponentially in the number o -expressions occurring in L(, τ (i) ) τ (i). However, we do not know an example or which the algorithm considers more than linearly many strategies. 6 The ν-method The τ -method, presented in the previous section, uses strategy iteration over - strategies to compute N (y). This could be expensive, as there may be exponentially many -strategies. Thereore, we derive an alternative generalization o Newton s method that in each step picks the currently most promising -strategy directly, without strategy iteration. Consider again a ixed easible min-max-msp whose least ixpoint we want to approximate. Assume that y is some approximation o µ. Instead o applying N to y, as the τ -method, we now choose a strategy σ Σ such that (y) = σ (y), and compute N σ(y), where N σ was deined in Section 5 as N σ(y) := µ(l( σ, y) y). In the ollowing we write N σ instead o N σ i is understood. Assume or a moment that is a max-msp and that there is a unique σ Σ such that (y) = σ (y). The approximant N σ (y) is the result o applying one iteration o Newton s method, because L( σ, y) is not only a linearization o σ, but the irst order Taylor approximation o at y. More precisely, L( σ, y)(x) = (y)+ (y) (x y), and N σ (y) is obtained by solving x = L( σ, y)(x). In this sense, the ν-method is a more direct generalization o Newton s method than the τ -method. Formally, we deine the ν-method by a sequence o approximates, the ν-sequence. Deinition 2 (ν-sequence). A sequence (ν (k) ) k N is called ν-sequence o a min-max- MSP i ν (0) σ(k) (ν (k) = 0 and or each k there is a strategy σ (k) ) and ν(k+1) = N σ (k) Σ with (ν (k) ) = (ν (k) ). We may drop the subscript i is understood. Notice the nondeterminism here i there is more than one -strategy that attains (ν (k) ). The ollowing proposition is analogous to Proposition 1 and states some basic properties o ν-sequences. Proposition 2. Let be a easible min-max-msp. The sequence (ν (k) ) is monotonically increasing, bounded rom above by µ, and converges to µ. More precisely, we have κ (k) ν (k) (ν (k) ) ν (k+1) µ or all k N. The goal o this section is again to strengthen Proposition 2 towards quantitative convergence results or ν-sequences. To achieve this goal we again relate the convergence o ν-sequences to the convergence o Newton s method or MSPs. I is an MSP, Lemma 4 allows to argue about the Newton operator N when applied to approximates x µ. To transer this result to min-max-msps we need an invariant like ν (k) µ σ(k) or ν-sequences. As a irst step to such an invariant we urther restrict the selection o the σ (k). Roughly speaking, the strategy in a component i is only changed when it is immediate that component i has not yet reached its ixpoint. 9

10 Algorithm 1 lazy ν-method procedure lazy-ν(, k) assumes: is a min-max-msp returns: ν (k), σ (k) obtained by k iterations o the lazy ν-method ν 0 σ any σ Σ such that (0) = σ (0) or i rom 1 to k do ν N σ (ν) σ lazy strategy update rom ν and σ od return ν, σ procedure N (y) assumes: is a min-msp, y R n 0 returns: µ(l(, y) y) g linear min-msp with g(d) = L(, y)(y + d) y u κ (n) g eg (eg 1,... j, eg n) where 0 i ui = 0 eg i = g i i u i > 0 d maximize x x n subject to 0 x eg(x) by 1 LP return y + d Deinition 3 (lazy strategy update). Let x σ (x) or a σ Σ. We say that σ Σ is obtained rom x and σ by a lazy strategy update i (x) = σ (x) and σ ( i ) = σ( i ) holds or all components i with i (x) = x i. We call a ν-sequence (ν (k) ) k N lazy i or all k, the strategy σ (k) is obtained rom ν (k) and σ (k 1) by a lazy strategy update. The key property o lazy ν-sequences is the ollowing non-trivial invariant. Lemma 7. Let (ν (k) ) k N be a lazy ν-sequence. Then ν (k) µ σ(k)π holds or all k N and all π Π. The ollowing example shows that lazy strategy updates are essential to Lemma 6 even or max-msps. Example 6. Consider the MSP (x, y) = ( 1 2 x, xy ). Let σ(0) ( 1 2 x) = 1 2 and σ (1) ( 1 2 x) = x. Then there is a ν-sequence (ν(k) ) with ν (0) = 0, ν (1) = N σ (0)(0) = ( 1 2, 0), ν(2) = N σ (1)(ν (1) ). However, the conclusion o Lemma 6 does not hold, because ( 1 2, 0) = ν(1) µ σ(1) = (0, 1 2 ). Notice that σ(1) is not obtained by a lazy strategy update, as 1 (ν (1) ) = ν (1) 1. Lemma 6 alls short o our subgoal to establish ν (k) µ σ(k), because Π \ Π might be non-empty. In act, we provide an example in [?] showing that ν (k) µ σ(k)π does not always hold or all π Π, even when σ(k)π is easible. Luckily, Lemma 6 will suice or our convergence speed result. The let procedure o Algorithm 1 summarizes the lazy ν-method which works by computing lazy ν-sequences. The ollowing lemma relates the ν-method or min-max- MSPs to Newton s method or MSPs. Lemma 8. Let be a easible min-max-msp and (ν (k) ) a lazy ν-sequence. Let m be the number o strategy pairs (σ, π) Σ Π with µ = µ σπ. Then m 1 and there is a constant k as N such that, or all k N, there exist strategies σ Σ, π Π with µ = µ σπ (kas +m k) and ν τ (k) σπ. In typical cases, i.e., i I ( σπ ) (µ) is invertible or all σ Σ and π Π with µ σπ = µ, Newton s method converges exponentially. The ollowing theorem captures the worst-case, in which the lazy ν-method still converges linearly. 10

11 Theorem 4. Let be a quadratic easible min-max-msp. The lazy ν-sequence (ν (k) ) k N converges linearly to µ. More precisely, let m be the number o strategy pairs (σ, π) Σ Π with µ = µ σπ. Then there is a k N such that ν (k +i m (n+1) 2 n) has at least i valid bits o µ or every i N. Next we show that N σ(y) can be computed exactly by solving a single LP. The right procedure o Algorithm 1 accomplishes this by taking advantage o the ollowing proposition which states that N σ(y) can be determined by computing the least ixpoint o some linear min-msp g. Proposition 3. Let y σ (y) µ. Then N σ(y) = y +µg or the linear min-msp g with g(d) = L( σ, y)(y + d) y. Ater having computed the linear min-msp g, Algorithm 1 determines the 0- components o µg. This can be done by perorming n Kleene steps, since (µg) i = 0 whenever (κ (n) g ) i = 0. Let g be the linear min-msp obtained rom g by substituting the constant 0 or all components g i with (µg) i = 0. The least ixpoint o g can be computed by solving a single LP, as implied by the ollowing lemma. The correctness o Algorithm 1 ollows. Lemma 9. Let g be a linear min-msp such that g i = 0 whenever (µg) i = 0 or all components i. Then µg is the greatest vector x with x g(x). The ollowing theorem is a direct consequence o Lemma 6 or the case where Π = Π. It shows the second major advantage o the lazy ν-method, namely, that that the strategies σ (k) are meaningul in terms o games. Theorem 5. Let Π = Π. Let (ν (k) ) k N be a lazy ν-sequence. Then ν (k) µ σ(k) holds or all k N. As (ν (k) ) converges to µ, the max-strategy σ (k) can be considered ɛ-optimal. In terms o games, Theorem 5 states that the strategy σ (k) guarantees the max-player an outcome o at least ν (k). It is open whether an analogous theorem holds or the τ -method. Application to the primaries example. We solved the equation system o Section 4 approximatively by perorming 5 iterations o the lazy ν-method. Using Theorem 5 we ound that Clinton can extinguish a problem (a) individual with a probability o at least X a = by concentrating on her program and her ready rom day 1 message. (More than 70 Kleene iterations would be needed to iner that X a is at least 0.49.) As ν (5) seems to solve above equation system quite well in the sense that (ν (5) ) ν (5) is small, we are pretty sure about Obama s optimal strategy: he should talk about Iraq. As ν (2) X 1 > 0.38 and σ (2) maps 1 to X1 2, Clinton s team can use Theorem 5 to iner that X a 0.38 by showing emotions and using her ready rom day 1 message. 7 Discussion In order to compare our two methods in terms o convergence speed, assume that denotes a easible min-max-msp. Since N (x) N σ (Lemma 3.5), it ollows 11

12 that τ (i) ν (i) holds or all i N. This means that the τ -method is as least as ast as the ν-method i one counts the number o approximation steps. Next, we construct an example which shows that the number o approximation steps needed by the lazy ν-method can be much larger than the respective number needed by the τ -method. It is parameterized with an arbitrary k N and given by (x 1, x 2 ) = ( x2 2, x x (k+1)). Since the constant 2 2(k+1) is represented using O(k) bits, it is o size linear in k. It can be shown (see [?]) that the lazy ν- method needs at least k steps. More precisely, ν τ (k) (1.5, 1.95). The τ -method needs exactly 2 steps. We now compare our approaches with the tool PReMo [12]. PReMo employs 4 dierent techniques to approximate µ or min-max-msps : It uses Newton s method only or MSPs without min or max. In this case both o our methods coincide with Newton s method. For min-max-msps, PReMo uses Kleene iteration, round-robin iteration (called Gauss-Seidel in [12]), and an optimistic variant o Kleene which is not guaranteed to converge. In the ollowing we compare our algorithms only with Kleene iteration, as our algorithms are guaranteed to converge and a round-robin step is not aster than n Kleene steps. Our methods improve on Kleene iteration in the sense that κ (i) τ (i), ν (i) holds or all i N, and our methods converge linearly, whereas Kleene iteration does not converge linearly in general. For example, consider the MSP g(x) = 1 2 x with µg = 1. Kleene iteration needs exponentially many iterations or j bits [3], whereas Newton s method gives exactly 1 bit per iteration. For the slightly modiied MSP g(x) = g(x) 1 which has the same ixpoint, PReMo no longer uses Newton s method, as g contains a minimum. Our algorithms still produce exactly 1 bit per iteration. In the case o linear min-max systems our methods compute the precise solution and not only an approximation. This applies, or example, to the max-linear system o [12] describing the expected time o termination o a nondeterministic variant o Quicksort. Notice that Kleene iteration does not compute the precise solution (except or trivial instances), even or linear MSPs without min or max. We implemented our algorithms prototypically in Maple and ran them on the quadratic nonlinear min-max-msp describing the termination probabilities o a recursive simple stochastic game. This game stems rom the example suite o PReMo (rssg2.c) and we used PReMo to produce the equations. Both o our algorithms reached the least ixpoint ater 2 iterations. So we could compute the precise µ and optimal strategies or both players, whereas PReMo computes only approximations o µ. 8 Conclusion We have presented the irst methods or approximatively computing the least ixpoint o min-max-msps, which are guaranteed to converge at least linearly. Both o them are generalizations o Newton s method. Whereas the τ -method converges aster in terms o number o approximation steps, one approximation step o the ν-method is cheaper. Furthermore, we have shown that the ν-method computes ɛ-optimal strategies or games. Whether such a result can also be established or the τ -method is still open. A direction or uture research is to evaluate our methods in practice. In particular, the 12

13 inluence o imprecise computation through loating point arithmetic should be studied. It would also be desirable to ind a bound on the threshold k. Acknowledgement. We thank the anonymous reerees or valuable comments. Reerences 1. A. Condon. The complexity o stochastic games. In. and Comp., 96(2): , J. Esparza, S. Kieer, and M. Luttenberger. Convergence thresholds o Newton s method or monotone polynomial equations. In Proceedings o STACS 2008, to appear. 3. K. Etessami and M. Yannakakis. Recursive Markov chains, stochastic grammars, and monotone systems o nonlinear equations. In STACS, pages , K. Etessami and M. Yannakakis. Recursive Markov decision processes and recursive stochastic games. In Proc. ICALP 2005, volume 3580 o LNCS. Springer, K. Etessami and M. Yannakakis. Eicient qualitative analysis o classes o recursive Markov decision processes and simple stochastic games. In STACS, pages , J. Filar and K. Vrieze. Competitive Markov Decision processes. Springer, T. Gawlitza and H. Seidl. Precise ixpoint computation through strategy iteration. In European Symposium on Programming (ESOP), LNCS 4421, pages Springer, T. Gawlitza and H. Seidl. Precise relational invariants through strategy iteration. In J. Duparc and T. A. Henzinger, editors, CSL, volume 4646 o LNCS, pages Springer, S. Kieer, M. Luttenberger, and J. Esparza. On the convergence o Newton s method or monotone systems o polynomial equations. In STOC, pages ACM, A. Neyman and S. Sorin. Stochastic Games and Applications. Kluwer Academic Press, J. Ortega and W. Rheinboldt. Iterative solution o nonlinear equations in several variables. Academic Press, D. Wojtczak and K. Etessami. PReMo: An analyzer or probabilistic recursive models. In Proc. o TACAS, volume 4424 o LNCS, pages Springer,