Incremental Quantitative Verification for Markov Decision Processes

Transcription

1 Incremental Quantitative Verification for Markov Deciion Procee Marta Kwiatkowka, David Parker and Hongyang Qu Department of Computer Science, Univerity of Oxford, Park Road, Oxford, OX1 3QD, UK {marta.kwiatkowka, david.parker, Abtract Quantitative verification technique provide an effective mean of computing performance and reliability propertie for a wide range of ytem. However, the computation required can be expenive, particularly if it ha to be performed multiple time, for example to determine optimal ytem parameter. We preent efficient incremental technique for quantitative verification of Markov deciion procee, which are able to re-ue reult from previou verification run, baed on a decompoition of the model into it trongly connected component (SCC). We alo how how thi SCC-baed approach can be further optimied to improve verification peed and how it can be combined with ymbolic data tructure to offer better calability. We illutrate the effectivene of the approach on a election of large cae tudie. Keyword-Quantitative verification; incremental verification; Markov deciion procee; performance analyi; probabilitic model checking I. INTRODUCTION In almot all apect of everyday life, we are reliant on computeried ytem: from the controller found in car and plane, to the computer network underlying our communication, tranport and finance ytem. The prevalence of uch ytem, combined with their increaing complexity, mean that effective method to aure their reliability and performance are eential. Model-baed analyi technique provide an effective way of achieving thi. The ytem to be analyed are often inherently tochatic: device component may be failure-prone; meage ent acro communication network may get lot or delayed; and wirele technologie uch a Bluetooth and ZigBee ue randomiation. Thu, model are typically probabilitic in nature; they are alo often extended with time and/or quantitie for reource. Approache to the analyi of uch model range from dicrete-event imulation, to analytical method, to numerical olution, and each have their own trength and weaknee. In thi paper, we focu on technique that exhautively contruct probabilitic model and then perform an exact analyi, baed on numerical computation. In particular, we conider quantitative verification, which i a formal approach for pecifying and checking quantitative propertie of a ytem model. The model itelf, typically a Markov chain or Markov deciion proce, i ytematically contructed from a formal decription in ome high-level modelling language. Wherea traditional formal verification technique are uually applied to check the correctne of ytem, quantitative verification can be ued to analye propertie uch a performance or reliability. A key trength of quantitative verification i that it yield exact reult, a oppoed to, for example, the approximation produced from imulation-baed analyi technique. In fact, for Markov deciion procee, which are the focu of thi paper, imulation-baed technique are inappropriate due to the preence of nondeterminim. Probabilitic temporal logic uch a PCTL [1], [2] and it variant are ued to formally pecify a wide range of ytem propertie, for example the probability that a data packet ha been uccefully tranmitted within 0.5 econd, the probability that both enor are imultaneouly non-operational or the expected time taken to execute the protocol. Probabilitic model checker, e.g. PRISM [3], have been ued to apply quantitative verification to a wide range of ytem, including communication protocol, ecurity protocol and dynamic power management cheme. Tool uch a PRISM alo facilitate invetigation of trend or variation in quantitative propertie, for example, to tudy how change in the failure probability of an individual component affect overall ytem reliability, or to elect the optimal value for a ytem parameter to maximie performance. Building on thee idea, quantitative runtime verification technique have been propoed [4], [5], which dynamically monitor a ytem behaviour and, through expoed control interface, enforce the atifaction of formally pecified contraint on performance or reliability at runtime. A oberved in [4], however, the need to repeat the verification proce for a range of parameter value (in order to configure ytem parameter appropriately) can incur ignificant time and memory overhead. In thi paper, we preent incremental quantitative verification technique, which offer improvement in efficiency by re-uing exiting reult when analying a model to which minor change have been made. We focu on Markov deciion procee (MDP), which generalie (dicrete-time) Markov chain and are well uited to modelling ytem uch a communication protocol. We target cenario in which a model need to be analyed repeatedly and where the probability of certain event occurring i ubject to change. The key idea behind our approach to incremental analyi i to ue a decompoition of the model into it trongly

2 connected component (SCC). Exploiting model tructure in thi way ha already been hown to be effective for an iolated intance of MDP verification [6] but the benefit for reducing work acro multiple verification ha not been conidered. Furthermore, we preent additional optimiation that can be applied when uing an SCC-baed analyi of an MDP, incrementally or otherwie. Firt, we how how to reduce the amount of precomputation performed: thi i an analyi of the underlying graph tructure of the MDP that need to be executed before numerical olution i applied. Secondly, we demontrate how analyi of the decompoed MDP i amenable to paralleliation. We have implemented our technique, uing explicit-tate data tructure, in an extenion of PRISM. Uing a election of large benchmark cae tudie, we demontrate that our incremental verification technique yield impreive peed-up, and that thee are further enhanced by our optimiation. When implementing verification technique in practice, there i a need not jut for improvement in term of peed, but alo memory conumption. In fact, calability i arguably the bigger challenge of the two. For thi reaon, tate-of-theart verification tool uch a PRISM often rely on ymbolic technique, employing data tructure uch a binary deciion diagram (BDD) or multi-terminal BDD (MTBDD). Thee exploit the regularity that i often preent in model to provide compact torage and efficient manipulation. In the latter part of thi paper, we preent a ymbolic implementation of SCC-baed MDP verification. The main difficulty i the crucial tep of identifying SCC in a model. The claic algorithm to do thi, due to Tarjan [7], i not well uited to a ymbolic (BDD-baed) implementation. Symbolic verion have been propoed [8], [9] but are difficult to adapt to thi etting: unlike Tarjan, they do not preerve information about the topological order, and thi information i expenive to obtain afterward with BDD. We preent a cutomied verion of Tarjan algorithm which reolve thi problem. Further experimental reult how that, for large model, thi new approach i fater than the exiting olution engine in PRISM, with only a limited increae in memory uage (linear in the ize of the tate pace). The remainder of thi paper i tructured a follow. Below, we briefly review ome related work. Section II cover background material on quantitative verification for MDP, including the SCC-decompoition technique of [6]. In Section III, we preent our SCC-baed optimiation for graph-baed computation and paralleliation. In Section IV, we decribe technique for SCC-baed incremental verification. Section V ummarie our experimental reult and Section VI dicue our ymbolic implementation of SCCbaed verification. Section VII conclude the paper. Related work. In addition to [6], SCC decompoition wa propoed for quantitative verification in [10], but for dicrete-time Markov chain and with an emphai on counterexample generation. We are not aware of any work on incremental verification for probabilitic model. For nonprobabilitic ytem, incremental technique have been propoed, e.g., [11] [13]; thee focu on peeding up tate pace generation or checking of functional propertie; quantitative propertie or numerical computation are not conidered. II. BACKGROUND We let Dit(S) be the et of all dicrete probability ditribution over S, i.e., the et of function µ : S [0, 1] uch that S µ() = 1. A. Markov Deciion Procee Markov deciion procee (MDP) are widely ued to model ytem that exhibit both probabilitic and nondeterminitic behaviour. Real-life ytem are often inherently tochatic, for example due to the preence of failure, unpredictable delay or randomiation. In addition, nondeterminim may be eential, for example to capture concurrency, i.e. the poible interleaving of multiple component operating in parallel, or underpecification, where a probability or other parameter i not known or i not relevant. Formally, an MDP i a tuple M = (S,, T, r) where: S i a finite et of tate, S i the initial tate, T : S 2 Dit(S) i a probabilitic tranition function, r : S Dit(S) R 0 i a reward function. The tranition probability function T map each tate S to a finite, non-empty et T () of probability ditribution. There are two tep to determine the ucceor of a tate in the MDP: firt, a ditribution µ i choen nondeterminitically from the et T (); econd, the next tate i choen randomly according to µ, i.e. the probability of moving to each tate i given by µ( ). For implicity, we do not include action label in MDP. Ditribution are, however, augmented with reward (ometime called impule reward). A path in an MDP, repreenting a poible execution of the ytem being modelled, i a non-empty (finite or infinite) µ 0 µ 1 equence of the form: where i S, µ i T ( i ) and µ i ( i+1 ) > 0 for all i 0. We ue ω(i) to denote the (i+1)th tate in the path ω, i.e. ω(i) = i, and tep(ω, i) i the ditribution taken in tate ω(i), i.e. tep(ω, i) = µ i. We let P ath denote the et of all (infinite) path tarting in tate. In order to reaon formally about the probabilitic behaviour of an MDP M, we require the notion of adverary (ometime called trategy, policy or cheduler), which i one poible reolution of the nondeterminitic choice in M. Formally, an adverary elect an available ditribution in each tate baed on the hitory of choice made o far. An adverary A retrict the behaviour of the MDP to a et of path Path A Path. It alo induce a probability 2

3 pace [14] Prob A over the path Path A. We ue Adv M to denote the et of all poible adverarie for M. B. Quantitative Verification of MDP Uually, propertie to be verified againt MDP are expreed in temporal logic, uch a PCTL [1], [2] and LTL [15]. Performing verification reduce to the computation of a few key propertie of MDP [2], [16]. The firt are the minimum or maximum reachability probabilitie, i.e. the minimum or maximum probability that a path through the MDP eventually reache a tate in ome target et F S, quantified over all poible adverarie: p min (F ) = inf p A (F ), p max (F ) = up p A (F ) A Adv M A Adv M where: p A (F ) = Prob A ({ω Path A i. ω(i) F }) Secondly, we may require the minimum or maximum expected reward accumulated until target F S i reached: e min (F ) = inf e A (F ), A Adv M where: e A (F ) = ω Path A e max (F ) = up e A (F ) A Adv M r F (ω) dprob A where r F (ω) give, for any path ω Path A, the total reward accumulated along ω until a tate in F i reached: r F (ω) = { nf i=1 r(ω(i 1), tep(ω, i 1)) if j. ω(j) F otherwie. and n F = min{j ω(j) F }. For implicity, in the remainder of thi paper, we will focu on the cae of maximum reachability probabilitie, i.e. computing p max (F ), but our technique adapt eaily to the cae of minimum probabilitie or expected reward (with the exception of Section III-A, which applie only to reachability probabilitie). Throughout the remainder of the paper, we will aume a fixed MDP M = (S,, T, r) and target et F. For clarity, we will abbreviate p max (F ) to jut. We ue p max to denote the vector of probabilitie p max p max for all tate S. Calculation of reachability probabilitie (or expected reward value) proceed in two tep. The firt tep, referred to a precomputation, execute an analyi of the underlying graph of the MDP to identify tate that have reachability probabilitie of 0 or 1. Second, numerical computation i performed to determine value for the remaining tate; thi can be done with a variety of tandard technique, including value iteration and linear programming. We decribe thi proce (for p max ) in more detail below. Precomputation. A graph-baed analyi i ued to partition the tate pace S into et S n, S y and S?, containing tate for which the probability p max i 0, 1 or in (0, 1), repectively. In fact, the analyi i performed uing two eparate algorithm, P rob0a and P rob1e: S n = P rob0a(f ), S y = P rob1e(f ), S? = S \ (S y S n ). Algorithm P rob0a [2] firt compute the et of tate with maximum probability greater than zero of reaching a tate in F. It then return the complement of thi et a S n. For each tate S n, the probability of reaching F i zero under any adverary A. Algorithm 1 P rob0a(f ) 1: R := F ; done := fale 2: while done = fale do 3: R := R { S µ T (). R. µ( )>0} 4: if R = R then done := true end if 5: R := R 6: end while 7: return S\R Algorithm P rob1e [17] ue two neted loop to determine the et S y of tate for which p A (F ) = 1 for ome adverary A. The outer loop identifie tate from which no adverary can make p A (F ) = 1, and remove thoe tate from S. The inner loop collect tate from which one cannot reach a tate in F without paing through a tate already removed from S. Algorithm 2 P rob1e(f ) 1: R := S; done := fale 2: while done = fale do 3: R := F ; done := fale 4: while done = fale do 5: R := R { S µ T (). ( S. µ( ) > 0 R) ( R. µ( ) > 0)} 6: if R = R then done := true end if 7: R := R 8: end while 9: if R = R then done := true end if 10: R := R 11: end while 12: return R Finally, we remark that, although P rob1e(f ) compute all tate for which p max = 1, thi i not trictly neceary. It uffice to ue any et of tate S y that atifie the condition F S y { S p max = 1}. In contrat, for S n, it may be eential (e.g. when uing linear programming) that the et contain all tate with p max = 0. Value iteration. One way to compute the probabilitie p max for the remaining tate S? i to ue value iteration, an iterative numerical method which can approximate the value up to ome deired accuracy. In practice, thi method i widely ued ince it cale well to large MDP. Value iteration work by computing a equence of vector p max,k for increaing k. Initially, i.e. for the cae k = 0, we et p max,0 to 1 if S y and 0 otherwie. Then, the kth 3

4 iteration of computation i defined, for each S, a: 1 S y p max,k := 0 S n max µ( ) p max,k 1 S?. µ T () S The equence of vector p max,k i guaranteed to converge eventually to p max. In practice, though, the computation i terminated when a pre-pecified convergence criterion i met. One common approach i to check that the maximum (abolute) difference between the correponding element of ucceive vector i below ome fixed threhold δ, i.e.: max S p max,k p max,k 1 < δ. Another i to check the maximum relative difference: max S (p max,k p max,k 1 )/p max,k < δ. Linear programming. An alternative to value iteration i to ue linear programming (LP) technique [16] [18]. Thi ha both advantage and diadvantage. One poitive i that LP yield an exact, rather than approximate, olution (ignoring the poibility of numerical error due to floatingpoint arithmetic). Secondly, wherea the running time for value iteration i enitive to the convergence criteria ued, the time required for LP i independent of the deired accuracy. For ome model, value iteration can converge lowly, making LP a fater alternative. On the other hand, LP typically doe not cale a well a value iteration o it uage i retricted to relatively mall model. A for value iteration, we know that p max i 1 for tate in S y and 0 for thoe in S n. For the remaining tate S?, we can compute p max by olving a linear optimiation problem over the et of variable {x S? }, letting x be p max and minimiing S x? under the contraint: x µ( ) x + µ( ) S? S y for all S? and all µ T (). C. SCC-baed Value Iteration We now decribe an optimiation for value iteration, firt preented in [6], baed on a decompoition of the MDP M to be analyed. The firt tep of thi proce i to remove maximal end component (MEC). An end component [17] of M i a pair (S, T ), where S S and T () T (), which i cloed and trongly connected, i.e.: 1) µ T (), S. (µ( ) > 0 S ) 2), S, there i a path in (S, T ) from to. A maximal end component i one for which there i no larger end component that contain it. It i known [17] that all tate within an MEC have the ame probability value. Furthermore, we can afely compre each MEC into a ingle tate [6]. In the ret of the paper, we aume that all MEC have already been compreed in thi way. p max Next, we identify trongly connected component (SCC) in the MDP. An SCC C i a et of tate that i trongly connected (there i a path between any two tate in C) and maximal (no uperet of C i alo trongly connected). SCC are particularly important in value iteration. Let C be an SCC, and P re (C) S\C be the et of tate that can reach C, but are not contained within it. Any change of a tate probability value in C affect probability value of all other tate in C, a well a thoe of tate in P re (C). Furthermore, until the probability value of the tate in C converge, the probability value of tate in P re (C) cannot converge. In fact, the computation of probability value for tate in P re (C) can be potponed until the probability value in C converge [6]. The et of SCC in M form a partition of it tate S. Let Π = {C 1,..., C m } be thi partition. The ucceor et Succ(C i ) of C i i the et of tate outide C i that are immediate ucceor of tate in C i. We ay that C i depend on C j if Succ(C i ) C j. A there i no cyclic dependence among SCC, we generate a revered topological order C among SCC uch that C j will appear before C i in C if C i depend on C j. SCC-baed value iteration procee each SCC eparately, according to the ordering C, and then terminate. For each SCC, a equence of approximation i computed, like for denote the value computed for in the kth iteration and p max,0 the initial value for. For any SCC, we et p max,0 to 1 if S y and 0 otherwie. We alo let p max denote the final value for. Conider now a particular SCC C i. The firt iteration i performed a follow. For each C i : { max µ( ) p pmax,0 <1 T () value iteration. For each tate in an SCC, p max,k p max,1 := µ T () S p max,0 otherwie. where T () = {µ T () Succ(C i ). ( µ( ) > 0 p max > 0) } and p i pmax if Succ(C i ), or p max,0 otherwie. In the remaining iteration, we only update probabilitie for tate that are affected by the previou iteration. Other tate imply keep their probability from the previou iteration. Algorithm 3 decribe the k-th iteration (for k > 1), where p = pmax if Succ(C i ); otherwie p = p max,k 1. The iteration on C i terminate at the k-th iteration when X in Algorithm 3 i empty. Note that Algorithm 3 alo work when we ue δ a a maximum relative difference, e.g., the condition p max,k 1 x p max,k 2 x δ in Algorithm 3 can be replaced by pmax,k 1 x p max,k 2 x δ. p max,k 2 x D. The Tarjan Algorithm for SCC Identification We conclude thi ection by decribing the proce of identifying SCC. A well-known and efficient method for thi i the Tarjan algorithm [7]. It time and pace complexity i linear in the ize of the model. The baic idea i to 4

5 Algorithm 3 The k-th iteration (for tate in C i ) 1: X := {x C i p max,k 1 x p max,k 2 x δ} 2: for all x X do 3: Y := {y C i p max,k 1 y < 1 and µ T (y). µ(x) > 0} 4: for all y Y do 5: T (y) := {µ T (y) µ(x) > 0} 6: p max,k y := max µ T (y) S µ( ) p 7: end for 8: end for execute a depth-firt earch (DFS) on the model, uing a tack to tore tate, which we denote tack. During the earch, each tate i aigned two value:.index for the order in which tate are viited, and.lowlink for the mallet index in the SCC containing the tate. The econd value i changed a more tate in the SCC are dicovered. The root tate i the one in which.index =.lowlink. Algorithm 4 i an improved verion of the algorithm, baed on [19]. In thi algorithm, the root node of each SCC i never puhed into the DFS tack in order to ave time and pace. Note that we convert the probabilitic tranition function T in an MDP into a non-probabilitic tranition relation E, i.e. (, ) E if and only if µ T (). µ( ) > 0. The Tarjan algorithm tart with a call to the recurive function tarjan with the initial tate and initial value 1 for the global variable index. Algorithm 4 tarjan() 1:.index := index;.lowlink := index 2: index := index + 1 3: for all (, ) E do 4: if.index i undefined then 5: tarjan( ) 6:.lowlink := min{.lowlink,.lowlink} 7: ele if tack then 8:.lowlink := min{.lowlink,.lowlink} 9: end if 10: end for 11: if.lowlink =.index then 12: while tack T OP (tack).index.index do 13: P OP (tack) and report 14: end while 15: ele 16: P USH(tack, ) 17: end if III. ACCELERATING SCC-BASED VALUE ITERATION The SCC-baed verion of value iteration, decribed in Section II-C above, ha already been hown to provide a peed-up in the time required for quantitative verification of MDP [6]. We begin by propoing two further improvement to the technique. Later, in Section V, we will illutrate that thee yield further gain in term of peed. A. Eliminating Precomputation The precomputation tep preented in Section II-B, which identifie the et S y and S n, often peed up value iteration, can reduce numerical error and, in the cae of S n, may be required for correctne. It can, however, be time-conuming. Here, we how that precomputation can be eliminated for SCC-baed value iteration, whilt retaining mot of it advantage. Firt, we conider S n, i.e. the identification of tate with p max = 0. Lemma 3.1: A tate in an SCC C ha p max = 0 if and only if p max = 0 for all tate in Succ(C). Proof:. Thi i trivial.. Suppoe that p max = 0 for ome C but p max 0 for ome Succ(C). There exit a tate C uch that µ T ( ). µ( ) > 0. Apparently, ha non-zero maximum probability. By the definition of an SCC, there exit a path 1... n in C uch that 1 = and n =. Working back along the path, we deduce that p max n 1 > 0, p max n 2 > 0,..., p max 1 = p max > 0. According to Lemma 3.1, and the algorithm of Section II-C, P rob0a can in fact be omitted. We imply take S n to be the empty et; tate that have maximum probability 0 will not be conidered beyond the firt iteration. Second, we conider S y, i.e. the identification of tate with p max = 1. The following lemma give u a ufficient check to identify ome tate with p max = 1. Lemma 3.2: Given an SCC C, let uc 0 be the et {x Succ(C) p max x < 1.0}. If either: 1) uc 0 i empty and Succ(C) i not, or 2) uc 0 i non-empty and there doe not exit a tate C uch that µ T (). uc 0. µ( ) > 0 then p max = 1 for all tate in C. Proof: Firt, recall that there are no MEC in the MDP. By removing every ditribution from tate in C uch that it ha a tranition reaching uc 0 with probability greater than zero, we obtain a partition of C where each block form a connected graph and there are no connection between block. In each block B C, each tate only ha tranition either leading to tate in the ame block or in uc 1 = Succ(C)\uc 0. For all tate B, if the maximum probability of reaching uc 1 i le than one, there exit an infinite path ω tarting at and only paing tate in B. Let inft(ω) be the et of tate-ditribution pair that occur infinitely often in ω. Then according to [17, Theorem 3.2, page 46], inft(ω) i an end component, which contradict the premie. By Lemma 3.2, we can alo omit P rob1e, replacing it with a impler check before the firt iteration for each SCC (having firt initialied S y to F ). Thi check i coniderably 5

6 impler than the computation required for P rob1e. Since Lemma 3.2 only give a ufficient condition, we will not identify all tate with p max = 1 but, a mentioned above, thi doe not affect correctne. Furthermore, in our experimental reult, thi approach alway yield the ame reult a P rob1e and, more importantly, run much fater. Finally, although we do not cover the cae of minimum reachability probabilitie in any detail, we briefly tate the following two lemma which, like Lemma 3.1 and 3.2 above, permit removal of the precomputation tep when calculating minimum probabilitie. Lemma 3.3: Given an SCC C and it ucceor et Succ(C), let uc 1 = {x Succ(C) p max x > 0}. If either: 1) uc 1 i empty, or 2) uc 1 i non-empty and there doe not exit a tate C uch that µ T (). uc 1. µ( ) > 0, then all tate in C have minimum probability zero. Lemma 3.4: A tate in an C ha minimum probability one iff all tate in the ucceor et Succ(C) of C, have minimum probability one. B. Parallel Computation SCC-baed value iteration alo preent opportunitie for paralleliation, which i particularly deirable to exploit, given the increaing prevalence of multi-core architecture in maintream CPU deign. The topological order among SCC provide a natural tructure for parallel computation. At any tep, an SCC can be proceed independently (and thu in parallel), a long a all of it ucceor et have been proceed. To achieve thi, we need a queue to tore SCC that are ready to be proceed. Initially, all SCC that have an empty ucceor et are put in the queue. Each computation thread take one SCC from the queue to proce, and when it i done, it put SCC that newly become ready into the queue. The whole proce terminate when the queue i empty. Let Succ(C) be a copy of the ucceor et Succ(C) of an SCC C in Π. Algorithm 5 how the procedure for parallel computation. Note that in the while loop, only line 5 can be executed in parallel. A an additional optimiation, MEC identification can alo be parallelied. Thi i done by by firt partitioning into SCC, then earching each one for MEC in parallel. IV. INCREMENTAL VALUE ITERATION Our main aim in thi paper i to develop incremental verification technique, which accelerate the proce of analying a model that ha undergone minor change, by exploiting the preence of exiting verification reult. Thi i a common cenario in practice, for example, when varying a parameter of a model to invetigate the effect that thi ha on overall model performance. Another ituation when incremental verification i particularly ueful i in the context of uing quantitative verification for online monitoring in a elf-adaptive framework [4]. Algorithm 5 Parallel proceing of SCC 1: Queue := {C i Π Succ(C i ) = } 2: Π := Π\Queue 3: while Queue do 4: cc := the head of Queue 5: compute maximum probabilitie for tate in cc 6: for all C Π. Succ(C) cc do 7: Succ(C) := Succ(C)\cc 8: if Succ(C) = then 9: Π := Π\{C} 10: Queue := Queue {C} 11: end if 12: end for 13: end while In thi paper, we target cae where the probabilitie of ome tranition in an MDP undergo change. We aume, though, that the tranition tructure of the model remain untouched. Thi mean that tranition with probability one or zero cannot be changed; otherwie, ome tranition with non-zero probability would be added or deleted from the model. We ue M = (S,, T, r) to denote the original MDP and M = (S,, T, r) for the modified one. Notice that only T i modified. When ome probabilitie in T are changed, it may be unneceary to recompute probability value for all tate. We firt identify the et Π of SCC that have been affected by the change. It can be generated uing Algorithm 6. Firt, Π i initialied to an empty et. Then, we can the SCC partition according to the revere topological order and add C i to Π if C i atifie one of two condition: 1) There exit a tate C i uch that one ditribution from i involved in the change; 2) There exit an SCC C Π that C i depend on. Algorithm 6 Generate Π 1: Π := 2: for all i 1,..., m do 3: if C i. T () T () or C Π. Succ(C i ) C then 4: Π := Π {C i } 5: end if 6: end for Let p max be the maximum probability for tate computed previouly on M and p max the one we need to compute after the change occur. The SCC-baed value iteration algorithm of Section II-C can be adapted to handle change in probabilitie by replacing Π by Π and initialiing 6

7 p max p max,0 := a follow: 1 S y 0 S n S? and C for ome C Π\Π 0 otherwie p max In addition, before we recompute the probability for an SCC C in Π, we perform a tet on it ucceor et Succ(C). Thi tet check the following condition: 1) for every tate Succ(C), it probability i not affected by the change, i.e.: Succ(C). p max = p max, (1) 2) all ditribution from a tate in C are not affected by the change, i.e.: C. T () = T (). If both condition hold, there i no need to perform recomputation in thi SCC, i.e.: C. p max = p max. Although the above tet can eliminate unneceary recomputation for SCC that might be affected by the change, condition (1) i quite retrictive ince it require all tate in the ucceor et to have the ame probability a before the change occurred. Recomputation i executed even if, for all tate in Succ(C), there are only tiny change, e.g., p max (0, ɛ) for ome mall ɛ > 0. In thi cae, the change in the probability for a tate in C i bounded by ɛ with repect to it original value. If ɛ i le than the required accuracy, we can ue p max a p max for tate in C, which peed up the recomputation by introducing a mall approximation error. Lemma 4.1 formalie thi idea. p max = p max in condi- < ɛ and the tet Lemma 4.1: tion (1) i replaced by p max ucceed, then: 1) If the condition p max x C. p max x p max p max x < ɛ. (2) 2) If condition p max = p max i replaced by pmax ɛ and the above tet ucceed, then: x C. pmax x p max x p max x p max p max < < ɛ. (3) p Proof: Conider the bae cae where 0 1 and 1 p 0 2. Let v 0, v 1 and v 2 be the probability value in tate 0, 1 and 2 repectively. Note that v = p v 1 + (1 p) v 2. Let v 0, v 1 and v 2 be the new probability value for 0, 1 and 2 repectively after ome probabilitie in the model are changed (but p i not changed). Here, we dicard the abolute value in formulae (2) and (3) and conider v 1 v 1 and v 2 v 2 only. Maximum abolute difference. If v 1 v 1 < ɛ and v 2 v 2 < ɛ, we have: v 0 v 0 = ( p v 1 + (1 p) v 2 ) ( ) p v1 + (1 p) v 2 = p (v 1 v 1 ) + (1 p) (v 2 v 2 ) < p ɛ + (1 p) ɛ = ɛ. Maximum relative difference. Aume v 1 v1 v 2 v2 v 2 < ɛ. We rearrange the inequalitie to obtain: v 1 < (1 + ɛ) v 1 and v 2 < (1 + ɛ) v 2. v 1 < ɛ and Then we have: ( ( ) p v 1 + (1 p) v 2) p v1 + (1 p) v 2 v 0 v 0 = p v 1 + (1 p) v 2 = p (v 1 v 1 ) + (1 p) (v 2 v 2 ) p v 1 + (1 p) v 2 < p ((1+ɛ) ) ( ) v 1 v 1 +(1 p) (1+ɛ) v2 v 2 p v 1 +(1 p) v 2 = p ɛ v 1 + (1 p) ɛ v 2 p v 1 + (1 p) v 2 = ɛ (p v 1 + (1 p) v 2 ) p v 1 + (1 p) v 2 = ɛ. Note that if 1 = or 2 =, the above reaoning till hold. For example, let 1 =. We have v = p v+(1 p) v 2, which i implified to v = v 2. If v 2 v 2 < ɛ, then v v < ɛ; if v 2 v2 v 2 < ɛ, then v v v < ɛ. Lemma 4.1 i proved by generaliation of the bae cae. In practice, we can ue δ, the maximum abolute difference or maximum relative difference, a ɛ, or a maller value than δ to increae the accuracy but poibly alo increae computation time. V. EXPERIMENTS We have implemented the technique decribed in the previou ection, uing explicit-tate data tructure, in an extenion of PRISM [3] and invetigated performance on everal cae tudie. The firt i a model of the Zeroconf network configuration protocol for allocating IP addree in a local network [20]. We compute the maximum probability of the protocol correctly configuring an IP addre within time T. The econd i a model [21] of the hared coin protocol ued in the randomied conenu algorithm of Apne & Herlihy [22]. The algorithm allow N procee in a ditributed network to reach a conenu. We compute the maximum probability of terminating without conenu being reached. The third cae tudy i a model of the IEEE Wirele LAN [23], featuring two tation ending data over a hared channel, each with a backoff counter of ize N. We compute the maximum probability of the backoff counter for both tation reaching their maximum value. All model can be found in the PRISM cae tudy 7

8 repoitory [24], along with detail of any model parameter not explained here (e.g. K for Zeroconf, K for Conenu) The experiment were performed on an AMD Phenom(tm) 9600B Quad-Core Proceor with 8GB memory running Fedora 12 x86 64 Linux. Our experimental reult are preented in two part. The firt cover the optimiation to accelerate SCC-baed value iteration from Section III; the econd focue on the incremental quantitative verification technique of Section IV. Accelerating SCC-baed value iteration. Firt, we compare running time, on the variou cae tudie, of our implementation of three approache: 1) the original verion of value iteration, 2) SCC-baed value iteration with precomputation [6], 3) SCC-baed value iteration without precomputation, including both equential and parallel verion (parallel ue 4 thread, one per core). The time are hown in the column Original, SCC pre, and SCC no-pre in Table I. For the firt, we report the time pent on the precompution and value iteration phae in addition to the total time required. For SCC pre, we only give the total running time, a the precomputation time i in general the ame a for Original. For the two SCC no-pre approache, i.e. equential and parallel, we give the time for computing SCC (including identification and compreion of maximal end component), which i required for both approache, and for SCC-baed value iteration. In all cae, our approach (SCC-baed value iteration without precomputation) i much fater than the original one. The time to generate the SCC partition pay off, even though it could take longer than SCC-baed value iteration itelf. In many example, though, the entire SCC-baed approach i fater than even jut the value iteration phae of the original. The precomputation phae account for a large part of the running time in both the original verion and the SCC-baed approach of [6]; thu eliminating it prove to be a good trategy. In general, the gain from our approach i more ignificant for larger tate pace. The parallel verion alo how improvement with repect to the equential one: in the bet cae, the peed up i about 2.5. Although thi i lower than the number of thread, numerical computation uch a value iteration are known to be hard to parallelie o thi remain a very encouraging reult. There are everal factor preventing further peedup for the parallel verion. The major one are: (1) at ome point in the proce, there are fewer independent SCC than thread; (2) the ynchroniation overhead i comparably heavy for SCC that contain only one tate (a, for example, in the Conenu model); In addition, the implementation could be further tuned to alleviate memory contention among thread. Incremental value iteration. To demontrate the incremental verification algorithm without bia, we randomly chooe three tate that are not in any MEC, and have a ditribution with probabilitic choice. For each tate, we pick uch a ditribution µ T () and modify the probability ditribution a follow. Aume there exit m (m > 1) tate 1,..., m S uch that µ( i ) > 0 for 1 i m. The new ditribution µ in a i uch that, for 1 i m 1, we keep half of the value, i.e., µ ( i ) = µ()/2; for i = m, we increae the value uch that µ ( m ) = µ( m ) + m 1 i=1 µ( i)/2. Time for the incremental value iteration algorithm, decribed in Section IV, are reported in the final two column of Table I. Thi include both the equential and parallel verion. We do not conider the time for SCC computation, ince thi doe not need to repeated. Even when ignoring thi, we ee that the time for incremental value iteration repreent ignificant peed-up compared to the nonincremental (SCC-baed) verion: they are alway fater, up to 50 time fater in ome cae. The equential verion work particularly well; for model where a mall number of SCC need to be updated, the gain for the parallel verion are le impreive. VI. SYMBOLIC SCC-BASED VERIFICATION In the previou ection, we have demontrated that the SCC-baed incremental verification can be very fat. A problem with the implementation, however, i that the explicittate data tructure ued to tore the tate pace and tranition relation can limit the ize of model that can be handled. A ucceful approach for alleviating thi in the context of verification i to ue ymbolic implementation, baed on binary deciion diagram (BDD) [25] and extenion uch a multi-terminal BDD (MTBDD). A problem here, though, i that the Tarjan algorithm for identifying SCC i known to be poorly uited to ymbolic implementation. Variou SCC decompoition algorithm have been propoed, pecifically for implementation with BDD [8], [9]. Unfortunately, they do not explore SCC in revere topological order, and it i very low to generate thi order once the SCC are tored a BDD. In thi ection, we adapt the Tarjan algorithm to the cae where model information i tored uing BDD. We omit here low-level detail of how BDD can be ued to repreent and manipulate et of tate and tranition relation (ee e.g. [26]). Here, it uffice to know that ome operation are efficient in thi form and other are not. For example, ome operation in the original Tarjan algorithm cannot be performed efficiently with BDD, notably aociation and update of an integer index to a tate. We propoe a novel hybrid adaption of the algorithm that combine ymbolic and explicit-tate data tructure. Keeping overhead to a minimum for efficiency i non-trivial. We maintain: the non-probabilitic tranition relation E and the union allcc of all viited SCC, tored a BDD; 8

9 Table I PERFORMANCE COMPARISON FOR SCC-BASED TECHNIQUES. Model Para- State Original SCC pre SCC no-pre Incremental meter Total Precomp. Val. iter. Total SCC comp. Sequential Parallel Sequential Parallel time () time () time () time () time () time () time () time () time () 1, , , , , , Zeroconf 2, 14 1,061, (K, T ) 3, , , 14 1,735,014 1,143 1, , , , , 14 2,288,771 1,768 1, , , 4 9, , 8 16, , 12 24, Conenu 2, 16 32, (N, K) 2, 20 40, , 1 72, , 2 1,418, , 3 2,259,817 1, , WLAN 3 96, (N) 4 345, ,295, a tack tack and hah table M, ued during depth-firt earch, whoe ize i linear in the number of tate. Generation of Π for incremental verification with BDD i not a imple tak either. Algorithm 6 i not efficiently implementable with BDD due to the condition C Π. Succ(C i ) C. It require that, in each iteration of the for loop in Algorithm 6, we can the intermediate Π to decide if C i need to be included in Π. For explicit-tate data tructure, thi i preferable becaue it ave memory with very little time cot. However, it i better to generate a (pare) matrix T to tore the relation between SCC. An entry T [i, j] = 1 mean that C i depend on C j. Thu, if C j i included in Π, all C i uch that T [i, j] = 1 are included in Π too. Note that T can alo be encoded ymbolically in order to ave pace. Thi need the following extra variable: A hah table M 2 to tore the root index of each SCC. In the hah table, the root index i the key, a it i unique among all SCC, and the pointer to BDD for the SCC i the value. An MTBDD M 3 to tore the root index (vlowlink) of each tate. A hah table can be ued for the ame purpoe, but would ue more pace than an MTBDD. The adapted Tarjan algorithm, which we call the hybrid Tarjan algorithm, begin with a call to the recurive function hybrid tarjan, hown in Algorithm 7, from the initial tate with index = 1. The line haded grey are ued to compute T. In Algorithm 7, x and y are integer, v, v w are BDD, each of which repreent a ingle tate, and cc i a BDD toring the et of tate in the current SCC. M[v] repreent the correponding value for the hah key v. Here we utilie a feature of mot BDD implementation (including CUDD, which we ue): equivalent BDD are guaranteed to have the ame pointer in memory. Thu, the pointer i ued a the hah key for the BDD v. M[v] = NULL mean that the key v cannot be found in the table and M[v] := NULL denote that the key and it value are deleted from the hah table. In the original Tarjan algorithm, each tate v i aociated with two value for the index of v and the minimum index lowlink among the tate in the SCC containing v. To reduce memory conumption, tate that have already been identified in ome SCC are tored in allcc, and the hah table M 2 only tore the attribute for the current tate and tate in the tack. A indicated in [19], only one attribute i actually needed in an elegant implementation. Indeed, only the value lowlink i tored in the hah table. For the current tate v, it attribute are tored in the local variable vindex and vlowlink; the value vlowlink from it ucceor tate i obtained from the return value of function hybrid tarjan. Theorem 6.1: The hybrid Tarjan algorithm partition a graph E into SCC correctly. Proof: The key idea i to prove that vlowlink i computed correctly with repect to Algorithm 4 when the for loop terminate. Firt of all, notice that M[v] and vlowlink for the current tate v in Algorithm 7 cannot be increaed once they are initialied. For each ucceor tate v, there are three poibilitie: v and v belong to different SCC and v i not explored before v. In thi cae, v i the root of the SCC it belong to. In Algorithm 4, thi i characteried a v.lowlink < v.lowlink, and therefore, v.lowlink := 9

10 Algorithm 7 hybrid tarjan(v) 1: vlowlink := index; M[v] := index 2: vindex := index; index := index + 1 3: ucc := 4: for all (v, v ) E do 5: x := 0 6: if M[v ] = NULL then 7: if v allcc then 8: x := hybrid tarjan(v ) 9: end if 10: ele 11: x := M[v ] 12: end if 13: if x > 0 vlowlink > x then 14: M[v] := x; vlowlink := x 15: ele 16: if x = 0 then ucc := ucc {M 3 [v ]} end if 17: end if 18: end for 19: if vlowlink = vindex then 20: vlowlink := 0; M[v] := NULL; cc := {v} 21: M 3 [v] := vindex 22: while tack M[T OP (tack)] vindex do 23: w := T OP (tack); P OP (tack) 24: for all k uch that T [M[w], k]=1 do 25: T [M[w], k] := 0; T [vindex, k] := 1 26: end for 27: M[w] := NULL; cc := cc {w} 28: M 3 [w] := vindex 29: end while 30: allcc := allcc cc 31: M 2 [vindex] := cc 32: ele 33: P USH(tack, v) 34: end if 35: for all k ucc do T [vindex, k] := 1 end for 36: return vlowlink v.lowlink in line 6. In Algorithm 7, x get value zero in line 8, a vlowlink for v i et to zero in line 20. Hence, vlowlink for v doe not change it value by the if tatement in line v and v belong to different SCC and v i explored before v. In Algorithm 4, v i not in S becaue the SCC it belong to wa deleted from S by the if tatement in line Thu, the value of v.lowlink i not changed, a v.lowlink i defined and v S. In Algorithm 7, thi i characteried a M[v ] = NULL v allcc, and therefore, vlowlink for v keep it value. v and v belong to the ame SCC, which indicate that vlowlink := 0 in line 20 of Algorithm 7 i not triggered (when proceing v ). Therefore, Algorithm 7 and 4 behave in the ame way: the condition v.index i undefined in Algorithm 4 i equivalent to M[v ] = NULL v allcc in Algorithm 7, and v S i equivalent to M[v ] NULL. Moreover, we have x > 0 under both condition, a vlowlink i not reet to zero, and therefore the if tatement in line mimic the function min in Algorithm 4. The SCC computed by the hybrid Tarjan algorithm are tored ymbolically, which make it impoible to adapt Algorithm 3 to compute each iteration efficiently. Thi i becaue it require acce to individual element of the SCC, which i inefficient for BDD-baed data tructure. Our approach i to generate a correponding explicit-tate data tructure: a pare matrix. Thi can be done relatively efficiently and i then amenable either to value iteration, or in fact olution via linear programming. Becaue, thee matrice are typically mall, here we chooe to olve an LP problem (in our implementation, we ue the ECLiPSe Contraint Logic Programming ytem with the COIN-OR CBC/CLP olver for thi). We alo employ an additional optimiation: we treat trivial SCC, containing a ingle tate without elf-loop, a a pecial cae. Probabilitie for thee can be computed quickly and eaily uing value iteration on the ymbolic data tructure. It i alo intereting to note that, to peed up SCC decompoition uing BDD, it i preferable to perform precomputation before applying the hybrid Tarjan algorithm; thi i the oppoite ituation to the explicit-tate data tructure cae. Thi i becaue precomputation i more efficient when uing BDD and reduce the number of tate that need to be explored by the hybrid Tarjan algorithm. A. Experimental Reult Table II how experimental reult for two of the three previou cae tudie: WLAN and Conenu 1. We compare the time for computing maximum reachability probabilitie uing the tandard verification engine in PRISM ( Spare, Hybrid, and MTBDD ), a well a for SCC-baed verification uing the hybrid Tarjan algorithm ( Hybrid Tarjan in the table). For the latter, we alo report the time pent generating SCC by the hybrid Tarjan algorithm, which i included in the total time. Model contruction time i the ame for all cae and therefore not reported in the table. Since the performance of SCC decompoition by the hybrid Tarjan algorithm i of independent interet, we compare it with two BDD-baed algorithm for SCC decompoition: SCC-Find [9] and Locktep [8]. The decompoition time (without conidering revere topological order) uing thee algorithm are given in column BDD SCC comp.. 1 We omit the Zeroconf model from thee experiment becaue ome SCC are too large to olve uing LP. 10

11 Table II PERFORMANCE COMPARISON FOR THE HYBRID TARJAN ALGORITHM. Model Parameter State Spare Hybrid MTBDD Hybrid Tarjan BDD SCC comp. Total Total Total SCC comp. Total SCC-Find Locktep time () time () time () time () time () time () time () 2 28, WLAN 3 96, (N) 4 345, ,295, ,007, , 4 9, , 8 16, , 12 24, , Conenu 2, 16 32, , (N, K) 2, 20 40, , , 1 729, , 2 1,418, , , 3 2,259, , , , 4 3,253, , , Table III PERFORMANCE COMPARISON FOR HYBRID TARJAN WITH REWARDS. N, K State Spare MTBDD Hybrid Tarjan Total Total SCC comp. Total time () time () time () time () 3, 2 1, , 4 2, , 8 5, , , 12 7, , , 16 9, , , 20 11, , 2 7, , 4 12, , 8 22, , 12 32, , 16 41, , 20 51, , 2 30, , 4 50, , 8 91, , , , , , , In addition to reachability probabilitie, we extend the hybrid Tarjan algorithm and SCC-baed LP computation to compute maximum expected reward propertie. A mentioned earlier, thi i a traightforward adaption. Experimental reult for the Conenu tudy and the property the maximum expected tep for the firt K round are hown in Table III. Note that: (1) the model i lightly different from the one ued for reachability probabilitie o the number of tate differ; (2) PRISM Hybrid engine doe not upport thi cla of propertie o i not compared to. We alo conider an additional cae tudy to illutrate computation of expected reward propertie: a model of the IEEE 1394 FireWire Root Contention Protocol [27]. Thi i a leader election algorithm for a multimedia bu; the property we check i the maximum expected time to elect a leader. The model ha two parameter, delay and fat. Figure 1 how the total running time for the hybrid Tarjan algorithm and PRISM pare engine, with three different value of fat and varying value of delay. The experimental reult for computation of both maximum reachability probabilitie and maximum expected reward clearly demontrate the advantage of the new approach, which outperform the other PRISM engine in mot cae. In Figure 1, we oberve trade-off between the contrating technique for different model parameter value. More preciely, the SCC-baed approach become more beneficial when value iteration i low to converge in the full model. Crucially, we are alo able to handle larger model than for the explicit-tate implementation preented earlier in the paper. Latly, we note that the hybrid Tarjan algorithm perform well irrepective of whether we are applying incremental verification: thi mean it i of independent benefit, for general quantitative verification purpoe. Finally, we note that the hybrid Tarjan SCC decompoition outperform the exiting BDD-baed algorithm implemented in PRISM. VII. CONCLUSION We have preented technique for optimiing quantitative verification of MDP, baed on a decompoition into trongly connected component. Thi i hown to reduce the amount of graph-baed computation required and to provide opportunitie for paralleliation. In particular, we alo focued on the applicability of thi to incremental verification: re-analying an MDP after mall change in it probability value, by re-uing exiting verification reult. In the future, we plan to develop thee technique further, for example, conidering alo the cae where the tructure of the model change. 11