Is There a Best Büchi Automaton for Explicit Model Checking?

Is There Best Büchi Automton for Explicit Model Checking? Frntišek Blhoudek Msryk University Brno, Czech Republic xblhoud@fi.muni.cz Alexndre Duret-Lutz LRDE, EPITA Le Kremlin-Bicêtre, Frnce dl@lrde.epit.fr Jn Strejček Msryk University Brno, Czech Republic strejcek@fi.muni.cz Mojmír Křetínský Msryk University Brno, Czech Republic kretinsky@fi.muni.cz ABSTRACT LTL to Büchi utomt (BA) trnsltors re trditionlly optimized to produce utomt with smll number of sttes or smll number of non-deterministic sttes. In this pper, we serch for properties of Büchi utomt tht relly influence the performnce of explicit model checkers. We do tht by mnul nlysis of severl utomt nd by experiments with common LTL-to-BA trnsltors nd relistic verifiction tsks. As result of these experiences, we gin better insight into the chrcteristics of utomt tht work well with Spin. Ctegories nd Subject Descriptors F.4.1 [Mthemticl Logic nd Forml Lnguges]: Mthemticl Logic temporl logic; D.2.4 [Softwre Engineering]: Softwre/Progrm Verifiction forml methods, model checking Generl Terms Theory, Algorithms, Verifiction Keywords Liner temporl logic, Büchi utomt, explicit model checking 1. INTRODUCTION The utomt-theoretic pproch to explicit model checking of Liner-time Temporl Logic (LTL) [25] cn be broken down into four steps: (1) build the stte spce, i.e., n utomton S representing ll the possible executions of the system to be verified, (2) trnslte n LTL formul ϕ representing desired property of the system into Büchi SPIN 14, July 21 23, 2014, Sn Jose, CA, USA This is the uthor s version of the work. It is posted here for your personl use. The definitive version ws published by ACM, http://dx.doi.org/10.1145/2632362.2632377.. Automton (BA) A ϕ tht ccepts ll words violting ϕ, (3) build the synchronous product S A ϕ of these two systems, nd finlly (4) check this product for emptiness. If S A ϕ ccepts word, it is n execution of S tht invlidtes ϕ, i.e., counterexmple. In typicl explicit model checker, the construction of the stte spce S nd its synchronous product with A ϕ re done one-the-fly, driven by needs of n emptiness check procedure. This ensures tht only the prt of the stte spce tht is comptible with A ϕ will be constructed. Further, the whole construction cn be stopped s soon s the emptiness check finds counterexmple, i.e., rechble cycle contining n ccepting stte. Here we focus on the influence of property utomton A ϕ on the steps (3) nd (4) of model checking procedure. There re mny lgorithms nd tools for trnslting n LTL formul into Büchi utomton, yet they produce vrious lnguge equivlent utomt. For instnce, Figure 4 shows severl Büchi utomt for the LTL formul GF GFb. Should one be preferred over the others? The intuition tht smller A ϕ produces smller synchronous product S A ϕ is not lwys correct. More importntly, it is not quite relevnt: ultimtely, only the prt of the product tht is explored by the emptiness check does mtter. Some uthors of utomt optimiztions or LTL-to- BA trnsltion improvements (e.g., Etessmi nd Holzmnn [10] nd Dx et l. [5]) provide lso running times of selected emptiness check executed on the product of obtined utomt nd either rndom stte spces or few relistic systems. Etessmi nd Holzmnn [10] even complined tht the reltion between the size of A ϕ nd the running time of the model checking procedure ws difficult to predict, especilly in the presence of counterexmple. In order to select n idel utomton for expressing formul, one should be wre of the inner workings of the emptiness check procedure tht will be used. Among the vrious existing emptiness checks, we hve decided to focus on the stndrd emptiness check of Spin, which is sequentil lgorithm bsed on Nested Depth-First Serch (NDFS) [17]. We look t concrete exmples of how formule re trnslted differently by existing tools to gin better insight into the chrcteristics of utomt tht work well with Spin. Our results should stimulte LTL-to-BA trnsltion

reserchers to focus on nother spects of produced utomt: not only their size nd determinism. The pper is orgnized s follows. The next section motivtes our reserch by experimentl results quntifying the influence of property utomt on the performnce of the explicit model checker Spin. Section 3 describes stndrd pproches to utomt optimiztion motivted by reduction of the product size. In Section 4, we discuss how property utomt cn ffect the performnce of the NDFS-bsed emptiness check of Spin. We ssume fmilirity with LTL nd Büchi utomt [3]. 2. MOTIVATION BY EMPIRICAL DATA First of ll, we present experimentl results showing how importnt the impct of Büchi utomt on Spin s performnce cn be. We use the following benchmrk, softwre, nd hrdwre. Benchmrk. The considered benchmrk set is bsed on the set of relistic model checking tsks BEEM [19]. In ddition to the originl 769 pirs of model in Promel nd corresponding specifiction formul we dded, to ech model describing some mutul exclusion lgorithm (ltogether 23 instnces of prmetric models clled nderson, peterson, nd bkery), three specifiction formule: 1. GF(P 0@CS) GF(P 0@NCS) mening tht if process P 0 spends infinitely mny steps in criticl section, then it lso spends infinitely mny steps in noncriticl section, 2. GF(P 0@NCS) GF(P 0@CS) mening tht if process P 0 spends infinitely mny steps in non-criticl section, then it lso spends infinitely mny steps in criticl section, 3. FG ( (P 0@CS P 1@CS) (P 0@CS P 2@CS) (P 1@CS P 2@CS) ) mening tht fter finitely mny steps, it never hppens tht two of the processes P 0, P 1, nd P 2 re in criticl section t the sme time. To sum up, we consider 769 + 3 23 = 838 verifiction tsks. All the benchmrks nd mesurements presented in this section re vilble t http://fi.muni.cz/~xstrejc/ publictions/spin2014.tr.gz. Softwre. We use five LTL-to-BA trnsltors presented in Tble 1: Spin nd LTL2BA re well estblished nd populr trnsltors, MoDeLL ws the first trnsltor focusing on determinism of produced utomt, nd LTL3BA nd Spot represent contemporry trnsltors. The lst two trnsltors re used in severl settings: the settings denoted by LTL3BA (det) nd Spot (det) im to produce more deterministic utomt, while the setting clled Spot (no jump) is explined in Section 4. The sme version of Spin (with its defult settings nd the mximl serch depth set to 100 000 000) is lso used in ll our experiments to perform ll model checking steps except the LTL-to-BA trnsltion. In prticulr, the prtil-order reduction, which severely limits the explortion of the stte-spce, is enbled. Hrdwre. All computtions re performed on n HP DL980 G7 server with 8 eight-core 64-bit processors Intel Xeon X7560 2.26GHz nd 448 GiB DDR3 RAM. Ech execution of Spin hs been restricted by 30 minutes timeout nd memory limit of 20GiB. Tble 1: Considered LTL-to-BA trnsltors, for reference. tool version commnd Spin [10, 16] 6.2.5 spin -f LTL2BA [12] 1.1 ltl2b -f MoDeLL [21] 1.5.9 mod2spin -f LTL3BA [1] 1.0.2 ltl3b -S -f LTL3BA (det) ltl3b -S -M -f Spot [7] 1.2.4 ltl2tgb -s Spot (det) ltl2tgb -s -D Spot (no jump) ltl2tgb -s -x degen-lskip=0 Originlly, we hve mesured the impct of Büchi utomt on Spin by its running time. Unfortuntely, our computtion server is shred with other users nd its vrible worklod hs led to enormous dispersion of mesured running times. We hve observed running time difference of over 300% on the sme input. Hence, insted on running times, we focus on the count of visited trnsitions, which is stble sttistic produced directly by Spin. The number of visited trnsitions ccumultes the numbers of product trnsitions explored in depth-first serches executed during run of the NDFS lgorithm (see Section 4 for brief description of NDFS). Hence, the number of visited trnsitions should be proportionl to the running time on dedicted mchine. For ech of the 838 considered verifiction tsks, we trnslte the negtion of the formul by ll the mentioned trnsltors nd we run Spin on the model with ech of the obtined utomt. Trnsltion of the negted formul to n utomton is instntneous (it tkes less thn 0.1s) in nerly ll cses: there is only one formul for which the trnsltor built in Spin needs couple of seconds to finish. For 823 tsks, Spin successfully finishes the computtion within the given limits for t lest two utomt obtined by different trnsltion tools. For ech such verifiction tsk, we find the mximl nd the miniml numbers of visited trnsitions nd we compute their rtio. Intuitively, the rtio represents how mny times slower Spin cn be if we choose the worst of the produced utomt compred to the best of those. Out of the 823 tsks, the rtio is exctly 1 only in 35 cses. In other words, in more thn 95% of the considered verifiction tsks, the choice of n LTL-to-BA trnsltor hs n influence on running time of Spin. In fct, the rtios significntly differ for verifiction tsks where the model stisfies given formul nd for those with counterexmple. Out of the 823 tsks, 731 tsks contin counterexmples while 92 tsks do not. The rtios for these two sets re presented by box-plots in Figure 1. One cn clerly see tht the selection of Büchi utomton hs bigger impct on the verifiction tsks with counterexmples (medin rtio is over 5.6) thn on the tsks without counterexmples (medin rtio i.4). Both sets contin extreme cses where the rtios exceed 10 6. Spin lso provides sttistics for stored sttes, which is the totl count of constructed nd stored product sttes nd should be proportionl to the memory consumed by Spin. If we compute rtios of mximl nd miniml numbers of stored sttes, we get the rtio 1 in 68 out of the 823 tsks.

10 8 731 tsks with counterexmple 92 tsks without counterexmple Rtio mx / min 10 6 10 4 10 2 10 0 5.67 4.9 1.4 1.37 (A 1) (A 2) (S) Figure 2: Two BA for GF nd stte spce. A 1 S hs 3 sttes wheres A 2 S hs 6. Note tht edges in the utomt re lbelled by Boolen formule over tomic propositions, where mens, stnds for true, nd used lter mens b. Formlly, n edge lbelled with formul ρ represents ll the trnsitions tht re lbelled with subset M of tomic propositions such tht M = ρ. trnsitions sttes trnsitions sttes Figure 1: Impct of the Büchi utomt on model checking. For ech verifiction tsk, we compute rtios between the mximum nd minimum number of trnsitions (or unique sttes) visited by Spin using ll vilble Büchi utomt. In ech column, box spns between the first nd third qurtiles, nd is split by the medin (whose vlue is given). The whiskers show the rnge of rtios below the first nd bove the third qurtile tht re not further wy from the qurtiles thn 1.5 times the interqurtile rnge. Other vlues re shown s outliers using circles. On Figure 1 one cn see tht the sitution is nlogous to rtios of visited trnsitions, but the rtios of stored sttes re slightly lower. To sum up, the choice of Büchi utomton is n importnt issue substntilly ffecting both running time nd memory needed for the explicit model checking process implemented in Spin. 3. STANDARD APPROACH TO OPTIMI- ZATION: HELPING THE PRODUCT Most of the work on optimizing the trnsltion of LTL formule to Büchi utomt hs focused on building Büchi utomt with the smllest possible number of sttes [e.g. 4, 12, 22, 15, 24]. This is motivted by the observtion tht the synchronous product of Büchi utomton A with stte spce S cn hve the sme number of sttes s their Crtesin product in the worst cse: S A S A. Therefore, decresing A lowers the upper bound on S A. However it is possible to build contrived exmples where smller A yield lrger product. For instnce, removing one stte in the utomton A 1 of Figure 2 doubles the size of its product with the stte spce S of the sme figure from 3 to 6 sttes. Of course, if S ws similr cycle of 2 sttes, the smller utomton A 2 would give smller product. Hence, one cnnot hope to build n optiml property utomton A without priori knowledge of the system S. With the introduction of LBTT [23], tool tht checks the output of different LTL-to-BA trnsltors by doing mny cross-comprisons, including some products with rndom stte spces, tool designers strted to evlute not only the size of the produced utomt, but lso the size of their products with rndom stte spces [e.g. 21, 8]. A recent clone of LBTT clled ltlcross [6] computes multiple products with rndom stte spces to lessen the luck fctor. Sebstini nd Tonett [21] used this product with rndom stte spce mesurement to benchmrk their trnsltor MoDeLL ginst other vilble trnsltors to support the clim tht producing more deterministic Büchi utomt might be more importnt thn producing smll Büchi utomt. Benchmrks bsed on the size of products my look like Tble 2. The tble shows tht MoDeLL genertes utomt tht re slightly bigger thn LTL2BA (its competitor in 2003) but when looking t the product, MoDeLL cuses fewer trnsitions to be built. If the number of trnsitions is proportionl to the running time of model checker nd the number of sttes is proportionl to its memory consumption, MoDeLL hs effectively trded memory for speed. MoDeLL s results do not pper to hold tody: more recent trnsltors such s LTL3BA or the trnsltor of Spot cn produce utomt tht re significntly smller nd yield smller products with rndom stte spces. These trnsltors lso hve options to produce more deterministic utomt, but the resulting products re not lwys better. The right prt of Tble 2 compres the trnsltors by the sizes of products of produced utomt with fixed set of rndom systems. For instnce, one cn observe tht even though Spot (6) produces the lowest ccumulted number of product trnsitions in this benchmrk, there re 30 formule where the generted products hve more trnsitions thn those obtined by LTL3BA (det) (5). Conversely, utomt from LTL3BA (det) produce products with more trnsitions thn those of Spot for 76 formule. It should be noted tht optimizing A to minimize S A is not equivlent to optimizing A for the model checking procedure, becuse the product S A is constructed onthe-fly by most emptiness check lgorithms. An emptiness check my explore prt of the product, nd my explore it severl times. Ultimtely, ny chnge to A should relly be mesured only by its effect on the model checker used. Such n evlution ws done for instnce by Dx et l. [5]: in ddition to explining how to build miniml wek deterministic Büchi utomt (WDBA) for subclss of LTL,

Tble 2: Trnsltion of 178 formule from the literture [9, 22, 11] using different LTL-to-BA trnsltors, with timeout of 60 seconds. Column n indictes how mny trnsltions re successful within the llocted time. The utomt columns show ccumulted vlues of stndrd utomt chrcteristics for ll successful trnsltions. Column ndst gives the number of non-deterministic sttes in the utomt. All produced utomt re synchronized with the sme 100 rndom systems, nd the medin number of sttes nd trnsitions of these products is kept. The products columns represent the medins ccumulted over ll successful trnsltions. The right-most prt of the tble counts the number of formule for which the trnsltor on the row produces n utomton with higher medin number of trnsitions in the products tht the trnsltor of the column. utomt products cses with product trns bigger thn... n sttes ndst edges trns sttes trns (1) (2) (3) (4) (5) (6) (7) (8) (1) Spin 161 1739 1474 9318 46252 260934 8892105 0 102 143 107 150 150 150 146 (2) LTL2BA 178 1003 802 3360 30159 191668 5556159 5 0 137 49 161 157 156 142 (3) MoDeLL 178 1297 647 4311 23874 216938 4193567 15 33 0 41 110 116 114 91 (4) LTL3BA 178 795 595 2209 21240 151373 4273646 0 23 126 0 149 153 152 140 (5) LTL3BA (det) 178 830 326 2405 14414 155716 2901474 0 0 10 5 0 76 75 63 (6) Spot 178 657 94 1615 10304 127792 2326271 1 6 15 5 30 0 1 1 (7) Spot (det) 178 662 88 1639 10414 128178 2328422 1 7 17 6 33 4 0 0 (8) Spot (no jump) 178 785 104 1874 12273 152592 2719360 12 28 40 27 70 61 57 0 they showed tht their miniml WDBA re smller thn the non-deterministic BA produced by other trnsltors. They lso show tht they improved the running times of Spin on few verifiction tsks. 1 We study how Spin s emptiness check cn be helped by chnging A in the next section. Improving the size of the product is one wy to improve the performnce of Spin (s the exmple of Section 4.5 illustrtes), but there re lso other spects. For exmple, the loction of ccepting sttes hve n influence too. 4. ANOTHER VIEW TO OPTIMIZATION: HELPING THE EMPTINESS CHECK 4.1 Emptiness Checks with Nested DFS To check the emptiness of S A ϕ, one should serch for cycle tht is rechble from the initil stte nd tht contins t lest one ccepting stte. The emptiness check procedure used in Spin by defult is bsed on two nested depthfirst serches [17]: the min DFS, which we shll cll blue, explores the product (on-the-fly) nd every time it would bcktrck from n ccepting stte s (i.e., ll successors of s hve been explored by the blue DFS) it strts second, red DFS from s. If the red DFS reches ny stte on the blue DFS serch stck then rechble nd ccepting cycle is found (since s is rechble from ll sttes on the blue DFS serch stck) nd the lgorithm reports it s counterexmple. Otherwise, the red DFS termintes nd the blue DFS cn continue. The two DFS lwys ignore sttes tht hve been completely explored by n instnce of the red DFS, so stte is never visited more thn twice. As n extr optimiztion, if the blue DFS hits its own serch stck by following trnsition tht is either going to or coming from n ccepting stte [13, 20], then n ccepting 1 We omitted their tool from our benchmrk becuse (1) it only supports subset of LTL, nd (2) their optimiztion is implemented in Spot nd both tools would therefore return the sme utomt. Besides, the subset of LTL does not include the formule studied in Sections 4.3 nd 4.5. (B 1) (B 2) Figure 3: Automt for G( X( X( X))). B 1 is inherently wek, B 2 is wek. cycle cn be reported without even strting ny red DFS. This cn be effectively pplied only on products with n ccepting cycle. When counterexmple exists in the product, the emptiness check my report it more or less rpidly depending on the order in which it hs explored the trnsitions of the product. With ny luck, the first trnsition selected t ech step of the DFS will led to n ccepting cycle. Conversely, the first trnsitions followed might led to huge component of the product tht just turns out to be ded-end, nd from which the emptiness check hs to bcktrck before finding the counterexmple. As the selected trnsition order in S A ϕ depends on the order of the trnsitions in the property utomton A ϕ, this explins some of the huge differences noticed in Figure 1. Note tht previous ttempts to explore reordering of the trnsitions of A to help the emptiness check hve been inconclusive [14], so we did not pursue this direction. (Furthermore the swrming techniques [18] used nowdys mkes this topic even less ttrctive: in these pproches severl threds compete to find counterexmple in S A ϕ using different, rndom trnsition order for A ϕ.) 4.2 Wek Automt The optimiztion we just described, where the blue DFS cn detect n ccepting cycle without running red DFS if it hits its own stck on (or from) n ccepting stte, suggests tht of the two utomt of Figure 3, B 2 should be preferred. Indeed when the blue DFS reches stte of its serch stck in the product S B 2, it is gurnteed to come from (nd go to) n ccepting stte, detecting the ccepting cycle without

strting red DFS. In the product S B 1 we might be less lucky if we close the cycle with the trnsition t the bottom of B 1: in tht cse the product hs to be explored second time by the red DFS. This exmple ctully illustrtes the distinction between wek utomt nd inherently wek utomt. An inherently wek utomton is n utomton in which strongly connected components (SCCs) cnnot mix ccepting cycles with non-ccepting cycles. A wek utomton is n inherently wek utomton in which the sttes of ech SCC re either ll ccepting or ll non-ccepting. Any inherently wek utomton cn evidently be trnsformed into n equivlent wek utomton [2]. Hving more ccepting sttes is not necessrily good from the point of view of the NDFS since red DFS is strted every time the blue DFS bcktrcks from n ccepting stte. However if n entire SCC is non-ccepting, the first red DFS will cover it fully, nd ech successive red DFS will immeditely return becuse it ttempts to process stte tht hs lredy been seen by previous red DFS. 4.3 Automt for GF GFb Figure 4 shows six different Büchi utomt for the formul GF GFb produced by the considered tools. Note tht if you ignore the exchnge of nd b (which hve symmetric purpose in the originl formul), utomt C 4 nd C 5 differ only in the initil stte nd thus cnnot be distinguished by ny determinism-bsed or size-bsed metrics. Tble 3 cptures dt bout Spin s runs on model of the bkery mutul exclusion protocol tken from BEEM nd the property utomt of Figure 4. The propositions nd b describe situtions tht (different) pirs of processes re in the criticl section t the sme time. The protocol prevents such sitution so neither nor b is ever true in the model. We observe tht in cse of products with utomt C 5 nd C 6 (both produced by Spot), Spin explores ech product twice becuse it triggers the red DFS from the initil stte of the product. This is not the cse for the other utomt. This yields the following hypothesis: When we suppose tht there is no ccepting cycle in the product, the utomton should keep its ccepting sttes s fr s possible from the initil stte. The further they re, the more chnce we hve tht the product will never rech the stte, nd therefore no red DFS will be triggered. For instnce, if we ignore the renming of tomic propositions, the utomton C 3 could be obtined from C 6 by unrolling the ccepting cycle by one step, so tht the cycle is entered on non-ccepting stte, nd the ccepting stte is ctully the lst one visited on the cycle. 2 This superfluous initil stte only mkes negligible difference on the product, nd does not incur ny noticeble difference for Spin compred to C 1, C 2, or C 4. Similrly, if we do not expect n ccepting cycle in the product, the inherently wek utomton B 1 of Figure 3 could be chnged by letting the right-most stte be ccepting insted of the middle one. 2 This is not ctully the reson why MoDeLL produces C 3. Internlly, MoDeLL trnsltes the formul into Büchi utomton with lbels on sttes nd hs to del with possibly multiple initil sttes. When it outputs n utomton, it lwys dds n extr initil stte with copies of the outgoing trnsitions of ll the originl initil sttes, even if the originl utomton hd only one initil stte. See lso D 3 of Figure 6 where nd were the originl initil sttes. 4.4 Trnsltion Differences Most LTL-to-BA trnsltors follow multi-steps procedure where they first trnslte given LTL formul into generlized Büchi utomton, often with trnsition-bsed cceptnce (TGBA), such s those of Figure 5. Trnsltors then degenerlize these utomt to obtin BA. Other simplifiction procedures my be pplied to these utomt, but it turns out tht the lst three utomt of Figure 4 were ll obtined by degenerlizing G 1 in Figure 5, nd their differences re due to choices mde in the degenerliztion procedure. When degenerlizing TGBA G with m cceptnce sets F 1,..., F m (the nd on the Figure 5), the structure of G is cloned m + 1 times. Let us cll ech of these clones level. For ech stte of level i m, ll trnsitions tht were originlly in F i hve their destintion redirected to the next level, the destintion of ll trnsitions in level m + 1 re redirected to level 1. Finlly, ll the sttes of the level m + 1 re mde ccepting. The initil stte cn be put on ny level. This procedure ensures tht words ccepted by the degenerlized utomton correspond to words recognized by runs of G tht visit ll cceptnce sets infinitely often. Accepting cycles in products involving these degenerlized utomt will lwys involve t lest m + 1 sttes. The degenerliztion pplied to G 1 with the initil stte on the lst level nd the cceptnce sets ordered s, then, produces the utomton C 6 of Figure 4. Recll tht the edge lbelled with corresponds to the four edges lbelled by, b,, nd b in the originl utomton G 1. An optimiztion introduced by Gstin nd Oddoux [12] consists in jumping levels. If trnsition of level i m belongs to F i... F j, its destintion cn be redirected directly to the level j + 1. Similrly, if trnsition from the level m+1 is in F 1... F j, it cn be redirected to the level j + 1. Implementing this optimiztion gives utomton C 5. Chnging the degenerliztion order to, then, nd putting the initil sttes on the first level would give utomton C 4. Often (but not in this exmple), jumping levels is wy to effectively void creting useless copies of some sttes. Another side effect of this optimiztion is tht some ccepting cycles my be shorter thn m + 1: the chnge effectively keeps the utomton s close to the ccepting level s possible. If we re looking for counterexmples, C 5 pper better thn C 6 becuse its ccepting cycles re shorter on the verge. We recll tht the initil stte of degenerlized utomton cn be put on ny level. For exmple, Ginnkopoulou nd Lerd [15] noticed tht by chnging the initil level, they could sometimes sve some sttes, so they try to use both the first nd the lst level nd keep the smllest utomton. In our exmple, C 4 nd C 5 differ only by the choice of the initil level (nd degenerliztion order but this is negligible s nd b re symmetric in our problem), there is no size difference, nd yet it mkes huge difference in the running time of Spin, s discussed in the previous section. Another trnsltion difference evidently comes from the difference between the generlized utomt obtined from the LTL formul. In our cse C 4, C 5, nd C 6 were obtined from G 1 while C 1 nd C 2 were obtined from G 2. (The difference with Spin (C 1) is tht it does no level jumping from the ccepting stte.) The difference between G 1 nd G 2 is cused

b b b b b s i b b b b b b b b b (C 1) Spin (C 2) LTL2BA & LTL3BA (C 3) MoDeLL (C 4) LTL3BA (det) (C 5) Spot & Spot (det) (C 6) Spot (no jump) Figure 4: Automt for GF GFb generted by different tools nd options. Tble 3: Sttistics bout generted utomt nd Spin s run on model bkery.7.pm nd formul GF GFb where neither nor b ever occurs in the model. The corresponding utomt re shown in Fig. 4. utomton size sttistics from Spin s execution sttes ndst edges trns stored sttes visited trns time C 1 Spin 3 2 6 17 27531713 95071k 88s C 2 LTL2BA & LTL3BA 3 3 8 20 27531713 95071k 99s C 3 MoDeLL 4 0 6 16 27531714 95071k 109s C 4 LTL3BA (det) 3 0 8 12 27531713 95071k 101s C 5 Spot & Spot (det) 3 0 8 12 27531714 190143k 211s C 6 Spot (no jump) 3 0 5 12 27531714 190143k 191s b (G 1) b b (G 2) b Figure 5: Two TGBA for GF GFb. Accepting runs must visit nd infinitely often. by choices mde during the trnsltion to fvor deterministic sttes in the cse of G 1. In our exmple of Tble 3, this improved determinism mkes no difference since nd b re never true in the model. 4.5 Automt for (GF GFb) We now focus on nother concrete cse: (GF GFb) on mutex protocols. The formul without negtion describes tht if some process visits infinitely often the criticl section, it infinitely often leves it this property holds in model peterson.4.pm nd therefore Spin hs to build the whole product to find tht it contins no ccepting cycle. Tble 4 shows series of experiments of verifiction of the model peterson.4.pm ginst this formul, using different tools to obtin Büchi utomton. In this cse, ech tool produces different utomton, s shown in the first prt of Figure 6. Note gin tht utomt D 2 nd D 4 cnnot be distinguished only by determinism nd size metrics (see Tble 4). They differ only in the trget of the outgoing edge of, yet we observe significnt difference in Spin s behviors. We ctully use 12 different utomt for this formul. The first seven of the tble re generted by the considered tools. The other re hndwritten by modifying the previous utomt to explore which spects of the utomt mke significnt difference in Spin s behvior s described further. D 8 is dpted from D 6 by chnging the degenerliztion level on which we enter the SCC. D 9 keeps the strong initil gurd of D 6 but then uses the ccepting SCC of D 2. D 10 is mix of D 6 nd D 2 to observe the influence of the gurds compred to. D 11 is version of D 2 in which the SCC is mde deterministic s in D 6. Finlly, D 12 fixes D 5 by removing the spurious s i. Bsed on Tble 4 we cn group these utomt in three ctegories, listed from the best to the worst with respect to Spin s performnce. Before we discuss these ctegories, it is importnt to notice tht in model where mens the process is in the criticl section nd b mens the process leves the criticl section, we cn expect most of the stte spce to be lbelled by. D 6, D 7, D 8, D 9 Automt with the smllest number of trnsitions. Note tht the no jump version (D 7) nd the one with non-deterministic SCC (D 9) both yields few more sttes nd trnsitions in the product, but the difference is not significnt. The key property of these utomt is tht they cn leve stte only by reding, wheres other utomt re more permissive. D 1, D 2, D 3, D 10, D 11 All these utomt exhibit more nondeterminism on stte nd will enter the ccepting SCC even fter reding. However when this hppens, they do not rech the ccepting stte before is red, so this limits the number of red DFS. D 4, D 5, D 12 These utomt go from to the ccepting stte ech time they red. This both mkes the product unnecessrily lrge, but it lso forces mny clls to the red DFS every time product stte with property utomton stte is bcktrcked. The nondeterminism in ccepting SCC of D 4 cuses it to visits only slightly more sttes thn the other two utomt. A comprison of utomt D 6 nd D 11 nd their impct on Spin s performnce show tht the hypothesis of Section 4.3 cnnot be used lone to select the best utomton.

s i s i s 3 (D 1) Spin (D 2) LTL2BA (D 3) MoDeLL (D 4) LTL3BA (D 5) LTL3BA (det) (D 6) Spot & Spot (det) (D 7) Spot (no jump) (D 8) (D 9) (D 10) (D 11) (D 12) Figure 6: Automt for the formul (GF GFb). Tble 4: Sttistics bout generted utomt nd Spin s run on the empty product between model peterson.4.pm nd formul (GF GFb). The corresponding utomt re shown in Fig. 6. utomton size sttistics from Spin s execution sttes ndst edges trns stored sttes visited trns time D 1 Spin 3 2 6 12 1577846 7680k 6.04s D 2 LTL2BA 3 3 6 12 1577440 7684k 5.95s D 3 MoDeLL 5 2 8 18 1580893 7670k 6.13s D 4 LTL3BA 3 3 6 12 2299250 15583k 12.10s D 5 LTL3BA (det) 4 1 7 14 2297625 15561k 12.00s D 6 Spot 3 1 6 9 848641 2853k 2.26s D 7 Spot (no jump) 3 1 5 9 852094 2863k 2.34s D 8 3 1 6 9 848641 2853k 2.43s D 9 3 3 6 11 852094 2878k 2.43s D 10 3 1 7 10 1575844 7658k 7.38s D 11 3 1 6 10 1577440 7657k 7.07s D 12 3 1 6 10 2297625 15561k 12.30s

Indeed, D 6 outperforms D 11 even if the distnce from the initil to the ccepting stte is shorter in D 6. Here the more restrictive lbel of trnsition (, ) in D 6 plys n importnt role s well. To sum up, if we suppose tht there is no ccepting cycle in the product, the utomton should 1. keep ccepting sttes s fr s possible from the initil stte (compre D 11 to D 12) nd 2. use more restrictive lbels (compre D 6 to D 12) in order to mke the ccepting sttes s hrd to rech s possible. Moreover, mking use of more restrictive lbels cn lso help to reduce the product. An pproprite metric tking these two fctors into ccount, s well s n LTL-to-BA trnsltion reflecting these hypotheses, re topics for our future reserch. 5. CONCLUSION LTL-to-BA trnsltors hve severl degrees of freedom when producing utomt. Some of these choices hve effects on the product with system to be verified nd lso to the emptiness check of the product. However, these effects re difficult to predict. So fr, most uthors of LTL-to-BA trnsltion tools hve mesured the performnce of their tools by looking t the size of the output, sometimes lso by looking t the size of products with rndom stte spces. While building smll product generlly helps the emptiness check, we hve provided evidence tht the size of A ϕ nd even the size of S A ϕ does not lwys correlte to the performnce of the emptiness check of S A ϕ. For instnce, s Spin uses Nested DFS, the loctions of ccepting sttes of A ϕ cn hve drmtic impct to Spin s running time. When system S stisfies ϕ, i.e., S A ϕ contins no ccepting cycle, the best utomton for Spin to verify it should hve ccepting sttes tht re hrd to rech from the initil stte, s it will lessen the chnce tht red DFS is strted. We observed tht such choice cn be mde during the degenerliztion procedure, or by unrolling some ccepting cycles. On the contrry, if S A ϕ contins n ccepting cycle, Spin cn find it fster if the ccepting sttes of A ϕ re esy to rech from the initil stte nd the ccepting cycles re short. Furthermore, the emptiness check cn use n optimiztion if the utomton is wek. We pln to exmine these suggestions nd potentilly integrte them in future versions of our trnsltors. Furthermore, we pln to devise set of heuristics to select the best utomton of given set of cndidtes. Clerly, LTL-to-BA trnsltors should tune their output ccording to the purposed use of the BA: BA used for bug finding need not to be the sme s BA used to prove correctness. Here we focused on the Nested DFS implementtion of Spin, but mny other emptiness checks exist. For instnce, some emptiness checks bsed on the enumertion of SCCs re insensible to the loction of ccepting sttes on cycle, so our suggestions should not be generlized blindly. Another point tht cn be influenced by the property utomton is the size of the counterexmple generted. The question of finding n utomton tht is optiml from this point of view is left open by Gstin et l. [13]. 6. ACKNOWLEDGMENTS Authors would like to thnk Vojtěch Rujbr for the initil inspirtion nd three nonynous referees for their suggestions. Fr. Blhoudek, M. Křetínský, nd J. Strejček hve been supported by The Czech Science Foundtion, grnt GBP202/12/G061. 7. REFERENCES [1] T. Bbik, M. Křetínský, V. Řehák, nd J. Strejček. LTL to Büchi utomt trnsltion: Fst nd more deterministic. In TACAS 12, vol. 7214 of LNCS, pp. 95 109. Springer, 2012. [2] B. Boigelot, S. Jodogne, nd P. Wolper. On the use of wek utomt for deciding liner rithmetic with integer nd rel vribles. In IJCAR 01, vol. 2083 of LNCS, pp. 611 625. Springer, 2001. [3] E. M. Clrke, O. Grumberg, nd D. A. Peled. Model Checking. The MIT Press, 2000. [4] J.-M. Couvreur. On-the-fly verifiction of temporl logic. In FM 99, vol. 1708 of LNCS, pp. 253 271, Sept. 1999. Springer. [5] C. Dx, J. Eisinger, nd F. Kledtke. Mechnizing the powerset construction for restricted clsses of ω-utomt. In ATVA 07, vol. 4762 of LNCS. Springer, Oct. 2007. [6] A. Duret-Lutz. Mnipulting LTL formuls using Spot 1.0. In ATVA 13, vol. 8172 of LNCS, pp. 442 445, Oct. 2013. Springer. [7] A. Duret-Lutz. LTL trnsltion improvements in Spot 1.0. Interntionl Journl on Criticl Computer-Bsed Systems, 5(1/2):31 54, Mr. 2014. [8] A. Duret-Lutz nd D. Poitrenud. SPOT: n Extensible Model Checking Librry using Trnsition-bsed Generlized Büchi Automt. In MASCOTS 04, pp. 76 83, Oct. 2004. IEEE Computer Society Press. [9] M. B. Dwyer, G. S. Avrunin, nd J. C. Corbett. Property specifiction ptterns for finite-stte verifiction. In FMSP 98, pp. 7 15, Mr. 1998. ACM Press. [10] K. Etessmi nd G. J. Holzmnn. Optimizing Büchi Automt. In CONCUR 00, vol. 1877 of LNCS, pp. 153 167. Springer, 2000. [11] K. Etessmi nd G. J. Holzmnn. Optimizing Büchi utomt. In Concur 00, vol. 1877 of LNCS, pp. 153 167, 2000. Springer. [12] P. Gstin nd D. Oddoux. Fst LTL to Büchi utomt trnsltion. In CAV 01, vol. 2102 of LNCS, pp. 53 65, 2001. Springer. [13] P. Gstin, P. Moro, nd M. Zeitoun. Minimiztion of counterexmples in SPIN. In SPIN 04, vol. 2989 of LNCS, pp. 92 108, Apr. 2004. [14] J. Geldenhuys nd A. Vlmri. More efficient on-the-fly LTL verifiction with Trjn s lgorithm. Theoreticl Computer Science, 345(1):60 82, Nov. 2005. [15] D. Ginnkopoulou nd F. Lerd. From sttes to trnsitions: Improving trnsltion of LTL formulæ to Büchi utomt. In FORTE 02, vol. 2529 of LNCS, pp. 308 326, Nov. 2002. Springer.

[16] G. J. Holzmnn. The Spin Model Checker: Primer nd Reference Mnul. Addison-Wesley, 2003. [17] G. J. Holzmnn, D. A. Peled, nd M. Ynnkkis. On nested depth first serch. In SPIN 96, vol. 32 of DIMACS. Americn Mthemticl Society, My 1996. [18] G. J. Holzmnn, R. Joshi, nd A. Groce. Swrm verifiction techniques. IEEE Trnsction on Softwre Engineering, 37(6):845 857, 2011. [19] R. Pelánek. BEEM: benchmrks for explicit model checkers. In SPIN 07, vol. 4595 of LNCS, pp. 263 267. Springer, 2007. [20] S. Schwoon nd J. Esprz. A note on on-the-fly verifiction lgorithms. In TACAS 05, vol. 3440 of LNCS, Apr. 2005. Springer. [21] R. Sebstini nd S. Tonett. More deterministic vs. smller Büchi utomt for efficient LTL model checking. In CHARME 03, vol. 2860 of LNCS, pp. 126 140, Oct. 2003. Springer. [22] F. Somenzi nd R. Bloem. Efficient Büchi utomt for LTL formulæ. In CAV 00, vol. 1855 of LNCS, pp. 247 263, 2000. Springer. [23] H. Turiinen nd K. Heljnko. Testing LTL formul trnsltion into Büchi utomt. Interntionl Journl on Softwre Tools for Technology Trnsfer, 4 (1):57 70, 2002. [24] X. Thirioux. Simple nd efficient trnsltion from LTL formuls to Büchi utomt. In FMICS 02, vol. 66(2) of ENTCS, July 2002. Elsevier. [25] M. Y. Vrdi. An utomt-theoretic pproch to liner temporl logic. In Bnff 94, vol. 1043 of LNCS, pp. 238 266, 1996. Springer.