Benchmarking Model Checkers with Distributed Algorithms. Étienne Coulouma-Dupont

Transcription

1 Benchmarking Model Checkers with Distributed Algorithms Étienne Coulouma-Dupont November 24, 2011

2 Introduction The Consensus Problem Consensus : application Paxos LastVoting Hypothesis The Algorithm Analysis of the Algorithm Simulator Motivations Benchmark Benchmarks Various parameters POEM ALCOOL 2/66

3 Introduction LastVoting Simulator Benchmarks 3/66

4 Panda Project Industrial Benchmark Airbus Code Academic Benchmark Paxos Algorithm 4/66

5 The Consensus Problem The Consensus Problem Context: Distributed Systems Consensus is the problem of making processes agree on a common value in spite of faults 5/66

6 The Consensus Problem The Consensus Problem An Algorithms solves Consensus if and only if it satifies the following conditions: Integrity: Any decision value is the proposed value of some process. Agreement: No two different values are decided. Termination: All process eventually decide. 6/66

7 Consensus : application Consensus Algorithm with industrial applications Chubby (Google, 2006) [Bur06][CGR07] implements a fault tolerant data-base: Fault-tolerance is achieved through replication Consistency is achieved using Paxos Algorithm 7/66

8 Paxos Paxos Family of algorithms built to solve Consensus First published by Leslie Lamport in 1998 [Lam98] Termination is not guaranteed without additional assumptions on liveness Safety is guaranteed 8/66

9 Paxos Several implementations in different models: ChandraToueg Attach to each process in the system a failure detector LastVoting Synchronous communication (rounds) 9/66

10 Paxos Related Works This algorithm has been intensively studied Fuzzati [Fuz08]: proof of Paxos and ChandraToueg with rewriting rules Küfner et al.: assisted proofs in Isabelle Tsuchiya & Schiper: model-checking of LastVoting SAT solver: up to 10 processes Traditional tools: up to 4 processes 10/66

11 Introduction LastVoting Simulator Benchmarks 11/66

12 Hypothesis LastVoting : Hypothesis Assumptions for the environment: with transmission faults : in each round, the adversary chooses a set of edges in which all messages will be correctly transmitted pseudo-synchronous: any message which is not received in the same round as the one during which it was send is thrown out by the process which receives it complete network: simplifies the way algorithms are expressed each process has a unique identity 12/66

13 Hypothesis LastVoting : Hypothesis We assume that each process p has at any time access to the following pieces of information: The round r in which it is The coordinator at round r 13/66

14 The Algorithm The Algorithm s Procedure rithm in a ition must odel such he collecordinators ing prediting algo- Phase i-1 Phase i Phase i+1 e: all pro- c i-1 c i c i+1 c i+2 c 1 c 2 c k c k+1 Figure: Transitions Figure 1. oftransitions configurations of configurations the phase level at(top) the and at the round level phase (bottom) level (top) [TS11]. and at the round level (bottom). (1) 0)) 14/66

15 The Algorithm The Algorithm s Procedure In each round r, each process p: can send a messages according to a sending function S r p, depending on the process state then can compute a new state according to a state transition function T r p, depending on the messages received 15/66

16 Selection (r = 4φ 3) Commit (r = 4φ 2) envoi Coord envoi Coord Ack (r = 4φ 1) Decision (r = 4φ) envoi Coord envoi Coord

17 The Algorithm Notations co = Coord(φ) D(φ) = {p Π d 4φ 3 p?} for all p Π and r [4φ 3, 4φ 1], RCV (p, r) is the set of processes from which p receives a non empty message in round r 17/66

18 The Algorithm Initialization : x p Val, initially v p // v p is the proposed value of p. vote p Val {?}, initially? // Val is the set of values that may be proposed. commit p a Boolean, initially false ready p a Boolean, initially false ts p N, initially 0 18/66

19 The Algorithm Round r = 4φ 3 : // #RCV (co, 4φ 3) > n/2 S r p : send x p, ts p to Coord(p, φ) T r p : if p = Coord(p, φ) and number of ν, θ received > n/2 then let be the largest θ from ν, θ received; vote p := one ν such that ν, θ is received; commit p := true; 19/66

20 The Algorithm Round r = 4φ 2 : // #{p Π co RCV (p, 4φ 2)} > n/2 S r p : if p = Coord(p, φ) and commit p then send vote p to all processes; T r p : if received v from Coord(p, φ) then x p := v; ts p := φ; 20/66

21 The Algorithm Round r = 4φ 1 : // #RCV (co, 4φ 1) > n/2 S r p : if ts p = φ then send ack to Coord(p, φ); T r p : if p = Coord(p, φ) and number of ack received > n/2 then ready p := true; 21/66

22 The Algorithm Round r = 4φ : // p Π \ D(φ) : co RCV (p, 4φ) S r p : if Coord(p, φ) and ready p then send vote p to all processes; T r p : if received v from Coord(p, φ) then DECIDE (v) if p = Coord(p, φ) then ready p := false; commit p := false; 22/66

23 Analysis of the Algorithm Terminology Given a partial execution, we define 0-Valence if the value 0 is to be decided 1-Valence if the value 1 is to be decided Univalence if one the above holds Bivalence if none of the above holds Formally, Univalence is defined for LastVoting as follows: v Val, P Π : #P n/2 P = {p Π pr p = v} ( q Π \ P : ts q < ts p ) 23/66

24 Analysis of the Algorithm Termination Termination holds iff P sync (φ) eventually holds: P sync (φ) φ > 0, co Π : ronde1 : #RCV (co, 4φ 3) > n/2 (CPROP(φ)) 24/66

25 Analysis of the Algorithm Termination Termination holds iff P sync (φ) eventually holds: P sync (φ) φ > 0, co Π : ronde1 : #RCV (co, 4φ 3) > n/2 (CPROP(φ)) ronde2 : #{p Π co RCV (p, 4φ 2)} > n/2 (BVOTE(φ)) 24/66

26 Analysis of the Algorithm Termination Termination holds iff P sync (φ) eventually holds: P sync (φ) φ > 0, co Π : ronde1 : #RCV (co, 4φ 3) > n/2 (CPROP(φ)) ronde2 : #{p Π co RCV (p, 4φ 2)} > n/2 (BVOTE(φ)) ronde3 : #RCV (co, 4φ 1) > n/2 (CACK (φ)) 24/66

27 Analysis of the Algorithm Termination Termination holds iff P sync (φ) eventually holds: P sync (φ) φ > 0, co Π : ronde1 : #RCV (co, 4φ 3) > n/2 (CPROP(φ)) ronde2 : #{p Π co RCV (p, 4φ 2)} > n/2 (BVOTE(φ)) ronde3 : #RCV (co, 4φ 1) > n/2 (CACK (φ)) ronde4 : p Π \ D(φ) : co RCV (p, 4φ) (BDEC(φ)) 24/66

28 Analysis of the Algorithm s 0 CP ROP F 01 s 1 F s 11 1 BV OT E F 12 s 2 F 13 s 2 CACK F 23 s 3 F s 33 3 BDEC F 34 s 4 s 4 25/66

29 Analysis of the Algorithm Execution-tree (graph... ) dec none A univ dec part dec all dec none/univ dec none/univ/dec part * B dec none/univ C dec part/dec all D dec all 26/66

30 Analysis of the Algorithm Example of execution φ = 1 p 1 p 2 p 3 pr 1 = 0 pr 2 = 0 pr 3 = 1 ts 1 = 0 ts 2 = 0 ts 3 = 0 d 1 =? d 2 =? d 3 =? s 0 27/66

31 Analysis of the Algorithm φ = 1 p 1 p 2 p 3 ho 1 = 1 ho 2 = 0 ho 3 = 1 pr 1 = 0 pr 2 = 0 pr 3 = 1 ts 1 = 0 ts 2 = 0 ts 3 = 0 d 1 =? d 2 =? d 3 =? s 1 CP ROP s 0 28/66

32 Analysis of the Algorithm φ = 1 p 1 p 2 p 3 ho 1 = 0 ho 2 = 0 ho 3 = 1 pr 1 = 0 pr 2 = 0 pr 3 = 1 ts 1 = 0 ts 2 = 0 ts 3 = 1 d 1 =? d 2 =? d 3 =? s 1 CP ROP F 12 s 0 s 2 29/66

33 Analysis of the Algorithm φ = 2 p 1 p 2 p 3 pr 1 = 0 pr 2 = 0 pr 3 = 1 ts 1 = 0 ts 2 = 0 ts 3 = 1 d 1 =? d 2 =? d 3 =? s 1 CP ROP F 12 s 0 s 2 30/66

34 Analysis of the Algorithm φ = 2 p 1 p 2 p 3 ho 1 = 0 ho 2 = 1 ho 3 = 1 pr 1 = 0 pr 2 = 0 pr 3 = 1 ts 1 = 0 ts 2 = 0 ts 3 = 1 d 1 =? d 2 =? d 3 =? s 1 CP ROP s 0 31/66

35 Analysis of the Algorithm φ = 2 p 1 p 2 p 3 ho 1 = 1 ho 2 = 0 ho 3 = 0 pr 1 = 1 pr 2 = 0 pr 3 = 1 ts 1 = 2 ts 2 = 0 ts 3 = 1 d 1 =? d 2 =? d 3 =? s 1 F 13 CP ROP s 0 s 3 32/66

36 Analysis of the Algorithm φ = 3 p 1 p 2 p 3 pr 1 = 1 pr 2 = 0 pr 3 = 1 ts 1 = 2 ts 2 = 0 ts 3 = 1 d 1 =? d 2 =? d 3 =? s 1 F 13 CP ROP s 0 s 3 33/66

37 Analysis of the Algorithm φ = 3 p 1 p 2 p 3 ho 1 = 1 ho 2 = 1 ho 3 = 0 pr 1 = 1 pr 2 = 0 pr 3 = 1 ts 1 = 2 ts 2 = 0 ts 3 = 1 d 1 =? d 2 =? d 3 =? s 1 CP ROP s 0 34/66

38 Analysis of the Algorithm φ = 3 p 1 p 2 p 3 ho 1 = 0 ho 2 = 1 ho 3 = 1 pr 1 = 1 pr 2 = 1 pr 3 = 1 ts 1 = 2 ts 2 = 3 ts 3 = 3 d 1 =? d 2 =? d 3 =? BV OT E s 1 CP ROP s 0 s 2 35/66

39 Analysis of the Algorithm φ = 3 CP ROP s 0 p 1 p 2 p 3 ho 1 = 1 ho 2 = 0 ho 3 = 1 pr 1 = 1 pr 2 = 1 pr 3 = 1 ts 1 = 2 ts 2 = 3 ts 3 = 3 d 1 =? d 2 =? d 3 =? s 2 BV OT E s 1 CACK s 3 36/66

40 Analysis of the Algorithm s 0 CP ROP φ = 3 p 1 p 2 p 3 ho 1 = 1 ho 2 = 0 ho 3 = 1 pr 1 = 1 pr 2 = 1 pr 3 = 1 ts 1 = 2 ts 2 = 3 ts 3 = 3 d 1 =? d 2 =? d 3 = 1 s 3 CACK s 2 BV OT E s 1 F 34 s 4 37/66

41 Analysis of the Algorithm s 0 CP ROP φ = 4 p 1 p 2 p 3 pr 1 = 1 pr 2 = 1 pr 3 = 1 ts 1 = 2 ts 2 = 3 ts 3 = 3 d 1 =? d 2 =? d 3 = 1 s 3 CACK s 2 BV OT E s 1 F 34 s 4 38/66

42 Analysis of the Algorithm φ = 4 p 1 p 2 p 3 ho 1 = 1 ho 2 = 1 ho 3 = 1 pr 1 = 1 pr 2 = 1 pr 3 = 1 ts 1 = 2 ts 2 = 3 ts 3 = 3 d 1 =? d 2 =? d 3 = 1 s 1 CP ROP s 0 39/66

43 Analysis of the Algorithm φ = 4 p 1 p 2 p 3 ho 1 = 1 ho 2 = 0 ho 3 = 1 pr 1 = 1 pr 2 = 1 pr 3 = 1 ts 1 = 4 ts 2 = 3 ts 3 = 4 d 1 =? d 2 =? d 3 = 1 BV OT E s 1 CP ROP s 0 s 2 40/66

44 Analysis of the Algorithm φ = 4 p 1 p 2 p 3 ho 1 = 1 ho 2 = 0 ho 3 = 1 pr 1 = 1 pr 2 = 1 pr 3 = 1 ts 1 = 4 ts 2 = 3 ts 3 = 4 d 1 =? d 2 =? d 3 = 1 CACK s 2 BV OT E s 1 CP ROP s 0 s 3 41/66

45 Analysis of the Algorithm s 0 CP ROP φ = 4 p 1 p 2 p 3 ho 1 = 1 ho 2 = 1 ho 3 = 0 pr 1 = 1 pr 2 = 1 pr 3 = 1 ts 1 = 4 ts 2 = 3 ts 3 = 4 d 1 = 1 d 2 = 1 d 3 = 1 s 3 CACK s 2 BV OT E s 1 BDEC s 4 42/66

46 Analysis of the Algorithm Comparison with the asynchronous version Each process has a infinite set of integers which is disjoint from the sets of the other processes This set is used to uniquely identify the consensus requests In the asynchronous version, the coordinator first broadcasts the request number of the new request The last broadcast is a flooding broadcast procedure 43/66

47 Analysis of the Algorithm Computability of Paxos Paxos does not terminate [FLP85][Fuz08] Optimal condition not known for consensus algorithms 44/66

48 Analysis of the Algorithm Fuzzati s proof of Paxos A general pattern of rule: Condition(s) (RULE) Action(s) or Event(s) Example of rule from Fuzzati s Paxos rules: S(i) = (a, r, p, b, (, ι), ) (CRASH) B, C, S B, C, S{i (a, r, p, b, (, ι), )} 45/66

49 Introduction LastVoting Simulator Benchmarks 46/66

50 Motivations Motivations Correct Algorithms Benchmark Algorithms performances 47/66

51 Benchmark Benchmark Banc d essai pour N = 3 Banc d essai pour N = 4 durée moyenne/test (µs) LVOpt version=1 LVOpt version=2 LVOpt version=3 MyLVOpt version=2 MyLVOpt version=3 durée moyenne/test (µs) LVOpt1 LVOpt2 LVOpt3 MyLVOpt2 MyLVOpt3 LVOpt1 LVOpt2 LVOpt3 MyLVOpt2 MyLVOpt3 LVOpt version=1 LVOpt version=2 LVOpt version=3 MyLVOpt version=2 MyLVOpt version=3 48/66

52 Introduction LastVoting Simulator Benchmarks 49/66

53 Motivations Benchmarking with Paxos Tsuchiya & Schiper: SAT Solvers: up to 10 processes NuSMV: up to 4 processes SPIN: up to 3 processes 50/66

54 Motivations Benchmarking with Paxos Tsuchiya & Schiper: SAT Solvers: up to 10 processes NuSMV: up to 4 processes SPIN: up to 3 processes Our work in progress: POEM (Peter Niebert, Marseille): frontend for different tools with partial order techniques preprocessing ALCOOL (CEA/List, Panda): Topological techniques 50/66

55 Various parameters Toward Benchmarking of Model-Checkers Number n of processes of the system 51/66

56 Various parameters Toward Benchmarking of Model-Checkers Number n of processes of the system Number of losses for the system:newline for one round for one phase for k phases for one execution Number of losses per process:newline for one phase for k phases for one execution 51/66

57 POEM POEM Checking is done with IF requests (first order logic): (d[0]<>not_def) and (d[1]<>not_def) and (d[0]<>d[1]) Current problem: false positives (Bug in the frontend?) 52/66

58 POEM First Tests duration (s) perte/ronde 1 perte/ronde number of processes 53/66

59 ALCOOL ALCOOL ALCOOL can detect deadlocks and forbidden zones We transform the properties which are to be verified into deadlock properties Example: B 0 and B 1 two n-ary synchronization barriers Agreement process p waits on B dp Current problem: false positive (feature) 54/66

60 ALCOOL Example 2 #mutex m processes: p1 = P(m).V(m) p2 = P(m).V(m) init: p1 p2 p1 V(m) P(m) P(m) V(m) p2 55/66

61 ALCOOL p1 V(m) P(m) P(m) V(m) p2 56/66

62 ALCOOL p1 Basically only 2 executions: p1.p2 p2.p1 V(m) P(m) P(m) V(m) p2 57/66

63 ALCOOL p1 V(m) p1.p2 : x=8 p2.p1 : V(m) p2 57/66

64 ALCOOL Example 3 p1 #mutex a b processes: p1 = P(a).P(b).V(b).V(a) p2 = P(b).P(a).V(a).V(b) init: p1 p2 V(a) V(b) P(b) P(a) P(b) P(a) V(a) V(b) p2 58/66

65 ALCOOL Example 4 #synchronization 2 b processes: p1 (W(b) + [x==1] + void) p2 init: p1 p2 p2 59/66

66 ALCOOL Example 4 #synchronization 2 b processes: p1 p2 init: p1 p2 W(b) p2 60/66

67 ALCOOL p1 p1 W(b) W(b) p2 61/66

68 ALCOOL Example 5 p1 #synchronization 2 b s processes: p1 = void.w(s).@(x,1). (W(b) + [x==1] + void) p2 init: p1 p2 W(b) p2 62/66

69 ALCOOL Example 5 p1 #synchronization 2 b s processes: p1 = W(s).@(x,1).W(b) p2 init: p1 p2 W(b) W(b) p2 63/66

70 ALCOOL p1 p1 W(b) W(b) p2 W(b) p2 64/66

71 ALCOOL Benchmarks: ALCOOL Analysis of numerical variables Analysis of potential deadlocks ALCOOL +POEM? 65/66

72 Introduction The Consensus Problem Consensus : application Paxos LastVoting Hypothesis The Algorithm Analysis of the Algorithm Simulator Motivations Benchmark Benchmarks Various parameters POEM ALCOOL 66/66

73 BURROWS, MIKE: The Chubby lock service for loosely-coupled distributed systems. In Proceedings of the 7th symposium on Operating systems design and implementation, OSDI 06, pages , Berkeley, CA, USA, USENIX Association. CHANDRA, TUSHAR D., ROBERT GRIESEMER and JOSHUA REDSTONE: Paxos made live: an engineering perspective. In Proceedings of the twenty-sixth annual ACM symposium on Principles of distributed computing, PODC 07, pages , New York, NY, USA, ACM. FISCHER, MICHAEL J., NANCY A. LYNCH and MICHAEL S. PATERSON: Impossibility of distributed consensus with one faulty process. J. ACM, 32: , April /66

74 FUZZATI, RACHELE: A formal approach to fault tolerant distributed consensus. PhD thesis, École Polytechnique Fédérale de Lausanne, Lausanne, LAMPORT, LESLIE: The part-time parliament. ACM Transactions on Computer Systems, 16: , TSUCHIYA, TATSUHIRO and ANDRÉ SCHIPER: Verification of consensus algorithms using satisfiability solving. Distributed Computing, 23: , /66