A Communication-Induced Checkpointing Protocol that Ensures Rollback-Dependency Trackability

Transcription

1 A Communication-Induced Checkpointing Protocol that Ensures Rollback-Dependency Trackability Roberto BALDONI Jean-Michel HELARY y Achour MOSTEFAOUI y Michel RAYNAL y Abstract Considering an application in which processes take local checkpoints independently (called basic checkpoints), this paper develops a protocol that forces them to take some additional local checkpoints (called forced checkpoints) in order that the resulting checkpoint and communication pattern satisfies the Rollback Dependency Trackability (RDT) property. This property states that all dependencies between local checkpoints are on-line trackable by using a transitive dependency vector. Compared to other protocols ensuring the RDT property, the proposed protocol is less conservative in the sense that it takes less additional local checkpoints. It attains this goal by a subtle tracking of causal dependencies on already taken checkpoints; this tracking is then used to prevent the occurrence of hidden dependencies. As indicated by simulation study, the proposed protocol compares favorably with other protocols; moreover, it additionally associates on-the-fly with each local checkpoint C the minimum global checkpoint to which C belongs. 1 Introduction A local checkpoint is a snapshot of a local state of a process and a consistent global checkpoint is a set of local states, one from each process, such that no message sent by a process after its local checkpoint is received by another one before its local checkpoint. The computation of consistent global checkpoints is an important task when one is interested in designing or implementing systems that have to ensure dependability of the applications they run. Many protocols have been proposed to select local checkpoints in order to form consistent global checkpoints (see the nice survey [5]). Remark that if local checkpoints are taken independently there is a risk that no consistent global checkpoint can ever be formed from them (this is the well-known unbounded domino effect, that can occur during rollback-recovery[9]). To avoid the domino effect, a kind of Universitá La Sapienza, Roma, Italy. baldoni@dis.uniroma1.it. y IRISA, Campus de Beaulieu, Université Rennes1, Rennes, France. fhelary,mostefaoui,raynalg@irisa.fr. coordination in the determination of local checkpoints is required. In [3, 6], the coordination is achieved at the price of synchronization by means of additional control messages. Another approach, namely, communication-induced checkpointing 1, achieves coordination by piggybacking control information on application messages. In that case, processes select local checkpoints independently (called basic checkpoints) and the protocol requires them to take 2 additional local checkpoints (called forced checkpoints) in order to ensure the progression of a consistent recovery line. Forced checkpoints are taken on the basis of control information piggybacked on messages. The classical use of consistent global checkpoints lies in the rollback mechanisms employed to resume a computation after a failure occurrence. Recently, it has been also used in the context of distributed debugging in which the determination of a sequence of consistent global checkpoints is crucial to track software errors and to correct distributed programs [1]. There are many other dependability problems whose solution rests on the determination of consistent global checkpoints containing a given set of local checkpoints and more particularly the minimum ( first ) and the maximum ( last ) ones. Examples of such dependability problems are: distributed software diagnosis, consistent deadlock recovery, definition of causal distributed breakpoints, output commit [13]. In general, the fact that two local checkpoints be not causally related is a necessary but not sufficient condition for them to belong to the same consistent global checkpoint [8]. They can have hidden dependencies (i.e., dependencies that cannot be tracked with transitive dependency vectors) that make impossible for them to belong to the same consistent global checkpoint. To solve this problem, Wang has defined the Rollback-Dependency Trackability (RDT) property [13]. A checkpoint pattern satisfies this property if all dependencies between local checkpoints can be online trackable (i.e., trackable by a simple use of a transitive dependency vector). RDT has two noteworthy properties. 1 We use the terminology introduced in [5]. 2 When a process selects one of its local state as a local checkpoint we say that it takes a local checkpoint.

2 (1) It ensures that any set of local checkpoints that are not pairwise causally related can be extended to form a consistent global checkpoint [8]. (2) It enjoys efficient calculations of the minimum and the maximum consistent global checkpoints that contain a given set of local checkpoints. As a consequence, the RDT property has applications in a large family of dependability problems such as the ones previously cited. Moreover, when combined with an appropriate message logging protocol [4], the RDT property allows to solve some dependability problems posed by nondeterministic computations as if these computations were piecewise deterministic (see [13] for a development of this approach). So, in this context and given a distributed computation, the main question is: how to ensure that local checkpoints satisfy the RDT property? Several communication-induced checkpointing protocols ensuring the RDT property have been proposed such as Checkpoint-Before-Receive, No- Receive-After-Send [10], Checkpoint-After-Send [12] etc. In [13], Wang introduces the FDAS checkpointing protocol (see Section 5.2) that, piggybacking a dependency vector as a control information, takes less forced local checkpoints than the aforementioned ones. Nevertheless, even FDAS is more conservative than necessary to ensure the RDT property: actually, it induces processes to take more forced checkpoints than necessary to guarantee that all dependencies between local checkpoints are on-line trackable. In this paper, we design a communication-induced checkpointing protocol that ensures the RDT property while forcing less checkpointing than any protocol of the FDAS family. This is achieved by a subtle tracking of causal dependencies on already taken checkpoints. This tracking is then used to prevent the occurrence of hidden dependencies; the prevention is done by forcing a process to take an additional local checkpoint only when necessary, according to its current knowledge of the past of the computation (this knowledge being encoded and piggybacked on application messages). The behavior of the resulting protocol is quantified by a simulation study. Moreover, the proposed protocol enjoys efficient on-the-fly calculation of the minimum consistent global checkpoint to which a local checkpoint C belongs. It follows that dependability problems whose solutions rest on the minimum consistent global checkpoint containing a given local checkpoint are easier to address; among these dependability problems, we find software error recovery, determination of causal distributed breakpoints and output commits. The paper is divided into 5 sections. Section 2 defines the computational model. Section 3 introduces definitions and elements of the Rollback-Dependency Trackability theory. Section 4 presents the communication-induced checkpointing protocol. Finally, Section 5 addresses related works, discusses the protocol and presents simulation results. 2 Consistent Global Checkpoints 2.1 Distributed Computations A distributed computation consists of a finite set P of n processes fp 1 P 2 ::: P n g that communicate and synchronize only by exchanging messages. We assume that each ordered pair of processes is connected by an asynchronous, reliable, directed logical channel whose transmission delays are unpredictable but finite. Each process runs on a processor; processors do not share a common memory; there is no bound for their relative speeds and they fail according to the fail-stop model. A process can execute internal, send and delivery statements. An internal statement does not involve communication. When P i executes the statement send(m) to P j it puts the message m into the channel from P i to P j. When P i executes the statement deliver(m), it is blocked until at least one message directed to P i has arrived; then a message is retrieved from one of its input channels and delivered to P i. Executions of internal, send and delivery statements are modeled by internal, sending and delivery events. Processes of a distributed computation are sequential,in other words, each process P i produces a sequence of events e i 1 :::e i s ::: This sequence can be finite or infinite. Every process P i has an initial local state denoted i 0. The local state i s (s>0) results from the execution of the sequence e i 1 :::e i s applied to the intial state i 0. More precisely the event e i s moves P i from the local state i s;1 to the local state i s. By definition we say that e i x belongs to j s ifi = j and x s. Let H be the set of all the events produced by a distributed computation. This computation is modeled by the partially ordered set H b hb = (H!), where! hb denotes the well-known Lamport s happened-before relation [7]. 2.2 Local and Global Checkpoints A local checkpoint C is a recorded state of a process. A local state is not necessarily recorded as a local checkpoint, so the set of local checkpoints is only a subset of the set of local states. Definition 2.1 A checkpoint and communication pattern is a pair ( b H C bh ) where b H is a distributed computation and C bh is a set of local checkpoints defined on b H. C i x represents the x-th local checkpoint of process P i ; x is called the index of this checkpoint. The local checkpoint C i x corresponds to some local state i s with x s. Figure 1.a shows an example of checkpoint and communication pattern. We assume that each process P i takes an initial local checkpoint C i 0 (corresponding to i 0 ), and after each event a checkpoint will eventually be taken.

3 P i Ci 0 Ci 1 Ci 2 Ci 3 Ci 0 Ci 1 Ci 2 Ci 3 m1 m2 m5 Cj 0 Cj 1 Cj 2 Cj 3 P j Ij 1 m3 m4 m6 m7 C k 3 Cj 0 Cj 1 Cj 2 Cj 3 C k 0 C k 1 C k 2 P k I k 1 I k 2 I k 3 C k 0 C k 1 C k 2 C k 3 a. b. Figure 1. A Checkpoint and Communication Pattern with the Corresponding R-Graph A message m sent by process P i to process P j is called orphan with respect to the ordered pair of local checkpoints (C i x,c j y ) if the delivery of m belongs to C j y while its sending event does not belong to C i x. An ordered pair of local checkpoints is consistent if and only if there are no orphan messages with respect to this pair. For example, Figure 1.a shows the pair (C k 1,C j 1 ) is consistent, while the pair (C i 2,C j 2 ) is inconsistent (because of orphan message m 5 ). A global checkpoint is a set of local checkpoints one from each process. For example, fc i 1 C j 1 C k 1 g and fc i 2 C j 2 C k 1 g are two global checkpoints depicted in the Figure 1.a. Definition 2.2 A global checkpoint is consistent if all its pairs of local checkpoints are consistent. For example, Figure 1.a. shows that fc i 1 C j 1 C k 1 g is a consistent global checkpoint, while fc i 2 C j 2 C k 1 g is not consistent (due to the inconsistent pair (C i 2 C j 2 )). 3 Rollback-Dependency Trackability The reader interested in the theory of Rollback- Dependency Trackability and its applications will consult [13]. 3.1 Rollback-Dependency Graph The sequence of events occurring at P i between C i x;1 and C i x (x > 0) is called checkpoint interval and is denoted by I i x. The Rollback-Dependency Graph (or R- graph) is defined as follows ([13]): each node represents a local checkpoint. a directed edge from C i x to C j y exists if and only if: 1. i = j and y = x +1,or 2. i 6= j and a message m is sent in I i x and delivered in I j y. Figure 1.b depicts the R-graph corresponding to the checkpoint and communication pattern depicted on Figure 1.a. As defined in [13], a R-path is a directed path in the R- graph. The R-path connecting the node C i x to the node C j y is denoted C i x! C j y. The operational meaning of the path C i x! C j y is the following: if P i has to be rolledback to a local checkpoint preceding C i x then P j has to be rolled-back to a local checkpoint preceding C j y. 3.2 Message Chains Definition 3.1 A message chain is a sequence of messages [m 1 m 2 ::: m q ](q 1) such that, for each 1 q ; 1, we have: delivery(m ) 2 I k s ^ send(m +1 ) 2 I k t ^ s t. To our knowledge this notion has been introduced for the first time by Netzer and Xu in [8] under the name zigzag path. In this paper, we restrict the use of the term path to paths of the R-graph. If a message chain [m 1 ::: m q ] is such that send(m 1 ) 2 I i x and delivery(m q ) 2 I j y we say that this chain is from C i x to C j y. Clearly, when there is a message chain from C i x to C j y there is a R- path C i x! C j y. However, there can be several message chains from C i x to C j y, corresponding to the same R-path C i x! C j y. In Figure 1.a, [m 3 m 2 ] 3 is a message chain from C k 1 to C i 2 (corresponding to the R-path C k 1! C i 2 ); [m 5 m 4 ] and [m 5 m 6 ] are two message chains corresponding to the R-path C i 3! C k 2. Definition 3.2 A message chain is causal if the delivery event of each message (but the last one) occurs before the send event of the next message in the chain. A message chain is non-causal if it is not causal. Of course, a message chain with only one message is causal. Every message chain is the concatenation of causal 3 The following notation will be used in the rest of the paper. Let and 0 be two message chains: =[m1] and 0 =[m2 m3]. These notations are equivalent: 0 and [m2 m3] and [m1] 0 and [m1 m2 m3].

4 message subchains. In Figure 1.a, [m 3 m 2 m 5 m 4 m 7 ] is a non-causal message chain; it is the concatenation of the causal message chains [m 3 ] [m 2 m 5 ], and [m 4 m 7 ]. When there are a non-causal message chain and one or several causal message chains from C i x to C j y, each of these causal message chains from C i x to C j y is called a causal sibling of the non-causal message chain. In Figure 1.a, [m 5 m 6 ] is a causal sibling of [m 5 m 4 ]. 3.3 Rollback-Dependency Trackability Definition A R-path C i x! C j y is on-line trackable if i = j ^ x y or if there is at least one causal message chain from C i x to C j y. Definition 3.4 A checkpoint and communication pattern ( b H C bh ) satisfies the Rollback-Dependency Trackability (RDT) property if and only if all R-paths are on-line trackable. When a checkpoint and communication pattern satisfies RDT, every non-causal message chain has a causal sibling. In other words, all the information related to this pattern can be tracked by causality. On-line trackability can be realized with the following simple mechanism [13]. Each process P i maintains a vector TDV i [1::n] (where n is the number of processes) called Transitive Dependency Vector. The entry TDV i [i] is initialized to 1, and incremented each time a new checkpoint is taken; so, its value is always equal to the index of the current checkpoint interval (which is also the index of the next local checkpoint). Every other entry TDV i [j] (j 6= i) is initialized to 0 and records the highest checkpoint interval index y of P j on which P i s current local state transitively depends (i.e., the index y is the highest index such that C j y! C i T DVi[i] is on-line trackable). When P i sends a message m, the current vector TDV i is piggybacked on m. When m is delivered, its receiver P j updates its vector TDV j to be the component-wise maximum of its current TDV j and the piggybacked vector m:t DV : for all k from 1 to n: TDV j [k] := max(tdv j [k] m:tdv[k]). When P i takes a local checkpoint C i x, the value of TDV i at that time is denoted TDV i x. It is easy to see that C i x! C j y is on-line trackable if and only if TDV j y [i] x. Remark. When considering a computation b H, it is interesting to remark that, from the point of view of local states, the set of local checkpoints C bh constitutes an abstraction of b H (this abstraction ignores all the local states that do not belong to C bh ). The 4 Though expressed differently, these definitions are equivalent to Wang s ones. important question is then is C bh a good abstraction? (i.e., an abstraction that allows an easy determination of consistent global checkpoints). If ( b H C bh ) satisfies the RDT property, the answer to this question is yes. The protocol developed in the next section ensures that ( b H C bh ) will always satisfy the RDT property. 4 An Efficient Checkpointing Protocol Ensuring the RDT Property According to the knowledge they acquire about the communication pattern of the past computation and about the dependencies between local checkpoints already taken, processes take communication-induced checkpoints in order that all R-paths be on-line trackable. When a process takes such a forced checkpoint, we say that it breaks a non-causal message chain. So, the protocol ensures the RDT property. The protocol adds no synchronization and no control messages to the computation and uses only the piggybacking of control information on application messages in order to take consistent decisions. Moreover, when a local checkpoint C is taken by a process (either basic or forced by the protocol) it is associated by the protocol with a vector of local checkpoint indexes, one per process, giving the minimum consistent global checkpoint containing C. The following subsection explains the core of the protocol, i.e., how it tracks non-causal message chains in order to break them. 4.1 Principle of the Protocol In order to ensure the RDT property, all non-causal message chains from C k z to C j y must be broken, if they have no causal siblings. It is the purpose of the additional local checkpoints to break such chains as explained below. Definition 4.1 : Breakable non-causal message chain A non-causal message chain is said breakable by a process P i if it contains a message whose delivery event occurs at P i after the send event of the next message in the chain. As an example, the non-causal message chain depicted on Figure 2 is breakable by P i : P i can break it by taking a local checkpoint between the send and the delivery event (depicted by a dotted box on Figure 2). As a result, the non-causal message chain is split into two message chains whose concatenation is no more a message chain. The corresponding R-path C k z! C i (x+1)! C j y is split into the two R-paths C k z! C i (x+1) and C i x! C j y that cannot be concatenated. Two cases will be examined in the next two sections: first the case when k 6= j, then the case when k = j.

5 P k P i C i x C k z m C i x+1 C k z C j y C i x+1 C k z C i x Ci x+1 C j y m 0 C j y P j b. the chain is not broken a. breakable non-causal message chain c. the chain is broken Figure 2. Non-Causal Message Chain Breakable by P i Breaking Non-Causal Message Chains from C k z to C j y, k 6= j. Consider the situation where a message m arrives at process P i. This message forms non-causal message chains with all messages sent by P i in the same checkpoint interval before the arrival of m (Figure 2). If P i decides to take a local checkpoint before the delivery of m, it breaks all such non-causal message chains. On the contrary, if P i does not take a local checkpoint before the delivery of m, none of these non-causal message chains is broken by P i. Thus, if, to the knowledge of P i, at least one of these noncausal message chains has no causal sibling, a safe strategy must force P i to take a local checkpoint before the delivery of m, to prevent the possible formation of such a non-causal message chain. This knowledge requires P i to fix the following points: i) Find all non-causal message chains that include m and that P i can break. ii) Find those non-causal message chains having causal siblings. Answering point (i) requires to answer the following two questions concerning non-causal message chains: where do they come from, where do they arrive? 1. The answer to the first question rests on the knowledge about the causal past of the message m. This knowledge is included in the array m:t DV ; in fact, for each k, either m:t DV [k] >TDV i [k] or not. a. If m:t DV [k] > TDV i [k] for some k, P i learns that there is an on-line trackable R-path C k m:t DV [k]! C i T DVi[i] and m is the first message bringing this information to P i. b. If m:t DV [k] TDV i [k] for all k, P i has previously received a message m 00 ending a causal message chain from C k m:t DV [k] to C i T DVi[i]. Upon the arrival of m 00, P i decided or not to take a local checkpoint, according to its knowledge at that time, and the arrival of m does not change anything to the previous decision. 2. The answer to the second question involves some knowledge on the future of the event send(m 0 ) (see Figure 2). But the only information available at P i when m arrives is the identity of the processes to which P i has sent messages in its current checkpoint interval. To that end, each process P i keeps an array of booleans sent to i such that, for all j (1 j n), sent to i [j] is true if and only if P i has sent a message to P j since its last local checkpoint. The set of non-causal message chains breakable by P i is thus determined by the set of pairs (P k P j ) such that (m:t DV [k] >TDV i [k]) ^ sent to i [j]. Answering point (ii) requires to check whether a noncausal message chain has a causal sibling. So, P i must be able to answer the following question: given two processes P k and P j, is there an on-line trackable R-path C k T DVi[k]! C j T DVi[j]? To answer this question, each process P i keeps a boolean matrix causal i, such that, for all (k j) (1 k j n), causal i [k j] is true if and only if, to the knowledge of P i, there is an on-line trackable R-path C k T DVi[k]! C j T DVi[j]. causal i is initialized to true on its diagonal, and nondiagonal entries are initialized to false. When P i takes a local checkpoint, all the entries causal i [i j] (j 6= i) are reset to false. When P i sends a message m, the matrix causal i is piggybacked on m. When a message m, sent by P j, is delivered to P i, causal i is updated as follows: 1. for each k such that m:t DV [k] > TDV i [k]: for every `, causal i [k `] := m:causal[k `]. In fact, P i must obtain its knowledge about causal message chains from the new checkpoint interval I k m:t DV [k]. 2. for each k such that m:t DV [k] =TDV i [k]: for every `, causal i [k `] :=causal i [k `] _ m:causal[k `]. In fact, P i adds to its current knowledge causal message chains issued from the checkpoint interval I k T DVi[k] that it was not yet aware of. Then (in both cases) causal i [j i] := true, and for every `, causal i [` i] := causal i [` i] _ causal i [` j] (transitive closure). As an example, let us consider the situation depicted Figure 3. It shows that the existence of the causal message chain 0, sibling of the non-causal message chain [m m 0 ], is known by P` (thanks to 00 ) upon the sending of m. Thus, the entry m:causal[k j] has the value true. The previous discussion shows that the test used by P i to decide whether it has to take a local checkpoint before delivering a message m is based on the following predicate:

6 C 1 9j :(sent to i [j]^ (9k :((m:t DV [k] >TDV i [k]) ^:m:causal[k j]))) This predicate means that, to the knowledge of P i, there exists a non-causal message chain from P k to P j, without causal sibling and breakable by P i. If it is evaluated to true, then the protocol forces P i to take a local checkpoint before the delivery of m. P k P l P i P j C i x; m 0 m C i x i j k l i j k l T T T T some values of m:causal Figure 3. A Causal Sibling Recorded in Matrix m:causal Breaking Non-Causal Message Chains from C k z to C j y k = j. An R-path C k z! C k y is on-line trackable if z y, by Definition 3.3. Consider the case z >y. Since y = TDV k y [k], wehave :(TDV k y [k] z) and thus the R-path C k z! C k y is not on-line trackable, whatever the checkpoint and communication pattern. It results from this observation that all such non-causal message chains must be broken somewhere in order to ensure the RDT property. Thus, it is necessary (and sufficient) to ensure that, for any k, all non causal message chains that can exist from C k z to C k z;1 are broken. P k C k z; P i Ci x;1 Ci x C k z Figure 4. Message Chain from C k z to C k z;1 Figure 4 depicts a non-causal message chain from C k z to C k z;1, breakable by process P i (with k 6= i). This chain is composed of the two sub-chains 0 and 00, where 0 is a message chain from C k z to C i x and 00 is a message chain from C i x to C k z;1 (hence the path C k z! C k z;1 is the concatenation of the two paths C k z! C i x and C i x! C k z;1 ). In the case where at least one of these two R-paths is not on-line trackable, either 0 or 00 (or both) is non-causal and has no causal sibling. Such a message chain is from C i x to C k z;1 (or from C k z to C i x ), with i 6= k; according to the discussion of Section 4.1 this chain is broken by some process on the chain that necessarily evaluates predicate C 1 to true. Consequently, this will break the non-causal chain. Thus, we have only to examine the case where both message chains 0 and 00 are causal, meaning that the non-causal message chain is breakable only by P i. It is easy to see that this situation occurs if and only if: 1. There is a causal message chain (namely 00 0 ) from C i x to C i x, and 2. This causal message chain includes a local checkpoint, in other words, one of the processes involved in this message chain has taken a local checkpoint between the delivery of a message and the sending of the next one (P k and C k z;1 in Figure 4). This situation can be causally tracked and detected by P i, thanks to the information carried by the message m 0 ending the chain 0 in the following way: point 1. above holds if and only if m 0 :T DV [i] = TDV i [i]. point 2. above holds if and only if messages carry an information indicating whether causal message chains include or not include intermediate local checkpoints. More precisely, we will say that a causal message chain [m 1 ::: m q ](q 1) is simple if for every (1 q ; 1) the event delivery(m ) occurs before and in the same checkpoint interval as the event send(m +1 ) (Figure 5). a. simple causal message chain b. non-simple causal message chain Figure 5. Simple and Non-Simple Causal Message Chains In order to track this information, each process P i keeps a boolean array simple i, such that, for all j (1 j n), simple i [j] is true if, to the knowledge of P i, all causal message chains from C j T DVi[j] to C i T DVi[i] are simple. The consistency of simple i is maintained by P i as follows: simple i [i] is permanently true. when P i takes a local checkpoint (including the initial one), it resets all the entries simple i [j] (with i 6= j)to false. When P i sends a message m, the array simple i is piggybacked on this message. When a message m from P j is delivered to P i,itcan be observed that each causal message chain [m], coming from some process P k, is simple if and only if m:simple[k]; in fact, m:simple[k] has the value of simple j [k] when P j has sent m. Thus, m:simple[k]

7 procedure take checkpoint is 8k do sent to i [k] :=false enddo; 8j 6= i do simple i [j] :=false; causal i [i j] :=false enddo; save the current local state and a copy of the array TDV i ; TDV i [i] :=TDV i [i] +1; (S0) initialization 8k do TDV i [k] :=0; 8` 6= k do causal i [k `] :=false enddo; causal i [k k] :=true; enddo ; simple i [i] :=true; take checkpoint; (S1) when P i sends a message to P j sent to i [j] :=true ; send(m T DV i simple i causal i ); (S2) when a message (m T DV simple causal) arrives to P i if C1 _C2 where C1 9j : (sent to i [j]^9k : ((m:t DV [k] >TDV i [k]) ^:m:causal[k j]), C2 ((m:t DV [i] =TDV i [i]) ^:m:simple[i]) then take checkpoint endif; % updating of control variables % 8k do case m:t DV [k]< TDV i [k]! skip m:t DV [k] >TDV i [k]! TDV i [k] :=m:t DV [k] ; simple i [k] :=m:simple[k] ; 8` do causal i [k `] :=m:causal[k `]enddo; m:t DV [k] =TDV i [k]! simple i [k] :=simple i [k] ^ m:simple[k]; 8` do causal i [k `] :=causal i [k `] _ m:causal[k `] enddo; endcase enddo ; %P s is the sender of m % causal i [s i] :=true; 8` do causal i [` i] :=causal i [` i] _ causal i [` s] enddo; deliver(m) Figure 6. The Protocol has the value true if and only if all the causal message chains received by P j from P k in P j s current checkpoint interval are simple, in which case, [m] is also simple. From this observation we get the rules P i has to observe to update simple i : 1. for every k such that m:t DV [k] > TDV i [k]: simple i [k] := m:simple[k] (recall that in that case, TDV i [k] :=m:t DV [k] is also performed). 2. for every k such that m:t DV [k] = TDV i [k]: simple i [k] :=simple i [k] ^ m:simple[k]. As a result, when a message m arrives at P i and when m:t DV [i] = TDV i [i], the value m:simple[i] indicates whether the causal message chains sent in I i T DVi[i] and ending with m are simple or not. The previous discussion shows that the test used by P i to decide whether it has to take a local checkpoint before delivering the message m is based on the following predicate: C 2 (m:t DV [i] =TDV i [i]) ^:m:simple[i] This predicate means that, to the knowledge of P i, there exists a non-causal message chain from some C k z to C k z;1, breakable only by P i. If it is evaluated to true, then the protocol forces P i to take a local checkpoint before the delivery of m. 4.2 Formal Description of the Protocol Each process P i is endowed with the following arrays whose semantics has been defined in the previous sections. TDV i : array[1::n] of integer simple i sent to i : array[1::n]of boolean causal i : array[1::n 1::n] of boolean The protocol is formally described in Figure 6. It is composed of statements performed by a process P i at initialization (S0), when it sends a message (S1), and when a message arrives (S2). The procedure take checkpoint is performed by P i when it takes a local checkpoint (basic or forced). 4.3 Proof of the Protocol To prove the correctness of the protocol, we show that every R-path C k z! C j y is on-line trackable. Thus, by

8 Definition 3.4, this will prove that the checkpoint and communication pattern including all the local checkpoints taken by the processes (basic or forced by the protocol) satisfies the RDT property. Lemma 4.1 addresses the case of non-causal message chains between two checkpoints belonging to the same process. Lemmas 4.2 and 4.3 address the other case (see Section 4.1). Due to space limitations, proofs are omitted. They can be found in [2]. Lemma 4.1 Let P i and P k be two processes, with i 6= k. There cannot exist two on-line trackable R-paths C i x! C k z;1 and C k z! C i x. Lemma 4.2 Let P i, P j, P k be three processes and x y z be three checkpoint indexes, such that: (a) there is a message m 0 from I i x to I j y (so, C i x! C j y ) and (b) there is an on-line trackable R-path C k z! C i x. Then the R-path C k z! C j y is on-line trackable. Lemma 4.3 Let P i, P j, P k be three processes and and x y z be three checkpoint indexes, such that: (a) there is an on-line trackable R-path C i x! C j y and (b) there is an on-line trackable R-path C k z! C i x. Then there is an on-line trackable R-path C k z! C j y. Theorem 4.4 (Proof of RD trackability) Every R-path C k z! C j y is on-line trackable. The following Corollary (the proof of which follows Theorem 5 in [13]) shows that our protocol gives the minimum consistent global checkpoint containing each local checkpoint without additional cost. Corollary 4.5 Let P i be a process, and x be a checkpoint index of this process. Then TDV i x defines the minimum consistent global checkpoint containing C i x. 5 Discussion 5.1 Variants of the Protocol Two weaker variants of the protocol are obtained if the array simple is omitted. The first one 5 consists in replacing the test :m:simple[i] used to evaluate C 2 with a test for any new dependency, i.e., 9k such that m:tdv [k] > TDV i [k] (like in C 1 ). So, C 2 is replaced with C 0 2 : m:t DV [i] =TDV i[i]^9k : m:t DV [k] >TDV i [k] It is easy to see that C 2 ) C 0 2 and thus, Lemma 4.1 still holds. Consequently, this protocol also achieves the 5 This variant has been suggested by Y. M. Wang in a private communication. RDT property. Note that C 1 _C 0 2 can be expressed as a single predicate, avoiding to evaluate twice the predicate 9k : m:t DV [k] >TDV i [k]. The second variant consists in replacing the predicate C 2 with the constant false. If the diagonal entries of matrices causal are maintained permanently to the value false (instead of true as previously), then the predicate C 1 is sufficient. In fact, it can be shown that Lemma 4.1 still holds: we first consider the case where the message chain has a single message m 1 (from C i x to C k z;1 ); then, upon the arrival of m 0 r, we have: sent to i [k] = true (due to m 1 ), m:t DV [k] > TDV i [k] and, by construction, :m:causal[k k]. So, the non-causal message chain 0 [m 1 ] is broken by P i. The general case, where q > 1, can be shown by induction, but, due to space limitation, we leave it to the reader. Both variants achieve Theorem 4.4 and thus ensure the RDT property, with less piggybacking but potentially forcing more local checkpoints to be taken. 5.2 A Comparison with FDAS All the protocols ensuring the RDT property define some rules that possibly force processes to take additional checkpoints upon the occurrence of some communication events. Let C P be the condition P tests to take a forced checkpoint. Given two protocols P and P 0 belonging to this class, we will say that P is more general (or less conservative) than P 0 if C P )C P 0. Several protocols belonging to this class have been previously proposed [11, 10, 12]. Wang [13] has introduced a method called Fixed-Dependency-After-Send (FDAS) and shown that the associated protocol is more general than the previous ones. We will show that our protocol (and its two variants) is more general than FDAS. In FDAS, vectors TDV are managed as in our protocol and each process maintains a single boolean variable after first send i. Its value is reset to false at the beginning of each checkpoint interval and set to true upon the first send event of the interval. Thus, its value is related with the value of our vector sent to i as follows: after first send i (9j : sent to i [j]). Before delivering a message, P i evaluates the predicate C FDAS (after first send i^9k : m:t DV [k] >TDV i [k]) Clearly, C 1 ) C FDAS ; also, C 2 ) C FDAS since m:t DV [i] = TDV i [i] ) after first sent i (m ends a causal message chain issued from I i x ) and :m:simple[i] ) 9k : m:t DV [k] > TDV i [k]. Thus (C 1 _C 2 ) )C FDAS. Similarly, (C 1 _C 0 2 ) )C FDAS and (C 1 _ false) )C FDAS. The price to be paid is in terms of increased size of piggybacked information. When compared to protocols that

9 Figure 8. R in Overlapping Group Communication Environments Client-server environment. Processes act like servers S 1 :::S n. An external client sends a request for a service to S 1 and waits for a reply. When it is delivered a request, S 1 either replies to the client or sends a message for a service to S 2 with probability 1=2. In the latter case, it waits for a reply. S 2 behaves like S 1, etc. If the request gets S n, this server processes the request and replies to S n;1. This environment is particularly interesting because the causal past of any message contains all the messages of the com-

10 Acknowledgements The authors would like to thank Yi-Min Wang (AT&T), Rob Netzer (Brown University), Mukesh Singhal (Ohio State University) and the reviewers whose comments helped improve the presentation of the paper. They also thank Paolo Fornara (University of Roma La Sapienza) for his help during simulation experiments. R. Baldoni was partially supported by the Scientific Cooperation Network of the European Community OLOS. References Figure 9. R in Client/Server Environments 6 Conclusion In this paper, we have designed a communicationinduced checkpointing protocol that ensures the RDT property while producing less forced checkpointing than any protocol of the FDAS family. This has been achieved by a subtle tracking of causal dependencies on already taken checkpoints. This tracking has been used to prevent the occurrence of hidden dependencies; the prevention is done by inducing a process to take a forced checkpoint only when necessary, according to its current knowledge of the past of the computation (this knowledge being encoded and piggybacked on application messages). The reduction of forced checkpoints taken by the proposed protocol with respect to FDAS has been quantified by a simulation study in different computational environment and it is never less than 10%. A nice feature of the proposed protocol is that it efficiently associates on-the-fly with each local checkpoint C the minimum consistent global checkpoint to which C belongs. [1] Babaoğlu, Ö., Fromentin, E. and Raynal, M., A Unified Framework for the Specification and Run-time Detection of Dynamic Properties in Distributed Computations, Journal of Systems Software, 33: , [2] Baldoni, R., Hélary, J.M., Mostefaoui, A and Raynal M., A Communication-Induced Checkpointing Protocol that Ensures Rollback-Dependency Trackability, IRISA Research Report 1076, January www access: ftp://ftp.irisa.fr:/techreports/1997. [3] Chandy, K.M. and Lamport, L., Distributed Snapshots: Determining Global States of Distributed Systems, ACM Transactions on Computer Systems, 3(1):63-75, [4] Cohen E., Wang, Y.M., Suri G., When Piecewise Determinism Is Almost True, Proc. Pacific Rim Int. Symp. on Fault-Tolerant Systems, 1995, pp [5] Elnozahy, E.N., Johnson, D.B. and Wang, Y.M., A Survey of Rollback-Recovery Protocols in Message-Passing Systems, Technical Report CMU-CS , Carnegie- Mellon University, [6] Koo, R., and Toueg, S. Checkpointing and Rollback- Recovery for Distributed Systems, IEEE Transactions on Software Engineering, 13(1):23-31, [7] Lamport, L. Time, Clocks and the Ordering of Events in a Distributed System, Communications of the ACM, 21(7): , [8] Netzer, R.H.B., and Xu, J., Necessary and Sufficient Conditions for Consistent Global Snapshots, IEEE Trans. on Parallel and Distributed Systems, 6(2): , [9] Randell, B. System Structure for Software Fault-Tolerance, IEEE Trans. on Soft. Engineering, SE1(2): , [10] Russell, D.L., State Restoration in Systems of Communicating Processes, IEEE Trans. on Software Engineering, SE6(2): , [11] Strom, R. E., Bacon, D. F. and Yemini, S. A., Volatile Logging in n-fault-tolerant Distributed Systems, Proc. IEEE Fault-Tolerant Computing Symp., pp.44-49, [12] Wu, K. L., and Fuchs, W. K., Recoverable Distributed Shared Virtual Memory, IEEE Trans. on Computers, 39(4): , [13] Wang, Y.M., Consistent Global Checkpoints That Contain a Given Set of Local Checkpoints, to appear in IEEE Transactions on Computers, 46(4), April 1997.