denition of distributed computations with synchronous than originally stated. Key words: Distributed computation { Message passing Communication

Transcription

1 Synchronos, asynchronos, and casally ordered commnication Bernadette Charron{Bost Laboratoire d'informatiqe, Ecole Polytechniqe, Palaisea Cedex, France Friedemann Mattern Department of Compter Science, Technical University of Darmstadt, Alexanderstr. 10, D Darmstadt, Germany Gerard Tel Department of Compter Science, University of Utrecht, P.O. Box , 3508 TB Utrecht, The Netherlands Bernadette Charron-Bost received her Ph.D. degree in compter science (1989) from Paris 7 University, France. She is a researcher at the CNRS (French National Center of Scientic Research), in the Compter Science Department of the Ecole Polytechniqe. Her research interests inclde program verication and semantics, distribted programming, and distribted algorithms. charron@lix.polytechniqe.fr Friedemann Mattern received the Diploma in compter science from Bonn University and the Ph.D. degree from the University of Kaiserslatern in 1983 and 1989, respectively. From 1991 to 1994 he was a professor of compter science at the University of Saarland in Saarbrcken. He is now a professor at the department of compter science of the Technical University of Darmstadt. His crrent research interests inclde programming of distribted systems, distribted applications, and distribted algorithms. mattern@informatik.th-darmstadt.de Gerard Tel received his M.Sc. and Ph.D. degree from Utrecht University. He is Assistant Professor at Utrecht University, teaching algorithms with emphasis on distribted algorithms. His main research interest is in message passing distribted algorithms. Dr. Tel is the athor of the book Introdction to Distribted Algorithms, Cambridge University Press, gerard@cs.r.nl Smmary. This article stdies characteristic properties of synchronos and asynchronos message commnications in distribted systems. Based on the casality relation between events in comptations with asynchronos commnications, we characterize comptations which are realizable with synchronos commnications, which respect casal order, or where messages between two processes are always received in the order sent. It is shown that the corresponding comptation The work of Gerard Tel was spported by the ESPRIT II Basic Research Actions Program of the EC nder contract no (proect ALCOM) classes form a strict hierarchy. Frthermore, an axiomatic denition of distribted comptations with synchronos commnications is given, and it is shown that several informal characterizations of sch comptations are eqivalent when they are formalized appropriately. As an application, we se or reslts to show that the distribted termination detection algorithm by Dikstra et al. is correct nder a weaker synchrony assmption than originally stated. Key words: Distribted comptation { Message passing { Synchronos commnication { Asynchronos commnication { Casal order { Distribted termination detection { Distribted system { Casality relation { Commnication 1 Introdction Messages, their transmission modes, and the semantics of commnication play an important role in distribted systems since messages are the only means by which processes can exchange data and can synchronize their actions. A priori, commnication in distribted systems is not reliable messages can be lost becase of commnication failres, dplicated becase of retransmissions, or their contents can be garbled or destroyed. Sophisticated techniqes and protocols have been devised which cope with these problems and garantee reliable message delivery to the application. However, even if one spposes that message commnication is reliable (as will be done in this article), distribted systems may behave in dierent ways. For example, messages may be transmitted synchronosly or asynchronosly, and messages might be received in the order in which they were sent or received ot of seqence. In order to help the programmer to implement ecient and well-strctred distribted programs and algorithms, distribted operating systems and program- 1

2 ming langages provide varios commnication mechanisms. Examples are selective receive statements, which block the receiver ntil a sitable message is available on one of several ports or channels, or remote procedre call abstractions, which block the sender ntil a retrn message is available. Non-blocking send instrctions, which do not force the sender to wait ntil the message is delivered at its destination, are also very common. Usally, non-blocking commnication mechanisms are called \asynchronos", whereas blocking commnication mechanisms are called \synchronos" [41]. 1 We will call distribted comptations that make exclsive se of asynchronos send instrctions A-comptations, and comptations where commnication is done solely in synchronos mode S-comptations. The distinction between synchronos and asynchronos refers only to the commnication between processes; the nderlying system model we consider here is in all cases asynchronos in the sense that we do not assme the existence of synchronized clocks or xed commnication delays. It is widely accepted that neither of the two commnication modes \synchronos" and \asynchronos" is generally sperior to the other. While asynchronos commnication is less prone to deadlocks and often allows a higher degree of parallelism (since the sender can proceed while the message is still being delivered), its implementation reqires complex ber management and ow control mechanisms. Frthermore, algorithms making se of asynchronos commnications are often more diclt to develop and to verify than algorithms working in a synchronos environment. It is a well-known fact, however, that it is possible to simlate one mode with the other. In synchronos mode, explicit bers or intervening ber processes can be sed to decople the sender of a message from the receiver, ths simlating asynchronos mode. 2 In asynchronos mode, synchronos mode can be simlated by waiting for an explicit acknowledgement immediately after asynchronosly sending a message. The control of A-comptations sally reqires more sophisticated algorithms than the control of S- comptations. This is de to their higher degree of parallelism and non-determinism, and de to messages that can be in transit in A-comptations. Examples of control problems for distribted comptations inclde achieving mtal exclsion, termination or deadlock de- 1 Sometimes, however, a dierence between the two notions \blocking" and \synchronos" (or \non-blocking" and \asynchronos", resp.) is made in the literatre. This is the case, for example, for the MPI message passing interface which is becoming a de facto standard for message-based commnication [24]. Here, a blocking send primitive does not retrn ntil the message has been copied ot of the sender's ber, whereas a non-blocking send can retrn immediately [16]. Since we abstract from implementation aspects sch as message bering, we are not concerned with this isse here. 2 Tre simlation reqires an nbonded amont of ber storage, however. tection, compting consistent snapshots, and collecting garbage obects. Two natrally arising qestions of practical importance are: 1. Are control algorithms that are correct for A- comptations in general also correct for the control of S-comptations? 2. And, conversely, is it possible to adapt distribted control algorithms for S-comptations to A- comptations? Informally, S-comptations are often regarded as a special case of A-comptations, namely comptations where the commnication channels always appear to be \empty". Ths, one wold expect that a distribted algorithm designed for the asynchronos case remains indeed correct when exected on a synchronos system or applied to an nderlying S-comptation. Formally, an assertional proof of the algorithm (showing an invariant implying its correctness) remains valid if the proof rles for A-comptations are replaced by proof rles for S-comptations (see Schlichting and Schneider [38]). However, one shold be aware of the fact that an algorithm working correctly in the asynchronos case might deadlock in the synchronos case. We will indeed show that not all A-comptations are realizable nder synchronos commnications. Clearly, the safety properties of a synchronos algorithm are sally not preserved when the algorithm is exected in an asynchronos environment (see Gribomont [23]). A good example is the well-known distribted termination detection algorithm of Dikstra, Feien, and van Gasteren [19]. This algorithm is safe if message commnication of the nderlying comptation is synchronos, 3 bt fails if it is asynchronos becase it does not consider messages in transit. Hence, disregarding the deadlock problem, it seems that A-comptations are more general than S-comptations in the sense that a distribted algorithm designed for an asynchronos environment will work in a synchronos environment bt not necessarily vice-versa. The preceding observations show that for the theory of distribted algorithms a precise characterization of synchronos and asynchronos commnication modes and an analysis of their relations is of great interest. This article shold contribte to the nderstanding of those fndamental concepts and relations. In particlar, we will formally dene the notion of a distribted comptation (Section 3.1) and demonstrate that the hierarchy of S-comptations and A-comptations indicated by the preceding remarks can be rened in a sensible way. Besides FIFO and non-fifo message passing disciplines, we will consider in particlar so-called casally ordered comptations (Section 3.2). 3 The athors se the term \instantaneos" instead of \synchronos". 2

3 These comptations respect the casality relation between send events and receive events in a similar way than do S-comptations bt they can be implemented withot blocking. It will be shown that they lie between S-comptations and FIFO-commnication based comptations in the hierarchy of distribted comptation classes (Section 3.3). We will give dierent characterizations of casally ordered comptations and prove their eqivalence (Section 3.4). However, it is eqally interesting to characterize S-comptations accordingly (Section 4), to formally dene the semantics of synchronos commnications (Section 5), and to analyze the conditions nder which A-comptations can be realized in synchronos mode. Parts of this article consist of a strctred synthesis of known (bt sometimes nproven) reslts, pt into a nifying formal framework. Only with sch a framework it is possible to axiomatically characterize and formally compare varios distribted comptation and commnication models. Even then, however, some proofs remain non-trivial and reqire a certain amont of technical sbtleties. They show, on the other hand, that intitive argments (sch as \message arrows in space-time diagrams of S-comptations are vertical") have a formal stication. Besides this, we also present several original reslts. Among other things we prove that a simpler and apparently stronger \message order" property is eqivalent to the classical casal order property (Section 3.4), we give a new criterion (the \crown criterion", Section 4.3) to decide whether a distribted comptation can be realized with synchronos commnications, and we show that a well-known termination detection algorithm remains correct nder weaker commnication reqirements than originally stated (Section 6). Before we do all this on a more formal basis, we shall rst introdce in the next section some fndamental concepts and characterize S-comptations and A-comptations from an informal point of view. 2 Distribted Comptations: an Informal View The intention of this section is to provide an intitive nderstanding of the main characteristics of synchronos and asynchronos message transmissions. The basic notions and main featres discssed in this article are introdced in an informal manner, sing diagrams and examples. Formal denitions and proofs as well as more precise frther characterizations are fond in later sections. 2.1 Events and Space-Time Diagrams A distribted system consists of seqential processes commnicating solely by messages. The behavior of each process is controlled by a local algorithm which determines the local seqence of actions and the reaction of the process to incoming messages. The concrrent (and coordinated) exection of all local algorithms forms a distribted comptation. In an abstract setting, a distribted comptation is determined by the types and the relative order of atomic actions called events occrring at the processes. Usally, events are classied into three types according to Lamport [28]: send events, receive events, and internal events. A send event cases a message to be sent, and a receive event cases a message to be received and the local state to be pdated by the content of the message. A send event s and a receive event r are said to correspond if the same message, sent by s, is received by r. Internal events case only a change of the local process state they are not of maor concern here since we are mainly interested in the casal relation of events on dierent processes, which is cased by the exchange of messages. P 1 P 2 P 3 c v bv v a dv v h 1 v v e g Fig. 1. A space-time diagram of a distribted comptation Lamport [28] pointed ot that distribted comptations can be visalized sing space-time diagrams, of which Fig. 1 shows an example. 4 The events of each process are depicted as dots located on a process line. On sch a line, an event e is drawn to the left of an event e 0 if and only if e \happens before" e 0 on the corresponding process. Ths \time" rns from left to right on a process line. Messages are depicted as arrows connecting send events with their corresponding receive events. In an A-comptation (where send events are decopled from their corresponding receive events) it seems obvios that the exection of an event e can casally aect another event e 0 if and only if there is a path in the diagram from e to e 0. This path mst follow the direction of the arrows and rn from left to right on process lines. The existence of sch a path indces a casal 4 \It is tre that there are certain implicit dangers in sing sch graphical representations, becase in every geometrical diagram time appears to be misleadingly spatialized. On the other hand, sch diagrams, provided we do not forget their symbolic natre, have a denite advantage..." This philosophical critiqe by Milic Capek [13] on Minkowski's relativistic space-time concept and its simplied models also applies here; as Lamport observes [28], those models are indeed strctrally very similar to spacetime diagrams as we se them. v f 3

4 relationship on the set of events. Since this casality relation is transitive and cycle-free (it is not possible that two events mtally depend on each other), it forms an irreexive partial order which will be denoted as \". Appropriately, e e 0 can be read \e can casally aect e 0 ", \e 0 potentially depends on e", \e happens before e 0 ", or \e 0 knows abot e". Message arrows need not be drawn in sch a way that receive events are necessarily located to the right of their corresponding send events. Sch a drawing, however, is always possible by introdcing a global time frame and drawing a time axis in parallel to the process lines. The vertical proection of an event on the time axis then represents the time at which the event is exected. Since the sending of a message never occrs later than its receipt, message arrows do not go from right to left in sch diagrams. 2.2 Synchronos and Casally Ordered Comptations In general, it is not immediately clear from a spacetime diagram whether it depicts a comptation with synchronos commnications (i.e., an S-comptation) or a comptation with asynchronos commnications (i.e., an A-comptation). Bt what, in fact, is the dierence between \synchronos" and \asynchronos" here? Literally, \synchronos" signies \at the same time" which wold mean that in an S-comptation a send event shold always happen simltaneosly with its corresponding receive event. Bt this is not possible in reality for physical reasons. 5 Therefore, in order to characterize S-comptations we conne orselves to \approximate" instantaneos message transmission in the sense that a comptation which exhibits (or which cold exhibit) a phenomenon that cannot be observed if message transmission were instantaneos is not \synchronos" and hence not called an S-comptation. This rles ot the comptation of Fig. 1: If message transmission were instantaneos, the rst message sent by P 2 (event e) mst be received at P 1 (event f) before a later message sent by P 2 to P 1 (event g) is received (event h). As a matter of fact, non-fifo-comptations are therefore not S-comptations. Consider now a distribted comptation formed by two main processes P 1 and P 2 and a control process P 3, connected as depicted in Fig. 2. Assme that P 1 conts the messages it sends to P 2 and P 2 conts the messages it receives from P 1. If message transmission were instantaneos, the two conters shold show the same vale at any instant in time. Therefore, if P 3 rst asks P 1 for the crrent vale of its send-conter S and 5 We qote again Milic Capek [12]: \There is an pper limit to the transmission of any casal action: this is the speed of electromagnetic waves. This is, as Pal Langevin said, the speed limit of casality. Ths there are no instantaneos transmissions in natre... the eect is never contemporaneos with its case." - - P - 1 v v So S P 2? SS? S SSw?? S?? P 3 Fig. 2. A controller process P3 and two main processes P1 and P2 then (after receipt of the reply) P 2 for the vale of its receive-conter R, one wold expect that (if message transmissions between P 1 and P 2 were instantaneos) all messages sent by P 1 when it replies to P 3 have been received at P 2 when P 2 receives P 3 's reqest message. Hence, P 3 shold always observe S R (note, however, that S = R cannot be garanteed). If message transmission is not instantaneos, however, P 3 cold observe S > R in the above-mentioned scenario. Figre 3 explains this phenomenon. The reqest message m 3 sent after the receipt of the reply message m 2 (sent after message m 1 ) arrives at P 2 before m 1. Ths, indirect commnication via P 3 was faster than direct commnication between P 1 and P 2 the comptation violates the \triangle ineqality", so to speak. A distribted comptation which does not show the eect that a message is indirectly (or directly) bypassed by a chain of other messages, is called a casally ordered comptation (or CO-comptation for short). This notion will be dened formally in Section 3.2. P 2 P 3 P 1 r 3 r 1 * 6m 3 m 1 r 2 s m s 1 s 2 Fig. 3. A comptation that is not casally ordered Obviosly, casal order is a generalization of FIFO in the sense that a CO-comptation is always a FIFOcomptation (bt not vice-versa as Fig. 3 shows). In CO-comptations, messages (i.e., their delivery seqences at the receiving processes) respect the (possibly indirect) casality relation among their send events. This is of importance to many practical applications where casal consistency (in the sense of consistent views or snapshots) mst be garanteed. Renesse [36] presents a collection of sch applications and also discsses the se of a generalization of the casal order principle to broadcast commnications. It shold be emphasized that in a non-co- 4

5 comptation there exists a single message that is overtaken by another message or a chain of messages (see Fig. 3 where m 1 is overtaken by m 2 / m 3 ). As the S-comptation depicted in Fig. 4 shows, it is qite possible that a chain of messages between two processes P 1 and P 2 is \overtaken" by a message (or another chain of messages) even if message transmission is instantaneos. The reason is that the intervening process P 3 may ber the messages for an arbitrary time and therefore essentially simlate asynchronos message transmission between P 1 and P 2. P 2 P 3 P 1 r 0 s x 6 r Fig. 4. A synchronos comptation with an intervening ber P3 Retrning to the system of Fig. 2, S > R shold also be nobservable in an S-comptation if P 3 rst asks P 1, and then (withot waiting for P 1 's reply) P 2 to take a local snapshot of their conters. If later these snapshots are compared, the vale of the snapshot of S shold always be smaller than or eqal to R's snapshot (becase P 1 and P 2 receive P 3 's reqest message instantaneosly). The space-time diagram of Fig. 5 depicts a scenario which shold therefore not happen in an S- comptation: The snapshot of S wold be 1 (when the \snapshot event" r 1 happens, the send event s 3 of the message from P 1 to P 2 did already happen locally) bt R's snapshot wold be 0 (the receive event r 3 of the message did not yet happen when the snapshot event r 2 of process P 2 happens). Notice that (if one abstracts from process names) Fig. 5 looks similar to Fig. 3, bt becase events s 3 and r 3 are swapped, there are no indirect message overtakings. Hence Fig. 5 depicts a COcomptation which is nevertheless able to prodce an eect not observable with S-comptations. The reason for the non-synchronos behavior is that in some sense (namely in \real time order") r 1 is \later" than r 2 althogh the order is reversed for the corresponding send events (i.e., s 1 happens earlier than s 2 ). This \message crossing" is disclosed by the message from P 1 to P 2 which is transmitted between the two receive events r 2 ; r 1. Even withot having yet precisely dened S- comptations, or discssion indicates that the hierarchy S-comptations CO-comptations FIFOcomptations general A-comptations is strict. We will come back to this frther down in Section 3. s P 1 P 2 P 3 r 2? 6r 3 s 1 s 2 s 3 r 1 * Fig. 5. A casally ordered comptation 2.3 Cycles and Schedles In order to characterize S-comptations more precisely, we shall now look at a denition given by Boge [10]: \A system has synchronos commnications if no message of a given type can be sent along a channel before the receiver is ready to receive (that is, in a state where the next action may be a reception of) a message of this type on the channel. For an external observer, the transmission then looks instantaneos and atomic. Sending and receiving a message correspond in fact to the same event.\ If the comptation of Fig. 3 wold be exected on a system with synchronos commnications as characterized by Boge, s 1 cold not take place, becase P 2 is not ready to receive the message. Process P 2 mst rst execte r 3, bt this event depends on s 3, which depends on r 2, which depends on s 2, which in trn depends on s 1. Hence, we have a deadlock sitation no message can be schedled rst becase m 1 can only be schedled after m 3, m 3 can only be schedled after m 2, and m 2 can only be schedled after m 1. The schedling of entire message transmissions (rather than individal events) seems to be a natral conseqence of the atomicity of a send and its corresponding receipt. A message schedle implies a schedle of events in which the corresponding commnication events are exected consectively, withot being separated by other events. The two events ths simlate a single commnication event, 6 distribted over sender and receiver. This notion will be formalized in Denition 3.6 ("RSC-comptations"). The comptation of Fig. 5 exhibits the same cyclic dependency on messages as the comptation of Fig. 3 (despite the fact that events s 3 and r 3 are swapped) and is therefore also not realizable in synchronos mode. Figre 6 depicts yet another (rather typical) deadlock sitation. Here, all processes are blocked in a send instrction since no process is ready to receive a message be- 6 Baeten and Weiland [3], for example, dene a synchronos commnication as the reslt of the simltaneos exection of two corresponding actions, together forming a single commnication action. This view is also implied by the semantics of CCS [32] and CSP [25] where the meaning of two matching commnication actions (a and a in CCS, or p!e and q?x in CSP) is given by a single \atomic" action (the silent action in CCS and the assignment statement x := e in CSP). 5

6 fore it has sent its message. It shold be noted, however, that the comptations depicted in Figres 3, 5, and 6 are not deadlocked when commnications are asynchronos. For example, breaking the atomicity of m 1 in Fig. 3 yields s 1 ; s 2 ; r 2 ; s 3 ; r 3 ; r 1 as a possible global schedle of events for the comptation. s 2?? s 1 r 2 s 3 r 1 * r 3 Fig. 6. Another casally ordered comptation Interestingly, for all three space-time diagrams st considered, it is possible to constrct the same cyclic dependency r 1 s 2 r 2 s 3 r 3 s 1 r 1 on events by reversing the direction of appropriate message arrows (e.g., m 1 in Fig. 3) and swapping the associated send-receive events. It shold be clear that if a cycle can be constrcted in this way, there is no rst message transmission which can be schedled as a whole. Hence, space-time diagrams containing sch cycles are invalid in the synchronos mode. Conversely, if sch a cycle cannot be constrcted, it shold be possible to nd a message schedle. We therefore come to the conectre that an A-comptation is realizable in synchronos mode (i.e., there exists an eqivalent S-comptation with the same events and messages) if and only if no sch cycle can be formed. Notice that a schedle of messages is consistent with the casal precedence order \" on events dened earlier. In fact, a message schedle directly yields a linearization of \" sch that the direct sccessor of a send event is its corresponding receive event. We shall come back to these characterizations of S-comptations in later sections; in particlar we will introdce the notion of a \crown" (Denition 4.1) which formalizes the idea of message schedling cycles. 3 Distribted Comptations In this section varios classes of distribted comptations are formally dened. Frthermore, it is shown that dierent reqirements on the message transmission disciplines with respect to the preservation of the casality relation yield a hierarchy of distribted comptation classes. As mentioned before, we consider reliable distribted systems only. 3.1 Comptations with Asynchronos Commnications A distribted system consists of processes P 1 ; : : :; P n, commnicating only by exchanging messages. We do not make any assmption on the relative speed of processes or on message transmission delays, hence we consider only so-called \asynchronos systems". A comptation on sch a system basically consists of a tple of local comptations, one for each process, and a casality relation on the set of events, which is indced by the exchange of message. Each process P i rns a seqential program, whose exection is modeled by a nite seqence C i of events. This seqence is called a local comptation (of P i ). A comptation C of a distribted system is composed of local comptations C 1 ; : : :; C n performed by P 1 ; : : :; P n, respectively. We write a b to denote that events a and b take place at the same process, i.e., a b if there is an i sch that a 2 C i and b 2 C i. For a given comptation C, let? = f(s; r) 2 C i C : s corresponds to rg denote the set of corresponding pairs of commnication events. Since we model pointto-point commnications, where only a single message is either sent or received in any commnication event, we often also say that r corresponds to s if s corresponds to r. We assme that a process does not commnicate with itself, i.e., corresponding send and receive events are located on dierent processes. 7 With point-to-point commnications, each commnication event has at most one corresponding commnication event. Since every message which is received mst have been sent, we will only consider n-tples C = (C 1 ; : : :; C n ) sch that, for every receive event r in C, there exists a send event s sch that (s; r) 2?. If, in addition, for each send event there is a corresponding receive event (i.e., send and receive events are in a one-to-one correspondence), then there is no message in transit at the end of the comptation. Sch comptations are called complete comptations. We shall now precisely dene the casality relation \" of an n-tple of local comptations. As each P i is a seqential process, the events of this process are totally ordered by their occrrence in the seqence C i. This order, which we denote by i, implies a casal order on the events of process P i in the sense that an earlier event may aect a later event on the same process. Clearly, i is an irreexive total order (on the events of a single process C i ). Casal dependencies between events at different processes are indced by message exchanges. If (s; r) 2?, then the receive event r casally depends on the corresponding send event s. All casal dependencies 7 This does not exclde application of or reslts to algorithms where processes send messages to themselves. Often, however, one wold model sch a message by a single internal event or a send event immediately followed by the corresponding receive event ths reecting the sal implementation shortct of self-sends. 6

7 between the events of C are indced by transitivity from the relations 1 ; : : :; n and?, as formalized in the following denition originally stated by Lamport [28] for his \happens before" relation. Denition 3.1 (Casality relation) The casality relation in C is the smallest relation that satises the following three properties. AS1 If a i b, then a b. AS2 If (s; r) 2?, then s r. AS3 If a b and b c, then a c. The reexive closre of will be denoted by, i.e., a b () (a b _ a = b). Two distinct events are said to be concrrent if they are not ordered by. When a distribted comptation is exected, an idealized global observer (who has an instantaneos global view of the comptation) can perceive the events only in an order which is consistent with the casality relation. In other words, if an event a cold aect another event b, then a mst happen before b in \real time". This explains why the casality relation of a distribted comptation mst be cycle-free and motivates the following denition. Denition 3.2 (A-comptation) A comptation with asynchronos commnications (or an A-comptation for short) of a distribted system consisting of processes P 1 ; : : :; P n is an n-tple C = (C 1 ; : : :; C n ) of local comptations together with a set? of corresponding commnication events, for which the casality relation is a partial order. It shold be observed that the set? is essential for a sitable denition of A-comptations. Withot it, an A-comptation wold st be a partially ordered set of events, together with a partitioning into totally ordered sbsets C i. In that case, however, Fig. 3 and a similar bt dierent scenario where message m 1 is removed (and events s 1 and r 1 ths replaced by internal events) wold represent the same comptation. This also shows that space-time diagrams are more than st Hasse diagrams 8 of the casality relation it is in fact not possible to apply the so-called transitive redction scheme becase this wold possibly remove messages and ths change the comptation. The crcial point in the denition of an A- comptation is that the transitive relation, as dened in Denition 3.1, is reqired to be a partial order. This garantees that it is cycle-free, a basic reqirement for any sensible notion of (potential) casality. It also garantees that space-time diagrams can be drawn in sch a way that the process axes and all message arrows go from left to right. 8 For a denition of Hasse diagrams we refer to appropriate textbooks on order theory (e.g., [17]). The term \asynchronos" in the denition of A- comptations is motivated by property AS2. This axiom is responsible for the basic asymmetry between send events and receive events receive events depend on their corresponding send events, bt not vice-versa. Hence, a receive event is blocking it cannot be exected before the corresponding send event is exected. This is not the case for a send event; send events can be exected withot waiting for the receiver. We will see in Section 5.1 that it is in fact only axiom AS2 which has to be replaced by a symmetric variant to formalize comptations with synchronos commnications (i.e., S-comptations) accordingly. 3.2 Order Preserving Properties and Constraints In practice, distribted comptations are often \less asynchronos" than allowed by Denition 3.2. This reslts from imposing certain constraints on the sending or receipt of messages. For example, it might be the case that a message can only be received if all messages sent earlier along the same commnication channel have already been received, or a message might only be sent if the previosly sent message has been acknowledged. Sch constraints cold arise at dierent levels: they cold reslt from a particlar implementation of a commnication protocol (e.g., no bering of messages), or they cold be imposed by the semantics of a distribted programming langage in order to simplify correctness proofs of distribted programs. Interestingly, it trns ot that the most important and commonly sed restrictions can be neatly stated as properties of the casality relation. This will be shown in the following, where we assme that all comptations are complete (i.e., for each send event there is a niqe corresponding receive event and vice-versa there are no messages in transit at the end of the comptation). In the design of distribted algorithms (e.g., the wellknown snapshot algorithm by Chandy and Lamport [14]) it is often assmed that any two messages exchanged between the same two processes are received in the order in which they have been sent. This assmption, referred to as the FIFO (i.e., \rst-in, rst-ot") assmption, is modeled by the following condition. Denition 3.3 (FIFO) A comptation C is called a FIFO-comptation if for all (s; r) and (s 0 ; r 0 ) 2? s s 0 r r 0 s s 0 9 = ; =) r r0 : The FIFO-property can be strengthened by dropping the s s 0 condition, thereby reqiring that all messages sent to the same process are received in an order consistent with the casal order of the corresponding send events. The order in which the messages are received 7

8 may be arbitrary in case the send events of the messages are concrrent. This property was called \casal ordering" by Schiper et al. [37]; for broadcast commnications it was dened by Birman and implemented in the ISIS system [6, 7]. Denition 3.4 (CO) A comptation C is called a COcomptation (Casally Ordered) if for all (s; r) and (s 0 ; r 0 ) 2? r r 0 s s 0 =) r r 0 : One wold expect that the CO-condition trivially holds for exections where a receive event always happens simltaneosly or \synchronosly" (i.e., at the same instant) with its corresponding send event. This is indeed the case, comptations which allow sch (idealistic) exections are called realizable with synchronos commnication (or RSC for short). Before we can formally dene them, the notion of an exection mst be formalized. Since events are abstractions of atomic actions, an exection of a distribted comptation can be modeled as a feasible schedling of the events of the comptation, which is a total order that extends the partially ordered set (or poset, for short) (C; ). Note that this means that instead of execting two casally nrelated events trly concrrently, they can always be exected seqentially in either order. In a natral way, an exection with instantaneos message transmission is modeled by an irreexive linear extension 9 where a receive event is the direct sccessor of its corresponding send event, as stated in the following denition. Denition 3.5 (Non-separated linear extension) A linear extension (C; <) of an A-comptation is called non-separated if for each pair (s; r) 2? the interval fx 2 C : s < x < rg is empty. The property of comptations where corresponding send and receive events may happen simltaneosly in an exection is now formally dened in the following way. Denition 3.6 (RSC) A comptation C is called an RSC-comptation (Realizable with Synchronos Commnication) if there exists a non-separated linear extension of the poset (C; ). Note that in sch a linear extension two corresponding commnication events always appear either both later than another event, or both earlier than another event. With respect to <, corresponding commnication events ths have a common past and a common ftre they appear as if they occrred simltaneosly. 9 Recall that a linear extension of a partial order (C;) is a total order (C; <) sch that a b ) a < b. 3.3 The Hierarchy of Distribted Comptation Classes After having formally dened the most important comptation classes, we now show that there is a strict hierarchy RSC CO F IF O A-comptations, as already indicated in Section 2.2. Indeed, the following theorem is an immediate conseqence of the varios de- nitions given in the previos section. Theorem 3.7 (Hierarchy) C is RSC ) C is CO ) C is FIFO. Proof. First assme C is RSC. Consider (s; r); (s 0 ; r 0 ) 2? for which s s 0. As C is RSC, there exists a nonseparated linear extension < of. As < extends, we have s < r and s < s 0. As < is total and non-separating, r < s 0 follows (note that if conversely s 0 < r wold hold, then (s; r) wold be \separated" by s 0 ). Becase s 0 r 0, this implies r < r 0, and as < extends and events of one process are linearly ordered, r r 0 implies r r 0. Ths C is CO. Next assme C is CO. From Denition 3.3 and Denition 3.4 it follows directly that C is FIFO. 2 Theorem 3.8 (Strict hierarchy) There exist A-comptations that are not FIFO. There exist FIFO-comptations that are not CO. There exist CO-comptations that are not RSC. Proof. Examples of sch comptations are shown in Figres 1, 3, and 5. It is left to the reader to verify that these comptations have (or do not have) the indicated properties Dierent Characterizations of CO- Comptations The only commnication disciplines which are traditionally considered for the design of distribted algorithms are synchronos commnications, FIFO, and asynchronos commnications. Since the CO-comptations fall between FIFO-comptations and comptations which are realizable with synchronos commnications (i.e., RSC-comptations), it is interesting to stdy the relevance of the CO-property. In Section 6, where we shall consider the problem of termination detection as an example of a distribted control problem, we will prove that the CO-property is in fact scient for the correctness of a classical termination detection algorithm originally designed for the class of S-comptations (which, as will be shown frther down, are in some sense eqivalent to RSC-comptations). This motivates frther investigations of the CO-property, of which some alternative characterizations will be discssed in this section. First, recall that the CO-property (Denition 3.4) was derived from the FIFO-property (Denition 3.3) st by 8

9 dropping the s s 0 condition, ths \globalizing" the casality preserving property of FIFO commnication channels. One cold now try to strengthen the COproperty frther by also removing the condition regarding the location of the receive events. In order to obtain a signicant class of comptations we allow two receive events on dierent processes to be concrrent if their corresponding send events are casally related. We reqire, however, that these receive events are not ordered conversely to their corresponding send events. The reslting type of comptations will be referred to as \message ordered". Denition 3.9 (MO) A comptation C is called an MO-comptation (Message Ordered) if for all (s; r) and (s 0 ; r 0 ) 2? s s 0 =) :(r 0 r): Note that in Denitions 3.3 (FIFO) and 3.4 (CO) the term r r 0 on the right-hand side of the implication can be replaced by :(r 0 r) (becase by r r 0 the receive events are linearly ordered) ths emphasizing the formal similarity of those denitions to the MO-denition. Srprisingly, the class of comptations dened by the MO-property is identical to the class of COcomptations, as shown by the following theorem. Theorem 3.10 (MO = CO) A comptation is CO if and only if it is MO. Proof. First assme C is a MO-comptation. Consider (s; r); (s 0 ; r 0 ) 2? for which s s 0 and r r 0. As C is MO, :(r 0 r), bt r r 0 implies r r 0 _r 0 r. Hence r r 0, which implies that C is CO. Next we shall show that if there are two messages violating the order reqired for the MO-property, then sch a pair can be fond for which the receive events occr on the same process, showing in this manner that the comptation is not CO. Assme C is a CO-comptation, bt C is not MO, i.e., there are two pairs (s; r) and (s 0 ; r 0 ) of corresponding events sch that s s 0 and r 0 r. It follows from the denition of that there exists a chain a 0 ; : : :; a k of events, sch that r 0 = a 0, r = a k, and for each < k, a i a +1 for some i, or (a ; a +1 ) 2?. 1. If no (a ; a +1 ) 2? occrs in this chain, then r 0 r. As C is a CO-comptation, this together with s s 0 implies r r 0, which contradicts the assmption. 2. Otherwise, let (a i ; a i+1 ) = (; ) be the last pair of? in this chain. Ths r and r, and from s s 0, (s 0 ; r 0 ) 2?, and r 0 we obtain s. As C is a CO-comptation and r, this implies r, which contradicts r. It follows that each CO-comptation is also MO. 2 The MO-characterization shows directly that in COcomptations a single message (s; r) cannot be overtaken by a chain of other messages, as already stated in Section 2.2. The reason is that for any message (s 0 ; r 0 ) which belongs to sch a hypothetical bypass chain, one has necessarily s s 0 r 0 r which, by transitivity, contradicts the MO-property. Conversely, if the MO-property does not hold, then there exist two messages (s; r) and (s 0 ; r 0 ) with s s 0 and r 0 r, and hence s s 0 r 0 r. This means that (s 0 ; r 0 ) is part of a chain which overtakes message (s; r). Informally, one cold ths say that CO-comptations are characterized by the fact that indirect commnications always take longer than direct commnications. The following corollary smmarizes these ndings. Corollary 3.11 (Triangle ineqality) A comptation is CO if and only if no message is bypassed by a chain of other messages. As an application of this characterization consider a distribted program written in a synchronos message passing langage. If the sender shold be decopled from the receiver, one might add to each process an inpt ber and an otpt ber, both realized by a FIFO message qee. A send event now corresponds to the insertion of a message into the otpt ber, and a receive event to its removal from the inpt ber. Clearly, becase of the FIFO property of the bers and becase message transmissions from the sender's otpt ber to the receiver's inpt ber are synchronos, no (indirect) message overtaking can take place. Hence, althogh from the application level perspective the comptation is no longer garanteed to be RSC, it still has the CO-property [31]. The denitions given above for FIFO, CO, and MO (Denitions 3.3, 3.4 and 3.9) have a similar strctre since they all concern the relative order of receive events corresponding to ordered send events. The denition of RSC (Denition 3.6) has a qite dierent appearance, as it is related to the existence of linear extensions of. We shall next prove a characterization (Proposition 3.13) of CO-comptations with a similar avor, namely based on the partial order (C; ) as a whole. An application of this characterization will be a dierent demonstration that all RSC-comptations are CO. We shall also give an interesting graphical interpretation of this characterization for space-time diagrams (Observation 3.14). Sbseqently (Corollary 3.15) we shall derive that COcomptations are precisely those comptations where the \casal ftre" of a send event is disoint from the \casal past" of the corresponding receive event. The following denition and proposition shows that CO is eqivalent to the empty interval (EI) property, which states that for no pair (s; r) 2? there are events \between" s and r. Denition 3.12 (EI) A comptation C is called an EI-comptation (Empty Interval) if for each pair (s; r) 2? the open interval hs; ri = fx 2 C : s x rg is empty. 9

10 Proposition 3.13 (EI = CO) A distribted comptation C is CO if and only if it is EI. Proof. It is rst shown that if C is not CO, then it is not EI. Assme that (s; r) and (s 0 ; r 0 ) are two pairs in? sch that s s 0, r r 0, and r 0 r. Since s 0 r 0, we have s s 0 r 0 r, hence hs; ri is not empty. It follows that EI implies CO. It is next shown that if C is not EI, then it is not CO. Let (s; r) be a pair of? sch that hs; ri contains an event x. As :(s r) and is transitive, :(s x) or :(x r) holds. In either case there exists a chain a 0 ; : : :; x; : : :; a k with s = a 0, r = a k, a a +1 for all, and at least one pair (a ; a +1 ) is a pair (; ) of corresponding send and receive events sch that 6= s and 6= r. Ths s, while r, which shows that C is not MO; by Theorem 3.10 the comptation is not CO. It follows that CO implies EI. 2 As an example, Fig. 3 depicts a comptation which is not EI: all for events s 2 ; r 2 ; s 3 ; r 3 are contained in the interval hs 1 ; r 1 i. For the comptations depicted in Fig. 5 and Fig. 6, however, all intervals formed by corresponding send-receive events are empty. This seems to be obvios for those pairs where the events are connected by vertical message arrows (since casality does not go from right to left, there is no space left for an event between the send event and the receive event), bt one also readily checks the condition for hs 1 ; r 1 i. The dierence between EI and RSC consists in the nderlying order for which the intervals hs; ri mst be empty. For EI the intervals of itself are empty, while for RSC a linear extension mst be fond with empty intervals. Clearly, if there exist events s, r, and x sch that s x r, then for each linear extension < of one has s < x < r. Conseqently, if there exists a linear extension with empty intervals, then the corresponding intervals of are empty, too. This shows again that every RSC-comptation is EI and hence CO, as already stated in Theorem 3.7. The EI-characterization of CO-comptations has a nice graphical interpretation. Note that for an empty interval hs; ri of a given comptation it is always possible to nd a linear extension < of for which the corresponding interval fx 2 C : s < x < rg is empty. Intitively, empty hs; ri-intervals in a linear extension correspond to vertical message arrows in space-time diagrams, becase one may move events s and r arbitrarily close together and in fact assign the same real-time instant to them. Conversely, a vertical message arrow from a send event s to a receive event r in a space-time diagram where no message arrow goes \backwards" implies that the corresponding hs; ri interval is empty with respect to < and. This yields the following characterization which is a conseqence of Proposition Observation 3.14 (Vertical message arrow criterion) A distribted comptation C is CO if and only if for each message m there exists a space-time diagram for C sch that m can be drawn as a vertical message arrow (and no other message arrows do go from right to left). An immediate conseqence of this criterion is that for each single message of a CO-comptation one can assme that this message is transmitted instantaneosly. 10 Clearly, sch an instantaneos message cannot be bypassed by another message or by a chain of messages, ths showing again the triangle ineqality (Corollary 3.11). An example of a comptation for which the criterion of Observation 3.14 does not apply is again Fig. 3 the arrow for message m 1 cannot be made vertical withot forcing m 2 or m 3 to move backwards in time. Note also that it is in general not possible to nd a single linear extension where all corresponding intervals are empty. An example is the comptation depicted in Fig. 5 althogh (for dierent space-time diagrams) each single message can be drawn vertically, no diagram can be drawn with all messages made vertical simltaneosly, a property which is characteristic for RSC-comptations as we shall see frther down (Observation 4.5). The following corollary characterizes CO-comptations by a so-called weak common past or weak common ftre property. It is merely a dierent formlation of Proposition 3.13 which, however, looks rather similar to Denition 3.9 (MO), in particlar if one observes that the implication in Denition 3.9 can be stated eqivalently as r 0 r =) :(s s 0 ). Corollary 3.15 (Weak common past/ftre) A distribted comptation C is CO if and only if for each pair (s; r) 2? and for each x 2 C or, eqivalently, if and only if x r =) :(s x) (W CP ) s x =) :(x r): (W CF ) It shold be noted that the stronger properties x r =) x s and s x =) r x do sally not hold, not even for RSC-comptations. (An example is the RSC-comptation depicted in Fig. 4.) We will see in Section 5, however, that an important role in the axiomatization of distribted comptations with synchronos commnications (i.e., S-comptations) is in fact played by sch \strong" common past and common ftre properties, where the whole \casal ftre" (or \casal past", resp.) of a commnication event is inclded in the \casal ftre" (\casal past", resp.) of its corresponding event. Obviosly, the converse implications x s =) x r and r x =) s x of the \strong" common past and common ftre properties are trivially tre for every comptation becase of AS2 (s r) and AS3 (transitivity). 10 This does not necessarily apply to any other message in that specic view, however. 10

11 4 Characterizations of RSC- Comptations Like CO-comptations, RSC-comptations can be characterized in varios ways, and in this section several eqivalent characterizations are presented and compared. Insight in properties satised by RSCcomptations may help to nd distribted control algorithms for this class which are simpler and more ecient than algorithms that cannot rely on these properties. Frthermore, eciently determining whether a comptation is RSC is sometimes of practical importance, for example when replaying a distribted comptation for debgging prposes. Unfortnately, testing whether a comptation is RSC by checking all linear extensions of the nderlying partially ordered set according to Denition 3.6 is very expensive and ths not practical. Therefore, two dierent characterizations of RSCcomptations are given in this section. The rst characterization is based on occrrences of a sbstrctre called \crown", and the main reslt of this section (Theorem 4.4) is that a comptation is RSC if and only if it does not contain a crown. We will show that eciently checking the existence of a crown is possible by considering a simple binary relation on pairs of corresponding commnication events (i.e., messages) and checking whether the graph of this relation is cycle-free. The second characterization, based on message schedling, follows from the rst one and has a simple graphical interpretation (Observation 4.5) a comptation is RSC if and only if it has a space-time diagram where all message arrows are vertical. 4.1 Crowns in Distribted Comptations Crowns captre the idea of cyclic dependencies of messages. We shall motivate their denition with two simple observations. First, in a comptation which is not CO, hence not RSC, there always exist two pairs (s; r) and (s 0 ; r 0 ) of corresponding send and receive events sch that s s 0 and r 0 r. As s 0 r 0, by transitivity of we get the symmetric property s r 0 and s 0 r: This, however, is the typical \deadlock sitation" considered in Section 2.3 which disallows the schedle of whole messages. Second, consider Figres 5 and 6 which show COcomptations that are not RSC, and Fig. 3 which shows a non-co comptation. For all these three examples of non-rsc-comptations we have s 1 r 2 ; s 2 r 3 ; s 3 r 1 : If here we \identify" corresponding send and receive events s i ; r i, which is the intition behind the idea of schedling whole messages or considering single commnication events in the synchronos case, we get again a cyclic dependency. Generalizations of these two cases sggest the following denition. Denition 4.1 (Crown) Let C be a comptation. A crown (of size k) in C is a seqence h(s i ; r i ); i 2 f1; : : :; kgi of pairs of corresponding send and receive events sch that s 1 r 2 ; s 2 r 3 ; ; s k?1 r k ; s k r 1 : Examples of \canonical" crowns with a characteristic wrapped-arond zigzag shape responsible for their name are given in Fig. 7. Figres 3 and 5 show, however, that not all crowns have this typical geometrical shape and that the send events (or the receive events) do not necessarily lie on dierent processes. Besides the crowns of size 3 indicated above, both comptations also have crowns of size 2. (For Fig. 3, s 1 r 2 and s 2 r 1 ; and for Fig. 5, s 1 r 3 and s 3 r 1.) The rst observation at the beginning of this sbsection shows that every non-co-comptation necessarily contains a crown of size 2. Notice that the comptation of Fig. 6 does not contain a crown of size 2 and is therefore CO. Comptations which contain a crown of size 2, however, can be CO as Fig. 5 proves. As indicated above, one can easily verify that every comptation that contains a crown is non-rsc. The main reslt of this section is that, conversely, any comptation with no crown is RSC, as stated in the Theorem 4.4 frther down. Hence, the existence of a crown flly characterizes non-rsc-comptations. The proof of this crown criterion relies on several axiliary reslts which we shall prove next. 4.2 The Redction Lemma In this section we show that the internal events of a comptation can be ignored in the crown criterion (Theorem 4.4). Let C be a comptation and dene a comptation ~ C, called the redced comptation of C, by removing all internal events of C (i.e., ~ C is the sbset of send and receive events). Corresponding pairs in ~ C are the same as in C. Clearly, the casality relation on ~ C is the redction on ~ C of the casality relation dened on the comptation C, and ~ C is a comptation according to Denition 3.2. Lemma 4.2 (Redction lemma) The redced comptation ~ C of C is RSC if and only if C is RSC. Proof. A linear extension L of C denes a linear extension ~ L of ~ C by removing all internal events from L. If L is a non-separated linear extension of C, then ~ L is a 11