Group Communication Protocol in Transaction-Oriented Applications

Transcription

1 Grou Communication Protocol in Transaction-Oriented Alications Tomoya Enokido, Takayuki Tachikaa, and Makoto Takizaa Det. of Comuters and Systems Engineering Tokyo Denki University feno, tachi, Abstract In grou communications, larger comutation and communication are consumed to causally order all the messages transmitted in the netork. Transactions in clients maniulate objects in servers by sending read and rite requests to the servers. In this aer, e define significant messages by using the relation among the transactions. We nely roose an object vector to causally order only the significant messages. The scheme of the object vector is invariant in the change of the grou membershi. We also sho a TBCO (transaction-based causally ordered rotocol hich adots the object vector, by hich the number of messages to be causally ordered are reduced. 1. Introduction In distributed alications, a grou of multile objects are cooerating. Here, the messages have to be reliably delivered to the destination objects in the grou. Many grou rotocols[2, 7, 8, 9] are discussed to causally order messages transmitted in the netork. Hoever, the grou rotocol imlies O(n to O(n 2 comutation and communication overhead. Cheriton[3] oints out that it is meaningless to causally order all the messages transmitted from the alication oint of vie. Agraal[5] discusses a rollback algorithm here only rocesses comuting significant requests are rolled back if the senders of the significant requests are rolled back. Tachikaa and Takizaa[13] define the object-based causality among messages based on the conflicting relation among oerations suorted by the objects in the resence of the nested invocation of oerations. Raynal[11] discusses a ay to relex the traditional messagebased causality based on the rite-semantics for a grou of the relicas. In this aer, e introduce a transaction[1] concet to define a causality among messages hich are significant for the alications. A transaction T in a client rocessor issues read and rite requests to server rocessors. Thus, the subtransactions of T are comuted in the servers. Data read from an object is ritten to another object if read and rite are comuted in a transaction. A message m 1 may carry information in m 2 if a transaction sends m 1 after receiving or sending m 2. Thus, the messages received and sent by one transaction are related. Suose one transaction T 1 sends m 2 after T 2 receives m 1 in the same rocessor. m 1 and m 2 are not related unless T 1 and T 2 maniulate same objects. In order to define messages to be causally ordered, e have to consider hat subtransactions send and receive messages. Hence, the grou is required to be comosed of multile subtransactions and servers. In this aer, e define ho read and rite messages are causally related in a context of transactions. In addition, each object is relicated in order to increase the reliability, availability, and erformance of the system. We define significant messages to be causally ordered based on transaction concet. In most grou rotocols, messages are causally ordered by using the vector clock[6]. Each time transactions are randomly initiated and terminated, the size of the vector clock is required to change. In this aer, e nely roose an object vector hose size is given by the number of objects. We discuss a TBCO (transaction-based causally ordered rotocol hich uses the object vector to causally order significant messages. In the TBCO rotocol, significant messages can be delivered earlier ithout aiting for insignificant messages receding the significant messages. We resent a system model in section 2. In section 3, e define significant messages. In section 4, e resent the object vector and the TBCO rotocol. In section 5, e evaluate the TBCO rotocol. 2. System Model A system is comosed of rocessors 1, : : :, n (n 1 interconnected by the communication netork. Each object o i is stored in one rocessor. In the communication netork, messages may be lost and may be delivered out of order. On receit of a request message m ith an oeration o i, o i comutes o i. On comletion of o i, o i sends back the

2 resonse message to the sender of m. We assume that each o i is relicated. Each relica is allocated in a rocessor. Let o t i denote a relica of o i in a rocessor t. A transaction T i in one client rocessor s issues read and rite requests to the server rocessors to maniulate relicas of an objects o i. A rocessor sends a read request to one relica. t sends back the resonse ith the data derived. s sends rite requests to all the relicas of x. That is, the read-one rite-all rincile is adoted. Here, o 1! Ti o 2 iff an oeration o 1 recedes o 2 in T i. T i commits (c i or aborts (a i at the end of T i. T i commits if all the oerations invoked by T i comlete successfully. If at least one oeration fails in T i, T i aborts. Thus, T i is an atomic sequence of read and rite oerations[1]. Let T be a set f T 1, : : :, T m g of transactions in the system. Let o i t (x denote an oeration o of T i on a relica x in t. o is r(read or (rite. A subsequence of oerations of T i on the relicas in a rocessor t is named a subtransaction T it of T i in t. o 1! Tit o 2 iff o 1 recedes o 2 in T it. The interleaved comutation of subtransactions T 1t, : : :, T mt in t is a local history H t of t. o 1 recedes o 2 in t ( o 1! Ht o 2 iff o 1 is comuted before o 2 in t. A global history H of T is a collection of the local histories H 1, : : :, H n. o 1 recedes o 2 in H o 1! H o 2 iff o 1! Ht o 2, o 1! Ti o 2, or o 1! H o 3! H o 2 for some o 3. H has to be serializable[1] to kee the relicas consistent. In order to make H serializable, the conflicting oerations, i.e. rite and read or rite and rite on the same object have to be comuted in the same order as some serial history. Each T i is given a unique identifer t(t i in order to totally order the transactions. In the time stam ordering (TO concurrency control[1], every air of conflicting oerations are comuted in the order of the transaction identifers, i.e. time stam. That is, if an oeration o i of T i conflicts ith o j of T j in t and t(t i < t(t j, o i is comuted before o j in t. In the to-hase locking[1], the relicas are locked before they are maniulated. 3. Significant Messages A rocessor t may not receive messages or may receive messages out of order. If t loses a message m sent by u, u is required to resend m to t. After t receives m, t has to ait for all messages causally receding m. [Examle] Suose 1 t (xt, 1 s (xs, and 1 u (xu are initiated by a request message m 1, r 2 t (x t is by m 2, 3 s(x s, 3 t (xt, and 3 u (xu are by m 3, and r 4 u (xu is by m 4 as shon in Figure 1. 3 t (x t is comuted after r 2 t (x t in t. Here, suose u receives m 3 before m 1. In the traditional grou rotocols, u has to ait for m 1 ithout delivering m 3. That is, u has to comute 1 s(x s, 3 u(x u, and r 4 u(x u in this sequence. r 4 u (xu reads x from 3 u (xu but not from 1 s (xs. Here, suose 1 u (xu, 3 u (xu, and r 4 u (xu are still stored in the receit queue RQ u of u due to the communication delay although the oerations comlete in s and t. Since r 4 u (xu reads from 3 u (xu, 1 u (xu does not need to be comuted, i.e. 1 s(x s can be omitted. 2 s 1 (x s s 3 (x s s m 1 m 2 t t 1 (x t r t 2 (x t m 3 m 3 t 3 (x t m 4 u u 1 (x u u 3 (x u r u 4 (x u Figure 1. Significant messages. [Examle] Suose t receives read requests rt, i r j t, and rt k from transactions T i, T j, and T k, resectively. t reads a value from the relica x and sends back the resonse ith data each time t receives a read. Here, t comutes read oerations on x three times. If t reads x once and sends back the resonse to T i, T j, and T k, the number of oerations comuted can be reduced. That is, r j t (x and rk t can be removed from RQ t if rt i is comuted. 2 The first examle shos that some rite requests can be omitted. Suose t(x i! H j t (x. If there is no read rt k(x such that i t (x! H rt k(x! H j t (x, t dose not need to receive t(x. i This means t can comute j t (x ithout aiting for t i(x. t can further reject t i(x if t receives t i(x after comuting j t (x. The second examle shos some read requests can be omitted. We discuss hat messages can be omitted. [Definition] A message m is insignif icant iff (1 if m = rt i(x, there is some read rj t (x in RQ t such that r j t (x recedes rt(x i and there is no rite t k (x beteen r j t (x and rt(x i in RQ t. (2 if m = t i(x, there is some rite j t (x in RQ t such that t i(x recedes j t (x and there is no read rt k(x beteen t(x i and j t (x in RQ t. 2 In Figure 1, suose u receives m 1 = u 1 (xu, m 3 = u(x 3 u, and m 4 = ru(x 4 u in RQ u. Here, m 3 and m 4 are significant in RQ u but m 1 is insignificant. Insignificant messages can be omitted from RQ u. [Theorem] Let R t be a sequence of messages in the receit queue RQ t of t. Let Rt 0 be a subsequence of R t hich includes only the significant messages. Here, the state of t and the sequence of outut values of read requests obtained

3 by comuting the oerations in R 0 t are the same as R t TBCO Protocol 4.1. Object vector Let snd t [m] and rcv t [m] denote a sending and receit events of a message m in a rocessor t, resectively. Lamort[4] defines that an event e 1 haens before e 2 ( e 1! e 2 iff (1 e 1 recedes e 2 in a rocessor t, i.e. e 1! Ht e 2. (2 e 1 = snd s [m] and e 2 = rcv t [m], or (3 e 1! e 3! e 2 for some event e 3. A message m 1 causally recedes m 2 ( m 1! m 2 iff snd s [m 1 ]! snd t [m 2 ][2]. t has to receive m 1 before m 2 if m 1! m 2. The vector clock[6] V = hv 1, : : :, V n i is used to causally order the messages received. Each V u is initially 0. t increments V t by one each time t sends a message m. m carries the vector clock m.v ( = V. On receit of m, t changes V as V u = max( V u, m.v u for u = 1, : : :, n and u 6= t. m 1! m 2 iff m 1.V < m 2.V. If the membershi of the grou changes, all the rocessors have to be synchronized[2, 3, 4, 5, 6, 13]. We consider the conflicting and recedence relations among transactions to order only significant messages. That is, a grou is considered to be comosed of subtransactions and servers. The membershi of the grou changes each time transactions are initiated and terminated. If the vector clock is used, the grou has to be frequently resynchronized. Here, the vector clock can be used to causally order messages sent by rocessors but not by transactions. In this aer, e roose an object vector to causally order the significant messages sent by the transactions. Each transaction T i is given a unique identifier t(t i satisfying the folloing roerties. (1 If T i starts before T j in a rocessor, t(t i < t(t j. (2 If a rocessor t initiates T i after receiving a message from T j, t(t i > t(t j. The transaction identifier is given by the linear clock[4]. t maniulates a variable tid shoing the linear clock hose initial value is 0. t maniulates tid as follos : (1 tid := tid + 1 if a transaction is initiated in t. (2 On receit of a message from T j, tid := max(tid, tid(t j. When T i is initiated in t, t(t i is given a concatenation of tid and the rocessor number no( t of t. Here, let tid(t i denote tid given to T i. For every air of T i and T j initiated at t and u, tid(t i > tid(t j if (1 t(t i > t(t j or (2 t(t i = t(t j and no( t > no( u. Each event e is given an event number no(e as follos : (1 If e 1 is the initial event of T i, no(e 1 = 0. (2 If e 2 is rite and e 1! e 2 in T i, no(e 1 < no(e 2. (3 If e 2 is read and there is no rite event e 3 such that e 1! Ti e 3! Ti e 2, no(e 1 = no(e 2. no i is incremented by one each time a rite event e occurs in T i. no(e := no i. Each event e in T i is given a global event number tno(e as the concatenation of t(t i and no(e. Every relica o j a of o a has the same version number v(o j a = v(o a. v(o j a is udated to be tno( each time (oj a is comuted. Here, if tno(o > v(o j a, o can be comuted. Otherise, o aborts because o is obsolete Message transmission and receit A message m is comosed of the folloing fields : m:src = sender rocessor of m, i.e. s. m:dst = set of destination rocessors. m:o = tye o of message, i.e. read, rite, commit, abort, or resonse. m:tno = global event number hm:t, m:noi, i.e. tno(o. m:o = object to be maniulated by m:o. m:v = object vector hv 1, : : :, V u i. m:sq = vector of sequence numbers hsq 1, : : :, sq n i. m:d = data. If m:o = read, m:dst denotes one rocessor hich has a relica o i a of o a. If m:o = rite, m:dst shos all the rocessors hich have the relicas of o a. If m 0 is a resonse message of a request message m, m 0.tno is m.tno. A rocessor s constructs a request message m ith o h from T h as follos : m.tno = h m.t, m.no i := h t(t h, no(o h i; m:o := o h ; m:o := o a ; m:src := s ; m:v a := V a (a = 1, : : :, u ; sq t := sq t + 1 for every t in m:dst; m:sq j := sq j (j = 1, : : :, n; s maniulates variables sq 1, : : :, sq n. Each time s sends a message to t, sq t is incremented by one. Then, s sends m to every destination t in m:dst. t can detect a ga beteen messages sent by s by checking the sequence number sq t, i.e. messages lost or unexectedly delayed. t maniulates variables rsr 1, : : :, rsq u, to receive message. On receit of a message m from T s in s, there is no ga if m:sq t = rsq s. If m:sq t > rsq s, there is a ga m 0 sent from s here m:sq t > m 0 :sq t rsq s. m is correctly received by t if t receives every message m 0 here m 0.sq t < m.sq t. That is, t receives every message hich s sends to s before m. t enqueues m in the receit queue RQ t. t maniulates the object vector V as V a = max(v a, m:v a for a = 1, : : : u. When a relica o t a is maniulated by m.o, t sends back a resonse m 0 of m to s. If m.o = read, m 0 carries data derived from o t a in.d and m m0 0 the version number v(o a of o t a. m 0 :t := m:t; m 0 :no := m:no; m 0 :o := o a ;

4 m 0 :o := resonse; m 0 :V a := v(o a ; T s has a vector of variables V = h V 1, : : :, V u i here each V a is initially 0. On receit of the resonse m 0 from t, T s maniulates V, i.e. V a : = m 0.V a (= v(o j a if V a < m 0.V a. Otherise, T s aborts Message delivery Messages in a receit queue RQ t are ordered according to the ordering rule. [Ordering rule] A message m 1 recedes m 2 (m 1 m 2 (1 if m 1 :V m 2 :V, (2 otherise (m 1 :V = m 2 :V, m 1 :t < m 2 :t. 2 s t u s t u < 0, 0 > < 0, 0 > < 0, 0 > < 0, 2v1 > < 0, 2v1 > < 0, 0 > TID =1s (x <1s0, 0 > (y <1s0, 1s1 > m 1 m 2 m 1 m 1.V < m 2.V TID =1s r s (y 0, 2v1 < > s (x <1s1, 2v1 > (1 (2 m Res TID =2u m 1 m 2 u (x <2u0, 0 > m 1.t < m 2.t Figure 2. Message ordering. TID = 1u [Examle] In Figure 2, h, i shos an object vector here and are the values of variables V x and V y shoing the versions of objects x and y, resectively. Initially h 0, 0 i in every rocessor. x and y are in every rocessor in Figure 2(1. A transaction T 1s is comuted in s here t(t 1s = 1s here s is the rocessor number of s and 1 is the local linear clock. T 1s first sends a rite request m 1 ith (x and then sends m 2 ith (y to s, t, and u. The global event number tno((x = 1s0. m 1.V = h 1s0, 0 i and m 1.t = 1s. Then, (y is comuted in s, t, and u. Here, tno((y = 1s1. T 1s sends m 2 ith m 2.V = h 1s0, 1s1 i and m 2.t = 1s. t receives m 2 after m 1 but u receives m 1 after m 2. Since m 1.V (= h 1s0, 0 i < m 2.V (= h 1s0, 1s1 i, m 1 m 2 by the ordering rule. t and u receive m 1 and m 2 in. After (x is comuted, h 1s0, 0 i in every rocessor. h 1s0, 1s1 i in every rocessor after (y. In Figure 2(2, T 2u is comuted after T 1u in u. x is in every rocessor but y is in s and t. Before T 1s is started, v(y is assumed to be 2v1. T 1s reads y and gets the version number 2v1 of y from t. T 1s and T 2u send m 1 ith s (x and m 2 ith u (x to every rocessor. tno( t (x = 1s1 in T 1s and tno( u (x = 2u0. Hence, m 1.V = h 1s1, 2v1 i and m 2.V = h 2u0, 0 i in T 2u. m 1.V and m 2.V are uncomarable but m 1.t (= 1s < m 2.t (= 2u. Hence, m 1 m 2. 2 [Theorem] m 1 causally recedes m 2 (m 1! m 2 if m 1 recedes m 2 in the ordering rule (m 1 m 2. [Proof] We assume that m 1 m 2 but m 1 6! m 2. In Figure 2 (1, m 1 m 2. Since m 1 is sent before m 2, m 1! m 2. It contradicts the assumtion. 2 Here, the messages in RQ t are ordered in by the ordering rule. m is a to message in RQ t iff there is no message m 1 in RQ t such that m 1 m. Each to message m 1 in RQ t still cannot be delivered because t may have not yet received some message m 2 causally receding m 1 due to the unexected delay. We discuss hat messages in RQ t can be delivered. [Definition] Let m be a message sent by u to t and stored in the receit queue RQ t. m is stable in t iff (1 there is a message m 1 from u in RQ t here m 1 :sq t = m:sq t + 1, and (2 t correctly receives a message m 1 in RQ t from every u here m!m 1. 2 The to message m in RQ t can be delivered if m is stable, because every message causally receding m from the transaction oint of vie is surely delivered in RQ t. [Definition] A to message m in RQ t is ready in t if t comutes no oeration conflicting ith m.o in a relica m.o. 2 In addition, only significant messages in RQ t are delivered from RQ t by the folloing rocedure in order to reduce time for delivering messages. [Delivery rocedure] While each to in RQ t is stable and ready, fm is dequeued from RQ t ; m is delivered if m is significant, otherise m is neglectedg. 2 [Examle] In Figure 3, to transactions T 1s and T 2s are comuted in s, and T 3t and T 4t are in t. T 1s, T 2s, and T 3t issue rite requests 1s (x, 2s (x, and 3t (x. 1s (x arrives at neither t nor u. 2s (x does not arrive at u. Since 1s (x and 2s (x are insignificant in u, they are omitted. 2 If t receives read requests r 1 (x, : : :, r h (x (h > 1. t comutes r(x once and then sends the resonse to all the source rocessors of r 1 (x, : : :, r m (x. Thus, the number of oerations comuted can be reduced. Each t has variables D 1, : : :, D u here each D a shos the version number of the object o a to deliver messages. Each time a message m is delivered, D a := m:t if m:o = o a and m.t D a. Here, if m:t < D a, m is omitted. If some u sends no message to t, the messages in RQ t cannot be stable. In order to resolve this roblem, each u sends a message ithout data to t if u had not sent any message to t for some redetermined time units. is defined to be roortional to delay time beteen the rocessors. t considers that t loses a message from u if

5 u t s 1s (x 1s 2s (x 2s r 2s (x 3t (x Res 3t : transactions r 4t (x 4t : omit Res Figure 3. Omission of rite oerations. time (1 t does not receive any message from u for time units or (2 t detects a ga in the receit sequence of messages. If t detects that m is lost, t requires u to resend m. u considers that t may lose m unless u receives the receit confirmation of m from t in 2 after u sends m to t. 5. Evaluation In the TBCO rotocol, messages are ordered by the object vector hose size is the number u of objects in the system. On the other hand, the size of the vector clock is given by the number of rocesses, i.e. servers and subtransactions in the grou. Messages sent and received by a transaction are related but messages exchanged among transactions may not be related. Hence, it is critical to consider a grou to be comosed of subtransactions to causally order only significant messages. Each time transactions are initiated and terminated, the scheme of the vector clock has to change, i.e. subtransactions and servers have to be resynchronized. Since the object vector only deends on the number of the objects, the rotocol can be adoted to transaction-oriented alications. Ratio of oerations Traditional TBCO rite / (rite + read Figure 4. Ratio of oerations. The TBCO rotocol does not causally order all the messages transmitted in the netork. We evaluate the TBCO rotocol in terms of the number of messages causally ordered comared ith the traditional message-based rotocol. The TBCO rotocol is realized in threads of Ultra Sarc stations suer server 6400 ith 10 Ultra Sarcs. Each rocessor is bound to one Ultra Sarc. TCP[14] is used to exchange messages among the rocessors. Each rocessor randomly creates transactions each of hich randomly maniulates objects by read and rite. There are three objects hich are fully relicated in all the rocessors. Each rocessor randomly initiates totally tenty transactions each of hich issues ten arbitrary kinds of oerations on arbitrary objects. Figure 4 shos the total number of requests comuted in the objects for the ratio of rite requests issued by the transactions for three rocessors. The more rite requests are issued, the more messages are sent to the relicas. The rite ratio 1.0 means that only rite requests are issued and 0.0 shos that only read requests are issued. The figure shos that the more rite messages are issued, the more messages can be omitted. For examle, only about 68% of the messages transmitted in the messagebased rotocol are transmitted in the TBCO rotocol in case that all the messages are rites, i.e. the rite ratio is 1.0 hile 50% for the rite ratio 0.4 and 90% for the rite ratio 0.0. Ratio of delivered messages :r = 9:1 Traditional TBCO Number of rocessors Figure 5. Ratio of messages. In Figures 5 and 6, the vertical axis indicates the ratio of the number of messages delivered to the number of messages transmitted in case of ten rocessors in the traditional message-based rotocol. The dotted line shos TBCO and the solid line indicates the traditional message-based rotocol. Figures 5 and 6 sho cases that 90% and 50% of the oerations issued by each transaction are rites, resectively. These figures sho the number of messages delivered

6 Ratio of delivered messages :r = 5:5 Traditional TBCO Number of rocessors Figure 6. Ratio of messages. can be reduced by the TBCO rotocol. About 30% and 20% of messages transmitted in the netork are reduced for the rite ratios 90% and 50%, resectively. 6. Concluding Remarks This aer has discussed hat messages have to be causally ordered in relicated objects ith read and rite from the alication oint of vie. We have roosed the novel object vector for causally receding messages based on the transaction concet. In the TBCO rotocol, only the messages to be meaningfully receded for the alications can be causally ordered. We have also discussed a ay for omitting messages hich are not significant for the alications. We have shon the TBCO rotocol imlies feer number of oerations comuted than the rotocols hich causally order all the messages transmitted in the netork. References [1] Bernstein, P. A., Hadzilacos, V. and Goodman, N., Concurrency Control and Recovery in Database Systems, Addison-Wesley,.25?45, [4] Lamort, L., Time, Clocks, and the Ordering of Events in a Distributed System, Comm. ACM, Vol.21, No.7, , [5] Leong, H. V. and Agraal, D., Using Message Semantics to Reduce Rollback in Otimistic Message Logging Recovery Schemes, Proc. of IEEE ICDCS- 14,.227?234, [6] Mattern, F., Virtual Time and Global States of Distributed Systems, Parallel and Distributed Algorithms (Cosnard, M. and Quinton, P. eds., North-Holland,.215?226, [7] Melliar-Smith, P. M., Moser, L. E., and Agraala, V., Broadcast Protocols for Distributed Systems, IEEE Trans. on Parallel and Distributed Systems, Vol.1, No.1,.17 25, [8] Nakamura, A. and Takizaa, M., Reliable Broadcast Protocol for Selectively Ordering PDUs, Proc. of IEEE ICDCS-11, , [9] Nakamura, A., Tachikaa, T., and Takizaa, M., Causally Ordering Broadcast Protocol, Proc. of IEEE ICDCS-14,.48?55, [10] Object Management Grou Inc., The Common Object Request Broker Architecture and Secification, Rev. 2.0, [11] Raynal, M. and Ahamad, M., Exloiting Write Semantics in Imlementing Partially Relicated Causal Objects, IRISA Research Reort, PI-1080, [12] Tachikaa, T. and Takizaa, M., Distributed Protocol for Selective Intra-grou Communication, Proc. of IEEE ICNP-95,.234?241, [13] Tachikaa, T. and Takizaa, M., Object-Based Message Ordering in Grou Communication, Proc. of the IEEE 3rd Int l Worksho on Object-oriented Realtime Deendable Systems (WORDS 97,. 315?322, [14] Transmission Control Protocol, RFC793,. 1 26, [2] Birman, K., Lighteight Causal and Atomic Grou Multicast, ACM Trans. on Comuter Systems,.272?290, [3] Cheriton, D. R. and Skeen, D., Understanding the Limitations of Causally and Totally Ordered Communication, Proc. of the ACM SIGOPS 93,.44?57, 1993.