Distriuted Systems Fö 5-1 Distriuted Systems Fö 5-2 TIME ND STTE IN DISTRIUTED SYSTEMS 1. Time in Distriuted Systems Time in Distriuted Systems euse eh mhine in distriuted system hs its own lok there is no notion of glol physil time. The n rystls on the n omputers will run t slightly different rtes, using the loks grdully to get out of synhroniztion nd give different vlues. 2. Lmport s Logil Cloks Prolems: 3. Vetor Cloks 4. Cusl Ordering of Messges 5. Glol Sttes nd their Consisteny Time triggered systems: these re systems in whih ertin tivities re sheduled to our t predefined moments in time. If suh tivities re to e oordinted over distriuted system we need oherent notion of time. Exmple: time-triggered rel-time systems 6. Cuts of Distriuted Computtion 7. Reording of Glol Stte Mintining the onsisteny of distriuted dt is often sed on the time when ertin modifition hs een performed. Exmple: mke progrm. Distriuted Systems Fö 5-3 Distriuted Systems Fö 5-4 Time in Distriuted Systems (ont d) The mke-progrm exmple When the progrmmer hs finished hnging some soure files he strts mke; mke exmines the times t whih ll ojet nd soure files were lst modified nd deides whih soure files hve to e (re)ompiled. Computer 1 (ompiler) Computer 2 (editor) 652 653 654 655 656 P.o reted 648 649 650 651 652 P. reted lthough P. is modified fter P.o hs een generted, euse of the lok drift the time ssigned to P. is smller. P. will not e reompiled for the new version! lol physil lok lol physil lok Solutions: Time in Distriuted Systems (ont d) Synhroniztion of physil loks Computer loks re synhronized with one nother to n hievle, known, degree of ury within the ounds of this ury we n oordinte tivities on different omputers using eh omputer s lol lok. Physil lok synhroniztion is needed for distriuted rel-time systems. Logil loks In mny pplitions we re not interested in the physil time t whih events our; wht is importnt is the reltive order of events! The mke-progrm is suh n exmple (slide 3). In suh situtions we don t need synhronized physil loks. Reltive ordering is sed on virtul notion of time - logil time. Logil time is implemented using logil loks.
Distriuted Systems Fö 5-5 Distriuted Systems Fö 5-6 Lmport s Logil Cloks Lmport s Logil Cloks (ont d) The order of events ourring t different proesses is ritil for mny distriuted pplitions. Exmple: P.o_reted nd P._reted in slide 3. Ordering n e sed on two simple situtions: 1. If two events ourred in the sme proess then they ourred in the order oserved following the respetive proess; 2. Whenever messge is sent etween proesses, the event of sending the messge ourred efore the event of reeiving it. If, we sy tht event uslly ffets event. The two events re uslly relted. There re events whih re not relted y the hppened-efore reltion. If oth e nd e re flse, then nd e re onurrent events; we write e. m 1 Ordering y Lmport is sed on the hppenedefore reltion (denoted y ):, if nd re events in the sme proess nd ourred efore ;,if is the event of sending messge m in proess, nd is the event of the sme messge m eing reeived y nother proess; If nd, then (the reltion is trnsitive). e P1, P2, P3: proesses;,,, d, e, f: events; d m2 f, d, e f,, d f, d, f, d, f,... e, e,... Distriuted Systems Fö 5-7 Distriuted Systems Fö 5-8 Lmport s Logil Cloks (ont d) Lmport s Logil Cloks (ont d) Using physil loks, the hppened efore reltion n not e ptured. It is possile tht nd t the sme time T > T (T is the physil time of ). Logil loks n e used in order to pture the hppened-efore reltion. logil lok is monotonilly inresing softwre ounter. There is logil lok C Pi t eh proess P i in the system. The vlue of the logil lok is used to ssign timestmps to events. C Pi () is the timestmp of event in proess P i. There is no reltionship etween logil lok nd ny physil lok. To pture the hppened-efore reltion, logil loks hve to e implemented so tht if, then C() < C() Implementtion of logil loks is performed using the following rules for updting the loks nd trnsmitting their vlues in messges: [R1]: C Pi is inremented efore eh event is issued t proess P i : C Pi := C Pi + 1. [R2]: ) When is the event of sending messge m from proess P i, then the timestmp t m = C Pi () is inluded in m (C Pi () is the logil lok vlue otined fter pplying rule R1). ) On reeiving messge m y proess P j, its logil lok C Pj is updted s follows: C Pj := mx(c Pj, t m ). ) The new vlue of C Pj is used to timestmp the event of reeiving messge m y P j (pplying rule R1). If nd re events in the sme proess nd ourred efore, then, nd (y R1) C() < C(). If is the event of sending messge m in proess, nd is the event of the sme messge m eing reeived y nother proess, then, nd (y R2) C() < C(). If nd, then, nd (y indution) C() < C().
Distriuted Systems Fö 5-9 Distriuted Systems Fö 5-10 Lmport s Logil Cloks (ont d) Prolems with Lmport s Logil Cloks 1 2 m 1 Lmport s logil loks impose only prtil order on the set of events; pirs of distint events generted y different proesses n hve identil timestmp. Computer 1 (ompiler) Computer 2 (editor) e 3 4 1 5 For the mke-progrm exmple we suppose tht proess running ompiltion notifies, through messge, the proess holding the soure file out the event P.o reted logil lok n e used to orretly timestmp the files. 652 653 654 655 656 1 2 P.o reted 648 649 650 651 652 1 d 3 4 m2 f P. reted lol physil lok lol physil lok For ertin pplitions totl ordering is needed; they onsider tht no two events n our t the sme time. In order to enfore totl ordering glol logil timestmp is introdued: - the glol logil timestmp of n event ourring t proess P i, with logil timestmp C Pi (), is pir (C Pi (), i), where i is n identifier of proess P i ; - we define (C Pi (), i) < (C Pj (), j) if nd only if C Pi () < C Pj (), or C Pi () = C Pj () nd i < j. (1,1) (2,1) e m 1 (3,2) (4,2) (1,3) (5,3) d m2 f Distriuted Systems Fö 5-11 Distriuted Systems Fö 5-12 Prolems with Lmport s Logil Cloks (ont d) Prolems with Lmport s Logil Cloks (ont d) Lmport s logil loks re not powerful enough to perform usl ordering of events. if, then C() < C(). However, the reverse is not lwys true (if the events ourred in different proesses): if C() < C(), then is not neessrily true. (it is only gurnteed tht is not true). (1,1) (2,1) e m 1 C(e) < C(), however there is no usl reltion from event e to event. (3,2) (4,2) (1,3) (5,3) y just looking t the timestmps of the events, we nnot sy whether two events re uslly relted or not. d m2 f send(m 1 ) 1 2 2 send(m 3 ) 3 4 1 M 2 M 1 4 1 3 send(m 2 ) M 3 3 We would like messges to e proessed ording to their usl order. We would like to use the ssoited timestmp for this purpose. Proess reeives messges M 1, M 2, nd M 3. send(m 1 ) send(m 2 ), send(m 1 ) send(m 3 ), send(m 3 ) send(m 2 ) M 1 hs to e proessed efore M 2 nd M 3. However hs not to wit for M 3 in order to proess it efore M 2 (lthough M 3 s logil lok timestmp is smller thn M 2 s).
Distriuted Systems Fö 5-13 Distriuted Systems Fö 5-14 Vetor Cloks Vetor Cloks (ont d) Vetor loks give the ility to deide whether two events re uslly relted or not y simply looking t their timestmp. Implementtion of vetor loks is performed using the following rules for updting the loks nd trnsmitting their vlues in messges: Eh proess P i hs lok CPi v, whih is n integer vetor of length n (n is the numer of proesses). The vlue of CPi v is used to ssign timestmps to events in proess P i. CPi() v is the timestmp of event in proess P i. C v Pi[i], the ith entry of C v Pi,orresponds to P i s own logil time. CPi[j], v j i,isp i s "est guess" of the logil time t P j. CPi v [j] indites the (logil) time of ourrene of the lst event t P j whih is in hppenedefore reltion to the urrent event t P i. [R1]: C v Pi is inremented efore eh event is issued t proess P i : C v Pi[i] := C v Pi[i] + 1. [R2]: ) When is the event of sending messge m from proess P i, then the timestmp t m = CPi() v is inluded in m (CPi v () is the vetor lok vlue otined fter pplying rule R1). ) On reeiving messge m y proess P j, its vetor lok CPj v is updted s follows: k {1,2,..,n}, CPj v [k] := mx(cv Pj [k], t m [k]). ) The new vlue of CPj v is used to timestmp the event of reeiving messge m y P j (pplying rule R1). Distriuted Systems Fö 5-15 Distriuted Systems Fö 5-16 (1,0,0) Vetor Cloks (ont d) (2,0,0) (3,0,0) Cusl Ordering of Messges Using Vetor Cloks (0,1,0) (2,2,0) (2,3,1) (2,4,1) d e f (0,0,1) (2,4,2) g For ny two vetor timestmps u nd v, we hve: u = v if nd only if i, u[i] = v[i] u v if nd only if i, u[i] v[i] u < v if nd only if (u v u v) u v if nd only if (u < v) (v < u) h The prolem hs een formulted on slide 12: We would like messges to e proessed ording to their usl order. If Send(M 1 ) Send(M 2 ), then every reipient of oth messges M 1 nd M 2 must reeive M 1 efore M 2. Two events nd re uslly relted if nd only if C v () < C v () or C v () < C v (). Otherwise the events re onurrent. With vetor loks we get the property whih we missed for Lmport s logil loks: if nd only if C v () < C v (). Thus, y just looking t the timestmps of the events, we n sy whether two events re uslly relted or not.
Distriuted Systems Fö 5-17 Distriuted Systems Fö 5-18 Cusl Ordering of Messges Using Vetor Cloks (ont d) Cusl Ordering of Messges Using Vetor Cloks (ont d) (0,0,0) (0,1,0) (0,1,1) (0,2,1) The events whih re of interest here re the sending of messges vetor loks will e inremented only for messge sending. (0,0,0) (0,0,0) (0,1,0) (0,1,0) (0,1,1) (0,1,1) (0,2,1) (0,2,1) messge delivery protool whih preforms usl ordering sed on vetor loks. si Ide: - messge is delivered to proess only if the messge immeditely preeding it (onsidering the usl ordering) hs een lredy delivered to the proess. Otherwise, the messge is uffered. We ssume tht proesses ommunite using rodst messges. (There exist similr protools for non-rodst ommunition too.) Implementtion of the protool is sed on the following rules: [R1]: ) efore rodsting messge m, proess P i inrements the vetor lok: C v Pi[i]:=C v Pi[i]+1. ) The timestmp t m = CPi v is inluded in m. [R2]: The reeiving side, t proess P j, delys the delivery of messge m oming from P i until oth the following onditions re stisfied: 1. CPj v [i] = t m [i] - 1 2. k {1,2,..,n} - {i}, CPj v [k] t m [k] Delyed messges re queued t eh proess in queue tht is sorted y their vetor timestmp; onurrent messges re ordered y the time of their rrivl. [R3]: When messge is delivered t proess P j, its vetor lok CPj v is updted ording to rule R2 for vetor lok implementtion (see slide 14). t m [i] - 1 indites how mny messges originting from P i preede m. Step R2.1 ensures tht proess P j hs reeived ll the messges originting from P i tht preede m. Step R2.2 ensures tht P j hs reeived ll those messges reeived y P i efore sending m. Distriuted Systems Fö 5-19 Distriuted Systems Fö 5-20 Glol Sttes The prolem is how to ollet nd reord onsistent glol stte in distriuted system. Why prolem? euse there is no glol lok (no oherent notion of time) nd no shred memory! Glol Sttes (ont d) Consider nk system with two ounts nd t two different sites; we trnsfer $50 etween nd. Ch1: empty $500 $200 Ch1: $50 $450 $200 $450 $250 Ch1: empty Ch1 C : onsistent NC: not onsistent 500 empty 200 C 500 50 200 NC 450 50 200 C 450 empty 200 NC 500 50 250 NC 450 50 250 NC 450 empty 250 C 500 empty 250 NC
Distriuted Systems Fö 5-21 Distriuted Systems Fö 5-22 Glol Sttes (ont d) Forml Definition In generl, glol stte onsists of set of lol sttes nd set of sttes of the ommunition hnnels. The stte of the ommunition hnnel in onsistent glol stte should e the sequene of messges sent long the hnnel efore the sender s stte ws reorded, exluding the sequene of messges reeived long the hnnel efore the reeiver s stte ws reorded. It is diffiult to reord hnnel sttes to ensure the ove rule glol sttes re very often reorded without using hnnel sttes. This is the se in the definition elow. LS i is the lol stte of proess P i. eside other informtion, the lol stte lso inludes reord of ll messges sent nd reeived y the proess. We onsider the glol stte GS of system, s the olletion of the lol sttes of its proesses: GS = {LS 1, LS 2,..., LS n }. ertin glol stte n e onsistent or not! send(m k ij ) denotes the event of sending messge mk ij from P i to P j ; re(m k ij) denotes the event of reeiving messge m k ij y P j. send(m k ij) LS i if nd only if the sending event ourred efore the lol stte ws reorded; re(m k ij) LS j if nd only if the reeiving event ourred efore the lol stte ws reorded. trnsit(ls i,ls j ) = {m k ij send(mk ij ) LS i re(mk ij ) LS j } inonsistent(ls i,ls j )={m k ij send(mk ij ) LS i re(mk ij ) LS j } Distriuted Systems Fö 5-23 Distriuted Systems Fö 5-24 Forml Definition (ont d) Forml Definition (ont d) glol stte GS ={LS 1, LS 2,..., LS n }isonsistent if nd only if: i, j: 1 i, j n :: inonsistent(ls i,ls j ) = In onsistent glol stte for every reeived messge orresponding send event is reorded in the glol stte. In n inonsistent glol stte, there is t lest one messge whose reeive event is reorded ut its send event is not reorded. LS 11 LS12 LS 21 LS 22 LS 23 glol stte GS ={LS 1, LS 2,..., LS n }istrnsitless if nd only if: i, j: 1 i, j n :: trnsit(ls i,ls j ) = ll messges reorded to e sent re lso reorded to e reeived. glol stte is strongly onsistent if it is onsistent nd trnsitless. strongly onsistent stte orresponds to onsistent stte in whih ll messges reorded s sent re lso reorded s reeived. LS 31 LS 32 LS 33 {LS 11, LS 22, LS 32 } is inonsistent; {LS 12, LS 23, LS 33 } is onsistent; {LS 11, LS 21, LS 31 } is strongly onsistent. Note: the glol stte, s defined here, is seen s olletion of the lol sttes, without expliitely pturing the stte of the hnnel.
Distriuted Systems Fö 5-25 Distriuted Systems Fö 5-26 Forml Definition (ont d) Ch1: empty $500 $200 Ch1: $50 $450 $200 Ch1: empty $450 $250 Cuts of Distriuted Computtion ut is grphil representtion of glol stte. onsistent ut is grphil representtion of onsistent glol stte. ut of distriuted omputtion is set Ct = { 1, 2,..., n }, where i is the ut event t proess Pi. ut event is the event of reording lol stte of the respetive proess. C : onsistent NC: not onsistent 500 200 {,}: strongly C 450 200 {,}: C (mess 1 sent) 500 250 (mess 1 reeived) {,}: NC 450 (mess 1 sent) 250 (mess 1 reeived) {,}: strongly C LS 11 LS12 1 6 LS 21 LS 22 LS 23 2 4 7 3 5 8 LS 31 LS 32 LS 33 fter registering of the reeive event(s) onsistent stte eomes strongly onsistent. It is onsidered to e norml (trnsient) sitution. Distriuted Systems Fö 5-27 Distriuted Systems Fö 5-28 Cuts of Distriuted Computtion (ont d) Cuts of Distriuted Computtion (ont d) Let e k denote n event t proess P k. utct ={ 1, 2,..., n } is onsistent ut if nd only if P i, P j, e i, e j suh tht (e i e j ) (e j j ) (e i i ) ut is onsistent if every messge tht ws reeived efore ut event ws sent efore the ut event t the sender proess. LS 11 1 e 1 6 LS12 Theorem ut Ct = { 1, 2,..., n } is onsistent ut if nd only if no two ut events re uslly relted, tht is: i, j :: ( i j ) ( j i ) set of onurrent ut events form onsistent ut. LS 11 1 e 1 6 LS12 LS 21 LS 22 LS 23 2 e 2 4 7 LS 21 LS 22 LS 23 2 e 2 4 7 3 5 8 LS 31 LS 32 LS 33 5 8 3 LS 31 LS 32 LS 33 { 1, 4, 5 } is not onsistent: (e 1 e 2 ) (e 2 4 ) (e 1 1 ) { 1, 2, 3 }: strongly onsistent (no ommunition line is rossed) { 6, 7, 8 }: onsistent (ommunition line is rossed ut no usl reltion). { 1, 4, 5 }: not onsistent; 1 4
Distriuted Systems Fö 5-29 Distriuted Systems Fö 5-30 Glol Stte Reording (Chndy-Lmport lgorithm) Glol Stte Reording (ont d) Some disussion on how to ollet glol stte: The lgorithm reords olletion of lol sttes whih give onsistent glol stte of the system. In ddition it reords the stte of the hnnels whih is onsistent with the olleted glol stte. Suh reorded "view" of the system is lled snpshot. We ssume tht proesses re onneted through one diretionl hnnels nd messge delivery is FIFO. We ssume tht the grph of proesses nd hnnels is strongly onneted (there exists pth etween ny two proesses). The lgorithm is sed on the use of speil messge, snpshot token, in order to ontrol the stte olletion proess. proess P i reords its lol stte LS i nd lter sends messge m to P j ; LS j t P j hs to e reorded efore P j hs reeived m. The stte SCh ij of the hnnel Ch ij onsists of ll messges tht proess P i sent efore reording LS i nd whih hve not een reeived y P j when reording LS j. snpshot is strted t the request of prtiulr proess P i, for exmple, when it suspets dedlok euse of long dely in essing resoure; P i then reords its stte LS i nd, efore sending ny other messge, it sends token to every P j tht P i ommunites with. When P j reeives token from P i, nd this is the first time it reeived token, it must reord its stte efore it reeives the next messge from P i. fter reording its stte P j sends token to every proess it ommunites with, efore sending them ny other messge. Distriuted Systems Fö 5-31 Distriuted Systems Fö 5-32 Glol Stte Reording (ont d) Glol Stte Reording (ont d) Wht out the hnnel sttes? Mye, you prefer this view: P i sends token to P j nd this is the first time P j reeived token P j immeditely reords its stte. ll the messges sent y P i efore sending the token hve een reeived t P j SCh ij :=. P j reeives token from P k,utp j lredy reorded its stte. M is the set of messges tht Pj reeived from P k fter P j reorded its stte nd efore P j reeived the token from P k SCh kj := M. i P i P k k token sending P k messges sent fter LS k reorded messges sent efore LS k reorded stte of hnnel Ch kj P j LS k token from P k messges reeived fter LS j reorded messges reeived efore LS j reorded LS j P j j stte of Ch kj Don t forget when you look to the piture: we ssumed tht messge pssing on hnnel onneting two proesses is FIFO. The lgorithm termintes when ll proesses hve reeived tokens on ll their input hnnels. The proess tht initited the snpshot should e informed; it n ollet the glol snpshot.
Distriuted Systems Fö 5-33 Distriuted Systems Fö 5-34 Glol Stte Reording (ont d) Summry The lgorithm Rule for sender P i : /* performed y the inititing proess nd y ny other proess t the reeption of the first token */ [SR1]: P i reords its stte. [SR2]: P i sends token on eh of its outgoing hnnels. Rule for reeiver P j : /* exeuted whenever P j reeives token from nother proess P i on hnnel Ch ij */ [RR1]: if P j hs not yet reorded its stte then Reord the stte of the hnnel: SCh ij :=. Follow the "Rule for sender". else Reord the stte of the hnnel: SCh ij := M, where M is the set of messges tht Pj reeived from P i fter P j reorded its stte nd efore P j reeived the token on Ch ij. end if. In distriuted system there is no ext notion of glol physil time. Physil loks n e synhronized to ertin ury. In mny pplitions not physil time is importnt ut only the reltive ordering of ertin events. Suh n ordering n e hieved using logil loks. Lmport s logil loks re implemented using monotoni integer ounter t eh site. They n e used in order to pture the hppened-efore reltion. The min prolem with Lmport s loks is tht they re not powerful enough to perform usl ordering of events. Vetor loks give the ility to deide whether two events re uslly relted or not, y simply looking t their timestmps. Distriuted Systems Fö 5-35 Summry (ont d) s there doesn t exist glol notion of physil time, it is very diffiult to reson out glol stte in distriuted system. We n onsider glol stte s olletion of lol sttes nd, possily, set of sttes of the ommunition hnnels. glol stte n e onsistent or not. ut is grphil representtion of glol stte. Using uts it is esy to elegntly reson out onsisteny of glol sttes. It is possile to reord lol sttes nd sttes of the hnnels, so tht together they provide onsistent view of the system. Suh view is lled snpshot.