3/4/14. Today's Lecture Distributed Systems. Why Global Timing? Impact of Clock Synchronization. Time Standards. Replicated Database Update

Transcription

1 Toy's Ltur Distriut Systms N or synhroniztion Tim synhroniztion thniqus Ltur 9 Tim Synhroniztion Lmport Cloks Vtor Cloks Why Glol Timing? Impt o Clok Synhroniztion Suppos thr wr glolly onsistnt stnr Woul hny Who got lst st on irpln? Who sumitt inl ution i or lin? Di ns mov or snp? Whn h mhin hs its own lok, n vnt tht ourr tr nothr vnt my nvrthlss ssign n rlir. 4 Rplit Dts Upt Upting rplit ts n lving it in n inonsistnt stt Tim Stnrs UT Bs on stronomil osrvtions Grnwih Mn Tim TAI Strt Jn, 958 Eh son is 9,9,63,770 yls o rition mitt y Csium tom Hs ivrg rom UT u to slowing o rth s rottion UTC TAI + lp sons to within 0.9s o UT Currntly 35 Most rnt: Jun 30, 0 5

2 Compring Tim Stnrs Coorint Univrsl Tim (UTC) UT "UTC Is rost rom rio sttions on ln n stllit (.g. GPS) Computrs with rivrs n synhroniz thir loks with ths timing signls Signls rom ln-s sttions r urt to out 0.-0 millison Signls rom GPS r urt to out miroson Why n't w put GPS rivrs on ll our omputrs? 8 Cloks in Distriut Systm Clok Synhroniztion Algorithms Ntwork Computr loks r not gnrlly in prt grmnt Skw: th irn twn th s on two loks (t ny instnt) Computr loks r sujt to lok rit (thy ount t irnt rts) Clok rit rt: th irn pr unit o rom som il rrn lok Orinry qurtz loks rit y out s in - ys. (0-6 ss/s). High prision qurtz loks rit rt is out 0-7 or 0-8 ss/s 9 Th rltion twn lok n UTC whn loks tik t irnt rts. 0 Toy's Ltur N or synhroniztion Tim synhroniztion thniqus Prt ntworks Mssgs lwys rriv, with propgtion ly xtly Lmport Cloks Vtor Cloks Snr sns T in mssg Rivr sts lok to T+ Synhroniztion is xt

3 Synhronous ntworks Synhroniztion in th rl worl Mssgs lwys rriv, with propgtion ly t most D Rl ntworks r synhronous Mssg lys r ritrry Snr sns T in mssg Rivr sts lok to T + D/ Synhroniztion rror is t most D/ Rl ntworks r unrlil Mssgs on t lwys rriv Cristin s Tim Syn Brkly lgorithm A srvr S rivs signls rom UTC sour Pross p rqusts in m r n rivs t in m t rom S p sts its lok to t + T roun / Aury ± (T roun / - min) : us th rlist S puts t in mssg m t is min tr p snt m r. th ltst ws min or m t rriv t p th y S s lok whn m t rrivs is in th rng [t+min, t + T roun - min] p m r m t T roun is th roun trip ror y p" min is n stimt minimum roun trip Tim srvr,s 5 Cristin s lgorithm - singl srvr might il, so thy suggst th us o group o synhroniz srvrs it os not l with ulty srvrs Brkly lgorithm (lso 989) An lgorithm or intrnl synhroniztion o group o omputrs A mstr polls to ollt lok vlus rom th othrs (slvs) Th mstr uss roun trip s to stimt th slvs lok vlus It tks n vrg (liminting ny ov vrg roun trip or with ulty loks) It sns th rquir justmnt to th slvs (ttr thn sning th whih pns on th roun trip ) Msurmnts 5 omputrs, lok synhroniztion 0-5 milliss rit rt < x0-5 I mstr ils, n lt nw mstr to tk ovr (not in oun ) 6 Th Brkly Algorithm () Th Brkly Algorithm () Th mon sks ll th othr mhins or thir lok vlus. Th mhins nswr

4 Th Brkly Algorithm (3) Th mon tlls vryon how to just thir lok. Ntwork Tim Protool (NTP) A srvi or th Intrnt - synhronizs lints to Rliility UTC Primry rom srvrs runnt r onnt pths, to sll, UTC sours uthntits sours Sonry srvrs r synhroniz to primry srvrs Synhroniztion sunt - lowst lvl srvrs in usrs omputrs Figur Th Ntwork Tim Protool (NTP) Uss hirrhy o srvrs Clss srvrs hv highly-urt loks onnt irtly to tomi loks, t. Clss srvrs gt rom only Clss n Clss srvrs Clss 3 srvrs gt rom ny srvr Synhroniztion similr to Cristin s lg. Moii to us multipl on-wy mssgs inst o immit roun-trip Aury: Lol ~ms, Glol ~0ms NTP Rrn Clok Sours (997 survy) In survy o 36,479 prs, oun,733 primry n kup xtrnl rrn sours 3 rio/stllit/mom primry sours 47 GPS stllit (worlwi), GOES stllit (wstrn hmisphr) 57 WWVB rio (US) 7 WWV rio (US) 63 DCF77 rio (Europ) 6 MSF rio (UK) 5 CHU rio (Cn) 7 mom srvi (NIST n USNO (US), PTB (Grmny), NPL (UK)) 5 othr (prision PPS sours, t.),50 lol lok kup sours (us only i ll othr sours il) For som rson or othr, 88 o th,733 sours ppr own t th o th survy Ul Mstr Tim Fility (MTF) (rom Jnury 000) Sptrom 870 WWVB Rivr Sptrom 883 GPS Rivr Sptrom 870 WWVB Rivr Sptrom 883 GPS Rivr Hwltt Pkr 05A Qurtz Frquny Stnr NTP Protool All mos us UDP Eh mssg rs stmps o rnt vnts: Lol s o Sn n Riv o prvious mssg Lol s o Sn o urrnt mssg Ripint nots th o ript T 3 (w hv T 0, T, T, T 3 ) Srvr T T Tim Hwltt Pkr 506A Csium Bm Frquny Stnr m m' Clint T 0 T 3 Tim 3 4 4

5 Aury o NTP How To Chng Tim Timstmps t 0 is th lint's stmp o th rqust pkt trnsmission, t is th srvr's stmp o th rqust pkt rption, t is th srvr's stmp o th rspons pkt trnsmission n t 3 is th lint's stmp o th rspons pkt rption. RTT = wit lint srvr_pro_ = (t 3 -t 0 ) (t -t ) Ost = ((t -t 0 ) + (t 3 -t ))/ = ((ost + ly) + (ost ly))/ Cn t just hng Why not? Chng th upt rt or th lok Chngs in mor grul shion Prvnts inonsistnt lol stmps NTP srvrs iltr pirs <rtt i, ost i >, stimting rliility rom vrition, llowing thm to slt prs Aury o 0s o milliss ovr Intrnt pths ( on LANs) 3/4/4 5 6 Importnt Lssons Toy's Ltur Cloks on irnt systms will lwys hv irntly Skw n rit twn loks Tim isgrmnt twn mhins n rsult in unsirl hvior Clok synhroniztion Rly on -stmp ntwork mssgs Estimt ly or mssg trnsmission Cn synhroniz to UTC or to lol sour Cloks nvr xtly synhroniz Otn inqut or istriut systms might n totlly-orr vnts might n millionth-o--son prision N or synhroniztion Tim synhroniztion thniqus Lmport Cloks Vtor Cloks 7 8 Logil Cptur just th hppns or rltionship twn vnts Disr th ininitsiml grnulrity o Corrspons roughly to uslity Logil n logil loks (Lmport 978) Evnts t thr prosss 30 5

6 Logil n logil loks (Lmport 978) Logil n logil loks (Lmport 978) Inst o synhronizing loks, vnt orring n us. I two vnts ourr t th sm pross p i (i =,, N) thn thy ourr in th orr osrv y p i, tht is th inition o: i. whn mssg, m is snt twn two prosss, sn(m) hppns or riv(m) 3. Th hppn or rltion is trnsitiv Th hppn or rltion is th rltion o usl orring (t ) (t ) us o lso us o 3 3 Logil n logil loks (Lmport 978) Not ll vnts r rlt y Consir n (irnt prosss n no hin o mssgs to rlt thm) thy r not rlt y ; thy r si to onurrnt writtn s Lmport Clok () 3 4 A logil lok is monotonilly inrsing sotwr ountr It n not rlt to physil lok. Eh pross p i hs logil lok, L i whih n us to pply logil stmps to vnts Rul : L i is inrmnt y or h vnt t pross p i Rul : () whn pross p i sns mssg m, it piggyks t = L i () whn p j rivs (m,t) it sts L j := mx(l j, t) n pplis rul or stmping th vnt riv (m) Lmport s lgorithm Eh pross i kps lol lok, L i Thr ruls:. At pross i, inrmnt L i or h vnt. To sn mssg m t pross i, pply rul n thn inlu th urrnt lol in th mssg: i.., sn(m,l i ) 3. To riv mssg (m,t) t pross j, st L j = mx(l j,t) n thn pply rul or -stmping th riv vnt Th glol L() o n vnt is just its lol For n vnt t pross i, L() = L i () Lmport Clok () 3 4 h o,, hs its logil lok initilis to zro, th lok vlus r thos immitly tr th vnt..g. or, or. or, is piggyk n gts mx(0,)+ =

7 Lmport Clok () implis L()<L( )" 3 4 Th onvrs is not tru, tht is L()<L(') os not imply ".g. L() > L() ut! 5 Lmport logil loks Lmport lok L orrs vnts onsistnt with logil hppns or orring I, thn L() < L( ) But not th onvrs L() < L( ) os not imply Similr ruls or onurrny L() = L( ) implis (or istint, ) os not imply L() = L( ) i.., Lmport loks ritrrily orr som onurrnt vnts 37 Totl-orr Lmport loks Toy's Ltur Mny systms rquir totl-orring o vnts, not prtil-orring Us Lmport s lgorithm, ut rk tis using th pross ID L() = M * L i () + i M = mximum numr o prosss i = pross ID N or synhroniztion Tim synhroniztion thniqus Lmport Cloks Vtor Cloks 40 Vtor Cloks Vtor Clok Algorithm Vtor loks ovrom th shortoming o Lmport logil loks" L() < L( ) os not imply hppn or " Gol Wnt orring tht mths uslity V() < V( ) i n only i Mtho Ll h vnt y vtor V() [,, n ] i = # vnts in pross i tht uslly pr Initilly, ll vtors [0,0,,0] For vnt on pross i, inrmnt own i Ll mssg snt with lol vtor Whn pross j rivs mssg with vtor [,,, n ]: St lol h lol ntry k to mx( k, k ) Inrmnt vlu o j 4 7

8 Vtor Cloks Vtor Cloks (,0,0) (,0,0) (,0,0) (,0,0) (,,0) (,,0) (,,0) (,,0) (0,0,) At " ours t (,0,0); ours t (,0,0) " piggyk (,0,0) on " At on ript o us mx ((0,0,0), (,0,0)) = (, 0, 0) n to own lmnt = (,,0) " Mning o =, <=, mx t or vtor stmps" ompr lmnts pirwis (,,) (0,0,) Not tht implis V()<V( ). Th onvrs is lso tru Cn you s pir o prlll vnts?" (prlll) us nithr V() <= V() nor V() <= V()" (,,) Importnt Points Cloks Cn kp losly synhroniz, ut nvr prt Logil Cloks Eno uslity rltionship Lmport loks provi only on-wy noing Vtor loks provi xt uslity inormtion Lst Ltur Clok Syn Importnt Lssons Cloks on irnt systms will lwys hv irntly Skw n rit twn loks Tim isgrmnt twn mhins n rsult in unsirl hvior Two pths to solution: synhroniz loks or nsur onsistnt loks Clok synhroniztion Rly on -stmp ntwork mssgs Estimt ly or mssg trnsmission Cn synhroniz to UTC or to lol sour 46 A sll xmpl A sll xmpl Four lotions: pithr s moun, irst s, hom plt, n thir s Tn vnts: : pithr throws ll to hom : ll rrivs t hom 3 : ttr hits ll to pithr 4 : ttr runs to irst s 5 : runnr runs to hom 6 : ll rrivs t pithr 7 : pithr throws ll to irst s 8 : runnr rrivs t hom 9 : ll rrivs t irst s 0 : ttr rrivs t irst s Pithr knows hppns or 6, whih hppns or 7 Hom plt umpir knows is or 3, whih is or 4, whih is or 8, Rltionship twn 8 n 9 is unlr 8

9 Wys to synhroniz Sn mssg rom irst s to hom? Or to ntrl kpr How long os this mssg tk to rriv? Synhroniz loks or th gm? Cloks rit million to on => son in ys Synhroniz ontinuously uring th gm? GPS, pulsrs, t Th sll xmpl rvisit y th mssg rul 0, us, y th mssg rul 4, y lol orring t hom plt 4 0, y th mssg rul Rpt trnsitivity o th ov rltions 8 9, us No pplition o th ruls yils ithr 8 9 or 9 8 Lmport on th sll xmpl Initilizing h lol lok to 0, w gt L( ) = L( ) = L( 3 ) = 3 L( 4 ) = 4 L( 5 ) = L( 6 ) = 4 L( 7 ) = 5 L( 8 ) = 5 L( 9 ) = 6 (pithr throws ll to hom) (ll rrivs t hom) (ttr hits ll to pithr) (ttr runs to irst s) (runnr runs to hom) (ll rrivs t pithr) (pithr throws ll to irst s) (runnr rrivs t hom) (ll rrivs t irst s) L( 0 ) = 7 (ttr rrivs t irst s) For our xmpl, Lmport s lgorithm sys tht th run sors! Lmport on th sll xmpl Initilizing h lol lok to 0, w gt L( ) = L( ) = L( 3 ) = 3 L( 4 ) = 4 L( 5 ) = L( 6 ) = 4 L( 7 ) = 5 L( 8 ) = 5 L( 9 ) = 6 (pithr throws ll to hom) (ll rrivs t hom) (ttr hits ll to pithr) (ttr runs to irst s) (runnr runs to hom) (ll rrivs t pithr) (pithr throws ll to irst s) (runnr rrivs t hom) (ll rrivs t irst s) L( 0 ) = 7 (ttr rrivs t irst s) For our xmpl, Lmport s lgorithm sys tht th run sors! Vtor loks on th sll xmpl Vtor Cloks () Evnt Vtor Ation [,0,0,0] pithr throws ll to hom [,0,,0] ll rrivs t hom 3 [,0,,0] ttr hits ll to pithr 4 [,0,3,0] ttr runs to irst s) 5 [0,0,0,] runnr runs to hom 6 [,0,,0] ll rrivs t pithr 7 [3,0,,0] pithr throws ll to st s 8 [,0,4,] runnr rrivs t hom 9 [3,,,0] ll rrivs t irst s 0 [3,,3,0] ttr rrivs t irst s Vtor: [p,,h,t] Conurrnt mssg trnsmission using logil loks. 54 9

10 Vtor Cloks () Vtor loks r onstrut y ltting h pross P i mintin vtor VC i with th ollowing two proprtis:. VC i [ i ] is th numr o vnts tht hv ourr so r t P i. In othr wors, VC i [ i ] is th lol logil lok t pross P i.. I VC i [ j ] = k thn P i knows tht k vnts hv ourr t P j. It is thus P i s knowlg o th lol t P j. 55 Vtor Cloks (3) Stps rri out to omplish proprty o prvious sli:. Bor xuting n vnt P i xuts VC i [ i ] VC i [i ] +.. Whn pross P i sns mssg m to P j, it sts m s (vtor) stmp ts (m) qul to VC i tr hving xut th prvious stp. 3. Upon th ript o mssg m, pross P j justs its own vtor y stting VC j [k ] mx{vc j [k ], ts (m)[k ]} or h k, tr whih it xuts th irst stp n livrs th mssg to th pplition. 56 Enoring Cusl Communition Lmport s Logil Cloks () Th "hppns-or" rltion n osrv irtly in two situtions: I n r vnts in th sm pross, n ours or, thn is tru. Enoring usl ommunition I is th vnt o mssg ing snt y on pross, n is th vnt o th mssg ing riv y nothr pross, thn Lmport s Logil Cloks () Lmport s Logil Cloks (3) Thr prosss, h with its own lok. Th loks run t irnt rts. Lmport s lgorithm orrts th loks

11 Lmport s Logil Cloks (4) Th positioning o Lmport s logil loks in istriut systms. Lmport s Logil Cloks (5) Upting ountr C i or pross P i. Bor xuting n vnt P i xuts C i C i +.. Whn pross P i sns mssg m to P j, it sts m s stmp ts (m) qul to C i tr hving xut th prvious stp. 3. Upon th ript o mssg m, pross P j justs its own lol ountr s C j mx{c j, ts (m)}, tr whih it thn xuts th irst stp n livrs th mssg to th pplition. 6 6 Lst Ltur Clok Syn Importnt Lssons Cloks on irnt systms will lwys hv irntly Skw n rit twn loks Tim isgrmnt twn mhins n rsult in unsirl hvior Two pths to solution: synhroniz loks or nsur onsistnt loks Clok synhroniztion Rly on -stmp ntwork mssgs Estimt ly or mssg trnsmission Cn synhroniz to UTC or to lol sour Distriut Prmis Th notion o is wll-in (n msurl) t h singl lotion But th rltionship twn t irnt lotions is unlr Cn minimiz isrpnis, ut nvr limint thm Rlity Sttionry GPS rivrs n gt glol with < µs rror Fw systms sign to us this 63 Glol Positioning Systm () Glol Positioning Systm () Rl worl ts tht omplit GPS It tks whil or t on stllit s position rhs th rivr. Th rivr s lok is gnrlly not in synh with tht o stllit. Computing position in two-imnsionl sp

12 Srvr popultion y strtum (997 survy) Clint popultion y strtum (997 survy) NTP - synhronistion o srvrs Cloks () Th synhroniztion sunt n ronigur i ilurs our,.g. primry tht loss its UTC sour n om sonry sonry tht loss its primry n us nothr primry Mos o synhroniztion: Multist A srvr within high sp LAN multists to othrs whih st loks ssuming som ly (not vry urt) Prour ll A srvr pts rqusts rom othr omputrs (lik Cristiin s lgorithm). Highr ury. Usul i no hrwr multist. Symmtri Pirs o srvrs xhng mssgs ontining inormtion Us whr vry high uris r n (.g. or highr lvls) Figur 6-. Computtion o th mn solr y Cloks () N or Prision Tim Figur 6-3. TAI sons r o onstnt lngth, unlik solr sons. Lp sons r introu whn nssry to kp in phs with th sun. Distriut ts trnstion journlling n logging Stok mrkt uy n sll orrs Sur oumnt stmps (with ryptogrphi rtiition) Avition tri ontrol n position rporting Rio n TV progrmming lunh n monitoring Intrur ttion, lotion n rporting Muli synhroniztion or rl- tlonrning Intrtiv simultion vnt synhroniztion n orring Ntwork monitoring, msurmnt n ontrol Erly ttion o iling ntwork inrstrutur vis n ir onitioning quipmnt Dirntit srvis tri nginring Distriut ntwork gming n trining 7 7

13 Vtor Cloks V i [ i ] is th numr o vnts tht p i hs stmp" V i [ j ] ( j i) is th numr o vnts t p j tht p i hs n t y" Vtor lok V i t pross p i is n rry o N intgrs. initilly V i [j] = 0 or i, j =,, N. or p i stmps n vnt it sts V i [i] := V i [i] + 3. p i piggyks t = V i on vry mssg it sns 4. whn p i rivs (m,t) it sts V i [j] := mx(v i [j], t[j]) j =,, N ( thn or nxt vnt s to own lmnt using rul ) 73 3