The Synchronization of Periodic Routing Messages

Transcription

1 The Synchrnizatin f Peridic Ruting Messages Sally Flyd and Van Jacbsn, 3 Lawrence Berkeley Labratry, One Cycltrn Rad, Berkeley CA 9472, flyd@eelblgv, van@eelblgv T appear in the April 994 IEEE/ACM Transactins n Netwrking Abstract The paper cnsiders a netwrk with many apparently-independent peridic prcesses and discusses ne methd by which these prcesses can inadvertently becme synchrnized In particular, we study the synchrnizatin f peridic ruting messages, and ffer guidelines n hw t avid inadvertent synchrnizatin Using simulatins and analysis, we study the prcess f synchrnizatin and shw that the transitin frm unsynchrnized t synchrnized traffic is nt ne f gradual degradatin but is instead a very abrupt phase transitin : in general, the additin f a single ruter will cnvert a cmpletely unsynchrnized traffic stream int a cmpletely synchrnized ne We shw that synchrnizatin can be avided by the additin f randmizatin t the traffic surces and quantify hw much randmizatin is necessary In additin, we argue that the inadvertent synchrnizatin f peridic prcesses is likely t becme an increasing prblem in cmputer netwrks Intrductin A substantial, and increasing, fractin f the traffic in tday s cmputer netwrks cmes frm peridic traffic surces; eamples include the peridic echange f ruting messages between gateways r the distributin f real-time audi r vide Netwrk architects usually assume that since the surces f this peridic traffic are independent, the resulting traffic will be independent and uncrrelated Eg, even thugh each ruting prcess might generate a packet at fied, 3 secnd intervals, the ttal ruting traffic bserved at any pint in the netwrk shuld be smth and unifrm since the prcesses are n separate ndes and started with a randm relative phase Hwever, many netwrk traffic studies [Pa93a, SaAgGuJa92, Ja92, BrChClP93] shw that the ttal traffic is nt unifrm but instead is highly synchrnized This paper argues that the architect s intuitin that independent surces give rise t uncrrelated aggregate traffic is simply wrng and shuld be replaced by epectatins mre in line with bserved 3 This wrk was supprted by the Directr, Office f Energy Research, Scientific Cmputing Staff, f the US Department f Energy under Cntract N DE-AC3-76SF98 This paper is an epanded versin f [FJ93]

2 reality There is a huge bdy f research n the tendency f dynamic systems t synchrnize in the presence f weak cupling [Bl88] As far back as the mid-seventeenth century, Huygens nticed that tw unsynchrnized pendulum clcks wuld keep in time if hung n the same wall, synchrnized by the barely-perceptible vibratins each induced in the wall As reprted in [Bl88], synchrnizatin has been studied in electrnic circuits, a wide range f mechanical bjects, and bilgical systems such as cell ppulatins and cmmunities f fireflies Mst f these systems ehibit a tendency twards synchrnizatin that is independent f the physical cnstants and initial cnditins f the system [En92] This research suggests that a cmple cupled system like a mdern cmputer netwrk evlves t a state f rder and synchrnizatin if left t itself Where synchrnizatin des harm, as in the case f highly crrelated, bursty ruting traffic, it is up t netwrk and prtcl designers t engineer ut the rder that nature tries t put in This paper investigates ne means by which independent surces f peridic traffic can becme synchrnized An analytic mdel is develped that shares many f the features bserved in simulatins and in real traffic measurements There are tw main results frm this mdel: The transitin frm unsynchrnized t synchrnized behavir is very abrupt The traffic des nt gradually clump up and becme mre synchrnized as netwrk parameters change Instead, fr each set f prtcl parameters and implementatin interactin strengths there eists a clearly defined transitin threshld If the number f surces is belw the transitin threshld, the traffic will almst certainly be unsynchrnized and, even if synchrnized by sme eternal frce 2 it will unsynchrnize ver time Cnversely, if the number f surces is abve the threshld, the traffic will almst certainly be synchrnized and, even if placed in an unsynchrnized state by sme eternal frce, will evlve t synchrnizatin ver time The amunt f randmness that must be injected t prevent synchrnizatin is surprisingly large Fr eample, in the Xer PARC internal netwrk, measurements [De93] shw their cisc ruters require rughly 3 ms t prcess a ruting message ( ms per rute times 3 rutes per update) Frm the results in Sectin 5, the ruters wuld have t add at least a secnd f randmness t their update intervals t prevent synchrnizatin There are many eamples f unanticipated synchrnized behavir in netwrks: TCP windw increase/decrease cycles A well-knwn eample f unintended synchrnizatin is the synchrnizatin f the windw increase/decrease cycles f separate TCP cnnectins sharing a cmmn bttleneck gateway [ZhCl9] This eample illustrates that unless we actively engineer t avid synchrnizatin, such as by injecting randmness int the netwrk, synchrnizatin is likely t be the equilibrium state As an eample f injecting randmness, the synchrnizatin f windw increase/decrease cycles can be avided by adding randmizatin t the gateway s algrithm fr chsing packets t drp during perids f cngestin[fj92] (This randmizatin has the advantage f aviding ther unintended phase effects as well) Synchrnizatin t an eternal clck Tw prcesses can becme synchrnized with each ther simply by bth being synchrnized t an eternal clck Fr eample, [Pa93a] shws DECnet traffic peaks n the hur and half-hur intervals; [Pa93b] shws peaks in ftp traffic as several users fetch the mst recent weather map frm Clrad every hur n the hur Client-server mdels Multiple clients can becme synchrnized as they wait fr service frm a busy r recvering server Fr eample, in the Sprite perating system clients check 2 Eg, by restarting all the ruters at the same time because f a pwer failure 2

3 with the file server every 3 secnds; in an early versin f the system, when the file server recvered after a failure, r after a busy perid, a number f clients wuld becme synchrnized in their recvery prcedures Because the recvery prcedures invlved synchrnized timeuts, this synchrnizatin resulted in a substantial delay in the recvery prcedure [Ba92] Peridic ruting messages Unlike the client/server mdel r the eternal clck mdel, the synchrnizatin f peridic ruting messages invlves seemingly-independent peridic prcesses There are many ruting prtcls where each ruter transmits a ruting message at peridic intervals Assuming that the ruters n a netwrk are initially unsynchrnized, at first glance it might seem that the peridic messages frm the different ruters wuld remain unsynchrnized This paper eplres hw initially-unsynchrnized ruting messages can becme synchrnized We eamine the details f ruter synchrnizatin t give a cncrete eample f inadvertent synchrnizatin, t underline the necessity f actively designing t avid synchrnizatin, and t emphasize the utility f injecting randmizatin as a methd f breaking up synchrnizatin When a particular instance f synchrnizatin is bserved, it is usually easy t suggest prtcl changes that culd prevent it This misses the pint Synchrnizatin is nt a small prblem caused by minr versights in prtcl design The tendency f weakly-cupled systems t synchrnize is quite strng and changing a deterministic prtcl t crrect ne instance f synchrnizatin is likely t make anther appear Varius frms f peridic traffic are becming an increasingly-large cmpnent f Internet traffic This peridic traffic includes nt nly ruting updates and traffic resulting frm the increasing use f peridic backgrund scripts by individual users [Pa93a], but realtime traffic (such as vide traffic) that has a peridic structure Althugh the peridic structure f vide traffic is generally nt affected by feedback frm the netwrk, there are still pssibilities fr synchrnizatin Fr eample, individual variable-bit-rate vide cnnectins sharing a bttleneck gateway and transmitting the same number f frames per secnd culd cntribute t a larger peridic traffic pattern in the netwrk As peridic traffic increases in the Internet, it becmes increasingly imprtant fr netwrk researchers t cnsider questins f netwrk synchrnizatin We use bth simulatin and analysis t eplre the synchrnizatin f peridic ruting messages The first gal f the analysis is t eamine the rle that randm fluctuatins in timing play in the synchrnizatin f ruting messages These randm fluctuatins cntribute t bth the frmatin f synchrnizatin and t the breaking up f synchrnizatin after it ccurs One way t break up synchrnizatin is fr each ruter t add a (sufficiently large) randm cmpnent t the perid between ruting messages A secnd gal f ur analysis is t investigate this eplicit additin f a randm cmpnent t the ruting timer, and t specify the magnitude f the randm cmpnent necessary t prevent synchrnizatin Sectin 2 gives eamples f peridic traffic patterns in the Internet; Sectin 3 describes ur mdel f peridic ruting messages n a netwrk Sectin 4 eplains the results f ur simulatins Sectin 5 describes a Markv chain used t analyze sme aspects f the Peridic Messages mdel Sectin 7 presents cnclusins and discusses alternatives fr preventing ruting message synchrnizatin 3

4 2 Peridic traffic patterns in the Internet This sectin gives an eample f synchrnized ruting messages, and several eamples f peridic traffic patterns in the Internet (sme f which are caused by peridic ruting messages) While we d nt have direct evidence f peratinal prblems in the Internet related t synchrnized ruting messages, we shw indirect evidence that such prblems culd eist In general, there are significant patterns f peridic packet drps and delays in the Internet We began this investigatin in 988 after bserving synchrnized ruting messages frm DECnet s DNA Phase IV (the DIGITAL Netwrk Architecture) [VMS88] n a lcal Ethernet at LBL (Lawrence Berkeley Labratry) Each DECnet ruter transmitted a ruting message at 2- secnd intervals Within hurs after bringing up the ruters n the netwrk after a failure, the ruting messages frm the varius ruters were cmpletely synchrnized In May 992, in the curse f investigating packet lss rates in the Internet, we cnducted eperiments sending runs f a thusand pings each, at rughly ne-secnd intervals, frm Berkeley and ther sites t destinatins acrss the Internet Fr all f the runs t destinatins at Harvard r MIT, at least three percent f the ping packets were drpped, regardless f the time f day Figure shws a particular run f a thusand pings frm Berkeley t MIT; the -ais shws the ping number and the y-ais shws the rundtrip time Drpped packets are represented by a negative rundtrip time Figure 2 shws the autcrrelatin functin fr the rundtrip times in Figure, where the drpped packets are assigned a rundtrip time f tw secnds (higher than the largest rundtrip time in the eperiment) The pattern f peridic packet drps at 9-secnd intervals is illustrated in bth figures Further eperiments determined that these packet drps were ccurring at the NEARnet (New England Academic and Research Netwrk) cre ruters Earlier investigatin f Internet behavir had als reprted a degradatin in service with a 9-secnd peridicity n paths t MIT [SaAgGuJa92] These packet drps were determined t be caused by IGRP (the Inter-Gateway Ruting Prtcl [He9]) ruting updates at the NEARnet ruters [Sc92] The ruters were unable t frward ther packets while large ruting updates were being prcessed The particular prblem f peridic packet lsses n NEARnet has since been reslved; the ruter sftware has been changed s that nrmal packet ruting can be carried ut while the ruters are dealing with ruting update messages Althugh it has been speculated that these packet drps were als cnnected with synchrnizatin, it is unclear, and there is n direct evidence [Sc92, L93] Peridic packet drps have been demnstrated assciated with RIP (the Ruting Infrmatin Prtcl [He88]) as well as with IGRP Figure 3 shws audi packet lsses during an audicast 3 f the December 992 Packet Vide wrkshp [Ja92] The -ais shws the time in secnds; the y-ais shws the duratin f each audi utage in secnds The little blips mre-r-less randmly spread alng the time ais represent single packet lsses The larger lss spikes are strngly peridic; they ccur every 3 secnds and last fr several secnds at a time During these events the packet lss rate ranges frm 5 t 85% and there are frequent single utages f -5 ms These peridic lsses are almst certainly due t the surce-ruted (tunneled) multicast packets cmpeting with ruting updates and lsing Because 3 secnds is the default update time fr RIP, these lng intervals f packet lsses are cnjectured t result frm RIP ruting updates; it is nt knwn if this prblem invlves synchrnizatin In ther instances peridic 3-secnd audi 3 Fr a reprt n the first such audicast, see [Ca92] 4

5 Rundtrip time (secnds) Ping number Figure : Peridic packet lsses frm IGRP ruting messages Autcrrelatin Pings Figure 2: The autcrrelatin f rundtrip times Audi lst (secnds) Time (secnds) Figure 3: Peridic packet lsses at 3-secnd intervals packet lsses have been cnclusively traced t RIP ruting updates [De93], and there is sme indirect evidence f synchrnizatin In ur ping eperiments f the Internet in May 992 we fund many eamples f peridic packet drps fr which we have n eplanatin Fr eample, we fund paths with packet drps 5

6 every 38 secnds, paths with packet drps every 5 secnds, and paths with large delays every 45 secnds We fund different peridic patterns n the lcal path frm LBL t the UC Berkeley campus at different times f the day Frm ur ping eperiments, we cnjecture that a significant number f packet drps in the Internet are assciated with peridic prcesses f ne type r anther 3 The Peridic Messages mdel This sectin describes a general mdel f peridic ruting messages n a netwrk; we call this the Peridic Messages mdel This mdel was initially patterned after DECnet s DNA Phase IV, but ther ruting prtcls that can cnfrm t this mdel include EGP (Eterir Gateway Prtcl) [M84], Hell [Mi83], IGRP, and RIP In these ruting prtcls, each ruter n a netwrk transmits a ruting message at peridic intervals This ensures that ruting tables are kept up-t-date even if ruting update messages are ccasinally lst The Peridic Messages mdel behaves as fllws: The ruter prepares and sends a ruting message In the absence f incming ruting messages, the ruter resets its timer T c secnds after step begins Other ruters receive the first packet f this ruter s ruting message T d secnds after step begins 2 If the ruter receives an incming ruting message (r the first packet f an incming ruting message) while preparing its wn utging ruting message, the ruter als prcesses the incming ruting message The ruter takes T c2 secnds t prcess an incming ruting message 3 After cmpleting steps and 2, the ruter sets its timer The time until the timer net epires is unifrmly drawn frm the interval [T p T r, T p T r ] secnds, where T p is the average perid and T r represents a randm cmpnent; this culd be a (small) randm fluctuatin due t unavidable variatins in perating system verhead r a (larger) fluctuatin due t a randm cmpnent intentinally added t the system When the timer epires, the ruter ges t step 4 If the ruter receives an incming ruting message after the timer has been set, the incming ruting message is prcessed immediately If the incming ruting message is a triggered update caused by a majr change in the netwrk such as the failure f a link, then the ruter ges t step, withut waiting fr the timer t epire Because the ruter resets its timer nly after prcessing its wn utging ruting message and any incming ruting messages, the timing f ne ruter s ruting messages can be affected by the ruting messages frm ther ndes This gives the weak cupling between ruters, allwing the synchrnizatin f ruting messages frm several ruters The Peridic Messages mdel ignres prperties f physical netwrks such as the pssibility f cllisins and retransmissins n an Ethernet The Peridic Messages mdel is nt intended t replicate the eact behavir f peridic ruting messages, but t capture sme significant characteristics f that behavir RIP and IGRP are intradmain ruting prtcls that use peridic ruting messages In RIP each ruter transmits peridic ruting messages every 3 secnds In IGRP, ruters send ruting messages at 9-secnd intervals EGP (Eterir Gateway Prtcl) is used in sme places between the NSFNET backbne and its attached reginal netwrks; EGP ruters send update messages every three minutes 4 In the 4 With BGP (Brder Gateway Prtcl), which runs n tp f TCP, incremental update messages are sent as the 6

7 988 LBL netwrk, DECnet ruters implementing DNA Phase IV sent ruting messages every tw minutes IGRP, RIP, and DECnet s DNA Phase IV all incrprate triggered updates, where ruting messages are sent immediately in respnse t a netwrk change such as the remval f a rute The first triggered update results in a wave f triggered updates frm neighbring ruters Nt all implementatins f these ruting prtcls crrespnd t the Peridic Messages mdel in this paper The RFC fr RIP [He88] mentins that when there are many gateways n a single netwrk, there is a tendency fr the peridic ruting messages t synchrnize The RFC specifies that in rder t avid this synchrnizatin, either the ruting messages must be triggered by a clck that is nt affected by the time required t service the previus message, r a small randm time must be added t the 3-secnd ruting timer each time, thugh the magnitude f the randm time is nt specified As an eample f implementatins that dn t cnfrm t the Peridic Messages mdel, in sme implementatins f IGRP and RIP ruters reset their ruting timers befre the utging ruting message is prepared, and ruters dn t reset their ruting timers after triggered updates [Li93] Thus the Peridic Messages mdel illustrates nly ne pssible mechanism by which ruting messages can becme synchrnized Wherever there are interactins between ruters, r between a ruter and the netwrk, there culd eist mechanisms that lead t synchrnizatin 4 Simulatins This sectin describes simulatins f the Peridic Messages mdel These simulatins shw the behavir f a netwrk with N ruting ndes n a single bradcast netwrk, fr N = 2 In the first set f simulatins the peridic ruting messages fr the N ndes are initially unsynchrnized; in the secnd set the peridic messages are initially clustered The simulatins shw that the behavir f the Peridic Messages system is determined by the randm verhead added t each nde s peridic timer As the level f randmizatin increases, the system s ability t break up clusters f synchrnized ruting messages als increases Definitins: T p,t r,t c,t c2, and T d The time T p is the cnstant cmpnent f the peridic timer and T r is the magnitude f the randm cmpnent Each ruter s ruting timer is drawn at each rund frm the unifrm distributin n [T p - T r, T p T r ] secnds Each ruter requires T c secnds f cmputatin time t prcess an utging ruting message, and T c2 secnds f cmputatin time t prcess an incming ruting message; each ruting message culd cnsist f multiple packets In this paper we assume that T c2 and T c are the same T d secnds after a ruter s ruting timer epires, ther ruters receive the first packet f the ruting message 2 Fr the simulatins in this sectin, T p is 2 secnds, T c is secnds, and T d is set t zer; fr the initial simulatins in this sectin T r is set t secnds The average timer-value f 2 secnds was chsen t give a minimum timer-value cmparable t the 2-secnd timer used by the DECnet ruters n ur lcal netwrk The value f secnds fr T c was chsen smewhat arbitrarily t mdel an estimated cmputatin time f secnds and transmissin time f secnds fr a ruter t cmpute and transmit packets fr an utging ruting message after a timer epiratin; these values are nt based n any measurements f actual netwrks Sectin 53 discusses hw the results scale with different values fr the varius parameters ruting table changes 7

8 Time (secnds) Time ffset (secnds) (826 runds in all) Figure 4: A simulatin shwing synchrnized ruting messages Time (secnds) Time ffset (secnds) : timer is set : timer epires A: B: Figure 5: An enlargement f the simulatin abve When a nde s ruting timer epires, the nde takes T c secnds t prepare and transmit its ruting message We call this time the busy perid Fr each ruting message received while a nde is in its busy perid, that nde s busy perid is etended by the T c2 = T c secnds required t prcess an incming ruting message Fr simplicity, in the simulatins in this sectin T d is set t zer; that is, when nde A s timer epires the ther ndes immediately receive the first packet f nde A s ruting message Thus in the simulatins, when nde A s timer epires nde A immediately spends T c secnds preparing and transmitting its ruting message, and at the same time the ther ruting ndes each spend T c2 = T c secnds receiving and prcessing the ruting message frm nde A This assumptin mst plausibly reflects a netwrk with lw prpagatin delay, where a ruter s ruting message cnsists f several packets transmitted ver a T c -secnd perid Sectin 54 shws the results f simulatins with T d > The first set f simulatins investigates the prcess by which initially-unsynchrnized ruting 8

9 Largest cluster Time (secnds) Figure 6: The cluster graph, shwing the largest cluster fr each rund messages becme synchrnized The ruting messages fr the N ndes are initially unsynchrnized; fr each nde the time at which the first ruting message is sent is chsen frm the unifrm distributin n [, T p ] secnds Fr the simulatin in Figure 4, T r is set t secnds Each jittery line in Figure 4 is cmpsed f hundreds f pints, and each pint represents ne ruting message sent by a ruting nde The -ais shws the time in secnds at which the ruting message was sent, and the y-ais shws the time-ffset, ie, the time mdul T, fr T = T p T c secnds This time-ffset gives the time that each ruting message was sent relative t the start f each rund The simulatin in Figure 4 begins with unsynchrnized ruting messages and ends with the N=2 ruters transmitting their ruting messages at essentially the same time each rund At the left-hand side f the figure the twenty jittery lines represent the time-ffsets f the transmit times fr the twenty ndes In the absence f synchrnizatin each ruter s timer epires, n the average, T p T c secnds after that ruter s previus timer epiratin These successive timer epiratins give a jittery but generally hrizntal line fr the timer epiratins fr a single ruter Hwever, as we eplain belw, when ruters becme synchrnized this increases the time interval between successive ruting messages frm a single ruter At the end f the simulatin the ruting messages are fully synchrnized, and all f the ndes set their timers at the same time each rund In this case each ruter has a busy perid f 2 3 T c secnds rather than f T c secnds, increasing the time interval between successive ruting messages Figure 5 is an enlargement f a small sectin f Figure 4 This figure illustrates the synchrnizatin f ruting messages frm tw ruters; each marks a timer epiratin, and each 2 marks the timer being reset In the first five runds f Figure 5 the tw ndes are independent, and each nde sets its timer eactly T c secnds after its previus timer epires Hwever, in the sith rund, nde A s timer epires, say, at time t, and nde A begins preparing its ruting message Befre nde A finishes preparing and sending its ruting message, nde B s timer epires; nde A has t finish sending its wn ruting message and t prcess nde B s ruting message befre it can reset its wn timer These tw tasks take T c T c2 = 2T c secnds, s nde A resets its timer at time t 2T c In ur mdel nde B begins prcessing nde A s ruting message at time t T d, and in the simulatin in Figure 5 T d is set t zer While nde B is receiving and prcessing nde A s ruting message, nde B s wn timer epires; nde B has t prepare and send its wn utging ruting message and finish prcessing nde A s ruting message befre resetting its timer These tasks take T c T c2 = 2T c secnds, s fr T d = nde B als resets its timer at time t T d 2T c = t 2T c At this pint nde A and nde B are synchrnized and we say that they frm a cluster; nde A and 9

10 nde B set their timers at the same time The tw ndes remain synchrnized, setting their timers at rughly the same time, as lng as the timers epire within T c T d secnds f each ther each rund The cluster breaks up again when, because f the randm cmpnent, nde A and nde B s timers epire mre than T c T d secnds apart Mre generally, a cluster f size i refers t a set f i ruting messages that have becme synchrnized Each f the i ndes in a cluster is busy prcessing incming ruting messages and preparing its wn utging ruting message fr it c secnds after the first timer in the cluster epires Fr T d =, the i ndes in a cluster reset their timers at eactly the same time One way t think f the simulatin in Figure 4 is as a system f N particles, each with sme randm mvement in a ne-dimensinal space Fr a particle in a lne cluster (a cluster f size ne), each timer-ffset differs frm the previus rund s timer-ffset by an amunt drawn frm the unifrm distributin n [-T r,t r ] secnds In Figure 4 the successive timer-ffsets fr an unsynchrnized ruting nde (the mvement f a single particle) are represented by a jittery but generally hrizntal line Fr particles (r ruting ndes) in a cluster f size i, T c ( i )T c2 = it c secnds are spent prcessing ruting messages after the first timer f the cluster epires; then the ndes in the cluster all reset their timers A cluster f i particles mves ahead a distance f rughly ( i ) c Tsecnds in each rund In Figure 4 the mvement f a cluster is represented by an irregular line with psitive slpe; the larger the cluster, the steeper the slpe When tw clusters meet, the ndes in the tw clusters all reset their timers at the same time; the tw clusters merge, fr the mment, int a larger cluster As Figure 4 shws, a cluster f i particles can smetimes break up int tw smaller clusters Even thugh the i ndes set their ruting timers at the same time, it is pssible fr ne nde s ruting timer t epire mre than T c T d secnds befre any f the ther ndes in the cluster, because f the randm cmpnent in the timer interval fr each nde When this happens, the first nde breaks ut f the cluster, as discussed further in Sectin 5 The break-up f a cluster can be seen in Figure 5 where a cluster f size tw frms and then breaks up again The first part f the simulatin in Figure 4 shws small clusters ccasinally frming and breaking up Twards the end f the simulatin a sufficiently-large cluster is frmed, mving rapidly acrss the space and incrprating all f the unclustered ndes that it encunters alng its path As the cluster size increases, the average perid f the cluster als increases; the larger the cluster, the mre quickly it bumps int and incrprates the smaller clusters A simulatin at any pint in time can be partially characterized by the size f the largest cluster f ruting messages Figure 6 shws a cluster-graph f the simulatin in Figure 4 The -ais shws time and the y-ais shws the size f the largest cluster in the current rund f N ruting messages Figure 7 shws a simulatin identical t that in Figure 4, ecept that the simulatin was started with a different randm seed Unlike the simulatin in Figure 4, the simulatin in Figure 7 ends with unsynchrnized ruting messages Fr the simulatin in Figure 7, a cluster as large as five ccasinally frms but each time the cluster breaks up again Figure 9 shws the cluster graphs frm several simulatins that start with unsynchrnized ruting messages The parameters are the same as the previus simulatins, ecept that the randm cmpnent T r ranges frm :6T c t : 4T c Nte that the time scale is different frm the cluster graphs n previus pages; in Figure 9 the simulatins run fr 7 secnds (5 days) instead f 5

11 Time (secnds) Time ffset (secnds) (826 runds in all) Figure 7: A simulatin shwing unsynchrnized ruting messages Time (secnds) Largest cluster Figure 8: The cluster graph, shwing the largest cluster fr each rund secnds (just ver day) As the randm cmpnent increases, the simulatins take lnger and lnger t synchrnize These simulatins d nt specifically include triggered updates, triggered by a change in the netwrk We can instead begin ur simulatins with synchrnized ruting messages, which can result frm triggered updates These simulatins are shwn in Figure ; the randm cmpnent T r ranges frm 2: 3T c t 2: 8T c As the randm cmpnent increases, the simulatins return mre quickly t the unsynchrnized state Our simulatin results are cnsistent with simulatins f the same mdel in [Tr92] In additin t simulatins, preliminary results frm eperiments by Treese have shwn synchrnizatin f systems n an Ethernet [Tr94] The eperiments use an algrithm similar t the Peridic Messages mdel The results suggest that the Peridic Messages mdel captures a realistic pssible behavir f real cmputer systems

12 Tr = 6 Tc Time Largest cluster e e synchrnizatin after 498 runds (7 hurs) Tr = Tc Time Largest cluster e e synchrnizatin after 7,798 runds ( days) Tr = 4 Tc Time Largest cluster e e n synchrnizatin Figure 9: Simulatins starting with unsynchrnized updates, fr different values fr T r Tr = 23 Tc Time Largest cluster e e synchrnizatin nt brken Tr = 25 Tc Time Largest cluster e e synchrnizatin brken after 4,79 runds (7 days) Tr = 28 Tc Time Largest cluster e e synchrnizatin brken after 3 runds ( hurs) Figure : Simulatins starting with synchrnized updates, fr different values fr T r 5 The Markv chain mdel This sectin uses a Markv chain mdel t further eplre the behavir f the Peridic Messages system The Markv chain eplres the behavir f a system f N ruters that each implement the Peridic Messages mdel described in the previus sectin The Markv chain mdel assumes that each ruter receives a peridic ruting message frm every ther ruter; this wuld be the case, fr eample, fr N ruters n a bradcast netwrk Sectin 56 discusses the issues in etending these results t N ruters cnnected in an arbitrary tplgy, Sectin 54 discusses the effects f a nnzer transmissin and prpagatin delay between ruters, and Sectin 6 discusses ther analytical appraches t synchrnizatin The Markv chain mdel is used t cmpute the epected time fr the system t mve frm an unsynchrnized state t a synchrnized state, and vice versa This Markv chain mdel uses several simplifying assumptins, and therefre nly apprimates the behavir f the Peridic Messages mdel Nevertheless, the Markv chain mdel illustrates sme significant prperties f the simulatins f the Peridic Messages mdel The Markv chain has N states; when the largest cluster frm a rund f N ruting messages is f size i, the Markv chain is defined t be in state i Figure shws the Markv chain, alng with the transitin prbabilities The transitin prbability p i;j is the prbability that a Markv chain in state i mves t state j in the net rund The Markv chain mdel is based n several simplifying assumptins: The first simplifying assumptin f the Markv chain mdel is that the future behavir f 2

13 p, p2,2 pi,i p N,N p,2 p2,3 p p i,i i,i pn,n size size size size 2 i N p2, p3,2 pi,i pi,i pn,n Figure : The Markv chain the system depends nly n the current state and is independent f past states This assumptin is clearly nt true fr the Peridic Messages mdel, where the future behavir f the system depends nt nly n the size f the largest cluster but n the transmit times f the ther ruting messages The secnd simplifying assumptin is that the size f the largest cluster changes by at mst ne frm ne rund t the net Again, this assumptin is nt strictly accurate, particularly fr large values f Nr T r Fr eample, in the Peridic Messages mdel it is pssible fr tw clusters f sizes i and 2 respectively t merge and frm a cluster f size i 2 in the net rund The analysis f the Markv chain mdel assumes that ecept fr the largest cluster f size i, all ther clusters are lne clusters f size ne; again, this cnservative assumptin is nt strictly accurate Given a cluster f size i, the fllwing cluster is defined as the cluster that fllws the cluster f size i in time At each rund, we assume that the distance between the largest cluster f size i and the fllwing lne cluster is given by an epnential randm variable with epectatin T p =( Ni ) This distance is defined as the wait between the time when the ndes in the cluster f size i set their timer and the time when the timer epires fr the nde in the fllwing lne cluster This epected value is based n the average distance between Ni clusters As in the Peridic Messages mdel, we assume that each nde s timer epires after a time drawn frm the unifrm distributin n [T p - T r, T p T r ] secnds Fr a nde in a cluster f size i, the nde takes T c ( i ) T c2 = it c secnds t prcess the incming and utging ruting messages in the cluster, and ther ndes receive the first packet f the ruting message T d secnds after the timer epires In this sectin we assume that T c < 2T r T d ; if nt, then a cluster never breaks up int smaller clusters The net tw sectins define the transitin prbabilities fr the Markv chain Given these transitin prbabilities, we cmpute the average time fr the Markv chain t mve frm state t state N, and the average time fr the Markv chain t mve frm state N back dwn t state This analysis shws that when T r is sufficiently large, the Markv chain mves quickly frm a synchrnized state t an unsynchrnized state 5 Cluster breakup and grwth This sectin estimates p i;i, the prbability that the Markv chain mves frm state i t state i in ne rund The secnd half f this sectin estimates p i;i In the Markv chain, a cluster f size i can break up t frm a cluster f size i either by breaking up int a cluster f size ne fllwed by a cluster f size i, r vice versa Because the first f the tw cases is mre likely, 5 fr simplicity we nly cnsider this case We say that the first nde breaks away frm the head f the cluster 5 The secnd f the tw cases ccurs nly if the last nde transmits its ruting message after it has had time t prcess ruting messages frm all previus ndes in the cluster 3

14 Thus p i;iis the prbability that the nde whse timer epires first, nde A, resets its timer befre it receives any ruting messages frm any f the ther i ndes in the cluster Fr i ndes in a cluster, the i timers are all set within T d secnds f each ther; in this analysis we estimate that the i timers are all set at the same time and the timers epire at i times unifrmly distributed in a time interval f length 2T r Let L be the time frm the epiratin f the first timer until the epiratin f the secnd f the i timers In the absence f incming messages, nde A resets its timer T c secnds after its timer epires, and receives ntificatin f a ruting message frm anther nde in the cluster L T d secnds after its timer epires Because we assume that T c <2T r T d, there is always a nnzer prbability that a cluster f size i breaks up int smaller clusters Frm [F6, p22], p i;i= Prb: ( T c <L T d ) = T c T d 2T r i () fr i > Nw we estimate p i;i, the prbability that the system mves frm state i t state i in ne rund We leave p ; 2 as a variable; p ; 2 depends largely n T r, the randm change in the timer-ffsets frm ne rund t the net Fr simplicity, this analysis assumes that T c = T c2 The prbability that a cluster f size tw r mre incrprates additinal ruting ndes, frming a larger cluster, depends largely n the fact that larger clusters have larger average perids than smaller clusters After sme time the larger cluster cllides with a smaller cluster, and the tw clusters merge Fr a cluster f size i, each nde in the cluster sets its timer T c ( i ) c2 T = it c secnds after the first timer in the cluster epires (r after it receives the first packet frm that nde s ruting message) Fr T d =, each f the i timer epiratins is unifrmly distributed in the interval [T p T r, T p T r ] Given i events unifrmly distributed n the interval [, ], the epected value f the smallest event is =( i ) [F6, p24] Thus the first f the i timers epires, n average, T p T r 2T r =( i ) = p TT r ( i ) =( i ) secnds after the timers are set The average ttal perid fr a nde in a cluster f size i is therefre T p T r ( i ) =( i ) c secnds it In ne rund the timer-ffset fr a cluster f size i mves an average distance f ( i ) c T T r ( i ) =( i ) secnds relative t the timer-ffset fr a cluster f size ne Fr simplicity, in estimating p i;iwe assume that the timer-ffset fr a cluster f size i mves in each rund eactly ( i ) c TT r ( i ) =( i ) secnds relative t the timer-ffset fr a cluster f size ne (This assumptin ignres the smewhat remte pssibility that a cluster f size i culd jump ver a smaller cluster) What is the prbability that, after ne rund, the timer-ffset fr a cluster f size i mves t within T c secnds f the timer-ffset fr a cluster f size ne? The Markv chain mdel assumes that the distance between a cluster f size i and the fllwing small cluster is an epnential randm variable with epectatin T p =( Ni ) Thus fr a Markv chain in state i, p i;i is the prbability that an epnential randm variable with epectatin T p =( Ni ) is less than ( i ) c T T r ( i ) =( i ) Fr 2 i N, this gives p i;i= e ((Ni)=Tp)(( i) ctr( T i) =( i) ) : (2) Fr all i, p i;i = p i;ip i;i: 4

15 52 Average time t cluster, and t break up a cluster This sectin investigates the average time fr the Markv chain t mve frm state t state N, and vice versa Definitins: t i; j and f( i) Leti;j t be the epected number f runds until the Markv chain mves frm state i t state j, given that it mves frm state i directly t state j Let f( i ) be the epected number f runds until the Markv chain first enters state i, given that the Markv chain starts in state We leave f ( 2) as a variable 2 We give a recursive definitin fr f ( i ) fr i >2 The epected number f runds t first reach state i equals the epected number f runds t first reach state i, plus the additinal epected number f runds, after first entering state i, t enter state i After state i is first reached, the net state change is either t state i 2, with prbability ( p i;i2) =( i;i2 p p i;i), r t state i, with prbability ( p i;i) =( i;i2 p p i;i) The epected number f runds t reach state i, after first entering state i 2, is f ( i ) f ( i 2) This leads t the fllwing recursive equatin fr f ( i ) : f ( i ) = f ( i ) p i;i2 ( p i;i2 p i;i2 t f ( i ) f ( i 2) ) i;i Thus fr c( i ) = i;i( t p i;i2=p i;i) i;i2, t p i;i t i;i: p i;i2 p i;i f ( i ) p i;i2 p i;i f ( i ) p i;i2 f ( i 2) = c( i ) : (3) p i;i p i;i Frm Appendi A, equatin 3 has the slutin 6 : ix f ( i ) = f (@ m Y i X m=3 m=3 c( k) m Y k=3 j=k j=2 p j; j p j; j p j; j p j; j AA A : (4) Cnsider t j; j, the epected number f runds t mve frm state j t state j, given that the Markv chain in fact mves frm state j t state j Let P j; be the prbability that the Markv chain in state j first mves t state j n rund, given that the Markv chain mves frm state j t state j The equatin fr t j; j is as fllws [R85, p37]: t j; j = X X = ( p j; j ) p j; j = = Similarly, the equatin fr t j; j is as fllws: t j; j = P j; = p j; j ( p j; j p j; j) 2 : p j; j ( p j; j p j; j) 2 : 6 This slutin culd als be verified by the reader by substituting the right-hand side f Equatin (4) int Equatin (3) 5

16 Net we investigate the average time fr the Markv chain t mve frm state N t state Definitins: g( i) Let g( i ) be the epected number f runds fr the Markv chain t first enter state i, given that the Markv chain starts in state N Thus g( N) = and g( i ) = g( i ) p i;i2 ( p i;i2 p i;i2 t g( i ) g( i 2) ) i;i p i;i t i;i: p i;i2 p i;i Fr d( i ) = i;i( t p i;i2=p i;i) i;i2, t this gives the recursive equatin g( i ) p i;i2 p i;i g( i ) p i;i2 g( i 2) = d( i ) : (5) p i;i p i;i Equatin 5 has the slutin belw: g( i ) N X = m=i N X d( k) k Y j=m p j; j p j; j A (6) The derivatin f this equatin is similar t that f f ( i ) Nte that this equatin des nt depend n the values f p ; 2 r f f ( 2) Cluster size Time (in secnds) t reach given cluster size, frm size (Slid line frm analysis, dtted lines frm simulatins) Figure 2: The epected time t reach cluster size i, starting frm cluster size, fr T r = : secnds The slid line in Figure 2 shws f ( i ), cmputed frm Equatin 4, fr N = 2, p T= 2 secnds, T c = : secnds, T r = : secnds, T d = secnds, and f ( 2) = 9 runds (This value fr f ( 2) is based bth n simulatins and n an apprimate analysis that is nt given here 7 ) The -ais shws the time in secnds, cmputed as ( T p T c ) f ( i ) The y-ais shws the cluster size i ;a 7 The dynamics fr mving frm a cluster f size tw r mre t a larger cluster is based largely n the fact that larger clusters have larger average perids In cntrast, the dynamics fr mving frm a cluster f size ne t a cluster f size tw depends n hw frequently tw clusters f size ne cllide, where all clusters f size ne have the same average perid; this requires a different analysis 6

17 Cluster size Time (in secnds) t reach given cluster size, frm size 2 (Slid line frm analysis, dtted line frm simulatins) Figure 3: The epected time t reach cluster size i, starting frm cluster size N, fr T r = : 3 secnds mark is placed at cluster size i when the system first reaches that cluster size The results f twenty simulatins are shwn by light dashed lines Each simulatin was started with unsynchrnized ruting messages, with the values fr N, T p, T c, T d, and T r described abve; these simulatins differ nly in the randm seed The heavy dashed line shws the results averaged frm twenty simulatins The slid line in Figure 3 shws g( i ), cmputed frm Equatin 6, fr the same parameters fr N, T p, T d, and T c as in Figure 2, and fr T r = : 3 secnds; fr the value f T r in Figure 2, the system takes a lng time t unsynchrnize, making simulatins unrealistic The heavy dtted line averages the results frm twenty simulatins Figures 2 and 3 shw that the average times predicted by the Markv chain are tw r three times the average times frm the simulatins This discrepancy is nt surprising, because the Markv chain is nly a rugh apprimatin f the behavir f the Peridic Messages system Aside frm the difference in magnitude, hwever, the functins predicted frm the Markv chain and cmputed frm the simulatins are reasnably similar Thus the Markv chain mdel des in fact capture sme essential prperties f the Peridic Messages system 53 Results frm the Markv chain mdel This sectin eplres the general behavir f the Markv chain mdel We cmpute the epected time fr the Markv chain t synchrnize and t unsynchrnize, fr a range f values fr N, T c, and T r, and cmpare these analytical results t the results f simulatins This cmparisn shws that the Markv chain mdel is eplanatry rather than predictive; the Markv chain mdel and the simulatins ehibit the same qualitative behavir, and the Markv chain mdel can be used t eplain the behavir f the simulatins, but the Markv mdel is nt sufficiently accurate t predict the eact results f the simulatins The analysis in this sectin, alng with the simulatins, shws that fr a wide range f parameters, chsing T r as a small multiple f T c ensures that the system is almst always unsynchrnized The analysis further shws that fr fied values fr T c and T r, the transitin t synchrnizatin is an abrupt functin f the number f ndes N Finally, in this sectin we cnsider a system f ruters 7

18 in an arbitrary tplgy, where each ruter nly receives peridic messages frm its immediate neighbrs We suggest that the mdel f synchrnized ruting messages in this paper is likely t hld in arbitrary tplgies nly fr cnnected subsets f ndes with similar degree Figure 4 cnsiders the epected time fr the Markv chain t synchrnize r t unsynchrnize, as a functin f the parameter T r Figure 4 gives f ( N), frm Equatin 4, and g( ), frm Equatin 6, fr T r ranging frm zer t 4: 5T c, given N = 2, T p = 2 secnds, T c = secnds, and T d = secnds The slid line n the right shws the epected time fr the Markv chain t mve frm state N t state ; the slid line n the left shws the epected time fr the Markv chain t mve frm state t state N The dtted line n the left was cmputed using values fr f ( 2) based n an apprimate analysis that is nt given here; the slid line n the left uses f ( 2) set t zer Fr T r < : 5( T c T d ), clusters never break up nce they have frmed, and the time t synchrnize depends largely n the time t first frm a cluster f size tw; this time increases as T r appraches 8 Nte that the y-ais is n a lg scale, and ranges frm less than 3 secnds (rughly 6 minutes) up t 2 secnds (ver 32 thusand years) e lw mderate high Average time e3 e9 e8 e7 e5 e Randm Nise Tr (as a multiple f Tc) Tc,Tc2= secnds, Td= secnds, N=2 Figure 4: Epected time t g frm cluster size t cluster size N, and vice versa, as a functin f T r Figure 4 can be used as a general guide in chsing a sufficiently large value f T r, given the values fr the ther parameters in a system, s that the system mves easily frm state N t state and rarely mves frm state back t state N The figure shws the regins f lw, mderate, and high randmizatin In the regin f lw randmizatin the system mves easily frm state t state N; in the regin f high randmizatin the system mves easily frm state N t state In the regin f mderate randmizatin the system takes a significant perid f time t mve either frm state t state N, r frm state N back t state In the lw and mderate regins f ( N), the epected time fr the Markv chain t mve frm state t state N, grws epnentially with T r The X marks n Figure 4 shw simulatins that start with unsynchrnized ruting messages and the marks shw simulatins that start with synchrnized ruting messages Figure 5 shws the same analytical results as in Figure 4 fr the number f ndes N ranging frm t 3, and fr a range f values fr T c These simulatins were perfrmed t check hw 8 As Figure 4 shws, fr etremely small values f T r there is little randmness in the system, and it can take sme time fr tw ndes t first frm a cluster 8

19 N=2 Average time e e9 e8 e7 e5 e4 e3 N=3 N= # # # N=2 N=3 # N= * * * * * * * * * * * Randm Nise Tr (as a multiple f Tc) Tc= secnds (N=: *, # N=2:, N=3: ) Average time e e9 e8 e7 e5 e4 e3 N= N=2 N=3 N= N=2 * * * N= * * * * * * * * * * * Randm Nise Tr (as a multiple f Tc) Tc= secnds (N=: *, - N=2:, N=3: ) Average time e e9 e8 e7 e5 e4 e3 N= N=3 # # # # N=2 N=2 # # N=3 # # * # # # # # # * * * N= * * * * * * * Randm Nise Tr (as a multiple f Tc) Tc=5 secnds (N=: *, # N=2:, N=3: ) Figure 5: Epected time t g frm cluster size t cluster size N, and vice versa, as a functin f N and f T r accurately the analytical results predict the simulatin results fr a range f parameters Nte that fr larger values f T c and f N, the analytical results significantly verestimate the time required by the simulatins t g frm state N t state The analytical results use the simplifying assumptin that the size f the largest cluster changes by at mst ne frm ne rund t the net 9

20 As the parameters T c and N increase, this assumptin becmes less applicable The figures shw that fr a wide range f parameters, chsing T r at least ten times greater than T c ensures that clusters f ruting messages will be quickly brken up Fr any range f parameters, chsing T r as T p =2 shuld eliminate any synchrnizatin f ruting messages This wuld be equivalent t setting the ruting timer each time t an amunt frm the unifrm distributin n the interval [: 5T p ; : 5T p ] secnds This intrduces a high degree f randmizatin int the system, yet ensures that the interval between ruting messages is never t small r t large 54 Incrprating delays between ruters The analysis and simulatins in the paper s far have assumed that T d = ; that is, that when a nde s timer epires, ther ruters are immediately ntified f the timer epiratin While small values fr T d accurately reflect a mdel f ruting ndes where prpagatin delay is lw and each ruting message cnsists f a number f packets, it is physically impssible fr T d t be zer In this sectin we eplre simulatins with small nnzer values fr T d Recall that, in the absence f incming ruting messages, ruter A resets its timer T c secnds after its timer epires, and ruter B is ntified f ruter A s incming ruting message T d secnds after ruter A s timer epires If T d >T c (fr eample, because ruter A resets its timer befre it transmits the first packet f the ruting timer), then there is little cupling between adjacent ruters In this case, if tw ruters timers epire at the rughly same time, then each ruter resets its timer befre receiving a ruting message frm the ther ruter, and clusters break up quickly In this sectin we eplre simulatins with <T d <T c This reflects a mdel where each ruting message cnsists f multiple packets, and neighbring ruters receive the first packet f a ruting message befre the surce ruter resets its timer e e9 Average time e3 e8 e7 e5 e Randm Nise Tr (as a multiple f Tc) Tc,Tc2= secnds, Td=2 secnds, N=2 Figure 6: Time t g frm a cluster f size t a cluster f size N, and vice versa, fr T d = : 2 secnds Figure 6 shws the results f simulatins with T d = : 2 secnds The lines shw the same analytical results given in Figure 4, but cmputed fr T d = : 2 secnds Frm the analysis in Sectin 5, p i;i, the prbability that a cluster f size i breaks int a cluster f size i in ne rund, can be estimated by p i;i= T c T d 2T r i (7) 2

21 e9 Average time e8 e7 e3 e e5 e Randm Nise Tr (as a multiple f Tc) Tc,Tc2= secnds, Td=4 secnds, N=2 Figure 7: Time t g frm a cluster f size t a cluster f size N, and vice versa, fr T d = : 4 secnds fr i > and T c T d < 2T r As this equatin describes, the main effect f increasing T d is t increase p i;i In general, five simulatins were run fr each value f T r, and each simulatin was terminated after 8 secnds The X marks shw simulatins that start with unsynchrnized ruting messages and the marks shw simulatins that start with synchrnized ruting messages Nte that with T d set t 2 secnds rather than t zer, the simulatins take lnger t synchrnize and less lng t unsynchrnize Nevertheless, the basic behavir f synchrnizatin is preserved As Figure 7 shws, increasing T d frm 2 secnds t 4 secnds further increases the time required fr synchrnizatin The simulatins and analysis shw that the time t synchrnize increases as T c T d increases After a nde s timer epires, this is the time between when ther ndes are ntified f the timer epiratin, and when the nde resets its wn timer (in the absence f incming ruting messages) This interval can be affected by a number f factrs, such as the prpagatin delay, the number f packets in the ruting message and the timing between the transmissin f these packets, and the prmptness with which ndes reset their ruting timers 55 Steady state behavir One quantity f interest is the fractin f time that the Markv chain spends with lw cluster sizes We were nly able t estimate the equilibrium distributin fr the Markv chain by further apprimating the transitin prbabilities Hwever, ne simple way t estimate the fractin f time that the Markv chain spends in synchrnized states is t cmpute g( ) =( f ( N) g( ) ) Recall that f ( N) is the epected number f runds fr the system t mve frm state t state N; fr mst f this time the system is largely unsynchrnized Similarly, g( ) is the epected number f runds fr the system t mve frm state N t state ; fr mst f this time the system is largely synchrnized In Figure 8 the -ais shws T r ; the ther parameters are N = 2, T p = 2 secnds, T d =, and T c = secnds The y-ais is g( ) =( f ( N) g( ) ), the estimated fractin f time fr which the system is synchrnized As Figure 8 shws, as T r is increased, the system makes a sharp transitin frm predminately-synchrnized t predminately-unsynchrnized The simulatins and analysis in Figure 5 shw that fr a wide range f values fr N and T c, the transitin frm 2