Adaptive Gravitational Gossip: A Gossip- Based Communication Protocol with User- Selectable Rates

Transcription

1 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, MANUSCRIPT ID Adaptive Gavitational Gossip: A Gossip- Based Communication Potocol with Use- Selectable Rates Kenneth Hopkinson, Membe, IEEE, Kate Jenkins, Kenneth Biman, James Thop, Fellow, IEEE, Gegoy Toussaint, Senio Membe, IEEE, and Manu Paasha, Membe, IEEE Abstact Gossip-based communication potocols ae attactive in cases whee absolute delivey guaantees ae not equied due to thei scalability, low ovehead, and pobabilistically high eliability. In ealie wok, a gossip-based potocol known as gavitational gossip was ceated that allows the selection of quality atings within subgoups based on wokload and infomation update fequency. This pape pesents an impoved potocol that adds an adaptive component that matches the actual subgoup communication ates with desied ates coping with netwok vaiations by modifying undelying gossip weights. The potocol is designed fo use in envionments whee many infomation steams ae being geneated and inteest levels vay between nodes in the system. The gossip-based potocol is able to allow subscibes to educe thei expected wokload in etun fo a educed infomation ate. The potocol is a good fit fo applications such as militay infomation systems, senso netwoks, and escue opeations. Expeiments wee conducted in ode to compae the meits of diffeent adaptation mechanisms. Expeimental esults show pomise fo this appoach. Index Tems Adaptive Communication, Epidemic Potocols, Publish/Subscibe Systems INTRODUCTION T HIS aticle intoduces adaptive gavitational gossip (AGG, a goup communication potocol that allows subgoups in the system to select tansmission quality levels, which ae maintained with high pobability. The potocol uses heuistics to uphold infomation dissemination tagets despite changes in netwok conditions. In a pevious wokshop aticle [], a static vesion of the gavitational gossip (GG potocol was intoduced based on a mathematical model of gossip pefomance. The AGG potocol builds on this ealie wok to allow gossip pobabilities to adapt ove time based on changing conditions. Expeimental esults show that the AGG potocol has geat pomise fo use in situations whee subgoups have diffeent infomation tagets, netwok conditions ae unstable, and scalability is an impotant consideation. The essence of gossip potocols is that nodes communicate by andomly and uneliably shaing infomation. K. Hopkinson is with the Depatment of Electical and Compute Engineeing, Ai Foce Institute of Technology, Wight-Patteson AFB, OH kenneth.hopkinson@afit.edu. K. Jenkins is with Akamai Technologies, Cambidge, MA kenkins@akamai.com. K. Biman is with the Depatment of Compute Science, Conell Univesity, Ithaca, NY ken@cs.conell.edu. J. Thop is with the Depatment of Electical and Compute Engineeing, Viginia Polytechnic Institute, Blacksbug, VA : sthop@vt.edu. G. Toussaint is with the Depatment of Electical and Compute Engineeing, Ai Foce Institute of Technology. Wight-Patteson AFB, OH Gegoy.Toussaint@afit.edu. M. Paasha is a consultant fo the Electic Powe Goup, Pasadena, CA 90 paasha@electicpowegoup.com. Manuscipt eceived May 28, The views expessed in this document ae those of the authos and do not eflect the official policy o position of the United States Ai Foice, Depatment of Defense, o the U.S. Govenment. xxxx-xxxx/0x/$xx x IEEE Gossip potocols ae also known as epidemic potocols because the spead of infomation mimics that of a disease. Each node peiodically chooses anothe at andom and sends a subset of the infomation that it has. No data is kept egading which nodes have which infomation, which yields a elatively stateless potocol compaed to taditional eliable goup communication. Messages ae sent uneliably using UDP. The pocess is shown in Fig.. The pocess is only pobabilistically eliable, but is poweful in pactice. Reliable goup communication potocols must ensue that all nodes eceive all messages, and this becomes inceasingly difficult as goup sizes gow. In lage netwoks, it is likely that some node is having a connectivity o congestion poblem at any given time, even if the identities of the poblem nodes vay. Gossip allows nodes not expeiencing difficulties to eceive all infomation with high pobability. Intemittently poblematic nodes get as much infomation as they can. Gossip also speads the wok to publish infomation acoss the goup's membeship, leading to bette scalability. The static GG potocol takes gossip futhe. In many settings, diffeent membes of a goup will have diffeent inteest levels in infomation. One can imagine a situation whee inteest in an infomation steam deceases with distance, as in Fig. 2. Nodes close to the infomation souce would like a highe pecentage of peiodic updates than those fathe away. Nodes eceiving less infomation should do less wok to maintain the oveall system. The use of adaptive weights in epidemic potocols is elatively new. Two notable instances of adaptive epidemic potocols focused on choosing the minimum gossip ate such that nealy all membes eceive infomation

2 2 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, MANUSCRIPT ID Fig.. In gossip, uneliable packets ae sent to andom destinations fom nodes with infomation duing each ound. Nodes without infomation ae light while nodes with the infomation ae dak. Aows epesent gossip messages. An X epesents a lost packet. Fig. 2. In illustation of gavitational gossip: inteest in infomation deceases with distance Allow nodes to only subscibe to a faction of the infomation that is being sent in a given steam. 2 Ensue that a publishe of infomation will have to do as little wok as possible as the system gows. 3 Receives will do thei fai shae of wok, but will do popotionately less if they eceive less infomation. A potential application fo the AGG potocol is infomation-centic militay opeations. The militay is moving towads a moe infomation-ich envionment fo its wafightes. This tend appeas in the Depatment of Defense s Joint Vision 2020 [4] and in the addition of Cybespace to the Ai Foce mission statement [5] [6]. Effots to bette manage lage numbes of publish-subscibe infomation steams in militay communication, like the JBI infomation system [7], point to a futue whee tade-offs will be equied in the infomation flow acoss bandwidth-constained battlefonts. Tools like AGG will allow a iche and moe adaptive set of dialable infomation settings, which will allow moe constained sections of the netwok to dial thei expected update ate down while othes dial ates up. Achieving simila esults using taditional potocols effectively means ceating a publish-subscibe goup fo each subgoup with a diffeent taget. This solution has at least as geat an ovehead on the sende (i.e. publishe of infomation as taditional eliable multicast potocols. AGG, by contast, speads the load of sending updates to all of the system s nodes. Membes eceiving fewe updates do less wok in tems of messages sent and eceived. Simila equiements exist in classes of senso netwoks and escue opeations. The aticle begins by descibing the basic GG potocol. A desciption of the new adaptive addition to the potocol follows. Expeimental esults ae pesented to show how the new potocol pefoms in contol-based scenaios. A summay and a set of mao conclusions follow. Fig. 3. A moe typical use of the potocol. Inteest levels may not coespond diectly to distance. Also, nodes ae clusteed based on both inteest and poximity to make adaptation easie. based on factos like netwok conditions and buffe sizes [2] [3]. AGG goes beyond pio wok by being the fist epidemic potocol to allow multiple subgoups to each specify desied infomation ates, and to adapt gossip settings to meet those ate tagets as netwok conditions change. Because conditions tend to be simila in a local aea, it is possible to have subgoups with simila taget infomation ates, but in diffeent locations, as in Fig. 3. The AGG potocol has many applications. These uses tend to fit a patten whee peiodic infomation updates ae boadcast, whee inteest in an infomation steam deceases by some citeion, often distance, and easonable actions can be taken with a subset of the published infomation. Thee is also a tendency to have numeous simultaneous infomation steams, making it impactical to fully subscibe to them all. Fo scalability easons, it becomes impotant to povide the following abilities. 3 STATIC GRAVITATIONAL GOSSIP 3. Gavitational Gossip Mathematical Model This section eviews the mathematical model, which foms the basis of the static gavitational gossip (GG potocol, fist intoduced in []. The GG potocol is based on an epidemic model incopoating vaiable subpopulation infectivity and susceptibility ates developed by Busenbeg and Castillo-Chavez in 99 [8]. In taditional epidemic potocols, nodes choose othes to send gossip messages to using a unifom pobability distibution. The potential eceives of a gossip ae the membes of a pocess goup, which is a set of nodes that subscibe to eceive such infomation. An intuitive way to envision this pocess is that when it is time fo a node within an epidemic-based pocess goup of size N to gossip to anothe node in the system, it olls an N-sided die to choose which othe node to send a gossip message to. This die is fai, which means that no egion of the die is moe likely than any othe. GG weights the gossip die so that some sides ae moe likely to appea than othes. Table summaizes GG s mao paametes. Nodes in a pocess goup ae divided into subgoups. Each subgoup is associated with a ating [0,] that equates to

3 HOPKINSON ET AL.: ADAPTIVE GRAVITATIONAL GOSSIP: A GOSSIP-BASED COMMUNICATION PROTOCOL WITH USER-SELECTABLE RATES 3 TABLE I UNITS IN GRAVITATIONAL GOSSIP Symbol Type Meaning Set by Use The taget infomation ate fo a subgoup in the ange [0, ], which equates to 0% to 00%. I Set by Use The infectivity (tendency to gossip fo a subgoup with taget infomation ate. In this a- S gossip weight Contolled Paamete (set by algoithm ticle, I = in all cases. The susceptibility (tendency to be gossiped to fo a subgoup with taget. Also called gossip weight. S. Contolled Paamete Synonymous with (set by algoithm timeout Set by Use The amount of time a message will be gossiped in the system befoe it expies. T Equation Vaiable A time value, not necessaily equal to timeout, which appeas in some equations. δ Set by Use How much eo can be toleated fo subgoups with a ate of 00%. (i.e. δ of 0.0 would be a taget ate of 99.9%. t Equation Vaiable Time used in gossip equations, expessed in ounds (i.e. t + is one ound highe than t. x ( t Equation Paamete Pecentage of subgoup membes at ating uninfected at time t. N Set by Use Numbe of nodes in the subgoup with ating. num_ nodes message_ pecent quality contibution OI S ( k Contolled Paamete (set by algoithm Contolled Paamete (set by algoithm Equation Paamete Contolled Paamete (set by algoithm Numbe of nodes to gossip to pe ound fom a given node. Pecent of available messages included pe gossip tansmission. Souce infectivity times Destination Susceptibility. The obseved infectivity (ate of infomation eceipt when a susceptibility of S is used. I Equation Paamete A subgoup with a taget infomation ate of 00%. γ Contolled Paamete (set by algoithm scales x ( t to expie at timeout. Used in a binay seach that the taget ate fo infomation updates. A highe ate yields moe infomation, but equies moe wok fom a subgoup's membes in tems of the expected numbe of messages that will be sent and eceived. Although nonunifom distibutions in epidemic potocols have been discussed fo ove 20 yeas in the liteatue, including an ealy aticle by Demes [9], GG s innovation is that it uses a mathematical model of gossip behavio to allow uses to intelligently choose the gossip pobabilities, also called gossip weights and susceptibility values in this aticle, to achieve the desied expected distibution of infomation by a given expiation time, called a timeout value. k Boowing fom epidemic teminology, each subgoup of nodes in a pocess goup has two paametes, its infectivity ( I and susceptibility ( S. A node that has not eceived a piece of infomation is said to be susceptible. Once that node has the infomation, it is infected and able to pass the new data along to othes. To be moe pecise, we have the following definitions. Define I = the infectivity of a subgoup with a taget infomation ate, which coesponds to the pobability that a membe of the subgoup with knowledge of a data item will gossip to othe goup nodes. I [0,]. Define S = the susceptibility of a subgoup with taget ate, which is the pobability that a subgoup membe will be tageted with a gossip message. S [0,]. The infectivity and susceptibility ates of two nodes i and multiplied togethe, I S i, gives the pobability that node i will send a gossip message to node, whee i and ae the two nodes subgoup atings. Because a publishe (aka sende of infomation will always have 00% of the infomation that it geneates, an infomation publishe will always be a pat of subgoup ( signifying 00%. The special tems I and S efe to the infectivity and susceptibility, espectively, fo this subgoup. While multiple publishes can be added with a simple extension to the fomulas pesented hee, we will concentate on the single publishe case in this aticle in ode to make the pesentation cleane. In GG, a ecuence elation was found to detemine the pobability that a node would not be infected afte t gossip ounds given fixed values of I and S. The lifetime of a gossip message is expessed in tems of a numbe of gossip ounds because nodes choose to gossip to othes in thei goup peiodically, although node clocks ae not assumed to be synchonized to the point whee a ound of gossip occus at the same time at all goup nodes. In this way, the lifetime of a message can be thought of as a discete value in tems of the numbe of gossip ounds emaining. Define N = the numbe of nodes with ating, and I, S espectively as the infectivities and susceptibilities of nodes with ating. Define x (t = pobability node m with ating is still susceptible at time t Then, x ( t + = x ( t P(each machine that gossiped to ating at time t + missed m num gossips to ating at time t + N S I N ( x ( t x ( t + = x ( t ( N ( ( ( x t+ x t N x By basic calculus [0], ( x = e as x, so x ( t S + x ( t e I N ( x ( t whee the sum of the exponents is calculated ove all of the diffeent atings. Fo puposes of appoximation, it is assumed that the andom vaiable in the exponent is

4 4 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, MANUSCRIPT ID equal to its expected value. This equation can be boken down into a seies of ecuence elations fo each of the system s subgoups. S I N ( x ( t x ( t + x ( t e S ( ( I N x t x( t+ x ( t e S ( ( 2 I N x t x ( t+ x ( 2 t e 2 S ( ( k I N x t x ( t+ x ( t e k k Given these equations, the faction of infected nodes fo the subgoup at ating in ound t is x ( t. An impotant popety was also deived that follows fom induction on the ecuence elations above. Namely, t, x ( t ( x ( t S / S This follows by induction since x ( t S I N ( x ( t + = x ( t e S I N ( x ( t S S S = ( x( t ( e S I N ( x ( t S S = ( x( t e S S S = ( x ( t + S S and 0 x (0 x ( fo N lage Each node within a given subgoup has the same desied ate of infomation,. Because is a eal numbe, it is possible that many o most nodes will be in subgoups by themselves unless cae is taken to make ating discete and to cluste nodes with close desied ates togethe. The deivation above depends on the law of lage numbes. A setting with many singleton subgoups seves as a wostcase fo the gavitational gossip potocol. 3.2 Gavitational Gossip Paamete Pediction The equations deived in the pevious section give a method fo detemining the numbe of infected nodes at ating at ound t given gossip paametes I and S. The invese poblem is often the one that is of most inteest. Given a set of taget subgoup atings, can the gossip paametes that will esult in nodes eceiving infomation at the pescibed ates be detemined mathematically? The equations deived in the pevious section ae in the fom of ecuence elations that do not lend themselves to a staightfowad invesion. This section biefly eviews the static algoithm [] fo finding gossip paametes that can be used to achieve the taget ating in expectation fo each subgoup. Mathematically, the goal of the invese algoithm is to find values S, S,..., S k given the taget infectivity values I I,..., I k input by the uses such that:, x ( timeout 0 x ( timeout ( timeout x k k and I / I =,..., I / I = k k so we will meet the goal that a machine at ating will initiate times as many gossips as a machine at ating, and a machine at ating will have a pobability of eceiving a message befoe it times out. I always includes the publishe, which has value.0 since it will have 00% of the infomation it geneates. The paamete pediction algoithm woks as follows: Fo any fixed paamete setting, thee is some smallest ound numbe T such that x ( T < δ whee δ is an abitay value asking fo how closely the expected value of the fist subgoup, which contains the sende (i.e. publishe, should be to. Fo example, a δ value of 0.0 would signify that the expected ate of infomation will be 99.9% of all published infomation in the fist subgoup of the system. Given the pevious section s equations, the faction of infected nodes fo the subgoup at ating in ound T is S ( ( ( S S S x T x T < δ. If we let S = and pick S such that x ( T < then this means that S should be S chosen so that δ =. Solving fo S yields: S = logδ ( = log2 ( log 2 ( δ, This equation gives the desied faction of infected nodes at each ating at ound T, but we want this to be achieved by ound timeout. To achieve this, we can globally scale the susceptibilities by some value, which we will call γ. No matte what the value of γ is, wheneve x ( T δ < the faction of susceptibles at the othe atings will satisfy the desied popotions. Vaying γ ust affects the time when this occus. Thus, we binay seach to find a value of γ fo which x ( timeout < δ, but x ( timeout δ. We test a given γ by unning the ecuence elations with the coesponding paametes. It should be noted that the tem time, in a slight abuse of notation, is used hee to efe to a numbe of ounds, which ae discete, athe than a continuous value. Time is used in a simila manne in the est of the aticle. 4 ADAPTIVE GRAVITATIONAL GOSSIP The static GG paamete pediction method woks well in stable netwok conditions, but many eal-wold applications un ove shaed netwoks with unpedictable levels of competing taffic, netwok congestion, and spoadic connectivity. The static potocol woks best in envionments with lage numbes of membes, high timeout values, low netwok communication eos, and low message latencies. Impotant applications, such as the wideaea monitoing and contol situations outlined in Section 2, equie a potocol that can adapt as conditions change. While othe adaptive gossip potocols exist that change thei gossip distibution based on goup membe infomation in esponse to use queies [, 2] and physical node locations [3, 4], the potocol in this aticle is able to allow the ceation of subgoups, each with thei own gossip ate tagets, and can adapt the oveall system as netwok conditions change. Each node keeps a ecod of the pecentage of messag-

5 HOPKINSON ET AL.: ADAPTIVE GRAVITATIONAL GOSSIP: A GOSSIP-BASED COMMUNICATION PROTOCOL WITH USER-SELECTABLE RATES 5 es eceived ove time. This can be done because messages ae assumed to be peiodic and the peiodicity is known to all membes of the goup, as mentioned in Section 4. This section begins by descibing a potocol famewok to suppot adaptive behavio. The potocol s cental pemise is that if infomation is known egading the pocess goup and subgoup membeship, the infectivities (the ate of infomation desied, and the publication ate fom the goup s sende, then appopiate susceptibilities (aka gossip weights can be found such that subgoups desiing moe infomation will gossip moe to maintain the oveall infomation flow within the goup. The sende must publish new infomation at a pedictable ate in the adaptive GG potocol in ode fo the goup membes to calculate the quality ate of infomation that they ae eceiving. The ate that the sende publishes new infomation is assumed to be peiodic in this aticle. The goal is to ensue that the desied ate of infomation fo each subgoup (its infectivity is equal to the eceived ate. (I.e. each node in a subgoup with an infectivity of 0.25 should eceive 25% of all published infomation, on aveage, by the time it expies. 4. A Basic Adaptive Potocol Famewok The pevious sections beg the question of how adaptive infomation is gatheed into and disseminated out fom adaptive weight adustment heuistics. A basic potocol famewok has been ceated in ode to illustate one way of implementing such a potocol and to allow expeiments to compae thee competing adaptive heuistics. Static gossip weight calculations ae tiggeed wheneve a new goup is ceated, when the set of the goup s subgoups changes, o when membeship changes occu. In the desciption of the goup communication potocol that follows, fo each node that belongs to a goup, it will be a membe of exactly one subgoup within that goup. In publish-subscibe teminology, all nodes in the goup would be consideed subscibes to an infomation steam. In the teminology used in this aticle, the sendes of infomation to a goup ae the goup s publishes. It is impotant to keep in mind that all membes in the goup ae likely to peiodically fowad infomation in the fom of gossip, but only the sendes (i.e. publishes ae souces of new infomation to the goup. In the basic adaptive potocol developed in this aticle, static weight calculations follow the mathematical model outlined in Section 3. The infectivity is equal to the desied ate of infomation fo each subgoup, whee the value anges fom 0 (no infomation to (00% of the published infomation. The susceptibility weight calculations fo each subgoup ae pefomed by the sende within a goup. (As mentioned peviously, a single sende is assumed in this aticle fo claity of pesentation. The same technique is easily extended to multiple sendes by keeping each sende in its own subgoup. A system invaiant is that the goup s sende belongs to its own subgoup. This is necessay due to the fact that the sendes inheently possess 00% of all messages that they send. So, fo example, if a sende wee placed in a subgoup with a susceptibility of 0.20 and fou othe nodes then the taget infomation ate would automatically be satisfied since 00%/5 nodes = 20%/node. Fo this eason, the sende will be in its own subgoup, and the fou emaining nodes will be placed in a sepaate subgoup. As the potocol pogesses, each node keeps a ecod of the pecentage of messages eceived ove time. This is possible because messages ae assumed to be peiodic with a known peiodicity, as mentioned in Section Oveview of Adaptive Potocol Messages All messages in the basic potocol famewok ae sent uneliably using UDP packets. The pupose and content of each message type is descibed in this subsection Gossip Messages Evey goup membe peiodically sends gossip messages. Gossip intevals ae unsynchonized between goup membes. Gossip cycles occu at oughly the same fequency at evey goup node, but the clocks and gossip tigge times vay fom node to node. The gossip fequency is a goup-wide paamete in the system. In the goup communication system, each new piece of infomation published to the goup is called a gossip fagment. Gossip fagments expie and ae no longe shaed afte a fixed time set on a goup-wide basis. A node will gossip to anothe node at andom accoding to the gossiping node s infectivity and the selected ecipient s susceptibility. Initial calculations fo the infectivity and susceptibility follow the method outlined in Section 3. Fom then on, infectivity and susceptibility values ae adapted to compensate fo changing netwok conditions using heuistics such as those in Section 5. When a gossip message is sent, the message consists of a seies of gossip fagments appended to each othe. The numbe of fagments is dictated by the fomula in Section 4.2. Fagments ae chosen at andom fom the sending nodes set of known unexpied published infomation Peiodic Gossip Weight Recalculation Denote the numbe of ounds befoe a message expies as T. In the adaptive gossip potocol, each goup membe is told how fequently each sende (i.e. publishe sends new infomation to the goup. The membes also know how long messages should be popagated befoe they expie and ae no longe of inteest. Given this infomation, goup membes ae able to calculate the pecentage of infomation that they ae eceiving based on the messages eceived in a given time peiod. Fo example, in the expeiments un in Section 6, the pecentage of messages eceived was calculated ove a timefame of T. In these expeiments, 20 messages wee published pe 00 ms ound and messages expied afte 20 ounds. Theefoe, simple division of the messages eceived by the lifetime of each message gives the quality measuement ove this time peiod. Fo example, if an aveage of 5 messages wee eceived pe ound then 5 messages / 20 messages published pe ound = a 25% quality ating. This esult can be aggegated with the othes acoss the T timefame to get an oveall aveage. So, if T is 4 then we can use the aveage quality ating fo each of the ms ounds between time δ and δ + 4. Once this value is calculated,

6 6 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, MANUSCRIPT ID infomation can be gatheed fom each subgoup to see if the aveage membe is eceiving the desied quality ate. Evey 2 T ounds, the message sende andomly chooses one node in each subgoup and sends a gossip quality equest. 2 T is chosen to allow ample time fo new gossip weights to popagate though the system befoe taking pefomance measuements. When a node eceives a subgoup quality equest, it elays a gossip message to each of the othe membes of its subgoup. The subgoup nodes esponses ae totaled and aveaged befoe they ae etuned to the sende. Weight adustments ae pefomed fo each esponding subgoup using heuistics such as those in Section 5. All messages ae sent uneliably. If a esponse is not eceived then the last known value is used. Weights eadustment esults ae distibuted in uneliable gossip messages sent to one andomly chosen membe of each subgoup. When a subgoup membe eceives a weight eadustment value, it sends an uneliable copy to all othe subgoup membes Recoveing When Weight Updates Do Not Aive at Goup Nodes The gossip weight update messages descibed in the pevious subsection ae sent uneliably so a mechanism is needed to allow nodes to eceive the new values when an oiginal update message is lost. To do so, the latest gossip weights and thei timestamp ae hashed togethe into a 32-bit value. The hash values ae included with all gossip messages thoughout the infomation dissemination pocess. In each gossip message, hash values ae included fo each goup/sende with at least one infomation fagment included in the message. When a node eceives a gossip message, the hash value(s in the message ae compaed against a hash of the node s cuent gossip weight values. If the two hashes ae not a match then a epai message is geneated fom the eceive to the souce of the gossip message. Upon eceipt of a epai message, if the eceiving node has newe weights than those contained in the message then those newe values ae etuned in a message. Once again, all messages use uneliable UDP packets. Message losses end the exchange and no attempt is made to detect o ecove fom them Potential Potocol Enhancements This potocol is elatively simple. It could be stengthened in a numbe of ways. Lage subgoups, subgoups with divese topologies, o subgoups with vaied communications chaacteistics could benefit fom a hieachical communications infastuctue o an automated mechanism to split nodes into smalle subgoups and then ecombine when appopiate. The adaptive potocol centalizes all subgoup gossip weight calculations at goup sendes, but heuistic othe than the initial adaptive do not equie this. A epesentative membe of each subgoup could do the ob instead. To make this wok, some type of leade election algoithm would be equied. 4.2 Conveting Gossip Weight to Tansmission Pobabilities The gossip weight calculations descibed in Section 3, and the adaptive heuistics descibed in Section 5, use a mathematical model of gossip behavio and heuistic algoithms to find gossip weights fo the subgoups affiliated with a goup sende/publishe. Based on the teminology in Section 3, the goal of these calculations is to pedict the susceptibility (gossip pobability value fo the membes of a subgoup in ode to maintain the desied infection ate (pecentage of published infomation eceived Calculating the Quality Contibution Facto In the gossip algoithm, each node possesses an infectivity ating, which is the desied pecentage of infomation eceived. To make an analogy with the spead of a disease, an individual s infectivity is thei tendency to spead disease. An individual s susceptibility is thei tendency to get ill aound an infected individual. Similaly, a highe susceptibility in gossip means a node is moe likely to be gossiped to, and a high infectivity means a node is moe likely to send gossip to othe individuals. (A goal of the potocol is fo membes with a highe desied infomation ate to do moe wok to maintain the goup. The souce infectivity multiplied by a destination subgoup susceptibility yields a aw value called the quality contibution fom a given node to a given subgoup Dividing Quality Contibution Factos into Gossip Pobabilities and Message Sizes Fo a given quality contibution value, a tade-off needs to be made between the pobability of gossiping to a node vesus the amount of infomation contained in each gossip message. (A quality contibution of 0.5 can be achieved by sending half of known infomation updates to one node, a quate to two nodes, etc. If the numbe of nodes gossiped to is minimized then the knowledge vaiance may be high fom one subgoup membe to anothe. At the same time, it is not desiable to send too many netwok messages. We have adopted the convention that the quality contibution should be divided so that the numbe of nodes to gossip to in a subgoup, called num_nodes hee, is calculated as follows quality _ contibution num _ nodes = taget _ subgoup _ ating and that the pecentage of messages included in each packet sent, called message_pecent hee, ae equal to quality _ contibution message _ pecent = num _ nodes Fo example, if subgoup A has taget infectivity 0.75 and susceptibility 0.25, and subgoup B has infectivity 0.5 and susceptibility 0.33 then fo A->B, num _ nodes = ( = so each ound one message will be sent fom each node in subgoup A to a andom membe of subgoup B. message _ pecent = ( = so each message will contain 49.5% of the known unexpied gossip fagments. The heuistic balances the numbe of gossip messages sent against the message sizes. The heuistic fo dividing a quality contibution into a pobability of gossip vesus the amount of infomation pe gossip message woked well in the expeiments un.

7 HOPKINSON ET AL.: ADAPTIVE GRAVITATIONAL GOSSIP: A GOSSIP-BASED COMMUNICATION PROTOCOL WITH USER-SELECTABLE RATES 7 5 ADAPTIVE GOSSIP HEURISTICS One shotcoming of the gavitational gossip algoithm in Section 3 is that thee is no closed-fom solution. It is expessed as a ecuence elation. The invese algoithm in Section 3.2 woks well, but cannot take netwok losses into account. This section descibes fou adaptive heuistics, which adapt gossip weights ove time to compensate fo changes in netwok losses ates and othe disuptions. The fou heuistics each un ove the distibuted famewok descibed in Section 4. At peiodic intevals, an adaptive algoithm will take in the aveage obseved infectivity in each subgoup since the last time a new susceptibility value was computed. The adaptive algoithm computes new susceptibility values pe subgoup based on the ate tagets (subgoup infectivity values. These new susceptibility values ae distibuted thoughout the communication goup using a gossip-based mechanism and the cycle stats again. A desciption of each of the adaptive mechanisms that we tested is povided below. 5. A Linea Heuistic fo Finding Gossip Weights Finding a heuistic method to find appopiate gossip weights (i.e. susceptibilities as netwok conditions change ove time is a challenge. This section descibes a simple linea heuistic, which begins with a numeical algoithm to solve the quality invesion poblem in Section 3.2 in the face of changing netwok conditions. To solve the invesion poblem numeically, an algoithm must find the oots of the mathematical model equations in Section 3 based on a given set of quality ates. The poblem is to find a set of susceptibility values S, fo each subgoup with taget ating, such that the infectivity values of each subgoup I will be equal to thei final ate of infection afte the last ound of gossip is ove. Each message has an expiation time, which tanslates into a fixed numbe of gossip ounds ove the message s lifetime, called timeout. The expected pecentage of uninfected nodes (i.e. nodes without knowledge of a piece of infomation by time t is given by the Section 3 equation S I N ( x( t x ( t+ x ( t e ( x ( t is the pecentage of uninfected nodes at time t. S is the susceptibility (gossip weight of the nodes in the subgoup with taget infomation ate. I is the infectivity fo subgoup with taget ate (the ate is the taget pecentage of nodes that will eceive the message befoe it expies. N is the numbe of infected nodes in subgoup by time t. The exponent is summed ove all subgoups, which is denoted by the subscipt. Equation ( is a ecuence elation fo finding the final pecent of infected nodes given a stating infectivity value I ove each subgoup with ating. Matlab s lsqcuvefit method [5] is a non-linea oot-finding algoithm in multiple dimensions. By specifying that susceptibility values ae only valid ove the ange [0,], lsqcuvefit can look fo subgoup susceptibilities such that the actual infomation ates will equal those desied. Equation ( gives good appoximations when the netwok has low congestion and low bit eo ates, but can be inaccuate in othe cicumstances. Non-linea ootfinding numeical methods like lsqcuvefit wok by testing many evaluations of a use-supplied function to find a vecto of values that esult in an answe that is close enough (as defined by the use to the tue solution. A good set of susceptibility values unde ideal netwok conditions can be found by supplying a function to evaluate equation ( at time timeout. In eal netwoks with shifting conditions, one could supply a function to lsqcuvefit that allowed a simulation o even a eal netwok to opeate long enough to be stable and then could use the esulting aveage infomation ate fo each subgoup as the etun value fo that function evaluation. Unfotunately, lsqcuvefit and othe simila methods can equie hundeds of function evaluations befoe conveging to a value. A long seies of evaluations is slow and is also susceptible to changing netwok conditions in the middle of a function evaluation cycle. The effects of small changes in gossip weights ae also likely to be lost in the noise caused by netwok fluctuations and sampling eo. A simple heuistic can capitalize on the Matlab-based non-linea least-squaes invesion method by assuming a linea continuum between susceptibility weights. Pseudocode fo this method is given below. All vaiables in the equation below ae [ num _ nodes] vectos, and division and multiplication ae to be pefomed tem by tem.. susceptibility least squaes calculation (based on the taget subgoup infectivity 2. obseved infectivity netwok potocol evaluation 3. modified susceptibility least squaes calculation (obseved infectivity based on calculated suceptibility susceptibility 4. tue susceptibility susceptibility modified susceptibility This algoithm calculates susceptibility weights based on equation ( (psuedocode line. Next, these weights ae used unde eal netwok conditions fo an evaluation peiod (4 seconds in the aticle s expeiments (line 2. Then, lsqcuvefit is used again to find the susceptibility values (modified susceptibility values in line 3 equied in an ideal netwok to achieve the infomation ate that was actually obseved in the netwok (obseved infectivity in line 3. In step 4, we assume that thee is a linea elationship between the input susceptibilities and output obseved infectivities. We take the popotionate diffeence between the obseved and desied infectivities and then scale the susceptibility values used by that facto. This simple heuistic has been shown to poduce an ode of magnitude gain in accuacy when the initial susceptibility estimate is fa fom the taget infectivity, but it is not a cue-all. Repeated applications can lead to divegence. Nonetheless, the expeimental gains make this a good stating place befoe additional weight efinement. 5.2 Newton s Method The Section 5. adaptive heuistic impoves the accuacy of the initial susceptibility weights. The adaptive heuistic could be followed by the use of futhe n-dimensional non-linea oot-finding iteations, but the numbe of function evaluations would be too high to be pactical. A heu-

8 8 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, MANUSCRIPT ID istic that woks well is using sepaate instances of a modified fom of Newton's Secant Method [6] in onedimension independently fo each of the subgoups. To biefly eview, Newton's Method fo one-dimensional oot-finding is based on a tuncated Taylo expansion aound the function to be minimized, which seaches fo a minimum value x using an iteative appoach. This seach is successful when the stating value is close enough to the solution. The seach uses equation f ( xk xk + = xk f '( xk The Secant Method is a modification of Newton's Method, which uses diffeence equations in place of explicit deivatives. Hence, the last equation is tansfomed into: x k + = x k f ( x k xk f ( x k xk f ( x k which conveges supelinealy. Advanced oot-finding algoithms exist, but a Secant method advantage is that it only depends on the pevious function evaluation. This is a mao stength as it helps shield the potocol fom eliance on outdated potocol pefomance measuements. The Secant method adapts gossip behavio by computing the next susceptibility values in ode to minimize the diffeence between the desied infomation ate (infectivity value and the obseved ate (obseved infectivity value. Peiodically (evey 4 seconds in the expeiments, each subgoup uses the secant value to compute the new susceptibility fo that subgoup to be used ove the next time peiod. Subgoup uses the infomation gatheed fom its membes, using the pocess in Section 4., to find new susceptibility values using the equation Sk Sk Sk+ = Sk OI( Sk OI ( S OI ( S k k whee Sk is the susceptibility value used in ound k and OI( Sk is the aveage obseved infectivity (infomation ate that was achieved duing the time peiod whee susceptibility S k was used. Two heuistics ae applied to the Secant Method to impove its pefomance. Fist, no new calculation is invoked if the desied and obseved infomation ates ae within 0.02 of each othe. Second, the etuned susceptibility value is foced to be in the ange [0.02,0.98 to ensue that it is within a sane ange. These exact bounds ae heuistics (i.e. [0.05, 0.95] might also wok. Limits above 0 and below.0 ae needed because it is difficult to measue the impact of a change in the input susceptibility on the output infectivity when the ange includes the exteme ends 0 and.0. Thee is no mathematical poof that this conveges, but expeimental evidence has shown that it does convege and does so apidly unde stable netwok conditions. 5.3 Popotional Contolle Newton s method woks well in pactice, but the esulting gossip weights (i.e., susceptibility values can oscillate. Wild steps can also occu when netwok conditions emain elatively constant between measuement intevals making the delta between eadings small. The outcomes measued will also dift natually fom one eading to the next. An altenative is to ceate a contolle fo the AGG potocol. As in Newton s method, one contolle is used pe subgoup to adapt its susceptibility values (i.e., gossip weights in esponse to changing netwok conditions. Fig. 4 shows a simplified block diagam model of a netwok subgoup that can be used to undestand how the contolle can stabilize the gossip weights. The contol model is only appoximate because it teats the netwok as a linea system and models the nonlineaities as an additive distubance. Moe advanced models ae possible, but this simplified appoach is sufficient to undestand how the feedback mechanism functions and it has been effective in the expeiments un. The meaning of the popotional contolle s mao components follows. Refeence ( The efeence signal to the system,, is equals to the subgoups taget infomation ate (i.e., infectivity ate. 2 Eo (e The eo, e, fom each iteation is the diffeence between the taget infectivity ate,, and the obseved infectivity ate in the netwok. The eo signal is sent to the contolle to ceate the next taget susceptibility value (gossip weight fo the netwok subgoup. 3 Contolle (K(s The contolle, K(s, is modeled as a popotional contolle that convets the eo signal into a taget susceptibility value. Popotional contol is a elatively simple appoach that allows the system to appoximately tack the efeence signal while also eecting the distubances. Fo this effot, the contolle was tuned to a constant value K P, which was set to 75% of the maximum susceptibility value. The maximum susceptibility value is the susceptibility value fo the subgoup when all subgoups desie a 00% infomation ate. At 00%, all subgoups eceive all infomation with high pobability. The susceptibility value was found using the Section 3 equations. 4 Plant Pocess (P(s and Distubance (d The plant pocess, P(s, epesents the behavio of the netwok modeled as a genealized linea system. The susceptibility (gossip weight fo the subgoup seves as an input to the netwok. As outlined in Section 4., the new subgoup susceptibility value will popagate thoughout the goup. Next, a use-defined amount of time will pass to allow peiodic newly published gossip fagments to be gossiped though the goup. Afte the use-defined time span, statistics ae collected fom the subgoup membes, as descibed in Section 4., and the obseved infectivity ate fo the subgoup is used as the system output. Thee ae two obsevations about the netwok that guide the model development fo the system. In geneal, it is not pactical to give an exact model of netwok behavio without specifying detailed opeating conditions. The fist obsevation is that highe levels of netwok congestion lead to lowe obseved infectivity ates. It is difficult to model the elationship between congestion levels and infectivity ates, but is likely nonlinea. The second obsevation is that the gossip potocol opeates in a netwok that cannot change its state instantaneously. That is, the netwok essentially has a memoy of past states and

9 HOPKINSON ET AL.: ADAPTIVE GRAVITATIONAL GOSSIP: A GOSSIP-BASED COMMUNICATION PROTOCOL WITH USER-SELECTABLE RATES 9 Distubance d Eo Refeence + e + Output K(s P(s y + Contolle Plant Pocess Fig. 4. Linea System Model with Distubance will not immediately move fom one congestion level to anothe. These obsevations indicate that netwok model must account fo the nonlinea pocesses and how the system esponds to equested changes. A simple appoach to accommodate these elements is to model the netwok with a fist-ode diffeential equation with additive distubances. The fist-ode diffeential equation captues the memoy featues of the netwok and the additive distubances epesent the nonlinea behavio. In tems of the model, shown in Fig. 4, the fist-ode diffeential equation tanslates into the tansfe function P(s = /(s+β, whee β chaacteizes how quickly the system esponds to changes. The distubance, d, is added to the output of the netwok model to yield the system output, y. When a subgoup s gossip weight (susceptibility is input into the netwok, the esult is a pecentage of nodes that will be infected on aveage. The esult must be between 0 (0% infected and (00% infected. 5 Output (y The system output of the system, y, is the obseved infectivity fom the netwok. The output is detemined though system measuements and used as a feedback signal to egulate the contolle. The fequency-domain (Laplace analysis below deives the elationship between the output signal, the efeence signal, and the distubances as depicted in Fig. 4. Y( s = Ds ( + PsKs ( ([ Rs ( Y(] s (2 Ys ( = Ds ( + PsKsRs ( ( ( PsKsYs ( ( ( (3 [ + PsKs ( ( ] Y( s = Ds ( + PsKsRs ( ( ( (4 Ds ( PsKsRs ( ( ( Y( s = + (5 + PsKs ( ( + PsKs ( ( Using a popotional contolle and the linea system model descibed above, we have: K( s = KP Ps ( = s + β Y( s KP Rs ( Ds ( s + β = + KP KP + ( s + + β K s+ ( PRs = + β s + β + K s+ β + K P If thee is no distubance, then the step esponse to a efeence input signal is given by: P KP Y( s = s( s+ β + KP and the coesponding output signal will be: KP ( β + P ( K t yt = e ut ( β + K P The output signal will appoach the faction K P /(β+k P, which, fo K P >> β, appoaches as a steady-state value. The expeiments un always had K P >> β, so the system would appoximately tack the efeence signal. If thee is no efeence signal and the distubance input is a step function, then the output signal is: s + β Y( s = s( s+ β + KP and the coesponding output signal will be: β KP ( β + P ( K t yt = + e ut ( β KP β K + + P The output signal will appoach the faction β/(β+k P, which, fo K P >> β, appoaches 0 as a steady-state value. Theefoe, the feedback system will attenuate the distubance at the output. In summay, if the popotional gain is lage enough elative to the paamete β, then the output signal will closely tack the efeence signal and the distubance will not play a significant ole in the output. This contol law is elatively simple and will yield easonable esults, but it can be impoved by combining integal feedback with the popotional feedback. 5.4 Popotional-Integal Contolle The popotional contolle does not have poblems associated with deivative calculations that occu in Newton's method, but it does not convege as quickly (expected linea convegence vesus supelinea convegence fo Newton's Secant method. The popotional contolle can be impoved by including an integation tem in the feedback contol law to incease the pefomance in tacking a efeence signal and in eecting a distubance. The new contolle is called a popotional-integal contolle and its coesponding tansfe function is given by KI Ks ( = KP + s whee K P and K I ae constants that detemine the elative weights of the popotional and integal components of the contol signal. Substituting the new contolle into the model in Fig. 4 and pefoming a simila analysis to the one in Section 5.2 yields the following output esponse: s( s+ β D( s + ( KPs+ KI R( s Y( s = 2 s + ( β + KP s+ KI With no distubances, the system esponse to a step input as the efeence signal is KPs+ KI Y( s = 2 s s + ( β + KP s + KI Using the Laplace tansfom final value theoem, it can be shown that as t, y(t, which indicates the system

10 0 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, MANUSCRIPT ID eventually tacks the efeence signal without eo. If only a distubance signal is applied to the system as a step function and thee is no efeence signal, then the system esponse is given by Y( s = s + β 2 s + ( β + KP s+ KI Again using the final value theoem, the output signal appoaches zeo as t, so the system will completely eect distubances given enough time to compensate fo initial eos. This aticle s test cases demonstate the pefomance advantages of the popotional-integal contol law compaed to the simple popotional contolle. In this aticle s expeiments, the values fo the popotional and integal contolle gains wee found though expeiments guided by Zeigle-Nichol's tuning ules [7]. The above contolle design consideed the model as a continuous-time system and used Laplace tansfom techniques in the analysis. The continuous-time epesentation is convenient and efficient fo analysis, but the system will be simulated in a discete-time envionment. The geneal esults ae valid fo a discete-time implementation, but the details of the analysis and contolle design ae slightly diffeent. The expeiments un accounted fo these diffeences and ensued the final system design maintained stability. An intoduction to discete-time contol and the associated z-tansfom can be found in Ogata's book, "Discete-Time Contol Systems" [8]. 6 EXPERIMENTAL RESULTS This section has fou subsections. The fist subsection descibes the expeimental setup. The second gives a desciption of the adaptive tests. The thid pesents expeiments that compae the behavio of the adaptive methods pesented in Section 5 using tials adapted fom standad tansient esponse tests. The fouth subsection compaes the pefomance of the adaptive methods against TCP and UDP in the same set of tansient esponse tests to illustate the adaptive algoithm's stengths. 6. Expeimental Setup Standad contol tests wee duplicated to compae the effectiveness of the diffeent adaptive algoithms. Detailed desciptions of the impulse, amp esponse, and sustained oscillation tests used can be found in intoductoy contol theoy texts including Ogata's book, "Moden Contol Engineeing" [7]. The tests pupose is to measue the stability and esponse times of adaptive heuistics in the face of changing input signals. Each test is biefly descibed in Section 6.2 along with its tanslation fom the signal domain to a netwok communication envionment. Expeiments wee conducted using Netwok Simulato 2 (ns2 vesion 2.b9a [9]. Theoetical simulations and non-linea least squaes calculations wee pefomed using Matlab vesion 5.3 Release with the optimization toolbox. The Matlab lsqcuve-fit outine was called at the beginning of the simulation and as-needed to calculate gossip weights fo each of the system s nodes using the static gossip algoithm in section 3. Simulations wee un to compae the Section 6.3 adaptive heuistics pefomance based on cost and accuacy. Each of the test sets wee adapted fom the contol theoetic tests in Section 6.2. Because contol theoy is taditionally applied to a signaling envionment, the tests in that domain ae based on input signals that vay ove time. The tests wee adapted to a netwok domain by vaying the bit eo ate occuing in the netwok links. In the tests un, bit eos wee inected into the system links to emulate vaious foms of netwok disuptions. Bit eo ates vaied between a low of 0%, coesponding to a zeo signal, and a high ate of 0%, coesponding to a one signal in the contol theoetic tests. The intent was to see how well the adaptive heuistics wee able to adapt to vaying levels of netwok disuption. Bit eo ate settings wee univesally applied to all links in the netwok. Inceased bit eos led to a ise in the numbe of dopped packets, which could be used to see how obust diffeent algoithms wee in the face of such disuptions. The 0% bit-eo ate is highe than expected in most wied netwoks, but is not uneasonable in many wieless netwoks, paticulaly when mobile. Also, if a potocol pefoms well with a 0% bit-eo ate then it indicates potocol obustness in othe challenged envionments. The tests in Section 6.4 compae the pefomance of the AGG potocol using the Section 5 s adaptive heuistics against TCP and UDP-based implementations. In the TCP implementation, the sende (publishe eliably sends all new infomation in the system to each of the goup membes via TCP. The UDP implementation sends all new infomation to the goup s nodes uneliably using UDP. Tests wee pefomed using fou 20-membe subgoups linked togethe in a mesh, as shown in Fig. 5. The sende (the diagam s Data Souce peiodically ceated new messages at a ate of 20 messages pe ound. The taget data ates of 25%, 50%, 75%, and 00% wee used in the tests descibed in Section 5.3. The same basic configuation was used in the tests in Section 5.4 except that all taget ates wee set to 00%. Tests stated at time 3.5 and ended at time The initial 3.5 and final 2 seconds of simulation time seved as amp-up/amp-down time whee no new messages enteed the system. The untime, amp-up, and amp-down peiods wee chosen to be lage enough to demonstate the stability, o instability, of the potocols tested. The buffes in each node, used to keep the node s known infomation updates, had a size of 400 messages pe goup the node subscibed to. Messages timed out and expied afte 2 seconds. Thee was a 00 millisecond inteval between gossip ounds. All nodes set thei initial gossip times andomly within a 00 millisecond inteval. 0 tials wee un fo each test configuation. Packet latency was set to millisecond pe link tavesed. Each link had a capacity of 500 Mbits/second to eliminate bandwidth as a facto in the tests conducted. When gossiping, nodes sent packets to at least one destination chosen at andom within each subgoup. If no infomation was available to include in a message, pehaps because the potocol had ust begun, then a weightsonly message would be sent. These weights-only messages wee sent in ode to make sue that gossip weight up-

11 HOPKINSON ET AL.: ADAPTIVE GRAVITATIONAL GOSSIP: A GOSSIP-BASED COMMUNICATION PROTOCOL WITH USER-SELECTABLE RATES Fig. 5. The fomat of the adaptive tests dates would popagate in a timely fashion. 6.2 A Desciption of the Contol Tests and Adaptive Heuistics Evaluated 6.2. Netwok Adaptations of Contol Tests A desciption of the adaptive tests follows. The impulse, amp esponse, and sustained oscillation tests have been adapted fom contol theoy to the domain of communication netwoks. The tests ae designed to measue the stability of a system. Unit step and unit step distubance tests ae not included due to space consideations. The esults wee consistent with those included. Impulse: In contol systems, an impulse involves applying a sudden spike fom a 0 input to a and then immediately cutting back to 0. In the communication tests, the netwok began with no bit eo ate on the links. At time 23.5 all links wee set to a 0% eo ate. At time the bit eo ate fell back to a 0% ate. The key measuements wee the time it took fo the adaptive heuistics to poduce gossip pobability values between subgoups so that they matched thei tagets. Ramp Response: This test involves inecting a amping signal input fom 0 to. In the expeiments, the amping begins at time 43.5 seconds. The signal ate slowly ises fom a 0% to a 0% eo ate at each link in a seies of 20 incements. The eo ate at time * t is (0.0 / 20.0 * t ove incements of t = 20. The key measuement is the system stability unde shifting conditions. Sustained Oscillation: An oscillating signal is applied to test the system s esponse. In the communication tests, at time t the eo ate is set to ( (.0 + ( t mod 20 on signal ise and ( (20.0 (.0 + ( t mod 20 on its fall. This continues fo 00 steps (duing which t anges fom Adaptive Heuistics Evaluated The mao aspects of the adaptive heuistics wee descibed ealie in Section 5. A bief desciption of each of the nine heuistics is descibed below. Adaptive: Uses the least-squaes heuistic, descibed in Section 5., when new gossip weights ae calculated. Base No Coection: Calculates gossip weights accoding to the static GG algoithm. Base No Coection does not adapt gossip weights afte the initial calculation. Base Coection: Calculates gossip weights accoding to the static GG algoithm. Calculates the gossip weights again, using the linea least-squaes heuistic, based on the diffeence between the static gossip weights computed and the aveage infomation ates achieved in 500 mini-tials. The mini-tials take place outside a netwok simulato. Evey gossip message is eliably deliveed to its destination and all gossip occus all at once pe ound. This coective step appeas to incease the accuacy of the gossip weight calculations. Base Coection does not adapt the gossip weights afte this second calculation. Base Refinement: Calculates gossip weights accoding to the Base Coection heuistic. Calculates the weights again, using the linea least-squaes heuistic, one time fou seconds (twice the message expiation time afte gossip begins. Base Refinement does not adapt the gossip weights afte this calculation. Newton Cut Bounds: Newton's Secant method is used once the gossip weights ae calculated via Base Refinement. Newton's method does not calculate new gossip weights if the obseved infectivity is within +/-.02 of the taget infectivity. If a calculated gossip weight is above o below 0 then it is mapped to o 0 espectively. P Contol: The Popotional Contolle is used once the gossip weights ae calculated using Base Refinement. PI Contol: The Popotional-Integal Contolle is used once the gossip weights have been calculated using the Base Refinement heuistic. The following two implementations wee used as a basis fo compaison against the AGG potocol using each of the peviously descibed heuistics. TCP: The sende establishes a TCP connection with each of the othe nodes in the system. New gossip fagments ae placed in each of the TCP send queues as soon as they ae ceated at the sende. The TCP potocol is only used in the tests in Section 6.4. UDP: New gossip fagments ae sent fom the sende to each of the othe nodes using point to point UDP messages. UDP is only used in the Section 6.4 tests. 6.3 The Adaptive Heuistics Pefomance The adaptive heuistics in Section 5 wee evaluated accoding to the contol-based tests in Section The tests consistently showed that the popotional and popotional-integal contolles outpefomed the othe adaptive heuistics. The heuistic based on Newton s method pefomed similaly to the contolles except that Newton s method had highe oveall eo in the scenaios un. Eo hee is the diffeence between the desied taget infomation ate, which anges fom 0% to 00%, and the actual eceived infomation ate. Eo ates ae computed using a Manhattan metic, which sums the diffeence between the desied and actual pecentage of infomation eceived fo each of the fou subgoups in the system. Two vaiations wee used. In the fist, the absolute values of all diffeences wee added togethe and in the second, the elative diffeences wee added. Fig. 6 depicts key points egading the impulse test using the absolute value metic. Fist, the Manhattan-metic sum of the diffeence between the desied and actual infomation ates tends to be vey good in steady-state fo the contolles and Newton heuistic, with a value of appoximately 0.02 pe subgoup. The gaphs show that the non-adaptive vesions of the algoithms do much wose than thei adaptive countepats. The adaptive algoithm, which uses the linea least squaes method, is bette than the non-adaptive altenatives, but can divege fom the taget ove time. Newton's method, the Popotional Contolle, and the Popotional-Integal Contolle pefom

12 2 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, MANUSCRIPT ID Fig. 6. Impulse Test Results based on an absolute value Manhattan metic. The gaph illustates the effect of vaying bit-eo ates. Fig. 7. The Fig. 6 impulse test, but using a elative diffeence Manhattan metic. The test illustates the impact of vaying bit-eo ates. best. Of these, Newton's method pefoms wost once the initial calibation phase ended. The Popotional Contolle and Popotional-Integal Contolle pefomed similaly. Both apidly convege to nea-zeo eo afte a netwok distubance, except that the delayed vesion stabilizes moe quickly unde shifting netwok conditions. Fig. 7 shows the same impulse tests using a Manhattan metic, which sums the elative, athe than absolute, diffeences between the desied and actual infomation ates. It shows that the contolles and Newton heuistic tend to be close to the desied values and quickly stabilizes unde a disuptive change fom 0% to 0% bit-eo ate acoss all links. Fig. 8 shows the esult of amp esponse tests using the elative Manhattan metic. As the impulse tests showed peviously, the contolles and Newton heuistic ae faily stable while the least-squaes heuistic oscillates and tends to emain fathe away fom the desied ates. The moe static, heuistics do not pefom well unde changing conditions. Fig. 9 shows sustained oscillation test esults using the elative Manhattan metic. The gaphs show the impact of oscillating bit-eo ates in tems of eo vesus time (top and cumulative eo vesus time (bottom. The esults show that the contolles ae elatively stable, both in tems of thei ates ove time and in tems of thei cumulative eo. The Newton heuistic also pefoms well. The linea least-squaes (adaptive heuistic emains within a elatively constant amount of eo fom the desied tagets, but the heuistic also has a tendency to ise o fall on a cumulative basis ove time. As expected, static heuistics pefom pooly unde changing conditions. 6.4 A Compaison of AGG Vesus TCP and UDPbased Implementations As mentioned in the expeimental setup section, the taget gossip pecentage was set to 00% fo all goups. One un took place with an eo ate of 0% pe link and anothe had an eo ate of 0% pe link. Only the cumulative eo gaphs fo 0% and 0% fault scenaios ae included in this section. The othe gaphs could not be included due to space limitations. In the 0% fault scenaio, thee wee a lage numbe of dops between nodes. Only 65% of the packets wee successfully eceived afte taveling though ust fou links. The key measuements wee the eo ate, which is expessed in tems of the diffeence between a pefect 00% eceipt of infomation and the actual eceipt ate, the latency between the time a new piece of infomation was published to the system and its eceipt (computed pe packet eceived, and the aveage message load in tems of the numbe of packets sent pe second. All nine types of adaptive heuistics wee used including the TCP and UDP implementations descibed ealie. Packets that aived late than 2.0 seconds counted towads the total eceived in TCP as the potocol does not allow packets to expie. Fig. 0 shows the diffeence between the desied ate of infomation (00% in all cases and the actual ate. Both gaphs ae cumulative. The top gaph shows that the adaptive heuistics, the UDP, and the TCP implementations all pefom well by this metic when thee ae no bit-eos on the links in the system. The bottom gaph depicts the cumulative eo ates when a 0% bit-eo ate is applied to the links. TCP s congestion mechanism pevents it fom having the thoughput to meet the demand fo published infomation. The UDP-based potocol loses infomation, due to dopped packets, and it also has pefomance poblems, though not as sevee as with the TCP-based potocol. The AGG potocols ae able to adapt to the changed conditions and still delive nealy 00% of the published infomation. It is impotant to note that all potocols pefom easonably well in the 0% loss case. Even the wost potocol of Base Refinement only has a cumulative loss of 6 ove 360 seconds. By contast, each of the gossip vaiants pefoms fa bette than the oughly 50 and 250 cumulative loss ates of UDP and TCP, espectively, fo the easons outlined above. This shows the obustness of gossip in heavy loss situations. Gaphs of the aveage numbe of messages pe second ae not shown due to space consideations. The aveage numbe of messages sent pe node emained steady in each of the potocols. One UDP packet is sent to each node in the system pe published piece of infomation so its ate emains at a constant 20 messages pe second in both the 0% bit-eo ate pe link and 0% bit-eo ate pe link cases. The AGG potocols also emain steady at a constant five messages pe node pe gossip ound, except duing adaptive gossip calibation, which take place evey fou seconds, when the numbe aveages ust ove 6 messages pe node. Duing the adaptive gossip calibation points, infomation must be gatheed and disseminated egading cuent system pefomance using gossip messages. TCP s congestion-contol mechanism esults in

13 HOPKINSON ET AL.: ADAPTIVE GRAVITATIONAL GOSSIP: A GOSSIP-BASED COMMUNICATION PROTOCOL WITH USER-SELECTABLE RATES 3 Fig. 8. This gaph shows the esult of amp-esponse tests. The test illustates the effect of vaying bit-eo ates. Fig. 9. These gaphs show the esult of sustained oscillation tests. The gaphs show the esult of oscillating bit-eo ates. The top gaph is a plot of elative Manhattan eos vesus time. The bottom gaph show cumulative eo vesus time. a lowe load on the system with an aveage load pe node of oughly packet pe second athe than the 2 that is pesent duing the 0% bit-eo scenaio. Gaphs depicting the latency acoss the 0% and 0% bit eo-ate scenaios ae not shown due to space constaints. The latency was extemely low fo UDP packets, aveaging unde 0 ms, because no ecovey mechanism was pesent. The AGG gossip-based potocols tended to have highe latency in good conditions, aveaging ust unde. seconds. Howeve, latency emained nealy constant as the bit-eo ate inceased. The TCP-based potocol, by contast, had a latency of unde 0 ms when no bit eo ates wee pesent. Howeve, it had high la- Fig. 0. The AGG, the UDP, and the TCP potocols all pefom well when no bit-eos on the links (top gaph. A 0% bit-eo ate pe link (bottom gaph is a mao poblem fo UDP and TCP while AGG delives close to 00% of the desied infomation. tency, with an aveage of 28.5 seconds when bit eo ates wee 0% pe link. Latency was the time diffeence between the moment an infomation update was fist ceated to the time it was eceived via a packet at a node. These esults show that AGG stikes a stong balance between deliveing infomation at a taget ate, the latency that such deliveies take, and the pe node aveage message load. Gossip can pefom bette with high bit ates because of its exponential spead of infomation and its ability to continue to opeate even when some nodes in the multicast goup ae eceiving data at a low ate. Pevious wok by the authos and othes has looked at the impact of ovelapping goups, of diffeent gossip distibutions, and of vaying goup sizes. Pio esults show that system ovehead deceases as subgoup sizes incease []. Efficiency also ises as the numbe of ovelapping goups gows because gossip infomation can often be piggy-backed into combined packets [20]. Notable studies show that non-unifom distibutions can impove pefomance when the opeating envionment is not unifom, pehaps due to geogaphy o netwok pefomance diffeences [, 3]. Othe chaacteistics, such as enegy use, may also be bette optimized unde othe distibutions [2]. These studies, as a whole, show that gossip can be tuned with domain knowledge fo bette pefomance and this could be a good avenue fo futue exploation. 7 CONCLUSION This aticle has pesented a new gossip-based goup communication potocol called adaptive gavitational gossip (AGG. AGG is tageted towads publish-