Privacy Preserving Randomized Gossip Algorithms. University of Edinburgh, UK KAUST, KSA Higher School of Economics, Russia.

Transcription

1 Prvacy Preservng Randomzed Gossp Algorhms Flp Hanzely Jakub Konečný Ncolas Lozou Peer Rchárk Dmry Grshchenko Unversy of dnburgh, UK KAUST, KSA Hgher School of conomcs, Russa June, 07 arxv: v mah.oc 3 Jun 07 Absrac In hs work we presen hree dfferen randomzed gossp algorhms for solvng he average consensus problem whle a he same me proecng he nformaon abou he nal prvae values sored a he nodes. We gve eraon compley bounds for all mehods, and perform eensve numercal epermens. Inroducon In hs paper we consder he average consensus AC problem. Le G V, be an undreced conneced nework wh node se V {,,..., n} and edges such ha m. ach node V knows a prvae value c R. The goal of AC s for every node of he nework o compue he average of hese values, c def n c, n a dsrbued fashon. Tha s, he echange of nformaon can only occur beween conneced nodes neghbours. The leraure on mehods for solvng he average consensus problem s vas and has long hsory 5, 5, 4, 30. The algorhms for solvng hs problem can be dvded n wo broad caegores: he average consensus algorhms 54 and he gossp algorhms 5, 49. The man dfference s ha he former work n a synchronous seng whle he gossp algorhms model he case of asynchronous seng. In he average consensus algorhms, all nodes of he nework updae her values smulaneously by communcae wh a se of her neghbours and n all eraons he same updae occurs. In gossp algorhms, a each eraon, only one edge s seleced randomly, and he correspondng nodes updae her values o her average. In hs work we focus on randomzed gossp algorhms and propose echnques for proecng nformaon of he nal values c, n he case when hese may be sensve. In hs work we develop and analyze hree prvae varans of he randomzed parwse gossp algorhm for solvng he average consensus problem. As an addonal requremen we wsh o preven nodes o learn nformaon abou he prvae values of oher nodes. Whle we shall no formalze he noon of prvacy preservaon n hs work, wll be nuvely clear ha our mehods ndeed make harder for nodes o nfer nformaon abou he prvae values of oher nodes.. Background The average consensus problem and randomzed gossp algorhms for solvng appear n many applcaons, ncludng dsrbued daa fuson n sensor neworks 55, load balancng 0 and clock synchronzaon 8. Ths subec was frs nroduced n 5, and was suded eensvely n he las decade; he semnal 006 paper of Boyd e al. 5 on randomzed gossp algorhms movaed a large amoun of subsequen research and generaed more han 500 caons o dae. In hs work, we focus on modfyng he basc algorhm of 5, whch we refer o as Sandard Gossp algorhm. In he followng, we revew several avenues of research he gossp algorhms were evolved. Whle we do no address any prvacy consderaons n hese sengs, hey can serve

2 as nspraon for furher work. For a survey of relevan work pror o 00, we refer he reader o revews n, 40, 45. The Geographc Gossp algorhm was proposed n, n whch he auhors combne he gossp approach wh a geographc roung owards a randomly chosen locaon wh man goal he mprovemen of he convergence rae of Sandard Gossp algorhm. In each sep, a node s acvaed, assumng ha s aware of s geographc locaon and some addonal assumpons on he nework opology, chooses anoher node from he res of he nework no necessarly one of s neghbours and performs a parwse averagng wh hs node. Laer, usng he same assumpons, hs algorhm was eended no Geographc Gossp Algorhm wh Pah Averagng 3, n whch conneced sequences of nodes were chosen n each sep and hey averaged her values. More recenly, n 9 and 0 auhors propose a geographc and pah averagng mehods whch converge o he average consensus whou he assumpon ha nodes are aware of her geographc locaon. Anoher mporan class of randomzed gossp algorhms are he Broadcas Gossp algorhms, frs proposed n and hen eended n 7, 53, 9. The dea of hs algorhm s smple: In each sep a node n he nework s acvaed unformly a random, followng he asynchronous me model, and broadcass s value o s neghbours. The neghbours receve hs value and updae her own values. I was epermenally shown ha hs mehod converge faser han he parwse and geographc randomzed gossp algorhms. Alernave approach o he gossp framework are so called non-randomzed Gossp algorhms 38, 7, 34, 56. Typcally, hs class of algorhms eecues he parwse echanges beween nodes n a deermnsc, such as pre-defned cyclc, order. T -perodc gosspng s a proocol whch spulaes ha each node mus nerac wh each of s neghbours eacly once every T me uns. Under suable connecvy assumpons of he nework G, he T -perodc gossp sequence wll converge a a rae deermned by he magnude of he second larges egenvalue of he sochasc mar deermned by he sequence of parwse echanges whch occurs over a perod. I has been shown ha f he underlyng graph s a ree, hs egenvalue s he same for all possble T -perodc gossp proocols. A dfferen approach, uses memory n he updae of he values each node holds, o ge Acceleraed Gossp algorhms. The nodes updae her value usng an updae rule ha nvolve no only he curren values of he sampled nodes bu also her prevous values. Ths dea s closely relaed o he shf regser mehods suded n numercal lnear algebra for mprovng he convergence rae of lnear sysem solvers. The works 6, 33 have shown heorecally and numercally, ha under specfc assumpons hs dea can mprove he performance of he Sandard Gossp algorhm. Randomzed Kaczmarz-ype Gossp algorhms. Very recenly has been proved ha popular randomzed Kaczmarz-ype mehods for solvng large lnear sysems can also solve he AC problem. In parcular, n 3 and 35 was shown how ha esng Randomzed Kaczmarz and Randomzed Block Kaczmarz mehods can be nerpreed as randomzed gossp algorhms for solvng he AC problem, by solvng a parcular sysem encodng he underlyng nework srucure. Ths approach was he frs me ha a connecon beween he wo research areas of lnear sysem solvers and dsrbued algorhms have been esablshed. In hs work we are neresed n he asynchronous me model 5, 4. More precsely, we assume ha each node of our nework has a clock whch cks a a rae of Posson process. Ths s equvalen of havng avalable a global clock whch cks accordng o a rae n Posson process and selecs an edge of he nework unformly a random. In general he synchronous seng all nodes updae he values of her nodes smulaneously usng nformaon from a se of her neghbours s convenen for heorecal consderaons, bu s no represenave of some praccal scenaros, such as he dsrbued naure of sensor neworks. For more deals on clock modellng we refer he reader o 5, as he conrbuon of hs paper s orhogonal o hese consderaons. Prvacy and Average Consensus. Fnally, he nroducon of noons of prvacy whn he AC problem s relavely recen n he leraure, and he esng works consder wo dfferen deas. The concep of dfferenal prvacy 3 s used o proec he oupu value c compued by all nodes n 8. In hs work, an eponenally decayng Laplacan nose s added o he consensus compuaon. Ths noon of prvacy refers o proecon of he fnal average, and formal guaranees are provded. A dfferen goal s he proecon of he nal values c he nodes know a he sar. In 36, 37, he

3 goal s o make sure ha each node s unable o nfer a lo abou he nal values c of any oher node. Boh of hese mehods add nose correlaed across ndvdual eraons, o make sure hey converge o he eac average. A formal noon of prvacy breach s formulaed n 37, n whch hey also show ha her algorhm s opmal n ha parcular sense.. Man Conrbuons In hs work we presen hree dfferen approaches for solvng he Average Consensus problem whle a he same me proecng he nformaon abou he nal values. All of he above menoned works focus on he synchronous seng of he AC problem. Ths work s he frs whch combnes he gossp framework wh he prvacy concep of proecon of he nal values. I s mporan o sress ha we provde ools for proecon of he nal values, bu we do no address any specfc noon of prvacy or a hrea model, nor how hese quanavely ranslae o any eplc measure. These would be hghly applcaon dependen, and we only provde heorecal convergence raes for he echnques we propose. The mehods we propose are all dual n naure. The dual approach s eplaned n deal n Secon. I was frs proposed for solvng lnear sysems n 3 and hen eend o he concep of average consensus problems n 35. The dual updaes mmedaely correspond o updaes o he prmal varables, va an affne mappng. One of he conrbuons of our work s ha we eacly recover esng convergence raes for he prmal eraes as a specal case. We now oulne he hree dfferen echnques we propose n hs paper, whch we refer o as Bnary Oracle, ɛ-gap Oracle and Conrolled Nose Inseron. The frs wo are, o he bes of our knowledge, he frs proposal of weakenng he oracle used n he gossp framework. The laer s nspred by and smlar o he addon of nose proposed n 37 for he synchronous seng. We eend hs echnque by provdng eplc fne me convergence guaranees. Bnary Oracle. The dfference from sandard Gossp algorhms we propose s o reduce he amoun of nformaon ransmed n each eraon o a sngle b. More precsely, when an edge s seleced, each correspondng node wll only receve nformaon wheher he value on he oher node s smaller or larger. Insead of seng he value on each node o her average, each node ncreases or decreases s value by a pre-specfed sep. ɛ-gap Oracle. In hs case, we have an oracle ha reurns one of hree opons, and s paramerzed by ɛ. If he dfference n values of sampled nodes s larger han ɛ, an updae smlar o he one n Bnary Oracle s aken. Oherwse, he values reman unchanged. An advanage compared o he Bnary Oracle s ha hs approach wll converge o a ceran accuracy and sop here, deermned by ɛ Bnary Oracle wll oscllae around opmum for a fed sepsze. However, n general wll dsclose more nformaon abou he nal values. Conrolled Nose Inseron. Ths approach s nspred by he works of 36, 37, and proecs he nal values by nserng nose n he process. Broadly speakng, n each eraon, each of he sampled nodes frs add a nose o s curren value, and an average s compued aferwards. Convergence s guaraneed because of correlaon n he nose across eraons. ach node remembers he nose added las me was sampled, and n he followng eraon, he prevously added nose s frs subraced, and a fresh nose of smaller magnude s added. mprcally, he proecon of nal values s provded by frs necng nose n he sysem, whch propagaes across nework, bu s gradually whdrawn o ensure convergence o he rue average. Convergence Raes of our Mehods. In Table, we presen summary of convergence guaranees for he above hree echnques. By we denoe he sandard ucldean norm. The wo approaches o resrcng he amoun of nformaon dsclosed, Bnary Oracle and ɛ-gap Oracle, converge slower han he sandard Gossp. In parcular, hese algorhms have sublnear convergence rae. A frs sgh, hs should no be surprsng, snce we ndeed use much less nformaon. However, n Theorem 7, we show ha f we had n a ceran sense perfec global nformaon, we could use o consruc a sequence of adapve sepszes, whch would push he capably of he bnary oracle o a lnear convergence rae. However, hs rae s sll m-mes slower han he sandard rae of he bnary gossp algorhm. We noe, however, ha havng global nformaon a hand s an 3

4 Man Resuls Randomzed Gossp Mehods Convergence Rae Success Measure Thm Sandard Gossp 5 New: Prvae Gossp wh Bnary Oracle New: Prvae Gossp wh ɛ-gap Oracle New: Prvae Gossp wh Conrolled Nose Inseron k αg m c k 5 / k mn k m e 6 /kɛ k k 0 ɛ k mn αg m, γ Dy Dy k m 9 Table : Compley resuls of all proposed gossp algorhms. mpraccal assumpon. Neverheless, hs resul hghlghs ha here s a poenally large scope for mprovemen, whch we leave for fuure work. These wo oracles could be n pracce mplemened usng esablshed secure mulpary compuaon proocols 8. However, hs would requre he sampled nodes o echange more han a sngle message n each eraon. Ths s nferor o he requremens of he sandard gossp algorhm, bu he concern of communcaon effcency s orhogonal o he conrbuon of hs work, and we do no address furher. The approach of Conrolled Nose Inseron yelds a lnear convergence rae whch s drven by mamum of wo facors. Whou gong no deals, whch of hese s bgger depends on he speed by whch he magnude of he nsered nose decays. If he nose decays fas enough, we recover he convergence rae of sandard he gossp algorhm. In he case of slow decay, he convergence s drven by hs decay. By αg we denoe he algebrac connecvy of graph G 6. The parameer γ conrols he decay speed of he nsered nose, see 0..3 Measures of Success Second maor dsncon o hghlgh s ha convergence of each of hese proposals naurally depend on a dfferen measure of subopmaly. All of hem go o 0 as we approach opmal soluon. The deals of hese measures wll be descrbed laer n he man body, bu we gve a bref summary of her connecons below. The sandard Gossp and Conrolled Nose Inseron depend on he same quany, bu we presen he laer n erms of dual values as hs s wha our proofs are based on. Lemma formally specfes hs equvalence. The bnary oracle depends on average dfference among drecly conneced nodes. The measure for he ɛ-gap Oracle depends on quanes ɛ m {, :, ɛ} whch s he number of edges, connecng nodes, values of whch dffer by more han ɛ. To draw connecon beween hese measures, we provde he followng lemma, proved n he Append. The dual funcon D s formally defned n Secon. Lemma. Relaonshp beween convergence measures Suppose ha s prmal varable correspondng o he dual varable y as defned n 9. Dual subopmaly can be epressed as he followng 3: Dy Dy c. Moreover, for any R n we have : 4

5 n e, e, e, c mn c, 3 αg c, 4 ɛ {, : ɛ}. 5.4 Oulne The remander of hs paper s organzed as follows: Secon nroduces he basc seup ha are used hrough he paper. A dealed eplanaon of he dualy behnd he randomzed parwse gossp algorhm s gven. We also nclude a novel and nsghful dual analyss of hs mehod as wll make easer o he reader o parse laer developmen. In Secon 3 we presen our hree prvae gossp algorhms as well as he assocaed eraon compley resuls. Secon 4 s devoed o he numercal evaluaon of our mehods. Fnally, conclusons are drawn n Secon 5. All proofs no ncluded n he man e can be found n he Append. Dual Analyss of Randomzed Parwse Gossp As we oulned n he nroducon, our approach o eendng he sandard randomzed parwse gossp algorhm o prvacy preservng varans ulzes dualy. The purpose of hs secon s o formalze hs dualy, followng he developmen n 3. In addon, we provde a novel and selfconan dual analyss of randomzed parwse gossp. Whle hs s of an ndependen neres, we nclude he proofs as her undersandng ads n he undersandng of he more nvolved proofs of our prvae gossp algorhms developed n he remander of he paper.. Prmal and Dual Problems Consder solvng he prmal problem of proecng a gven vecor c 0 R n ono he soluon space of a lnear sysem: def mn Rn{P 0 } subec o A b, 6 where A R m n, b R m, 0 R n. We assume he problem s feasble,.e., ha he sysem A b s conssen. Wh he above opmzaon problem we assocae he dual problem def ma Dy b A 0 y y Rm A y. 7 The dual s an unconsraned concave bu no necessarly srongly concave quadrac mamzaon problem. I can be seen ha as soon as he sysem A b s feasble, he dual problem s bounded. Moreover, all bounded concave quadracs n R m can be wren n he as Dy for some mar A and vecors b and 0 up o an addve consan. Wh any dual vecor y we assocae he prmal vecor va an affne ransformaon: y 0 + A y. I can be shown ha f y s dual opmal, hen y s prmal opmal. Hence, any dual algorhm producng a sequence of dual varables y y gves rse o a correspondng prmal algorhm producng he sequence def y. We shall now consder one such dual algorhm. 5

6 . Sochasc Dual Subspace Ascen SDSA 3 s a sochasc mehod for solvng he dual problem 7. If we assume ha b 0, he eraes of SDSA ake he form y + y S S AA S S A 0 + A y, 8 where S s a random mar drawn from ndependenly a each eraon from an arbrary bu fed dsrbuon D, and denoes he Moore-Penrose pseudonverse. The correspondng prmal eraes are defned va: def y 0 + A y 9 The relevance of hs all o average consensus follows hrough he observaon, as we shall see ne, ha for a specfc choce of mar A as defned n he ne subsecon and dsrbuon D, mehod 9 s equvalen o he sandard randomzed parwse gossp mehod. In ha case, SDSA s a dual varan of randomzed parwse gossp. In parcular, we defne D as follows: S s a un bass vecor n R m, chosen unformly a random from he collecon of all such un bass vecors, denoed {f e e }. In hs case, SDSA s a randomzed coordnae ascen mehod appled o he dual problem. For general dsrbuons D, he prmal mehods obaned from SDSA va 9 bu whou observng ha arses ha way was fs proposed and suded n under a full rank assumpon on A. Ths assumpon was lfed, and dualy eposed and suded as we eplan here, n 3. For deeper nsghs and connecons o sochasc opmzaon, sochasc fed pon mehods, sochasc lnear sysems and probablsc nersecon problems, we refer he reader o 48. The mehod can be eended o compue he nverse 5 and pseudonverse 4 of a mar, n whch case has deep connecons wh quas-newon updaes 5. In parcular, can be used o desgn a sochasc block eenson of he famous BFGS mehod 5 and appled o he emprcal rsk mnmzaon problem arsng n machne learnng o desgn a fas sochasc quas-newon ranng mehod..3 Randomzed Gossp Seup: Choosng A We wsh A, b o be an average consensus AC sysem, defned ne. Defnon. 35 Le G V, be an undreced graph wh V n and m. Le A be a real mar wh n columns. The lnear sysem A b s an average consensus AC sysem for graph G f A b ff for all,. Noe ha f A b s an AC sysem, hen he soluon of he prmal problem 6 s necessarly c, where c n n 0. Ths s eacly wha we wan: we wan he soluon of he prmal problem o be c for all : he average of he prvae values sored a he nodes. I s easy o see ha a lnear sysem s an AC sysem precsely when b 0 and he nullspace of A s { : R}, where s he vecor of all ones n R n. Hence, A has rank n. In he res of hs paper we focus on a specfc AC sysem; one n whch he mar A s he ncdence mar of he graph G see Model n 3. In parcular, we le A R m n be he mar defned as follows. Row e, of A s gven by A e, A e and A el 0 f l / {, }. Noce ha he sysem A 0 encodes he consrans for all,, as desred..4 Randomzed Parwse Gossp We provde boh prmal and dual form of he sandard randomzed parwse gossp algorhm. The prmal form s sandard and needs no lenghy commenary. A he begnnng of he process, node conans prvae nformaon c 0. In each eraon we sample a par of conneced nodes, unformly a random, and updae and o her average. We le he values a he remanng nodes nac. 6

7 Algorhm Prmal form Inpu: vecor of prvae values c R n Inalze: Se 0 c. for 0,,..., k do. Choose node e, unformly a random. Updae he prmal varable: + l { +, l {, } l, l / {, }. end reurn k The dual form of he sandard randomzed parwse gossp mehod s a specfc nsance of SDSA, as descrbed n 8, wh 0 c and S beng a randomly chosen sandard un bass vecor f e n R m e s a randomly seleced edge. I can be seen 3 ha n ha case, 8 akes he followng form: Algorhm Dual form Inpu: vecor of prvae values c R n Inalze: Se y 0 0 R m. for 0,,..., k do end reurn y k. Choose node e, unformly a random. Updae he dual varable: y + y + λ f e where λ argma λ Dy + λ f e. The followng lemma s useful for he analyss of all our mehods.i descrbes he ncrease n he dual funcon value afer an arbrary change o a sngle dual varable e. Lemma 3. Defne z y + λf e, where e, and λ R. Then Proof. The clam follows by drec calculaon: Dz Dy λ λ. 0 Dy + λf e Dy Ac y + λf e A y + λf e + Ac y + A y λfe A c + A y }{{} λ A f e }{{} λ λ. The mamzer n λ of he epresson n 0 leads o he eac lne search formula λ / used n he dual form of he mehod..5 Compley Resuls Wh graph G {V, } we now assocae a ceran quany, whch we shall denoe β βg. I s he smalles nonnegave number β such ha he followng nequaly holds for all R n : We wre, o ndcae sum over all unordered pars of verces. Tha s, we do no coun, and, separaely, only once. By, we denoe a sum over all edges of G. On he oher hand, by wrng, we are summng over all unordered pars of verces wce. 7

8 β.,, The Laplacan mar of graph G s gven by L A A. Le λ L λ L λ n L λ n L be he egenvalues of L. The algebrac connecvy of G s he second smalles egenvalue of L: αg λ n L. We have λ n L 0. Snce we assume G o be conneced, we have αg > 0. Thus, αg s he smalles nonzero egenvalue of he Laplacan: αg λ + mn L λ+ mn A A. As he ne resul saes, he quanes βg and αg are nversely proporonal. Lemma 4. βg n αg. The followng heorem gves a compley resul for sandard randomzed gossp. Our analyss s dual n naure see he Append. Theorem 5. Consder he randomzed gossp algorhm Algorhm wh unform edge-selecon probables: p e /m. Then: Dy Dy k αg k Dy Dy 0. m Theorem 5 yelds he compley esmae O m αg, log/ɛ whch eacly maches he compley resul obaned from he prmal analyss 3. Hence, he prmal and dual analyses gve he same rae. Randomzed coordnae descen mehods were frs analyzed n 3, 39, 46, 47. For a recen reamen, see 4, 43. Dualy n randomzed coordnae descen mehods was suded n 50, 44. Acceleraon was suded n 3, 5,. These mehods eend o nonsmooh problems of varous flavours 4, 7. Wh all of hs preparaon, we are now ready o formulae and analyze our prvae gossp algorhms; we do so n Secon 3. 3 Prvae Gossp Algorhms In hs secon we nroduce hree novel prvae gossp algorhms, complee wh eraon compley guaranees. In Secon 3. we proec prvacy va a bnary communcaon proocol. In Secon 3. we communcae more: besdes bnary nformaon, we allow for he communcaon of a bound on he gap, nroducng he ɛ-gap oracle. In Secon 3.3 we nroduce a prvacy-proecon mechansm based on a procedure we call conrolled nose nseron. 3. Prvae Gossp va Bnary Oracle We now presen he gossp algorhm wh Bnary Oracle n deal and provde heorecal convergence guaranee. The nformaon echanged beween sampled nodes s consraned o a sngle b, descrbng whch of he nodes has he hgher value. As menoned earler, we only presen he concepual dea, no how eacly would he oracle be mplemened whn a secure mulpary proocol beween parcpang nodes 8. We wll frs nroduce dual verson of he algorhm. 8

9 Algorhm Dual form Inpu: vecor of prvae values c R n, sequence of posve sepszes {λ } 0 Inalze: Se y 0 0 R m, 0 c for 0,,..., k do. Choose node e, unformly a random. Updae he dual varable: 3. Se y { y + λ f e, <, y λ f e,. { + λ, <, λ, {. λ, <, + λ. + l l l {, } end reurn y k The updae of prmal varables above s equvalen o se + as prmal pon correspondng o dual erae: + c + A y + + A y + y. In oher words, he prmal eraes { } assocaed wh he dual eraes {y } can be wren n he form: { + + λ A e:, <, λ A e:,. I s easy o verfy ha due o he srucure of A, hs s equvalen o he updaes above. Snce he evoluon of dual varables {y k } serves only he purpose of he analyss, he mehod can be wren n he prmal-only form as follows: Algorhm Prmal form Inpu: vecor of prvae values c R n, sequence of posve sepszes {λ } 0 Inalze: Se 0 c for 0,,..., k do. Choose node e, unformly a random. Se + + { + λ, <, λ, {. λ, <, + λ. + l l l {, } end reurn k Gven a sequence of sepszes {λ }, wll be convenen o defne α k def k 0 λ and β k def 9

10 k 0 λ. In he followng heorem, we sudy he convergence of he quany Theorem 6. For all k we have Moreover: mn L 0,,...,k L def m k 0 e, If we se λ λ 0 > 0 for all, hen U k Dy Dy 0 λ 0 k+ + λ λ α k L U k def Dy Dy 0 α k + βk α k. 4 Le R be any consan such ha R Dy Dy 0. If we f k, hen he choce of sepszes {λ 0,..., λ k } whch mnmzes U k correspond o he consan sepsze rule λ R k+ for all 0,,..., k, and U k R k+. If we se a/ + for all 0,,..., k, hen U k Dy Dy 0 + a logk + 3/ + log a k + logk O k The par of Theorem 6 s useful n he case f we know eacly he number of eraons before runnng he algorhm, provdng n a sense opmal sepszes and rae O/ k. However, hs mgh no be he case n pracce. Therefore par s also relevan, whch yelds he rae Ologk/ k. These bounds are sgnfcanly weaker han he sandard bound n Theorem 5. Ths should no be surprsng hough, as we use sgnfcanly less nformaon han he Sandard Gossp algorhm. Neverheless, here s a poenal gap n erms of wha rae can be praccally achevable. The followng heorem can be seen as a form of a bound on wha convergence rae s possble o aan wh he Bnary Oracle. However, hs s aaned wh access o very srong nformaon needed o se he sequence of sepszes λ, lkely unrealsc n any applcaon. Ths resul pons a a gap n he analyss whch we leave open. We do no know wheher he sublnear convergence rae n Theorem 6 s necessary or mprovable whou addonal nformaon abou he sysem. Theorem 7. For Algorhm wh sepszes chosen n eraon adapvely o he curren values of as λ m, we have c k αg k m c 0 Comparng Theorem 7 wh he resul for sandard Gossp n Theorem 5, he convergence rae s worse by facor of m, whch s he prce we pay for he weaker oracle. An alernave o choosng adapve sepszes s he use of adapve probables 9. We leave such a sudy o fuure work. 3. Prvae Gossp va ɛ-gap Oracle Here we presen he gossp algorhm wh ɛ-gap Oracle n deal and provde heorecal convergence guaranees. The nformaon echanged beween sampled nodes s resrced o be one of hree cases, based on dfference n values on sampled nodes. As menoned earler, we only presen he concepual dea, no how eacly would he oracle be mplemened whn a secure mulpary proocol beween parcpang nodes 8. We wll frs nroduce dual verson of he algorhm. 0

11 Algorhm 3 Dual form Inpu: vecor of prvae values c R n ; error olerance ɛ > 0 Inalze: Se y 0 0 R m ; 0 c. for 0,,..., k do. Choose node e, unformly a random. Updae he dual varable: y + ɛ y + f e, y ɛ f e, y, < ɛ < ɛ, oherwse. 3. If 4. If ɛ hen + ɛ hen + + ɛ ɛ and + ɛ and + + ɛ end reurn k Noe ha he prmal eraes { } assocaed wh he dual eraes {y } can be wren n he form: + ɛ + A e:, < ɛ ɛ A e:, < ɛ,, oherwse. The above s equvalen o seng + + A y + y c + A y +. Snce he evoluon of dual varables {y } serves only he purpose of he analyss, he mehod can be wren n he prmal-only form as follows: Algorhm 3 Prmal form Inpu: vecor of prvae values c R n ; error olerance ɛ > 0 Inalze: Se 0 c. for 0,,..., k do. Se +. Choose node e, unformly a random 3. If 4. If ɛ hen + ɛ hen + + ɛ ɛ and + ɛ and + + ɛ end reurn k Before sang he convergence resul, le us defne a quany he convergence wll naurally depend on. For each edge e, and eraon 0 defne he random varable {, ɛ, eɛ def 0, oherwse. Moreover, le ɛ def m eɛ. 5

12 The followng Lemma bounds he epeced ncrease n dual funcon value n each eraon. Lemma 8. For all 0 we have Dy + Dy ɛ 4 ɛ. Our compley resul wll be epressed n erms of he quany: δ k ɛ def k ɛ k 0 k ɛ. 6 k 0 Theorem 9. For all k we have δ k ɛ 4 Dy Dy 0 kɛ. Noe ha f k ɛ 0, does no mean he prmal erae k s opmal. Ths only mples ha he values of all pars of drecly conneced nodes are dffer by less han ɛ. 3.3 Prvae Gossp va Conrolled Nose Inseron In hs secon, we presen he Gossp algorhm wh Conrolled Nose Inseron. As menoned n he nroducon, he approach s smlar he echnque proposed n 36, 37. Those works, however, address only algorhms n he synchronous seng, whle our work s he frs o use hs dea n he asynchronous seng. Unlke he above, we provde fne me convergence guaranees and allow each node o add he nose dfferenly, whch yelds a sronger resul. In our approach, each node adds nose o he compuaon ndependenly of all oher nodes. However, he nose added s correlaed beween eraons for each node. We assume ha every node owns wo parameers nal magnude of he generaed nose σ and rae of decay of he nose φ. The node nsers nose w o he sysem every me ha an edge correspondng o he node was chosen, where varable carres an nformaon how many mes he nose was added o he sysem n he pas by node. Thus, f we denoe by he curren number of eraons, we have n. In order o ensure convergence o he opmal soluon, we need o choose a specfc srucure of he nose n order o guaranee he mean of he values converges o he nal mean. In parcular, n each eraon a node s seleced, we subrac he nose ha was added las me, and add a fresh nose wh smaller magnude: w φ v φ v, where 0 φ <, v 0 and v N0, σ for all eraon couners k 0 s ndependen o all oher randomness n he algorhm. Ths ensures ha all nose added nally s gradually whdrawn from he whole nework. Afer he addon of nose, a sandard Gossp updae s made, whch ses he values of sampled nodes o her average. Hence, we have lm c lm φ n v lm φ n v n n lm n lm 0, φ φ v σ n n lm σ lm φ φ v as desred. I s no he purpose of hs paper o defne any quanfable noon of proecon of he nal values formally. However, we noe ha s lkely he case ha he proecon of prvae value c wll be sronger for bgger σ and for φ closer o.

13 For smplcy, we provde only he prmal algorhm below. Algorhm 4 Prmal form Inpu: vecor of prvae values c R n ; nal varances σ R + and varance decrease rae φ such ha 0 φ < for all nodes. Inalze: Se 0 c; n 0, v v vn 0. for 0,,... k do. Choose node e, unformly a random. Generae v 3. Se N0, σ and v N0, σ w w φ v φ v φ v φ v end reurn k 4. Updae he prmal varable: 5. Se +, w + + w, l, : + l l We now provde resuls of dual analyss of Algorhm 4. The followng lemma provdes us he epeced decrease n dual subopmaly for each eraon. Lemma 0. Le d denoe he number of neghbours of node. Then, Dy Dy + αg m m Dy Dy + 4m φ v + φ d σ v. φ 7 We use he lemma o prove our man resul, n whch we show lnear convergence for he algorhm. For noaonal smplcy, we decded o have ρ ρ,.e. superscrp of ρ denoes s power, no an eraon couner. Theorem. Le us defne he followng quanes: ρ ψ def αg m, def n d σ d σ d φ m. Then for all k we have he followng bound Dy Dy k ρ k Dy Dy 0 d σ + 4m k ρ k ψ. 3

14 Noe ha ψ s a weghed sum of -h powers of real numbers smaller han one. For large enough, hs quany wll depend on he larges of hese numbers. Ths brngs us o defne M as he se of ndces for whch he quany d m φ s mamzed: M arg ma { d } φ m. Then for any ma M we have ψ n d σ d σ M d φ m M d σ n d σ d ma φ m ma, whch means ha ncreasng φ for M wll no subsanally nfluence convergence rae. Noe ha as soon as we have ρ > d φ m for all, he rae from heorem wll be drven by ρ k as k and we wll have Dy Dy k Õ ρ k 9 One can hnk of he above as a hreshold: f here s such ha φ s large enough so ha he nequaly 8 does no hold, he convergence rae s drven by φ ma. Oherwse, he rae s no nfluenced by he nseron of nose. Thus, n heory, we do no pay anyhng n erms of performance as long as we do no h he hreshold. One mgh be neresed n choosng φ so ha he hreshold s aaned for all, and hus M {,..., n}. Ths movaes he followng resul: 8 Corollary. Le us choose def φ γ 0 d for all, where γ d mn. Then Dy Dy k αg mn m, γ k n Dy Dy 0 d σ + k. m 4m As a consequence, φ αg d s he larges decrease rae of nose for node such ha he guaraneed convergence rae of he algorhm s no volaed. Whle he above resul clearly saes he mporan hreshold, s no always praccal as αg mgh no be known. However, noe ha f we choose nd mn n γ d mn, we have mn αg m, γ m αg m snce αg n d mn n γ, where eg denoes graph edge connecvy: he mnmal number of edges o be removed so ha he graph becomes dsconneced. Inequaly αg n n d mn s a well known resul n specral graph heory 6. As a consequence, f for all we have φ hen he convergence rae s no drven by he nose. n d mn nd, 4 Numercal valuaon We devoe hs secon o epermenally evaluae he performance of he Algorhms, 3 and 4 we proposed n he prevous secons, appled o he Average Consensus problem. In he followng epermens, we used he followng popular graph opologes. 4

15 Cycle graph wh n nodes: Cn. In our epermens we choose n 0. Ths small smple graph wh regular opology s chosen for llusraon purposes. Random geomerc graph wh n nodes and radus r: Gn, r. Random geomerc graphs 4 are very mporan n pracce because of her parcular formulaon whch s deal for modelng wreless sensor neworks 6, 5. In our epermens we focus on a -dmensonal randomzed geomerc graph Gn, r whch s formed by placng n nodes unformly a random n a un square wh edges beween nodes whch are havng eucldean dsance less han he gven radus r. We se hs o be o be r rn logn/n s well know ha he connecvy s preserved n hs case 6. We se n 00. An llusraon of he wo graphs appears s n Fgure. a Cycle Graph: C0 b Random Geomerc Graph: Gn, r Fgure : Illusraon of he wo graph opologes we focus on n hs secon. Seup: In all epermens we generae a vecor wh of nal values c from a unform dsrbuon over 0,. We run several epermens and presen wo knds of fgures ha helps us o undersand how he algorhms evolve and verfy he heorecal resuls of he prevous secons. These fgures are:. The evoluon of he nal values of he nodes. In hese fgures we plo how he raecory of he values k of each node evolves hroughou eraons. The black doed horzonal lne represens he eac average consensus value whch all nodes should approach, and hus all oher lnes should approach hs level.. The evoluon of he relave error measure q def 0. We run each mehod for several parameers and for a pre-specfed number of eraons no necessarly he same for each epermen. In each fgure we have he relave error, boh n normal scale or logarhmc scale, on he vercal as and number of eraons on he horzonal as. To llusrae he frs concep, we provde a smple eample wh he evoluon of he nal values k for he case of he Sandard Gossp algorhm 5 n Fgure. The horzonal black doed lne represens he average consensus value. I s he eac average of he nal values c of he nodes n he nework. 5

16 a Cycle Graph b Random Geomerc Graph Fgure : Traecores of he values for he Sandard Gossp algorhm for Cycle Graph and a random geomerc graph. ach lne corresponds o for some. In he res of hs secon we evaluae he performance of he novel algorhms we propose, and conras wh he above Sandard Gossp algorhm, whch we refer o as Baselne n he followng fgures labels. 4. Prvae Gossp va Bnary Oracle In hs secon we evaluae he performance of Algorhm presened n Secon 3.. In he algorhm, he npu parameers are he posve sepszes {λ } 0. The goal of he epermens s o compare he performance of he proposed algorhm usng dfferen choces of λ. In parcular, we use decreasng sequences of sepszes λ / and λ /, and hree dfferen fed values for he sepszes λ λ {0.00, 0.0, 0.}. We also nclude he adapve choce λ 4m whch we have proven o converge wh lnear rae n Theorem 7. We compare hese choces n Fgures 4 and 6, along wh he Sandard Gossp algorhm for clear comparson. a λ λ 0.00 b λ λ 0.0 c λ λ 0. d λ k e λ k f λ Adapve Fgure 3: Traecores of he values of for Bnary Oracle run on he cycle graph. 6

17 a Lnear Scale b Logarhmc Scale Fgure 4: Convergence of he Bnary Oracle run on he cycle graph. In general, we clearly see wha s epeced wh he consan sepszes ha hey converge o a ceran neghbourhood and oscllae around opmum. Wh smaller sepsze, hs neghbourhood s more accurae, bu akes longer o reach. Wh decreasng sepszes, Theorem 6 suggess ha λ of order / should be opmal. Fgure 6 demonsraes hs, as he choce of λ / decreases he sepszes oo quckly. However, hs s no he case n Fgure 4 n whch we observe he oppose effec. Ths s due o he cycle graph beng small and smple, and hence he dmnshng sepsze becomes problem only afer relavely large number of eraons. Wh he adapve choce of sepszes, we recover lnear convergence rae as predced by Theorem 7. The resuls n Fgure 6 show one surprsng comparson. The adapve choce of sepszes does no seem o perform beer han λ /. However, we verfed ha when runnng for more eraons, he lnear rae of adapve sepsze s presen and converges sgnfcanly faser o hgher accuraces. We chose o presen he resuls for 6000 eraons snce we found overall more clean. a λ λ 0.00 b λ λ 0.0 c λ λ 0. d λ e λ f λ Adapve Fgure 5: Traecores of he values of for Bnary Oracle run on he random geomerc graph. 7

18 a Lnear Scale b Logarhmc Scale Fgure 6: Convergence of he Bnary Oracle run on he random geomerc graph. 4. Prvae Gossp va ɛ-gap Oracle In hs secon we evaluae he performance of he Algorhm 3 presened n Secon 3.. In he algorhm, he npu parameer s he posve error olerance varable ɛ. For epermenal evaluaon. we choose hree dfferen values for he npu, ɛ {0., 0.0, 0.00}, and agan use he same cycle and random geomerc graphs. The raecores of he values are presened n Fgures 7 and 9, respecvely. The performance of he algorhm n erms of he relave error s presened n Fgures 8 and 0. The performance s eacly machng he epecaon wh larger ɛ, he mehod converges very fas o a wde neghbourhood of he opmum. For a small value, converges much closer o he opmum, bu requres more eraons. a ɛ 0.00 b ɛ 0.0 c ɛ 0. Fgure 7: Traecores of he values of for ɛ-gap Oracle run on he cycle graph. a Lnear Scale b Logarhmc Scale Fgure 8: Convergence of he ɛ-gap Oracle run on he cycle graph. 8

19 a ɛ 0.00 b ɛ 0.0 c ɛ 0. Fgure 9: Traecores of he values of for ɛ-gap Oracle run on he random geomerc graph. a Lnear Scale b Logarhmc Scale Fgure 0: Convergence of he ɛ-gap Oracle run on he random geomerc graph. 4.3 Prvae Gossp va Conrolled Nose Inseron In hs secon we evaluae he performance of Algorhm 4 presened n Secon 3.3. Ths algorhm has wo dfferen parameers for each node. These are he nal varance σ 0 and he rae of decay, φ, of he nose. To evaluae he mpac of hese parameers, we perform several epermens. As earler, we use he same graph srucures for evaluaon: cycle graph and random geomerc graph. The algorhm converges wh a lnear rae dependng on mamum of wo facors see Theorem and Corollary. We wll verfy ha hs s ndeed he case, and for values of φ above a ceran hreshold, he convergence s drven by he rae a whch he nose decays. Ths s rue for boh dencal values of φ for all, and for varyng values as per 0. We furher demonsrae he laer s superor n he sense ha enables nseron of more nose, whou sacrfcng he convergence speed. Fnally, we sudy he effec of varous magnudes of he nose nsered nally Fed varance, dencal decay raes In hs par, we run Algorhm 4 wh σ for all, and se φ φ for all and some φ. We sudy he effec of varyng he value of φ on he convergence of he algorhm. In boh Fgures b and 4b, we see ha for small values of φ, we evenually recover he same rae of lnear convergence as he Sandard Gossp algorhm. If he value of φ s suffcenly close o however, he rae s drven by he nose and no by he convergence of he Sandard Gossp algorhm. Ths value s φ 0.98 for cycle graph, and φ for he random geomerc graph n he plos we presen. Lookng a he ndvdual runs for small values of φ n Fgure 4b, we see some varance n erms of when he asympoc rae s realzed. We would lke o pon ou ha hs does no provde addonal nsgh no wheher specfc small values of φ are n general beer for he followng reason. The Sandard Gossp algorhm s self a randomzed algorhm, wh an nheren uncerany n he 9

20 convergence of any parcular run. If we ran he algorhms mulple mes, we observe varance n he evoluon of he subopmaly of smlar magnude, us as wha we see n he fgure. Hence, he varance s epeced, and no sgnfcanly nfluenced by he nose. a φ 0.00 b φ 0.0 c φ 0. d φ 0.5 e φ 0.9 f φ 0.98 Fgure : Traecores of he values of dfferen values of φ. for Conrolled Nose Inseron run on he cycle graph for a Lnear Scale b Logarhmc Scale Fgure : Convergence of he Conrolled Nose Inseron run on he cycle graph for dfferen values of φ. 0

21 a φ 0.00 b φ 0.0 c φ 0. d φ 0.5 e φ 0.9 f φ Fgure 3: Traecores of he values of for Conrolled Nose Inseron run on he random geomerc graph for dfferen values of φ. a Lnear Scale b Logarhmc Scale Fgure 4: Convergence of he Conrolled Nose Inseron run on he random geomerc graph for dfferen values of φ Varance and dfferen decay raes In hs secon, we perform smlar epermen as above, bu le he values φ be vary for dfferen nodes. Ths s conrolled by he choce of γ as n 0. Noe ha by decreasng γ, we ncrease φ, and hus smaller γ means he nose decays a a slower rae. Here, due o he regular srucure of he cycle graph, we presen only resuls for he random geomerc graph. I s no sraghforward o compare hs seng wh he seng of dencal φ, and we reurn o n he ne secon. Here we only remark ha we agan see he esence of a hreshold predced by heory, beyond whch he convergence s domnaed by he nsered nose. Oherwse, we recover he rae of he Sandard Gossp algorhm.

22 a γ 0. b γ 0. c γ 0.3 d γ 0.5 e γ f γ Fgure 5: Traecores of he values of for Conrolled Nose Inseron run on he random geomerc graph for dfferen values of φ, conrolled by γ. a Normal Scale b Logarhmc Scale Fgure 6: Convergence of he Conrolled Nose Inseron run on he random geomerc graph for dfferen values of φ, conrolled by γ Impac of varyng φ In hs epermen, we demonsrae he praccal uly of leng he rae of decay φ o be dfferen on each node. In order o do so, we run epermen on he random geomerc graph and compare he sengs nvesgaed n he prevous wo secons he nose decay rae drven by φ, or by γ. In frs place, we choose he values of φ such ha ha he wo facors n Corollary are equal. For he parcular graph we used, hs corresponds o γ 0.7 wh φ αg d. Second, we make he facors equal, bu wh consran of havng φ o be equal for all. Ths corresponds o φ for all. The performance for a large number of eraons s dsplayed n lef sde of Fgure 7. We see ha heabove wo choces ndeed yeld very smlar praccal performance, whch also evenually maches he rae predced by heory. For complee comparson, we also nclude performance of he Sandard Gossp algorhm. The mporan message s conveyed n he hsogram n he rgh sde of Fgure 7. The hsogram shows he dsrbuon of he values of φ for dfferen nodes. The mnmum of hese values s wha we needed n he case of dencal φ for all. However, mos of he values are sgnfcanly hgher.

23 Ths means, ha f we allow he nose decay raes o depend on he number of neghbours, we are able o ncrease he amoun of nose nsered, whou sacrfcng praccal performance. Ths s benefcal, as more nose wll lkely be benefcal for any formal noon of proecon of he nal values. Fgure 7: Lef: Performance of he nose oracle wh nose decrease rae chosen accordng o Corollary. Rgh: Hsogram of of dsrbuon of φ 5 Concluson In hs work we addressed he Average Consensus problem va novel asynchronous randomzed gossp algorhms. We propose algorhmc ools for proecon of he prvae values each node n he nework holds nally. However, we do no quanfy any formal noon of prvacy proecon achevable usng hese ools; hs s lef for fuure research. In parcular, we propose wo ways o acheve hs goal. Frs, whch we beleve s he frs of s knd, weakens he oracle used n he gossp framework, o provde only caegorcal or even bnary nformaon o each parcpang node abou he value of he oher node. In he second approach, we sysemacally nec and whdraw nose hroughou he eraons, so as o ensure convergence o he average consensus value. In all cases, we provde eplc convergence raes and evaluae praccal convergence on common smulaed nework opologes. References Zeyuan Allen-Zhu, Zheng Qu, Peer Rchárk, and Yang Yuan. ven faser acceleraed coordnae descen usng non-unform samplng. In Proceedngs of The 33rd Inernaonal Conference on Machne Learnng, pages 0 9, 06. Tuncer Can Aysal, Mehme rcan Yldz, Anand D Sarwae, and Anna Scaglone. Broadcas gossp algorhms for consensus. I Transacons on Sgnal processng, 577:748 76, Florence Bénéz, Aleandros G Dmaks, Parck Thran, and Marn Veerl. Order-opmal consensus hrough randomzed pah averagng. I Transacons on Informaon Theory, 560: , Dmr P Bersekas and John N Tsskls. Parallel and dsrbued compuaon: numercal mehods, volume 3. Prence hall nglewood Clffs, NJ, Sephen Boyd, Arpa Ghosh, Bala Prabhakar, and Devavra Shah. Randomzed gossp algorhms. I Transacons on Informaon Theory, 4SI: , Mng Cao, Danel A Spelman, and dmund M Yeh. Acceleraed gossp algorhms for dsrbued compuaon. In Proceedngs of he 44h Annual Alleron Conference on Communcaon, Conrol, and Compuaon, pages ,

24 7 Anonn Chambolle, Mahas J. hrhard, Peer Rchárk, and Carola-Bbane Schoenleb. Sochasc prmal-dual hybrd graden algorhm wh arbrary samplng and magng applcaons. arxv: , Ronald Cramer, Ivan Berre Damgård, and Jesper Buus Nelsen. Secure Mulpary Compuaon and Secre Sharng. Cambrdge Unversy Press, Domnk Csba, Zheng Qu, and Peer Rchárk. Sochasc Dual Coordnae Ascen wh Adapve Probables. ICML 05, George Cybenko. Dynamc load balancng for dsrbued memory mulprocessors. Journal of Parallel and Dsrbued Compung, 7:79 30, 989. Aleandros DG Dmaks, Anand D Sarwae, and Marn J Wanwrgh. Geographc gossp: ffcen averagng for sensor neworks. I Transacons on Sgnal Processng, 563:05 6, 008. Aleandros G Dmaks, Soummya Kar, José MF Moura, Mchael G Rabba, and Anna Scaglone. Gossp algorhms for dsrbued sgnal processng. Proceedngs of he I, 98: , Cynha Dwork, Aaron Roh, e al. The algorhmc foundaons of dfferenal prvacy. Foundaons and Trends R n Theorecal Compuer Scence, 93 4: 407, Olver Fercoq and Peer Rchárk. Smooh mnmzaon of nonsmooh funcons by parallel coordnae descen. arxv: , Olver Fercoq and Peer Rchárk. Acceleraed, parallel and promal coordnae descen. SIAM Journal on Opmzaon, 5:997 03, Mroslav Fedler. Algebrac connecvy of graphs. Czechoslovak mahemacal ournal, 3:98 305, Mauro Franceschell, Alessandro Gua, and Carla Seazu. Dsrbued averagng n sensor neworks based on broadcas gossp algorhms. I Sensors Journal, 3:808 87, 0. 8 Nkolaos M Frers and Anasasos Zouzas. Fas dsrbued smoohng of relave measuremens. In I 5s Annual Conference on Decson and Conrol CDC, pages I, 0. 9 Valero Fresch, manuele Laanz, and Alessandro Boglolo. Accelerang dsrbued averagng n sensor neworks: Randomzed gossp over vrual coordnaes. In Sensors Applcaons Symposum SAS, 06 I, pages 6. I, Valero Fresch, manuele Laanz, and Alessandro Boglolo. Fas dsrbued consensus hrough pah averagng on random walks. Wreless Pers Commun, do:0.007/s : 5, 07. Rober Mansel Gower, Donald Goldfarb, and Peer Rchárk. Sochasc block BFGS: squeezng more curvaure ou of daa. In Proceedngs of The 33rd Inernaonal Conference on Machne Learnng, pages , 06. Rober Mansel Gower and Peer Rchárk. Randomzed erave mehods for lnear sysems. SIAM Journal on Mar Analyss and Applcaons, 364: , Rober Mansel Gower and Peer Rchárk. Sochasc dual ascen for solvng lnear sysems. arxv: , Rober Mansel Gower and Peer Rchárk. Lnearly convergen randomzed erave mehods for compung he pseudonverse. arxv preprn arxv:6.0655, 06. 4

25 5 Rober Mansel Gower and Peer Rchárk. Randomzed quas-newon updaes are lnearly convergen mar nverson algorhms. arxv preprn arxv: , Pyush Gupa and Panganmala R Kumar. The capacy of wreless neworks. I Transacons on Informaon Theory, 46: , Fenghua He, A Sephen Morse, J Lu, and Shaoshua Mou. Perodc gosspng. IFAC Proceedngs Volumes, 44: , 0. 8 Zhenq Huang, Sayan Mra, and Ger Dullerud. Dfferenally prvae erave synchronous consensus. In Proceedngs of he 0 ACM workshop on Prvacy n he elecronc socey, pages ACM, 0. 9 Ye Yu Jun and Mchael Rabba. Performance comparson of randomzed gossp, broadcas gossp and collecon ree proocol for dsrbued averagng. In I 5h Inernaonal Workshop on Compuaonal Advances n Mul-Sensor Adapve Processng CAMSAP, pages I, Davd Kempe, Aln Dobra, and Johannes Gehrke. Gossp-based compuaon of aggregae nformaon. In Foundaons of Compuer Scence, 003. Proceedngs. 44h Annual I Symposum on, pages I, Yn Ta Lee and Aaron Sdford. ffcen acceleraed coordnae descen mehods and faser algorhms for solvng lnear sysems. In Foundaons of Compuer Scence FOCS, 03 I 54h Annual Symposum on, pages I, Denns Levenhal and Adran S. Lews. Randomzed mehods for lnear consrans: Convergence raes and condonng. Mahemacs of Operaons Research, 353:64 654, J Lu, Bran D.O. Anderson, Mng Cao, and A Sephen Morse. Analyss of acceleraed gossp algorhms. Auomaca, 494: , J Lu, Shaoshua Mou, A Sephen Morse, Bran DO Anderson, and Changbn Yu. Deermnsc gosspng. Proceedngs of he I, 999:505 54, Ncolas Lozou and Peer Rchárk. A new perspecve on randomzed gossp algorhms. In 4h I Global Conference on Sgnal and Informaon Processng GlobalSIP, Ncolaos Manara and Chrsoforos N Hadcoss. Prvacy-preservng asympoc average consensus. In Conrol Conference CC, 03 uropean, pages I, Yln Mo and Rchard M Murray. Prvacy preservng average consensus. I Transacons on Auomac Conrol, 6: , Shaoshua Mou, Changbn Yu, Bran DO Anderson, and A Sephen Morse. Deermnsc gosspng wh a perodc proocol. In Decson and Conrol CDC, 00 49h I Conference on, pages I, Yur Neserov. ffcency of coordnae descen mehods on huge-scale opmzaon problems. SIAM Journal on Opmzaon, :34 36, Reza Olfa-Saber, J Ale Fa, and Rchard M Murray. Consensus and cooperaon n neworked mul-agen sysems. Proceedngs of he I, 95:5 33, Mahew Penrose. Random geomerc graphs. Number 5. Oford Unversy Press, Zheng Qu and Rchárk. Coordnae descen wh arbrary samplng I: algorhms and compley. Opmzaon Mehods and Sofware, 35:89 857, 06. 5

26 43 Zheng Qu and Rchárk. Coordnae descen wh arbrary samplng II: epeced separable overappromaon. Opmzaon Mehods and Sofware, 35: , Zheng Qu, Peer Rchárk, and Tong Zhang. Quarz: Randomzed dual coordnae ascen wh arbrary samplng. In Advances n Neural Informaon Processng Sysems 8, We Ren, Randal W Beard, and lla M Akns. Informaon consensus n mulvehcle cooperave conrol. I Conrol Sysems, 7:7 8, Peer Rchárk and Marn Takáč. Ieraon compley of randomzed block-coordnae descen mehods for mnmzng a compose funcon. Mahemacal Programmng, 44: 38, Peer Rchárk and Marn Takáč. Parallel coordnae descen mehods for bg daa opmzaon. Mahemacal Programmng, 56: , Peer Rchárk and Marn Takáč. Sochasc reformulaons of lnear sysems: algorhms and convergence heory. arxv preprn arxv: , Devavra Shah. Gossp algorhms. Foundaons and Trends R n Neworkng, 3: 5, Sha Shalev-Shwarz and Tong Zhang. Sochasc Dual Coordnae Ascen Mehods for Regularzed Loss Mnmzaon. Journal of Machne Learnng Research, 4: , 0. 5 John Tsskls, Dmr Bersekas, and Mchael Ahans. Dsrbued asynchronous deermnsc and sochasc graden opmzaon algorhms. I ransacons on auomac conrol, 39:803 8, John Nkolas Tsskls. Problems n decenralzed decson makng and compuaon. Techncal repor, DTIC Documen, Shaochuan Wu and Mchael G Rabba. Broadcas gossp algorhms for consensus on srongly conneced dgraphs. I Transacons on Sgnal Processng, 66: , Ln Xao and Sephen Boyd. Fas lnear eraons for dsrbued averagng. Sysems & Conrol Leers, 53:65 78, Ln Xao, Sephen Boyd, and Sanay Lall. A scheme for robus dsrbued sensor fuson based on average consensus. In Proceedngs of he 4h nernaonal symposum on Informaon processng n sensor neworks, page 9. I Press, Changbn Brad Yu, Bran DO Anderson, Shaoshua Mou, J Lu, Fenghua He, and A Sephen Morse. Dsrbued averagng usng perodc gosspng. I Transacons on Auomac Conrol, 07. 6