Communication-Efficient Algorithms For Distributed Optimization

Transcription

1 Communication-Efficient Agorithms For Distributed Optimization Submitted in partia fufiment of the requirements for for the degree of Doctor of Phiosophy in Eectrica and Computer Engineering João F. C. Mota M. S., Eectrica and Computer Engineering, Instituto Superior Técnico, Portuga B. S., Eectrica and Computer Engineering, Instituto Superior Técnico, Portuga Carnegie Meon University Pittsburgh, PA October 2013

2 ii Doctora Dissertation Committee: Professor Pedro Aguiar (Advisor), Instituto Superior Técnico, Technica University of Lisbon Professor José M. F. Moura, Carnegie Meon University Professor Markus Püsche (Advisor), ETH Zurich & Carnegie Meon University Professor Aejandro Ribeiro, University of Pennsyvania Professor João Xavier (Advisor), Instituto Superior Técnico, Technica University of Lisbon

3 ... to my parents, Luís and Heena.

4 iv

5 Abstract This thesis is concerned with the design of distributed agorithms for soving optimization probems. The particuar scenario we consider is a network with P compute nodes, where each node p has excusive access to a cost function f p. We design agorithms in which a the nodes cooperate to find the minimum of the sum of a the cost functions, f f P. Severa probems in signa processing, contro, and machine earning can be posed as such optimization probems. Given that communication is often the most energy-consuming operation in networks and, many times, aso the sowest one, it is important to design distributed agorithms with ow communication requirements, that is, communication-efficient agorithms. The two main contributions of this thesis are a cassification scheme for distributed optimization probems of the kind expained above and a set of corresponding communication-efficient agorithms. The cass of optimization probems we consider is quite genera, since we aow that each function may depend on arbitrary components of the optimization variabe, and not necessariy on a of them. In doing so, we go beyond the commony used assumption in distributed optimization and create additiona structure that can be expored to reduce the tota number of communications. This structure is captured by our cassification scheme, which identifies particuar instances of the probem that are easier to sove. One exampe is the standard distributed optimization probem, in which a the functions depend on a the components of the variabe. A our agorithms are distributed in the sense that no centra node coordinates the network, a the communications occur excusivey between neighboring nodes, and the data associated with each node is aways processed ocay. We show severa appications of our agorithms, incuding average consensus, support vector machines, network fows, and severa distributed scenarios for compressed sensing. We aso propose a new framework for distributed mode predictive contro, which can be soved with our agorithms. Through extensive numerica experiments, we show that our agorithms outperform prior distributed agorithms in terms of communication-efficiency, even some that were specificay designed for a particuar appication. v

6 vi

7 Acknowedgments This thesis is the resut of a compicated sequence of events. I woud ike to take the opportunity to thank here some of the peope who, directy or indirecty, infuenced, changed, or caused those events. The direct causers of the main events were, undoubtedy, my advisors: João Xavier, Pedro Aguiar, and Markus Püsche. The three of them gave me the support, the insight, and the knowedge that made this thesis possibe. I earned a ot from them, both academicay and non-academicay. Most importanty, no matter how busy they were, they coud aways find time to answer my questions, to take care of bureaucracy that invoved me, and to meet in our reguar meetings. Aso, I want to say that I had ots of fun in those yeary (work!) trips to severa towns in Portuga. Thank you for a of that! I woud ike to thank my thesis committee members, José Moura and Aejandro Ribeiro, for a the insight and suggestions. During my PhD, and especiay during the years I spent in CMU, José was aways very supportive. On the few occasions that we discussed research, José showed me how to ook at my research from a different perspective. I woud aso ike to thank Aejandro for arranging everything when I visited him in Phiadephia. The person who convinced me to enter the CMU/Portuga PhD program was João Pauo Costeira. He has aways been in the background, doing whatever is needed to make this program great, and providing a comfortabe ayer between a the bureaucracy that ies under such a big program and the students (incuding mysef). He has aso put me in contact with peope and projects from the rea word! Another eary causer of the events that ed to this thesis was Victor Barroso, who invited me to participate in research meetings at ISR, and subsequenty introduced me to 2/3 of my future advisors. During my PhD, I had the opportunity to coaborate and to discuss research with severa peope. I woud ike to thank them for that. Some of these peope are Michae Rabbat, João Miranda Lemos, Gabriea Hug, André Martins, Mário Figueiredo, Petros Boufounos, Qing Ling, Ricardo Lima, Bruno Sinopoi, Stephen Boyd, Soummya Kar, Aurora Schmidt, Pedro Guerreiro, Ricardo Cabra, Christian Conte, Stefan Richter, Pau Gouart, Christian Berger, Jerónimo Rodrigues, vii

8 Caudia Soares, Brian Swenson, Dusan Jakovetić, Dragana Bajovic, Sabina Zejniovic, Pinar Oguz, Dario Figueira, June Zhang, Qixing Liu, Matthias Athoff, Aysson Bessani, Pauo Oiveira, Bruce Krogh, Marija Iić, Susana Brandão, Nichoas O Donoughue, Nikos Arechiga, Kyri Baker, Aiaksei Sandryhaia, Marek Tegarsky, Augusto Santos, Bernardo Pires, Ceyhun Eksin, Divyanshu Vats, Akshay Rajans, Ehsan Zamanizadeh, Jhi-Young Joo, Sanja Cvijic, Luís Brandão, Franz Franchetti, Xiahui Wang (Eeyore), Luca Paroini, Rohan Chabukswar, Joe Harey, Rodrigo Beo, Joya Deri, Jim Weimer, and Sérgio Pequito. Aso, thank you to Danie McFarin and Vas Cheappa for heping me out with severa issues with GNU/Linux and MPI. Some of the experiments shown in this thesis were run on a computer custer kindy provided by Forin Manoache. Conducting research whie jumping back and forth over a arge ocean is not possibe without proper funding and an exceent team taking care of things. So I woud ike to thank the CMU/Portuga program and Fundação para a Ciência e Tecnoogia (FCT) for the grant SFRH/BD/33520/2008, provided through the Carnegie Meon/Portuga Program and managed by the Information and Communication Technoogies Institute (ICTI). Some work was partiay funded by the FCT grants CMU-PT/SIA/0026/2009 and PEst-OE/EEI/LA0009/2009. I am aso gratefu to a the staff invoved at CMU, IST, and the CMU/Portuga program, especiay to Ana Mateus, Caroyn Patterson, Susana Santana, Aexandra Araújo, Ana Santos, Fiomena Viegas, Lori Spears, Caire Bauere, Tara Moe, Eaine Lawrence, and Samantha Godstein. No singe piece of this thesis woud be possibe without the eary support of both of my parents, Luís and Heena, who aways encouraged me no matter what direction I chose. Their support has been a constant throughout my entire education and, for this and other reasons, the minimum I can give back is to dedicate this thesis to them. My sister, Renata, has aso aways provided constant encouragement, kept me in a good mood, and was a source of inspiration. My unce Manue and aunt Fátima, and my unce Aexandre and aunt Lii, and cousin Afonso have aways encouraged me in my studies and instied in me an interest in science from an eary age. I woud aso ike to thank the famiy friends Lourdes and Francisco Ceestino for their support and friendship. Finay, I have no words to describe my gratitude to my wife, Kate. Thank you for a your support, kindness, and ove. Thank you aso for making sure that, during the writing of this thesis, I had proper nutrition, rest, and aso fun; thanks for proofreading some parts of the thesis aso. Now I promise that I do my homework for our piano essons. viii

9 Contents Abstract Acknowedgments v vii 1 Introduction Overview Goas of the thesis A cassification scheme for distributed optimization Communication network Variabe cassification Contributions Organization Background and Reated Work Buiding bocks: non-distributed, parae agorithms Decomposition methods Bock-coordinate minimization methods Augmented Lagrangian methods Distributed agorithms Goba cass Star-shaped cass Mixed cass Goba Cass Probem statement Appications Inference probems Sparse soutions of inear systems ix

10 3.3 Agorithm derivation Experimenta resuts Average consensus Row partition: BP and BPDN Coumn partition: reversed asso SVM Connected and Non-Connected Casses Probem statement Appications Distributed mode predictive contro Reversed asso with a row partition Network utiity maximization Network fow probems State estimation in the power grid Agorithm derivation Connected variabe Non-connected variabe Experimenta resuts Network fow probems D-MPC Concusions and Future Work Major contributions Current imitations Future work A ADMM-based Agorithms For The Goba Cass: Derivation 115 A.1 Network identities A.2 Derivation of Agorithm A.3 Derivation of Agorithm B Some Conjugate Functions 123 C ADMM-based Agorithm For The Connected Cass: Derivation 125 Bibiography 129 x

11 List of Tabes 3.1 Network modes Network parameters, average degree, and number of coors Networks used in the D-MPC experiments xi

12 xii

13 List of Figures 1.1 Iustration of the main probem of the thesis The two steps of a distributed agorithm Two instances of (P) for a variabe with 3 components Iustration of how the agorithms operate according to the cooring of the network Comparison of the performance of a proposed agorithm with prior agorithms for the average consensus probem Exampe of a generic connected variabe and a mixed connected variabe Our cassification scheme for the variabe of probem (P) Row partition and coumn partition of A into P bocks Construction of a directed graph from the cooring scheme of an undirected graph Comparison of severa agorithms for the average consensus probem in a geometric network with P = 2000 nodes Resuts for the average consensus probem for a the networks of Tabe Resuts of the simuations for BP, reversed asso, BPDN, and SVM Two D-MPC scenarios. Soid ines represent inks in the communication network and dotted arrows represent system interactions Exampe of a geometrica pattern used in formation for minimizing the effect of drag forces or for escorting a moving object. Soid ines indicate direct communication, whie dashed ines indicate dynamic couping, but not necessariy direct communication (a) Exampe network with 3 source nodes s 1, s 2, and s 3, that use predetermined routes to send packets to three recipient nodes r 1, r 2, and r 3 ; (b) Bipartite graph obtained from (a): each ink from (a) with a capacity associated is represented as a circuar node in (b) xiii

14 4.4 Exampe of a network fow probem. Each edge has associated both a variabe x ij and function of that variabe, φ ij (x ij ). The goa is to the sum of a the functions, whie satisfying conservation of fow constraints Iustration of five connected areas in a power network Exampe of an optima Steiner tree Resuts of our experiments for the network fow probems Resuts for D-MPC with a connected variabe Resuts for D-MPC with a non-connected variabe xiv

15 Chapter 1 Introduction Optimization theory has contributed to many fieds in engineering by providing efficient agorithms that sove nontrivia rea word probems. Notabe exampes can be found in signa processing, contro engineering, and machine earning [1, 2, 3]. On the other hand, over the ast years, some computation patforms on which these agorithms may be executed have become distributed. For exampe, computers are now equipped with severa processing devices, aowing for parae computation. Aso, compex systems such as power grids or water distribution systems are composed of severa interconnected components, each with some processing power, and thus are distributed by nature. In addition, the data to be processed is often generated at different ocations as, for exampe, in sensor networks, or in the internet. A these factors ask for agorithms that process data or contro systems in a distributed way. However, it is chaenging to design optimization agorithms that are matched to distributed resources. Part of the reason is that efficient centraized agorithms, such as for exampe interior-point methods, cannot be easiy adapted to distributed scenarios. The high-eve goa of this thesis is to advance the design of distributed agorithms for soving optimization probems. 1.1 Overview Figure 1.1 wi hep us describe the main probem addressed by this thesis. The figure shows a network with 10 nodes, where each node p hods a function f p. Our goa is to make a nodes cooperate in order to find a r of the sum of a the functions: x R n f 1 (x S1 ) + f 2 (x S2 ) + + f P (x SP ). (P) 1

16 2 1. Introduction f 1 1 f f f 7 f 2 f f 4 f f 5 9 f 9 Figure 1.1: Iustration of the main probem of the thesis: each node in the network hods a private function and the goa is to the sum of a the functions. where x R n is the optimization variabe. Each function f p in (P) depends on the components of the variabe x that are indexed by the set S p {1,..., n}, and we use x Sp to denote those components. For exampe, if the function at node 3 depends on components x 1, x 5, x 8, and x 10, then S 3 = {1, 5, 8, 10} and f 3 (x S3 ) = f 3 (x 1, x 5, x 8, x 10 ). We require each function f p to be private to node p, i.e., no other node in the network has access to it. The edges of the network represent communication inks; this means, for exampe, that node 3 in Figure 1.1 can communicate ony with its neighbors: nodes 2, 4, and 8. Given such a network, an agorithm that soves (P) is considered distributed if it uses no centra node, no a-to-a communications, and if the privacy requirement for each function f p is satisfied. In this thesis, we aim to sove (P), and reated probems, with distributed agorithms that are communication-efficient, i.e., that use a minima amount of communication. Communication-efficiency is an essentia requirement, for exampe, when the nodes are battery-operated devices, such as in sensor-networks, since communication is usuay very energy-demanding. Simpe exampe. Consider an inference probem on a sensor network [4, 5], and suppose that each function f p depends on a the components of the optimization variabe x R n, i.e., S p = {1,..., n}, for a p or, more compacty, P p=1 S p = {1,..., n}. Whie each node in the network represents a sensor with computing abiities, each edge indicates direct sensor communication, for instance, through a wireess connection. We want to estimate a parameter θ R n (e.g., a set of environmenta parameters [6]), by using noisy measurements from a nodes. Let θ p be the measurement of θ taken at node p. Assuming the noise is independent across nodes, finding the maximum og-ikeihood estimate of θ can be written as (P) with P p=1 S p = {1,..., n}. For exampe, if the noise is Gaussian with zero mean and its covariance is the identity matrix, each f p (x) is given by (1/2) x θ p 2, and the resuting probem is known as the average consensus probem [7]. In this case, the soution to (P) is simpy x = (1/P) P p=1 θ p, that is, the maximum og-ikeihood

17 1.1. Overview 3 (a) Computation (b) Communication Figure 1.2: The two steps of a distributed agorithm. The nodes iterativey perform (a) computations and (b) broadcast the resuts of those computation to their neighbors. estimation of θ is the average of a the measurements. However, in our distributed scenario, node p is the ony node who knows θ p, and this makes computing the above average chaenging. This simpe exampe shows that to compute a soution of (P) the nodes have to communicate, either by exchanging their private data or by exchanging their estimates of the probem s soution. What they exchange and how they do it is determined by the distributed agorithm they use. Distributed agorithms. A distributed agorithm computes a soution x of (P) whie satisfying the requirement that each function f p remains private to node p. Typicay, each iteration of a distributed agorithm consists of the two steps shown in Figure 1.2: (a) a computation step, and (b) a communication step. In the computation step, a nodes update their estimates of the components of x. Usuay, each node p updates its estimates by combining information given by its private function f p with information given by the estimates of its neighbors from the prior communication step. A these estimates are then exchanged in the subsequent communication step. Athough a nodes in Figure 1.2 are performing each of the two steps in parae, this is not required for a distributed agorithm. Actuay, as we wi see, in environments such as wireess networks it might be impossibe to perform the communication step (b) in parae, because of packet coisions. In the average consensus exampe given above, a popuar choice for the computation step (a) is to ineary combine the estimate of node p with the estimates of its neighbors N p. That is, the estimate of node p, x p, is updated as x k+1 p = a pp x k p + j N p a pj x k j, (1.1) where each a pj is a positive number, a pp + j N p a pj = 1, and k denotes the iteration number. The computation scheme (1.1) impies that the nodes exchange their estimates x k p at each communication

18 4 1. Introduction f 1 (x 1, x 2, x 3 ) 1 f 6 (x 1, x 2, x 3 ) 6 f 5 (x 1, x 2, x 3 ) 2 5 f 2 (x 1, x 2, x 3 ) f 1 (x 1, x 2 ) 1 f 6 (x 2 ) 6 2 f 2 (x 2, x 3 ) f 5 (x 1, x 2 ) 5 3 f 3 (x 1, x 2, x 3 ) 4 f 4 (x 1, x 2, x 3 ) 3 f 3 (x 1, x 2, x 3 ) 4 f 4 (x 1, x 3 ) (a) Goba variabe (b) Non-connected variabe Figure 1.3: Two instances of (P) for a variabe with 3 components, x = (x 1, x 2, x 3 ). In (a), the variabe is goba (and thus connected) because a the functions depend on a the components. In (b), the variabe is non-connected because x 1 induces a subgraph that is not connected. step (Figure 1.2(b)). This famiy of agorithms for the average consensus probem has been widey studied in the iterature [7, 8, 9, 10, 11, 12]. In this thesis, we propose agorithms that sove not ony the average consensus probem, but the entire cass (P). We wi see that this cass contains severa other probems that are reevant in signa processing, contro theory, machine earning, and other areas. Soving (P) in fu generaity, however, is chaenging because the sets S p are arbitrary. Our approach consists of identifying particuar cases of (P) that are easier to sove, designing agorithms for those cases, and then generaizing them to the most difficut cases. To do that, we introduce a scheme to cassify instances of (P), as overviewed next. The outcome of our approach wi be an agorithm soving (P) in fu generaity. Despite its generaity, our agorithm achieves performances better than prior distributed agorithms, even incuding some that were designed for a particuar appication. Cassification scheme. The most popuar instance of (P) is iustrated in Figure 1.3(a): each function depends on a the components of the variabe, P p=1 S p = {1,..., n}. Rewriting (P) for this case, we have x f 1 (x) + f 2 (x) + + f P (x), which is the instance of (P) for which most distributed agorithms have been designed. In our cassification scheme, formay introduced ater in Section 1.3 and visuaized in Figure 1.7, we say that probem (G) has a goba variabe. Athough many appications can be written as (G), many others are instances of (P) with a non-goba variabe. In fact, our main motivation for considering the generic probem (P) stems from its abiity to mode probems where each node is interested ony in a subset of the probem s parameters or variabes, rather than in a of them. This is typica in arge-scae systems, for exampe, in arge pants, in the power grid, and in the internet. (G)

19 1.1. Overview 5 A fundamenta assumption we make is that if node p depends on components x Sp, then that node is interested in computing the optima vaue for those components ony, and not for any of the other components. For exampe, node 4 in Figure 1.3(b) depends on components x 1 and x 3, which means that it wi compute the optima vaue for these components, but not for x 2. The fexibiity introduced in (P) by the sets S p, however, produces instances that are difficut to sove, given the previous assumption. Figure 1.3(b) shows an exampe: the component x 1 appears in the functions of nodes 1, 3, 4, and 5, but not in the functions of nodes 2 and 6. This means that node 1 is isoated from a the other nodes that aso depend on x 1 ; indeed, nodes 2 and 6 are not interested in computing an optima vaue for x 1, et aone exchanging estimates of it. In other words, the subgraph of the nodes that depend on x 1 is not connected and, for this reason, we say that the variabe in this case is non-connected. Of course, computing an optima soution of (P) in this case wi invariaby require seecting one of the nodes 2 or 6 to retransmit estimates of x 1, so that a the nodes depending on this component can agree on an optima vaue for it. In the sma exampe of Figure 1.3(b), it is indifferent to seect either node 2 or node 6 for this task, but in arger networks, and for arbitrary sets S p, we shoud seect the nodes in such a way that the tota number of communications is d. Our soution for this probem invoves computing Steiner trees and is expained in Chapter 4. The concepts of goba variabe and non-connected variabe are concepts of the cassification scheme we introduce in this thesis. These concepts and the ones of connected, mixed, and starshaped variabe wi be formay defined in Section 1.3, but their reation can be visuaized in Figure 1.7. Roughy, the variabe of (P) is divided into two casses: connected and non-connected. These are, in fact, the most reevant casses in our cassification scheme for two reasons: they form a partition of the a the instances of the variabe of (P), and addressing them requires competey different techniques. These two casses thus comprise the first eve of our cassification scheme. The second eve consists of the foowing subcasses: goba, star-shaped, and mixed. These subcasses neither are mutuay disjoint nor do they cover a instances of the variabe of (P). However, they are reevant both because they are much simper instances of (P), and because they have been soved with severa distributed agorithms. Most of the agorithms that sove these subcasses, however, cannot be easiy generaized to sove the entire connected and non-connected casses. In this thesis, we propose an agorithm that soves (P) for a casses and subcasses of variabes. Overview of some appications. In this thesis we wi consider severa appications that arise in distributed contexts and that can be written as instances of (P). The recent fied of compressed sensing [13, 14] provides a rich coection of such probems: basis pursuit (BP) [15], basis pursuit denoising (BPDN) [15], and the east absoute shrinkage and seection operator (asso) [16], among others. These compressed sensing probems are convex and provide heuristics for finding sparse

20 6 1. Introduction soutions of inear systems. Athough finding the sparsest soution of a inear system is NP-hard, compressed sensing theory estabishes conditions under which the previous probems find an optima (i.e., sparsest) soution. There is an increasing interest in soving compressed sensing in distributed scenarios, where either the coumns or the rows of the matrix defining the inear system are spread over severa nodes. We reformuate the above compressed sensing probems as (P), some with a goba variabe and others with a mixed one; some of these reformuations are nove and are presented in this thesis for the first time. We wi see that training a support vector machine (SVM)[17, Ch.7] requires soving an optimization probem that can be easiy recast as (G). Roughy, given a database with two casses of datapoints, the goa in training an SVM is to find the hyperpane that best separates the two casses of datapoints. When the datapoints are distributed among severa sites, training an SVM arises naturay as a distributed optimization probem. Therefore, soving this probem with a distributed agorithm has the advantages of not requiring the transmission of the private databases to a remote ocation, and of providing more robustness (if one node fais, the remaining nodes can sti train the SVM, yet, with ess data). Many systems can be modeed as networked dynamica systems [18]. Specificay, each system is seen as the node of a network and has associated a state, a contro input, or both. The state of a given node is infuenced not ony by its own state and contro input (or simpy, input), but aso by the states and inputs of its neighbors. An effective contro strategy for this type of systems is distributed mode predictive contro (D-MPC) [19], which consists of the foowing. First, at each time instant, each node senses its own state; then, the nodes coectivey sove an optimization probem that finds the best set of contro inputs for a future time-horizon. These inputs are computed in such a way that their appication to the systems wi ead the nodes states to a given goa and, at the same time, they wi some energy function. Athough the nodes know an optima set of inputs for a the time instants in the time-horizon, they wi ony use the input for the next time instant. The reason is to mitigate the impact of modeing and sensing errors. So, in the next time instant, after appying the previousy computed input, each node senses its state and cooperates with the other nodes to sove the D-MPC optimization probem, now with new data. This procedure is repeated at each time instant. In this thesis, we provide a new framework for formuating D-MPC probems, and aso communication-efficient agorithms to sove them. We aso mention that severa network fow probems can be recast as (P) with a star-shaped variabe. These are optimization probems formuated on directed networks where physica items can fow through the edges of the network. As a consequence, certain conservation aws have to be satisfied and are typicay written as probem constraints. Network fow probems arise in severa contexts [20], for exampe, in determining best energy poicies in the power grid. After

21 1.1. Overview (a) Step 1 (b) Step 2 (c) Step 3 Figure 1.4: Iustration of how the agorithms operate according to the cooring of the network. The cooring scheme has three coors: nodes 1, 3, and 5 have coor 1, nodes 4 and 6 have coor 2, and node 2 has coor 3. some reformuations, network fow probems can be recast as (P) and, hence, can be soved with the agorithms we propose here. Overview of the proposed agorithms. Probem reformuation pays a key roe in the design of distributed optimization agorithms. In fact, we wi see throughout this thesis that it impacts significanty the fina agorithm. Our strategy for soving instances of (P), and utimatey (P) in fu generaity, consists of reformuating those instances into a format such that we-known centraized optimization agorithms become naturay distributed. Our reformuations make use of a concept that has rarey appeared in high-eve distributed agorithms, such as the ones considered in this thesis. That concept is network cooring, an assignment of coors to the nodes of a network such that no two neighboring nodes have the same coor (for convenience, instead of coors, we just use natura numbers). Assuming that a cooring scheme is avaiabe beforehand is reaistic in many distributed scenarios, especiay in wireess networks. For exampe, wireess networks require protocos known as media access contro (MAC) to avoid packet coisions, i.e., that one node receives two messages at the same time and in the same frequency (assuming there is ony one receive antenna). Some MAC protocos, such as time division mutipe access (TDMA), rey on network cooring. Figure 1.4 shows how the agorithms we propose work as a function of the cooring scheme. The network in this figure has three coors: nodes 1, 3, and 5 have coor 1, nodes 4 and 6 have coor 2, and node 2 has coor 3. The agorithms we propose are iterative, and each iteration is divided into a number of steps equa to the number of coors. Figure 1.4 thus has 3 subfigures, each one corresponding to a step. In each step, a the nodes with the same coor perform the same tasks in parae, as iustrated in subfigures 1.4(a), 1.4(b), and 1.4(c). These subfigures show the communication pattern occurring in each step. From an high-eve point of view, the tasks

22 8 1. Introduction performed by node p consist of: 1. finding new estimates for the components that f p depends on, by soving f p ( ) + quadratic term, that is, node p s the sum of f p and a quadratic term. That quadratic term depends on the network structure as we as on previous estimates of the neighbors of node p. Soving the above optimization probem corresponds to evauating the proximity operator of the function f p and, many times, this can be done in a simpe way. 2. sending the new estimates to the neighboring nodes. Finay, we note that the concept of network cooring required by our agorithms coincides with the concept of network cooring commony used in ow-eve communication protocos, namey, MAC protocos [21, Ch.6]. The goa of MAC protocos is to avoid packet coisions due to the hidden node and the exposed node probems [21, 6.2.2]. For exampe, in Figure 1.4(a), node 6 is receiving simutaneous messages from nodes 1 and 5. If the messages are in the same frequency and node 6 has one antenna ony, this resuts in a packet coision and the nodes have to retransmit their messages. Time division mutipe access (TDMA), for exampe, is a MAC protoco that avoids packet coisions by using a second-order cooring scheme: each node cannot have the same coor as its neighbors and as its neighbors neighbors. Such a cooring scheme works for our agorithms as we and, for this reason, the high-eve structure of our agorithms is not atered by ow-eve protocos when they are impemented in networks that use TDMA as a MAC protoco. The strategy we use to derive our distributed agorithms consists of reformuating the probems we want to sove in such a way that we can appy we known centraized optimization agorithms. Regarding our choice for these agorithms, we wi focus on the Aternating Direction Method of Mutipiers (ADMM), more specificay on an extended version of it: the muti-bock, or extended, ADMM [22]. ADMM was proposed in the seventies by [23, 24] to sove ineary constrained optimization probems, using a divide-and-conquer approach. In the eighties and nineties, ADMM was shadowed by the popuar interior point methods, which sove sma- and medium-sized probems very efficienty, but in a centraized way. Latey, ADMM has regained attention from the optimization community, because of its wide appicabiity and its abiity to dea with arge-scae and distributed scenarios. Notaby, ADMM has been appied to sove some of the probems addressed in this thesis, in particuar, instances of (P) with a goba variabe and with a star-shaped variabe. In spite of that, itte is sti known about its behavior. For instance, partia resuts on the convergence rate of ADMM, or a proof of the convergence of the muti-bock ADMM, were estabished ony very recenty.

23 1.2. Overview 9 Reative error [11] Proposed [26] [25] Communication steps Figure 1.5: Comparison of the performance of a proposed agorithm with prior agorithms for the average consensus probem. The network is a randomy generated geometric network with 2000 nodes. Exampe of performance resuts. Figure 1.5 shows as exampe the performance of the agorithm we propose for (G) when appied to the average consensus probem (scaar case, i.e., n = 1). The pot shows the reative error of the soution estimates versus the number of communication steps. We say that a communication step has occurred whenever a the nodes have updated their estimates and transmitted them to their neighbors. This is equivaent to saying that the number of communication steps is the tota number of communications divided by 2E, twice the number of edges. A communication, in this case, is defined as an edge usage. For exampe, in Figure 1.4, there are 6 communications in (a), 5 communications in (b), and 3 communications in (c). And one communication step in that figure is comprised of steps 1, 2, and 3. The number of communication steps, therefore, provides a direct measure of communication-efficiency. The network we used in the experiments of Figure 1.5 has P = 2000 nodes and was randomy generated as a geometric network with parameter og(p)/p The figure shows that, among a agorithms, the proposed one required the east amount of communications (i.e., communication steps) to achieve any error between 10 0 and The other agorithms in the figure are [25, 26], which sove the entire goba probem cass (G), and [11], which is considered the most efficient consensus agorithm [9], but it can ony sove the average consensus probem and not any other probem in the cass (G). Actuay, if we consider the convergence rate, i.e., the sopes of the error ines, the proposed agorithm and [11] have roughy the same performance. In fact, they have the same sope, but [11] exhibits an offset, since it requires a specia initiaization. A the other agorithms were initiaized with zeros. This experimenta resut reveas the surprising fact that, athough the agorithms we propose sove an entire probem cass, they can sometimes achieve the same performance as the best agorithms for a particuar appication. This is particuary surprising for the average consensus probem, since it is the simpest and the most thoroughy studied distributed probem.

24 10 1. Introduction 1.2 Goas of the thesis We next summarize the goas of the thesis and then we expain each requirement in detai. We aim to design, anayze, and impement agorithms that sove optimization probems of the form (P) on networks. The agorithms shoud be Distributed: no node has compete knowedge about the probem data and no centra node is aowed; aso, each node communicates ony with its neighbors; Communication-efficient: the number of communications they use is d; Network-independent: the agorithms run on networks with arbitrary topoogy and their output is independent of the network. Distributed. A distributed agorithm ony makes sense in an environment where both the data and the computing power are distributed. In such an environment, an agorithm is considered distributed if it fufis three requirements. First, oca data to a given node shoud remain private to that node. This enforces oca computations, since any computation invoving a piece of data has to be performed at the node where that data beongs to. In our probem (P), data of node p is encoded in the function f p and, thus, we require f p to be private to node p; this means that no other node has fu knowedge of f p at any time during and before the execution of the agorithm. The second requirement is that there shoud not exist any centra or specia node. Such a node woud coordinate a the other nodes and woud make the data at a given node reachabe to any other node in a very sma number of hops. Actuay, an agorithm that satisfies the first requirement but not the second one is usuay caed a parae agorithm [27]. Indeed, according to [27], parae agorithms run on systems where computing devices are at a sma distance of each other and may be controed by a centra entity. Distributed agorithms, in contrast, run on systems where computing devices are ocated far apart, making centraized coordination inconvenient; in the atter, there is aso itte contro on the network topoogy. Finay, the third requirement is that each node communicates ony with neighboring nodes. Athough this is equivaent to forbidding a centra node, we expicity state this requirement in order to excude patforms that aow a-to-a communications. For exampe, an agorithm running on a computer custer and using function cas from a message passing interface (MPI) [28] impementation, such as MPI_Bcast or MPI_Reduce, cannot be considered distributed; at most, it is parae. We mention that sometimes distributed agorithms are aso referred to as decentraized agorithms.

25 1.3. A cassification scheme for distributed optimization 11 Communication-efficient. In a centraized agorithm, the execution time and the cosey reated foating-point operation (FLOP) count are the most common performance metrics: the ower these metrics are, the more efficient an agorithm is. In distributed scenarios, however, other metrics arise. For exampe, computing accurate soutions is chaenging in scenarios where there is communication noise. In that case, sower agorithms that are noise-resiient may be preferabe to faster agorithms that are noise-sensitive. Another exampe is energy consumption. In many distributed scenarios, e.g., sensor networks, nodes rey on batteries and therefore have a imited source of energy. In these situations, increasing the ifespan of the network becomes the main priority. As communication in battery-operated devices is currenty the most energy-consuming operation [6, 29], this priority transates into having agorithms with ow communication requirements. The performance metric adopted in this thesis wi then be the number of communications: the ower the number of communications an agorithm uses, the more efficient that agorithm wi be. Hence, our goa wi be to design distributed optimization agorithms that use the fewest communications possibe. Network-independent. The ast requirement we impose on distributed agorithms is network independence. This simpy means that the output of the agorithm, i.e., the estimate of the soution returned by the agorithm, shoud be independent of the network topoogy. For instance, the agorithms shoud output the same soution estimate whether they are run on a densey or on a sparsey connected network. Naturay, the performance of the agorithms, i.e., the number of iterations or communications they use to compute that estimate, wi in genera vary with the network topoogy. 1.3 A cassification scheme for distributed optimization Our strategy for achieving the goas of this thesis is a divide-and-conquer one: first, we identify instances of (P) that are easier to sove; then, we combine the soutions we designed for the simper instances to sove (P) in fu generaity. For convenience, we reproduce (P) here: x R n f 1 (x S1 ) + f 2 (x S2 ) + + f P (x SP ). (P) In this section, we formay introduce our cassification scheme for the variabe of probem (P). Before doing that, however, we need the concept of communication network.

26 12 1. Introduction Communication network The communication network is the physica network through which the computing devices, seen as network nodes, communicate. We represent the communication network with an undirected graph G = (V, E), where V and E are the set of nodes and the set of edges, respectivey. The cardinaity of these sets, i.e., the number of nodes and the number of edges, wi be denoted with P = V and E = E, respectivey. Figure 1.1 shows an exampe of a graph representing a communication network with P = 10 nodes and E = 21 edges. An edge beongs to the communication network, say (i, j) E, if and ony if nodes i and j communicate directy. For exampe, nodes 1 and 10 in Figure 1.1 are neighbors: this means they can exchange messages with each other, because there is a communication ink connecting them. We use the foowing convention: if (i, j) E, then i < j. Throughout this thesis, we wi assume that the communication network G is connected and that its topoogy does not vary with time. Functions associated to nodes. Associated with each node, there is a function depending on the components of a variabe x R n. The function at node p is denoted with f p : R np R {+ }, where n p is the cardinaity of the set S p. As expained before, we use S p {1,..., n} to denote the components of x R n that function f p depends on. Of course, 1 n p = S p n. We assume that each node p is interested in computing the optima vaue ony of the components of x that are indexed by S p. We wi see that in some situations, however, it is difficut, or even impossibe, to sove instances of (P) without forcing some nodes to receive and transmit components of x that are not indexed by their sets S p. To make our probem we-defined, we assume that each component of the variabe appears in at east one of the nodes, that is, P p=1 S p = {1,..., n}. Uness otherwise stated, we that assume f p is cosed and convex [1, 3, 2, 30, 31], and not identicay +. Note that our definition for each f p aows it to take the vaue + ; as a consequence, node p can impose constraints on the variabe x impicity, via indicator functions. An indicator function of a given set S R n is defined as i S : R n R {+ }, 0, x S i S (x) = +, x S. Incuding an indicator function i S (x) in the objective of a minimization probem forces x S, since otherwise the optima (minima) vaue is +. Each function f p is private, i.e., at a times during and before the execution of the agorithm, ony node p knows f p. As expained before, this privacy rue formaizes our wish to derive a distributed agorithm by enforcing oca computations; namey, a computations invoving f p have to be done at node p. This makes sense in scenarios where each f p encodes a database that shoud be known ony at node p, or simpy to make use of a the

27 1.3. A cassification scheme for distributed optimization 13 f 1 (x 1, x 2 ) 1 f 6 (x 1, x 2 ) 6 f 5 (x 1, x 3 ) 2 5 f 2 (x 1, x 2, x 3 ) f 1 (x 1, x 2 ) 1 f 6 (x 1, x 2 ) 6 f 5 (x 1, x 3 ) 2 5 f 2 (x 1, x 2, x 3 ) f 1 (x 1, x 2 ) 1 f 6 (x 1, x 2 ) 6 f 5 (x 1, x 2 ) 2 5 f 2 (x 1, x 2, x 3 ) 3 f 3 (x 2, x 3 ) 4 f 4 (x 3 ) 3 f 3 (x 2, x 3 ) 4 f 4 (x 3 ) 3 f 3 (x 1, x 3 ) 4 f 4 (x 1 ) (a) Generic connected variabe (b) Subgraph induced by x 2 (c) Mixed connected variabe Figure 1.6: Exampe of (a) a generic connected variabe and (c) a mixed connected variabe. (b) highights the subgraph induced by the component x 2 in (a). The communication network is the same in a cases. In (c), the component x 1 is goba, and x 2 and x 3 induce connected subgraphs. distributed computing resources as, for exampe, in a sensor network, where each sensor has some processing power avaiabe for computation Variabe cassification Athough each function is uniquey associated to a singe node, the same does not happen for each component of x R n, the optimization variabe. This creates an additiona structure and motivates our cassification scheme. Essentia to our cassification scheme is the concept of induced subgraph. Induced subgraph. Let x R denote the th component of the optimization variabe x R n. We define the subgraph induced by x simiary to how [32, Ch.1] defines the subgraph induced by a set of nodes. In our case, these nodes are the ones whose functions depend on x. To be more concrete, given a communication network G, the subgraph induced by x is the subgraph G = (V, E ) G, where V is the set of nodes whose functions depend on x, and an edge (i, j) beongs to E ony if (i, j) E and both nodes i and j beong to V. As an exampe, Figure 1.6(b) highights the subgraph induced by the component x 2 in the setting of Figure 1.6(a): the set of nodes and the set of edges of this induced subgraph G 2 are, respectivey, V 2 = {1, 2, 3, 6} and E 2 = {(1, 2), (1, 6), (2, 3), (2, 6)}. Note that neither f 4 nor f 5 depend on x 2. Component-wise cassification of x. We cassify each component x according to its induced subgraph G the foowing way: x is connected if G is a connected subgraph, and is non-connected otherwise; goba if its induced subgraph coincides with the communication network, i.e., G = G; star-shaped if G is a star graph.

28 14 1. Introduction connected star-shaped goba non-connected mixed Figure 1.7: Our cassification scheme for the variabe x R n of probem (P). The variabe is either connected or non-connected. Goba and star-shaped variabes are particuar instances of a connected variabe, and a mixed variabe can be connected or non-connected. By star graph we mean a graph in which there exists a node who is a neighbor of a the other nodes; the remaining nodes can aso be neighbors between themseves. For exampe, the subgraph induced by variabe x 2 in Figure 1.6(b) is a star, because every node is a neighbor of node 2; therefore, x 2 is star-shaped. If a component is star-shaped, it can be handed in a centraized way, since the node in the center of the star can act as a centra node. It can be checked that component x 1 in Figure 1.6(a) is aso star-shaped, with node 6 in the center, but component x 3 is not. A the components of the variabe in that figure, however, are connected, since the respective subgraphs are connected. Naturay, a star-shaped variabe is aways connected. An exampe of a goba component is given in Figure 1.6(c): the subgraph induced by x 1 coincides with communication graph and, thus, x 1 is goba. In other words, a the functions in Figure 1.6(c) depend on x 1. Again, a goba component is aways connected, since its induced subgraph coincides with the communication network, which we assume connected. Uness the communication network is a star, a goba variabe is never starshaped. We had aready iustrated a non-connected component in Figure 1.3(b): the subgraph induced by x 1 in that network is not connected, and thus x 1 non-connected. Cassification of x. With the component-wise cassification of components, we are now in conditions to cassify the fu optimization variabe x R n in (P). The proposed cassification scheme is shown in Figure 1.7. There, the variabe x is either connected if a the components x of x R n are connected, for = 1,..., n; or non-connected if x has at east one non-connected component. For exampe, whie the variabe in Figure 1.6(a) is connected, because x 1, x 2, and x 3 are connected, the variabe in Figure 1.3(b) is non-connected, because x 1 is non-connected (in spite of x 2 and x 3 being connected). Note that the connected and non-connected casses partition the entire cass of the variabe x (see Figure 1.7). This distinction between a connected and a non-connected variabe is the most important one in our cassification scheme. In fact, we wi see in Chapter 4 that they have to be addressed with different techniques.

29 1.3. A cassification scheme for distributed optimization 15 To the best of our knowedge, no agorithm has ever been designed (purposefuy) to sove (P) with a generic connected or non-connected variabe. However, there are agorithms soving it with goba, mixed (connected), and star-shaped variabes, defined as foows. The variabe x R n is goba if a its components are goba: P p=1 S p = {1,..., n}; mixed if it has at east one goba component and at east one non-goba component: P p=1 S p { }, {1,..., n} ; star-shaped if a its components are star-shaped. We have aready seen an exampe of a goba variabe in Figure 1.3(a). When (P) has a goba variabe, it can be written simpy as x f 1 (x) + f 2 (x) + + f P (x). (G) Because of the assumption that the communication network is aways connected, a goba variabe is aso aways connected. A mixed variabe, in turn, can be either connected or non-connected. It is connected if a the non-goba components are connected, and non-connected if at east one of the non-goba components is non-connected. Figure 1.6(c) shows an exampe of a mixed variabe that is connected, since the non-goba components x 2 and x 3 are connected. Probem (P) with a mixed variabe can be written as x=(y,z) R n f 1 (y, z S1 ) + f 2 (y, z S2 ) + + f P (y, z SP ), (M) where the variabe x was decomposed into its goba components y and into its non-goba components z. Finay, a the components of a star-shaped variabe are ike x 2 in Figure 1.6(b). Note that in Figure 1.7 the star-shaped cass intersects with the goba cass; this happens when the variabe is goba and the communication network is a star. Summarizing, our cassification scheme partitions the variabe of probem (P) into two casses, shown as rectanges of Figure 1.7: connected and non-connected. These casses are the most fundamenta ones, since they require different soution methods. The subcasses shown as eipsoids in Figure 1.7 identify easier instances of (P). Aso, each one of these subcasses has been addressed with prior distributed optimization agorithms. In reaity, whie severa agorithms have been proposed for the goba and the star-shaped subcasses, we ony found one distributed agorithm, in [33], soving an instance of (P) with a mixed variabe. That instance is actuay a very particuar one: the variabe is connected and a the non-goba components are star-shaped. The cassification scheme of Figure 1.7 wi aso guide us throughout the thesis: we first address the goba subcass,