Orchestrating Massively Distributed CDNs

Transcription

1 Orhestrating Massively Distributed CDNs Joe Wenjie Jiang Prineton University Prineton, NJ prineton.edu Stratis Ioannidis Tehniolor Palo Alto, CA tehniolor.om Laurent Massoulié INRIA Paris, Frane inria.fr Fabio Pioni Tehniolor, Paris, Frane tehniolor.om ABSTRACT We onsider a ontent delivery arhiteture based on geographially dispersed groups of last-mile CDN servers, e.g., set-top boxes loated within users homes. These servers may belong to administratively separate domains, suh as multiple ISPs. We propose a set of salable, adaptive mehanisms to jointly manage ontent repliation and request routing within this arhiteture. Relying on primal-dual methods and fluid-limit tehniques, we formally prove the optimality of our design. We further evaluate its performane on both syntheti and trae-driven simulations, based on real BitTorrent traes, and observe a redution of network osts by more than 5% over traditional mehanisms suh as LRU/LFU with losest request routing. Categories and Subjet Desriptors C.4 [Computer-Communiation Networks]: [Distributed Systems] Keywords Distributed Content Distribution Networks, Content Plaement, Request Routing 1. INTRODUCTION The total Internet traffi per month was already in exess of 1 19 Bytes in 211 [1]. Video-on-demand traffi alone is predited to grow to three times this amount by 215. Existing ontent providers suh as Youtube and Netflix, whih represent a large fration of today s Internet video traffi, use ontent delivery networks (CDNs) to repliate, ahe, and stream videos at many servers aross the world. Nevertheless, the large volumes of traffi exiting the CDN infrastruture inur signifiant operational osts for the ontent providers. This state of affairs prompts a rethinking of the urrent ontent delivery arhiteture. The author is urrently at Google In. Permission to make digital or hard opies of all or part of this work for personal or lassroom use is granted without fee provided that opies are not made or distributed for profit or ommerial advantage and that opies bear this notie and the full itation on the first page. To opy otherwise, to republish, to post on servers or to redistribute to lists, requires prior speifi permission and/or a fee. CoNEXT 12, Deember 1 13, 212, Nie, Frane. Copyright 212 ACM /12/12...$15.. A promising evolution from today s approah onsists of extending the CDN to the last mile of ontent delivery by inorporating small servers lose to the edge of the network. For instane, this approah leverages devies within users homes, suh as set-top boxes or broadband gateways, as advoated by the Nano Dataenter [2] onsortium, network attahed storage (NAS) suh as Boxee [3], or applianes promoted by business initiatives suh as AppleTV. Part of or the entire storage and bandwidth apaities of these devies an be leased to a CDN or a ontent provider. The latter leverage these resoures to store ontent and serve download requests effetively in a managed peer-to-peer fashion. Suh an arhiteture has onsiderable advantages for both providers as well as users. First, it harnesses available bandwidth and storage resoures of applianes already deployed at users homes. Seond, it enables serving requests loally, e.g., from a devie within the same ISP or even within the same neighborhood. In addition to reduing lateny, this alleviates CDN server traffi while also reduing ross-traffi among ISPs, thereby signifiantly dereasing the operational ost of the CDN. In pratie, the servie operator an make a ombined use of the traditional CDN and distributed CDNs to optimize performane and redue ost simultaneously. Ideally, one would like to optimize the design of this diffuse loud of nano-servers so that suh ross-traffi osts are minimized. There are two degrees of freedom in this design: (a) ontent plaement, i.e., identifying what ontent to ahe in eah devie, and (b) request routing, identifying whih devie is to serve an inoming request for ontent. As devies have limited resoures, the storage and bandwidth onstraints of eah devie need to be taken into aount in the above optimization. Also, optimal plaement and routing deisions depend on the demand for ontent aross users; to redue traffi osts, it is preferable to ahe ontent loser to where it is most frequently requested. Nevertheless, this optimization raises several important hallenges. First, while traditional CDN design has addressed ontent ahing and request routing separately, a joint optimization is required as both deisions impat eah other in the ase of massively distributed last-mile servers. Seond, as the optimization is to be performed over millions of devies, salability is a ruial issue. Third, the system omprises devies dispersed aross multiple ISPs and their heterogeneity in terms of bandwidth and storage resoures, as well as the relative osts of serving requests needs to be addressed. In addition, managing many boxes over different ISPs requires a distributed and ollaborative solution that does not reveal the internal struture of ISPs. Finally,

2 though optimal plaement and routing deisions depend on demand, the latter may be a priori unknown or time-variant. Hene, adaptive plaement and routing shemes that reat to hanges in user demand, are preferred. In this work, we address these hallenges by developing a solution for effiient traffi management in next-generation CDNs. We make the following ontributions: We propose a distributed, adaptive ontent plaement and routing arhiteture, designed to sale over millions of heterogeneous devies. Building upon primal-dual deomposition tehniques, we formally establish that our adaptations jointly optimize both ontent plaement and routing, and prove that they minimize CDN traffi osts. To establish these results, we haraterize the asymptoti loss probability of the uniform-slot servie assignment poliy (determining whih devie serves a request inside an ISP), a ruial omponent of our design. We show that although muh simpler to implement than prior art, it exhibits the same asymptoti behavior. As part of this design, we propose a novel ontent plaement sheme that hanges ahe ontents, and prove that it is order-optimal: it reahes a targeted repliation within a onstant fator from the optimal number of write operations neessary. We ondut simulations driven by BitTorrent traes, with all the assoiated features of real traffi (burstiness, long-tail distribution), and show that our mehanism redues network osts by more than half, as ompared to traditional solutions based on LRU ahe management and nearest-neighbor routing. The remainder of this paper is organized as follows. We first review related work (Setion 2) and desribe our problem setup (Setion 3). In Setion 4 we overview our system arhiteture. The theoretial guarantees of our adaptation, request routing and ontent plaement algorithms are presented in Setions 5 and 6. Finally, in addition to this theoretial underpinning, our arhiteture is further validated experimentally in Setion RELATED WORK Peer-assistane of CDNs has been studied from several perspetives. Reent work has shown that it an redue CDN server traffi [4] and energy onsumption [5] by more than 6%. Early researh ompared the effiieny of prefething poliies for peer-assisted VoD [6], and bandwidth alloation aross different P2P swarms [7]. Peer inentivization, e.g., through rebates or servie fee redutions, has also been studied [8 1]. Moreover, the use of dediated home devies as an extension of a CDN s infrastruture has been the model of at least one reent start-up [11] and has been the subjet of several reent papers [5,12]. We build and extend upon these works by providing a formal framework for joint plaement and routing optimization. Minimizing ross-traffi is extensively studied in the ontext of ISP-friendly P2P system design and is known to redue both ISP ross-traffi and download delays [13, 14]. Typially, the seletion of download soures is biased towards nearby peers; peer proximity an be inferred either through lient-side mehanisms [15] or through a servie offered by the ISP [16 18]. In the latter ase, the ISP an expliitly reommend whih neighbors to download ontent from by solving an optimization problem that minimizes ross-traffi [18]. In the ontext of peer-assisted CDNs, an objetive that minimizes a weighted sum of ross-traffi and the load on the ontent server an also be onsidered [17]. Prior work on ISP-friendliness redues ross traffi solely by performing servie assignment to suitable peers. We add a ontrol knob on top of servie assignment, namely ontent plaement: our optimization selets not only where requests are routed, but also where ontent is stored. In the ooperative ahing problem, lients generate a set of requests for items, that need to be mapped to ahes that an serve them; eah lient/ahe pair assignment is assoiated with an aess ost. The goal is to deide how to plae items in ahes and assign requests to ahes that an serve them, so that the total aess ost is minimized. The problem is NP-hard, and a 1-approximation algorithm is known [19]. Motivated by CDN topologies, Borst et al. [2] obtain lower approximation ratios as well as ompetitive online algorithms for the ase where ahe osts are determined by weights in a star graph. A polynomial algorithm is known in the ase where ahes are organized in a hierarhial topology and the replia aessed is always the nearest replia [21]. We signifiantly depart from the above studies by expliitly dealing with bandwidth onstraints, assuming a stohasti demand for items and proposing an adaptive, distributed algorithm for joint ontent plaement and servie assignment among multiple lasses of idential ahes. Finally, reent work [22 24] has onsidered ahe management speifially in the ontext of P2P VoD systems, and is in this sense lose to our work. However, the heterogeneous (e.g., aross multiple ISPs) aspet of our system is not present in any of the above works. As in [24], we apture box servie behavior through a loss model; our Thm. 2 extends their analysis, by establishing that a simple servie assignment poliy (the uniform slot poliy), has asymptotially the same behavior as the one proposed in [24] (see also Setion 6.1). 3. PROBLEM FORMULATION In the system we onsider, a ontent provider suh as YouTube or Netflix delivers ontent (e.g., videos or musi) to home users subsribing to its servies. Users aess the servie through devies installed at their home, suh as set-top boxes (providing ommon Internet onnetivity and limited storage) or network-attahed storage (NAS) devies. Part of the storage and upload apaities of these boxes is leased to the ontent provider. The latter uses these resoures to serve requests for its ontent, alleviating the load on its CDN. Users are geographially dispersed aross ISPs. Serving them inurs a ost depending on the ross-traffi they generate. The ontent provider s goal is to determine (a) where to ahe ontent and (b) how to serve user requests for ontent, in order to minimize this ost. The remainder of this setion formalizes our setup; our key notation is in Table Box Classes We represent the users home devies by a set B, the set of boxes, where B = B. We partition B into D lasses B d, d D = {1,..., D}, with size B d = B d. Suh partitioning may orrespond, e.g., to grouping together boxes managed by the

3 s Existing CDN infrastruture. B Set of all boxes in the system. C Content atalog. D Set of all set-of-box lasses. B d Boxes in lass d D. M d Storage apaity of boxes in B d. U d Upload apaity of boxes in B d. F b Cahe ontent of box b. p d Repliation ratio of ontent in B d. λ d Request rate for ontent C in B d. w dd Cost of a transfer from d D {s} to d D. r dd Rate of requests for forwarded from d D to d D {s}. Aggregate rate of inoming requests for to d D. R d R d = B d U d, the total upload apaity in lass d. Table 1: Summary of key notation same ISP. Different levels of aggregation or granularity may be used: for example, eah lass may omprise boxes within the same ity or even the same ity blok. Throughout the text, we use the index d for a lass in D and the index s (as in server ) to indiate the existing CDN infrastruture. Through the CDN, boxes gain aess to the provider s ontent olletion, suh as, e.g., movies, lips or shows. We denote this olletion by C and all it the ontent atalog. We assume that items in C have idential size if not, the original ontent an be partitioned into fixed-size hunks, and C viewed as a olletion of hunks. Classes are heterogeneous: storage and bandwidth apaities as well as traffi osts assoiated with serving requests differ aross lasses. Nevertheless, as desribed below, boxes within the same lass have the same apaities and inur the same osts. 3.2 Storage and Bandwidth Capaity Part of the boxes storage is alloated to and managed by the CDN. Eah box in B d has M d storage slots, used by the CDN to store ontent items. We all M d the storage apaity of boxes in B d. For eah b B d, we denote the set of items ahed in box b by F b C, where F b = M d. Note that F b is determined by the CDN, not the user. Users may store (or delete at will) ontent they retrieve in private storage devies, or even at the spare storage of their box, but suh replias are not managed or shared by the CDN. Boxes an serve inoming requests for ontent they store in this designated spae. We model this servie behavior through a loss model [25], rather than a queueing model. A box in B d an upload at most U d ontent items onurrently, eah at a fixed rate. We refer to U d as the upload apaity of boxes in lass d. Alternatively, eah box has U d upload slots : if a box reeives a request for a ontent it stores and has a free upload slot, this slot is used to serve the request and upload the requested ontent. The servie time, i.e., the duration of an upload, is assumed to be exponentially distributed with one-unit mean. Slots remain busy until the upload terminates, at whih point they beome free again. The use of a loss model ensures that inoming requests are immediately served by a box at a guaranteed rate and no queuing delays are inurred. Moreover, most of today s ontent servies, suh as video streaming, require a onstant bit-rate and do not onsume additional bandwidth, so partitioning uplink bandwidth into slots makes sense. 3.3 Request Load Users (boxes) generate ontent requests at varying rates aross different lasses. We model requests for ontent generated by eah box b B d through a Poisson proess with rate λ d. Hene, the aggregate request proess for in lass d is also Poisson with rate λ d = λ d B d, whih sales proportionally to the lass size. When a box b B d storing C (i.e., F b ) generates a request for, it is served by the loal ahe no downloading is neessary. Otherwise, the request must be served by either the CDN s pre-existing infrastruture or some other box in B. Though our analysis assumes that the request generation proess is stationary and Poisson (Thms. 1-3) we relax this assumption in Setion 7, evaluating our system over requests of time-varying intensity generated by real-life P2P users. 3.4 Minimizing Traffi Costs Serving a user request from lass d using either the existing CDN infrastruture or another lass d requires transferring ontent aross the lass boundaries. In general, rosstraffi osts are ditated by the transit agreements between peering ISPs and may vary from one lass to the next. As suh, we denote by w dd, d, d D, the traffi ost for serving a request from lass d by a box in lass d. Similarly, we denote by w ds as the traffi ost of serving a request from lass d by the CDN s existing infrastruture. The ontent provider that manages the boxes pays inurred ross-traffi osts to ISPs. Hene, it is in its interest to minimize suh osts. In partiular, the servie provider needs to determine (a) the ontent F b plaed in eah box b, and (b) where the request generated by eah box should be direted to, so that its aggregate traffi osts are minimized. Solving this problem over millions of devies, while satisfying the onstraints imposed by the limited storage and bandwidth apaities at eah box, poses a signifiant salability hallenge. Further, deiding where to plae ontent and how to route requests is, in general, a omputationally intratable ombinatorial problem [19]. In addition, managing boxes from different ISPs raises the need for a distributed solution that does not require the ISPs to reveal their internal struture to eah other. Finally, the optimal plaement and routing sheme depends on the demands λ d ; these may dynami and a priori unknown to the CDN. As suh, an adaptive sheme, that measures and reats to user demand, is preferable. We present a system design that addresses these hallenges in the next setion. 4. SYSTEM ARCHITECTURE To address the hallenges above, we propose a distributed, adaptive sheme for joint plaement and routing. Through a ombination of asymptoti results, we show our design is optimal, in the sense that it minimizes the CDN s aggregate traffi ost when the number of boxes is large. 4.1 Overview Rather than entralizing the management of the distributed CDN, our solution delegates management to one devie per lass, termed the lass traker. Class trakers are deployed by the ontent provider, either as separate servers or as designated boxes within eah lass. They manage (a) ontent plaement within their lasses, (b) the routing of requests either generated or served by boxes in their lasses.

4 Eah traker has a full view of the state of boxes within its lass, knowing, e.g., the ontents of eah box s ahe and the number of its free upload slots. Nevertheless, the traker does not have aess to the same information about boxes in other lasses. In fat, its knowledge about the state in other lasses is limited to lightweight ongestion signals exhanged periodially between trakers. Overall, trakers perform the following operations: Adaptation. For eah ontent C, the traker maintains the desirable repliation ratio p d, i.e., the fration of boxes in the lass that store. In addition, it also maintains the desirable forwarding rates r dd, d D {s}, whih orrespond to the rate of outgoing requests for, forwarded to other lass trakers as well as the CDN infrastruture (denoted by s). These variables are stored loally by the traker and updated periodially, at fixed time intervals (e.g., one every day). The updated values are used as inputs to the traker s plaement and routing algorithms within the next adaptation round. Updating these variables allows the trakers to adapt both their plaement and routing deisions, in a way that the system reahes a global objetive (namely, the minimization of aggregate osts). Content Plaement. At the termination of eah adaptation round, after deiding the repliation ratios p d of eah ontent item in lass d, the traker alloates the ontent items to boxes: for eah box b B d, it determines F b in a manner so that the fration of boxes storing is indeed p d. Request Routing. Trakers are responsible for routing requests either generated or served by boxes in their lasses. We separate the routing of requests into two phases: request forwarding and servie assignment. Request forwarding determines to whih lass a request generated by a loal box is forwarded, so that the desirable forwarding rates r dd are maintained. Upon reeiving a request, the traker of the lass selets a box within its lass to serve the request; we refer to this seletion as servie assignment. In the remainder of this setion, we formally define the optimization performed by the traker through adaptation and desribe our request routing algorithm in detail. We also outline our distributed adaptation and ontent plaement algorithms; their full speifiation is in Se. 5 and Traker Information As stated above, eah lass traker has a omplete view of the urrent state of every box inside its own lass. In partiular, it knows (a) whih ontent items are stored in eah box, and (b) how many free upload slots it has. The trakers also ollet traffi statistis: the lass d traker maintains estimates of λ d, C, the rate with whih requests for ontent are generated within the lass. It also maintains estimates of the inoming rate of requests for ontent in lass d. All the above are measured and maintained loally at the traker; this is possible preisely beause it manages both ontent plaement and request routing within its lass. Nevertheless, trakers are a-priori unaware the states of boxes in other lasses: they learn about ongestion in other lasses through the exhange of appropriate light-weight ongestion signals, at the end of eah adaptation round. 4.3 Repliation and Forwarding Poliies In addition to the above information pertaining to their lasses, trakers maintain C loal variables p d [, 1], for C, d D. We all p d the repliation ratio of item in lass d, and the vetor p d = [p d ] C the repliation poliy of lass d. At any point in time, the repliation ratios satisfy: p d = b B d 1 Fb /B d, C, d D, (1) i.e., the repliation ratio p d equals the fration of boxes in B d that store ontent C. Also, by summing (1) in terms of, it is easy to see that, when all ahes are full, C pd = M d, d D. (2) The repliation poliy of a traker serves as an input to its ontent plaement algorithm. That is, the traker updates it repliation poliy (or, its desired repliation ratios) at the end of every adaptation round. Subsequently, the repliation poliy is used to determine the plaement of ontent to boxes in B d, so that (1) is indeed satisfied. In addition to its repliation poliy, the traker maintains ( D + 1) C additional loal variables r dd, d D {s}, C. We all these the forwarding rates of lass d, and the vetor r d = [r dd ] d D {s}c of these values the forwarding poliy of d. At any point in time eah variable r dd R +, for d D, equals the rate of requests for ontent that the traker forwards from lass d to d. Similarly, eah variable r ds R + equals the rate of requests for forwarded by the traker diretly to the CDN s existing infrastruture. The forwarding poliies satisfy the equalities: r ds + d D rdd = λ d (1 p d ), C, d D, (3) i.e., requests not immediately served by loal ahes are forwarded to the CDN or a box in another lass. The traker also updates its forwarding poliy at the end of an adaptation round. The updated values are subsequently used as inputs to the traker s request forwarding algorithm for the next round. In partiular, the traker implements a routing sheme that ensures that the rate of requests for item forwarded to d C {s} is preisely r dd. 4.4 Request Routing and Loss Probabilities As mentioned above, the routing of requests in our sheme onsists of two phases, request forwarding and servie assignment. We desribe these in detail below. Request Forwarding. In the request forwarding phase, a box b B d generating a request for an item C first heks if it already stores, i.e., F b. If so, the request is served immediately and no downloading is neessary. If not, the box ontats the lass traker; the latter determines whether the request should be forwarded to (a) another box within the lass, (b) a box in another lass, or () served diretly by the CDN s infrastruture. If the traker determines that the request is to be forwarded to lass d (ase (b)), it routes the request to the traker managing this lass. To selet among these 3 outomes, the traker uses r d as follows: it forwards a request to d D {s} with a probability proportional to r dd. As a result, provided that (3) is satisfied, requests forwarded from lass d to d form independent Poisson proesses with rates r dd. Servie Assignment. In the servie assignment phase, the lass d traker assigns a request for a ontent to the box in its lass that is to serve it. Requests an be loal, i.e.,

5 generated by a box in B d and deemed to be served loally during the forwarding phase, or external, i.e., generated by a box in a different lass d and forwarded to the lass d traker by the traker of d. To assign requests to boxes, the traker follows a uniform slot poliy. Under this poliy, an inoming request for ontent is assigned to a box seleted among all boxes urrently storing and having an empty upload slot. Eah suh box is seleted with a probability proportional to the number of its empty slots. Equivalently, the request is mathed to a slot seleted uniformly from all free upload slots of boxes storing : for X b the number of free slots of box b B d, an inoming request for ontent is mapped to a slot seleted uniformly at random among the b B d : F b X b slots of boxes that an serve this request. Loss Probabilities. It is possible that no free upload slots in the lass exist when the request for arrives (i.e., b B d : F b X b = ). In suh a ase, a request is re-routed to the CDN s infrastruture. Hene, not all requests for ontent that arrive at lass d are served by boxes in B d. Let ν d be the loss probability of item in lass d, i.e., the probability that a request for annot be served and is rerouted to the infrastruture. We say that requests for item are served with high probability (w.h.p.) in lass d, if lim B d ν d (B d ) =, (4) i.e., as the total number of boxes inreases, the probability that a request for ontent fails goes to zero. Two neessary onstraints (see, e.g., [24]) for (4) to hold in lass d D are: where C < B d U d, (5) < B d U d p d, C, (6) is the aggregate request rate for ontent reeived by lass d. Constraint (5) states that the aggregate traffi load imposed on lass d should not exeed the total upload apaity over all boxes; (6) states that the traffi imposed on d by requests for should not exeed the total apaity of boxes storing. = d D rd d 4.5 Content Plaement Our ontent plaement algorithm is presented in detail in Setion 6. In summary, the algorithm is exeuted at the end of every adaptation round. It reeives as input the repliation poliy p d and generates a plaement {F b } b B d that is onsistent with (1). Implementing the new plaement requires opying new ontents to box ahes; transferring ontent inurs traffi osts. Our design ensures these osts are small in two ways. First, poliy adaptations are smooth, i.e., hanges in r d, p d are gradual. Seond, we implement the new plaement by performing as few hanges to box ontents as possible. Cruially, our plaement also satisfies the following property: when uniform slot servie assignment is used, all requests are satisfied w.h.p. In partiular, our plaement is suh that (5) and (6) are not only neessary but also suffiient for (4) to hold (.f. Thms. 2 and 3). As suh, our ontent plaement ensures that, asymptotially, almost no requests are re-routed to the CDN. 4.6 Global Optimization Reall that the ost inurred when a request originating from lass d is served by d D {s} is w dd. Moreover, the rate of requests for ontent forwarded from d to d is r dd. Hene, the total traffi ost is [ w ds r ds + ( w dd r dd (1 ν d ) + w ds r dd d Cd D ν d This is beause a fration ν d requests for ontent arriving at lass d are re-routed to the CDN. In general, this is not a onvex funtion, due to the loss probabilities ν d. However, given that (4) holds, the ontribution of these losses beomes negligible for large system sizes. The total system osts an thus be approximated as d D F d (r d ), where F d (r d ) = ( C w ds r ds + ) d D wdd r dd (7) is the total traffi ost generated by lass d. Hene, the operator s minimal ost is a solution to the linear program: GLOBAL Min. d D F d (r d ) (8a) subj. to C pd = M d, d D d D rdd +r ds )]. (8b) =λ d (1 p d ), C, d D (8) C < R d, d D (8d) < R d p d, C, d D (8e) r dd, r ds var. r d, p d, d D, 1 p d, C, d, d D where R d = B d U d is the total upload apaity in lass d. The objetive of this optimization problem is to minimize the total ost inurred by ontent transfers. Constraints (8b) and (8) orrespond to equations (2) and (3); they state that the full storage apaity of eah lass is used and that all requests are eventually served, respetively. Constraints (8d) and (8e) orrespond to (5) and (6), respetively. GLOBAL is a linear program in r d, p d, d D. Our distributed, adaptive method for updating the routing and plaement poliies, is presented in detail in Setion 5. It is designed in a way so that (a) trakers measure and adapt their poliies to user demand, and (b) poliy updates are omputed in a distributed fashion, thus saling well as the number of boxes inreases. Most importantly, the poliies of our design onverge to a solution of GLOBAL (see Thm. 1), i.e., our design minimizes aggregate traffi osts. The formal properties of our design are shown in the next two setions. In Setion 5, we speify how trakers update their poliies so that they onverge to a solution of (8). In Setion 6, we haraterize ontent plaements under whih requests assigned by a uniform slot poliy are served w.h.p. Moreover, we show that if (8d) and (8e) hold, suh a ontent plaement exists, and give an algorithm implementing it. 5. POLICY ADAPTATION We now present how the trakers solve GLOBAL and determine their repliation and forwarding poliies in a distributed fashion. In short, trakers exhange ongestion signals and update p d, r d over several rounds. We ensure that both are updated in a smooth fashion, i.e., hanges between two rounds are inremental and the system does not osillate wildly. We proeed by first disussing the hallenges in solving (8) in a distributed fashion with lassial methods, and then presenting our distributed implementation.

6 5.1 Standard Dual Deomposition Consider the partial Lagrangian of (8). L(r, p; α, β) = d F d (r d ) + d β d (,d rdd R d ) + ( ) d αd d rdd R d p d where α d, β d are the dual variables (Lagrange multipliers) assoiated with the onstraints (8d) and (8e), respetively. Observe that L is separable in the primal variables, i.e., it an be written as L(r, p, α, β) = d Ld (r d, p d ; α, β) where L d (r d, p d, α, β) =F d (r d ) β d R d αd R d p d + d ( β d rdd + αd r dd This suggests a standard dual deomposition algorithm [26] for solving GLOBAL. Reall that a dual deomposition algorithm runs in multiple rounds t =, 1,.... The lass d traker maintains the primal variables r d, p d, as well as the dual variables α d = [α] d C, β d, assoiated with the oupling onstraints (8d) and (8e). At the end of eah round, the traker updates the dual variables α, d β d, inreasing them when the respetive onstraints (8d) and (8e) are violated or dereasing them when the onstraints are loose. Subsequently, eah traker broadasts its urrent dual variables with all other trakers. Having all dual variables α, β in the system, the trakers adapt their primal variables, reduing traffi forwarded to ongested lasses and inreasing traffi forwarded to non-ongested ones. This an be performed by eah traker solving the linear program: (r d, p d )(t + 1) = argmin (r d,p d ) I d L d (r d, p d ; α(t), β(t)). (9) where I d is the set of pairs (r d, p d ) defined by (8b) and (8) as well as the non-negativity onstraints. Suh adaptations are known to onverge to a maximizer of the primal problem when the funtions L d are stritly onvex (see, e.g., [26] Setion 3.4.2, pp ). Unfortunately, this is not the ase in our setup, as L d are linear in r d, p d : onvergene to optimal poliies does not readily follow. In pratie, the lak of strit onvexity makes r d, p d osillate wildly with every appliation of (9). This is disastrous: wide osillations of p d imply that a large fration of boxes in B d need to hange their ontent in eah round. This is both impratial and ostly; ideally, we would like eah round to hange the ontents of eah lass smoothly, so that the ost of implementing these hanges is negligible. 5.2 A Smooth Distributed Implementation To address these issues, we use an interior point method that deals with the lak of strit onvexity alled the method of multipliers [26]. Applied to GLOBAL this implementation yields the algorithm summarized in Fig. 1. The following theorem, proved in App. A, follows from the analysis in [26] and establishes that this algorithm indeed solves (8): Theorem 1. Assume that the traker in lass d orretly estimates λ d, in eah round, and that {θ(t)} t N is a nondereasing sequene of non-negative numbers. Then, under the adaptation algorithm in Fig. 1, r d (t), p d (t) onverge to an optimal solution of (8). Cruially, the algorithm in Fig. 1 performs smooth adaptations of r d, p d. Though the theorem assumes that trakers ). Traker d at the end of round t: Obtain estimates of λ d,, C. // Update dual variables s d tot 1 ( D C + yd R d) β d β d + θsd tot for eah ontent s d 1 ( D + z d Rd p d ) α d α d + θs d end for Broadast ( α d, β d, s d, s d tot) to other trakers d D Reeive dual variables from all other trakers d D // Update primal variables (r d, p d, z d, y d ) argminlocal d (r d, p d, z d, y d, α, β, s, s tot) I d Figure 1: Deentralized solution to the global problem GLOBAL. orretly estimate the request rates λ d, whih are stationary, we relax both assumptions in Setion 7, evaluating our adaptive approah under real-life, time-varying traffi. We desribe the operations performed and messages exhanged by eah traker below. The lass d traker maintains r d (t), p d (t), α d (t), β d (t), the primal and dual variables of (8), as well as the slak variables y d, z d = [z d ] C, resulting from onverting of (8d) and (8e) to equality onstraints: C + y d = R d, d D (1a) + z d = R d p d, C, d D (1b) y d, z d, C, d D (1) In addition, for every C, the traker maintains an estimate of λ d, i.e., the request rate of from boxes within its own lass, as well as an estimate of,i.e., the request rate for ontent served by boxes in B d. These an be estimated through appropriate ounters or through more sophistiated moving-average methods (suh as, e.g., EWMA). Using these estimates, the primal and dual variables are updated as follows. At the end of round t, the traker in lass d uses the estimates of to see whether onstraints (1a) and (1b) are violated or not. In partiular, the traker omputes the quantities: s d tot(t) = ( (t) + y d (t) R d) / D ( ) s d (t) = (t) + z d (t) R d p d (t) / D, and updates the dual variables as follows: β d (t) = β d (t 1) + θ(t)s d tot(t) α d (t) = α d (t 1) + θ(t)s d (t), C C where {θ(t)} t N are positive and non-dereasing. Subsequently, the traker broadasts to every other traker in D its ongestion signals α d (t), β d (t), s d (t), s d tot(t). This entails the exhange of 2 D ( D + 1) values, in total. For any d, d D, let G dd tot (r d, y d ) = rdd + 1 d=d y d, and G dd (r d, p d, z d ) = r dd + 1 d=d (z d R d p d ). Intuitively, these apture the ontribution of the primal variables of lass d to the onstraints (1a) and (1b) of lass d. After the traker in lass d has reeived all the messages sent by other trakers, it solves the following quadrati program:

7 LOCAL d (r d (t), p d (t), z d (t), y d (t), α(t), β(t), s(t), s tot(t)) Min. F d (r d ) + d + d, + θ(t) 2 + β d (t)g dd tot (r d, y d ) α d (t)g dd (r d, p d, z d ) d (G dd [ ( ( G dd tot r d r d (t), y d y d (t) ) ) 2 + s d tot(t) ( r d r d (t), p d p d (t), z d z d (t) ) ) ] 2 + s d (t) s.t. (r d, p d, z d, y d ) J d, d D var r d, p d, z d, y d, d D where J d is the set of quadruplets (r d, p d, y d, z d ) defined by (8b) and (8) as well as the non-negativity onstraints. LOCAL d thus reeives as input all the dual variables α, β, the ongestion variables s d, s d tot, as well as all the loal primal variables at round t. The last four are inluded in the quadrati terms appearing in the objetive funtion, and ensure the smoothness of the hanges to the primal variables from one round to the next. 6. CONTENT PLACEMENT The previous setion establishes that, through an appropriate exhange of ongestion signals, trakers an solve (8) in a distributed fashion. Nevertheless, the objetive (8a) is only an approximation of the atual traffi; this is beause requests reahing a lass may be dropped and redireted to the CDN infrastruture. In this setion, we desribe our ontent plaement sheme and show that it ensures that all inoming requests are satisfied w.h.p Conditions for Non-Rediretion We begin by establishing the neessary and suffiient onditions for requests to sueed w.h.p., when the trakers implement the uniform slot servie assignment. Consider a olletion of ontents F C suh that F = M d. Let B d F = {b B d : F b = F} be the set of boxes in the lass that store exatly F. These sets partition B d into sub-lasses, eah omprising boxes that store idential ontents. Let the number of boxes B = B go to infinity, while saling both the request arrival rates and the size of the sublasses BF d = BF d proportionally to B. That is, the quantities r.d /B, BF/B d are onstants that do not depend on B as the latter inreases. This saling is onsistent with our design: as B inreases, the aggregate ontent demand and the storage and upload apaities grow proportionally with B. The following theorem, proved in App. B, haraterizes the onditions under requests sueed w.h.p. Theorem 2. Assume that requests are assigned aording to the uniform slot poliy. Then, requests for every ontent C are served w.h.p. if and only if A < F:F A Bd FU d, for all A C, (11) Condition (11) stipulates that for any set of items A C the arrival rate of requests for these items does not exeed the total upload apaity of lass d boxes storing these items. It is relatively straightforward to see that (11) is neessary for (4) to hold (see [24]). It has reently shown that it is also suffiient when the servie assignment poliy used is the so-alled repaking poliy [24]. At the arrival of a request, repaking re-assigns requests already served in boxes in the system in order to aommodate this request. Performing this repaking requires finding a maximum mathing in a bipartite graph of 2B d U d nodes. Our uniform slot poliy is thus easier to implement than repaking; moreover, Thm. 2 establishes that, despite its simpliity, it exhibits the same asymptoti performane. The theorem implies that, to serve all requests in a lass w.h.p., the ontent plaement should be suh that ondition (11) is satisfied. Unfortunately, this ondition onsists of a number of inequalities that is exponential in the atalog size C, and is not a priori lear how to onstrut a ontent plaement sheme. We address this in the next setion. 6.2 Plaement Algorithm In this setion, we show that if the onditions (8d) and (8e) of GLOBAL hold, there exists a simple ontent plaement sheme that satisfies (11). This has the following immediate impliations. First, it simplifies our design, as we only need to ensure that the O( D C ) onstraints (8d) and (8e) hold, rather than the exponentially many onstraints in (11); indeed it is only these onstraints that our adaptation sheme of Setion 5 takes into aount. Seond, by Thm. 2, implementing the plaement in this setion along with a uniform slot assignment ensures that all requests are served w.h.p. Below, we first desribe this plaement sheme, i.e. the mapping of ontents to boxes ahes, that satisfies (11). We then present an algorithm that, at eah round, reshuffles ahe ontents in the lass to reah this plaement with as few item transfers as possible. Designated Slot Plaement. We now show that if (8d) and (8e) hold, there exists a simple plaement sheme i.e., a set of ahe ontents {F b } b B d that satisfies (11). For every box b B d, we identify a speial storage slot whih we all the designated slot. We denote the ontent of this slot by D b and the remaining ontents of b by L b = F b \ {D b }. For all C, let E d = {b B d : D b = } be the set of boxes storing in their designated slot. The following lemma implies that if a suffiient number boxes store in their designated slot, then (11) is satisfied. Lemma 1. If E d > /U d then (11) holds. Proof. As E d are disjoint, we have F:F A Bd FU d = b B d :F b A U d = b B d :D b A U d + b B d :D b A U d A E d U d > A, for all A C. Hene, to ensure that (11) is satisfied, it suffies that at least /U d boxes store in their designated slot. We all suh a plaement sheme a designated slot plaement. On the other hand, the fration of boxes that store in any slot must not exeed p d. The following lemma states that is possible to plae ontents in eah designated slot to ensure that both onstraints are satisfied when (8d) and (8e) hold: Lemma 2. Given a lass d, onsider and p d, C, for whih (8d) and (8e) hold. There exist q d [, 1], C, suh that qd = 1 and Moreover, suh q d /B d U d < q d p d 1, C. (12) an be omputed in O( C log C ) time. The proof an be found in App. C. In summary, if (8d) and (8e) hold, ensuring that requests for all ontents are served

8 Input: Initial plaement {F b } b B and target ratios q, p Let A + := { C : q > q }, A := { : C : q < q }; while there exists b B s.t. D b A + and L b A Pik L b A, and swap it loally with the ontent of D b. Update q, π, A +, A aordingly while there exists b B s.t. D b A + and L b A = Pik A and plae in the designated slot D b ; Update q, π, A +, A aordingly Let C + := { : π > π }; C+ := { : π < π }; C := C \ (C + C ). Let G := { b B s.t. C + L b and C \ (D b L b ) }; while (G ) or (there exists C s.t. (π π )B 2) if (G ) then Pik any b G Replae some C + L b with some C \ (D b L b ); else Pik C s.t. (π π )B 2. Find a box b that does not store. Pik C L b and replae with. update G, π, C +, C, C aordingly. Figure 2: Plaement Algorithm w.h.p. in lass d is ahieved onstruting a designated slot plaement. Suh a plaement stores ontent in the designated slot of at least q d B d boxes, where q d B d are determined as in Lemma 2; the remaining slots are used to ahieve an overall repliation ratio of p d within the lass. Below, we desribe an algorithm that, given ratios q d and p d, plaes ontent in lass d in a way that these ratios are satisfied. For simpliity, we drop the supersript d in the remainder of this setion, referring to ontent plaement in a single lass. Construting a Designated Slot Plaement. We now desribe how to hange ahe ontents at the end of eah adaptation round. The traker is aware of the initial ontent plaement {F b } b B over B boxes in set B, prior to the adaptation, as well as the target (i.e., adapted) repliation ratios p and q, C, satisfying (12); the former are given by the adaptation algorithm, and the latter by Lemma (2). The plaement algorithm, outlined in Fig. 2, reeives these as inputs and outputs a new ontent plaement {F b} b B in whih q B boxes store in their designated slot, while approximately p B boxes store overall. Cruially, the plaement requires as few ahe hanges as possible. We assume that q B and p B are integers for large B, this is a good approximation. Let q, p be the orresponding designated slot and overall frations in the input plaement {F b } b B. Let π = p q, π = p q. A lower bound on the ahe modifiation operations needed to attain the target repliation ratios q and p is given by B(α + β)/2, where α = q q, β = π π. We also express the number of operations performed in terms of these quantities. The omplexity and orretness of the algorithm are established in the following theorem, whose proof is in App. D: Theorem 3. The ontent plaement algorithm in Fig. 2 leads to a ontent repliation {F b} b B in whih exatly q B boxes store in their designated slot, and p B boxes store overall, where p p B < 2M, and p p B 1, for all C. The total number of write operations is at most B[α + (M 1)(α + β)]/2. In summary, our algorithm produes a plaement in whih at most 2M items are either under or over-repliated, eah by only one replia. Most importantly, the plaement is ahieved with at most O(B(α + β)) write operations, whih is order-optimal. If repliation ratios hange gradually (and, thus, α and β are small), as ensured by our poliy adaptation, the algorithm does not perform a large number of ahe hanges. We desribe the algorithm in more detail below. The algorithm proeeds in three phases. In the first phase, the algorithm modifies the designated slots to reah the desired ratios q. To do so, the algorithm piks any overrepliated ontent in set A + = { : q > q }. For any user holding in its designated slot, it heks whether it holds in its normal slots an under-repliated ontent A = { : q < q }. If suh ontent exists, it renames the orresponding slot as designated and the slot holding as normal. This is repeated until an under-repliated ontent annot be found within the normal ahe slots of boxes storing some A +. If there still are over-repliated items in A +, some A is seleted arbitrarily and overwrites within the designated slot. At the end of this phase, the repliation rates within the designated slots have reahed their target B q, and the resulting ahes are free of dupliate opies. Also, after these operations, the intermediate repliation rates π within the normal ahe slots verify Bπ Bπ Bq Bq. In the seond phase, the algorithm begins transforming these intermediate repliation rates π into π. To this end, we distinguish ontents that are over-repliated, underrepliated and perfetly repliated by introduing C + = {: π >π }, C = {:π <π }, and C = {:π =π }. C + = { : π > π }, C = { : π < π }, C = { : π = π }. For any box b, if there exists C + L b, and C \ (D b L b ), the algorithm replaes by within L b. We all the orresponding operation a greedy redution. Greedy redutions are repeated until the algorithm arrives at a onfiguration where no suh hanges are possible; this terminates the seond phase. At that point, for any box b suh that C + L b is not empty, neessarily C (L b D b ). Hene, the size of C is at most M 1. If any of the elements in C is under repliated by at least two replias, the algorithm enters its third phase. In this phase, the algorithm piks some ontent that is under-repliated by at least 2 replias, and finds a user b whih does not hold, i.e. C \(D b L b ). It also selets some ontent within C L b : suh ontent must exist, sine C M 1, and C L b C \ { } has size stritly less than M 1, the size of L b ; the remaining ontent must belong to C sine otherwise we ould have performed a greedy redution. The algorithm then replaes ontent by ontent. We all this operation a swith. This augments the size of set C : indeed ontent is now under-repliated (one replia missing). The algorithm then tries to do a greedy redution, i.e., a replaement of an over-repliated ontent by if possible. If not, it performs another swith, i.e., by identifying some ontent under-repliated by at least 2, and reating a new replia in plae of some perfetly repliated item, thereby augmenting the size of C. Hene, in at most M 1 steps, the algorithm inflates the size of C to at least M, at whih stage we know that a greedy redution an be performed. This alteration between greedy redutions and swithes is repeated until the size of C is at most M 1, and eah suh ontent is missing exatly one replia.

9 Dropping Probability P(B) M=1 M=2 M=4 M= Number of Boxes B Figure 3: Dropping probability dereases fast with uniform slot strategy. Simulation in a single lass with a atalog size of C = 1. Global Cost Empirial Numerial Optimal Timeline Figure 4: Deentralized optimization, ontent plaement sheme and uniform-slot poliy, under parameters C = 1, D = 1, B = 1, Ū = 3, M = PERFORMANCE EVALUATION In this setion, we evaluate our algorithms under synthesized traes and a real-life BitTorrent traffi trae. We implement an event-driven simulator that aptures box-level ontent plaement and servie assignment. In partiular, we implement the solutions we proposed in Setion 5 (deentralized optimization) and Setion 6 (Designated Slot Plaement). We also implement lass trakers that exeute the deentralized solution in Fig. 1. In the rest of this setion, all evaluations are performed using a ost matrix with random lass-pairwise sampled uniformly from [, 1]. The download ost from a box in the same lass is, while the ost of downloading from the infrastruture is set to 3 for all lasses. 7.1 Simulation on Synthesized Trae First, we show that the uniform-slot poliy ahieves loseto-optimal servie assignment, given that ontent-wise apaity onstraints are respeted. We fous on a single ISP and assign requests to individual boxes under the uniformslot poliy. Fig. 3 shows the loss probability under various settings of B and M. We utilize a synthesized trae with Poisson arrivals and exponentially distributed servie time of mean one. Content popularity follows a Zipf distribution. We sale the total request rate to be proportional to the number of boxes. As predited by the theory, the dropping probability quikly vanishes as B grows. For the same B, the dropping probability is higher when M is larger. When the storage is rih, our sheme tries to alloate popular ontent in ahes in order to minimize ost. The loally absorbed requests are not ounted in alulating the dropping probability. In fat, the effetive requests ome from unpopular ontents, whih is expeted to generate a higher drop rate. We next show the optimality of our full solution, again utilizing a synthesized trae generated as above. Content popularities are heterogeneous in different lasses. Fig. 4 illustrates the average empirial ost per request, ompared to (a) the fluid predition under distributed optimization, when λ d and r dd are estimated perfetly, and (b) the optimal ost omputed offline. Though the number of boxes is finite, and the above quantities are estimated empirially, the distributed algorithm onverges lose to the global optimum after a handful of rounds. 7.2 Simulation on BitTorrent Trae We next employ a real-life trae olleted from the global Vuze network, one of the most popular BitTorrent lients. Our main motivation for using the BitTorrent trae is to validate our solution under realisti ontent popularity and aess patterns. Trae Colletion and Evaluation Setup. Vuze lients issue a put eah time they start downloading a file, and route the put to the 2 nodes whose IDs are losest to the file identifier. To ollet these traes, we ran 1 DHT nodes with randomly-hosen IDs, and logged all DHT put messages routed to our nodes. Therefore, our traes show all downloads for those files whose identifiers are lose to the ID of one of our 1 DHT nodes. Sine the Vuze DHT has around 1 million nodes, and eah file download is observed by 2 nodes, by running 1 nodes we observe around 1 6 /2/1, i.e., 2% of all download requests. During 3 days we traed the downloads of around 2 million unique files by 8 million unique IP addresses. We determined the ountry of eah IP using Maxmind s GeoLite City database [27]. We use the ountry-level geo-loality of BT users to organize them into lasses. We do not model interlass ost, e.g., lateny, on geo-loality, as measuring and estimating suh osts are outside the sope of this paper. To limit the runtime of our event-driven simulations, we trim the traes by onsidering only the top 1, most popular files, whih ontribute 52% of the total downloads. Given a fixed number of boxes and ahe size, there are very few opies of unpopular ontents ahed in these boxes. As a result, adding more ontents will not signifiantly hange the system onfiguration and the overall ost. Fig. 5(a) illustrates the total number of download events and unique IPs, grouped by ountries in a dereasing order of total downloads during the 3-day period, and the umulative ounts in Fig. 5(b). We selet the top 2 ountries as lasses in our evaluation, omprising over 8% of all downloads. The trae shows that one user issues, on average, approximately one ontent request. We use 2 upload slots and 2 storage slots for eah user (box) in our simulation. Previously we have shown the effiieny of our algorithm given fixed request rates. In pratie, the ontent popularity may hange over time. The ontent demand should remain suffiiently stable in order for our algorithm to work well and not osillate wildly. In our evaluation, we measure ontent demands based on a 24-hour interval, and simply use the demand from the previous day to projet the request rates for the next day. This undoubtedly only provides a onservative estimate for our solution, as a more aurate traffi predition will further improve the performane of our approah. Fig. 5() shows the relative differene between the predited rate and the true rate, grouped by ontent in dereasing order of popularity. Eah data point is averaged over the