A Simple Congestion-Aware Algorithm for Load Balancing in Datacenter Networks

Transcription

1 IEEE/ACM TRANSACTIONS ON NETWORKING A Smpe Congeston-Aware Agorthm for Load Baancng n Datacenter Networs Mehrnoosh Shafee, Student Member, IEEE, and Javad Ghader, Member, IEEE Abstract We study the probem of oad baancng n datacenter networs, namey, assgnng the end-to-end data fows among the avaabe paths n order to effcenty baance the oad n the networ. The soutons used today rey typcay on ECMP (Equa Cost Mut Path) mechansm whch essentay attempts to baance the oad n the networ by hashng the fows to the avaabe shortest paths. However, t s we nown that ECMP performs poory when there s asymmetry ether n the networ topoogy or the fow szes, and thus there has been much nterest recenty n aternatve mechansms to address these shortcomngs. In ths paper, we consder a genera networ topoogy where each n has a cost whch s a convex functon of the n congestons. Fows among the varous source-destnaton pars are generated dynamcay over tme, each wth a sze (bandwdth requrement) and a duraton. Once a fow s assgned to a path n the networ, t consumes bandwdth equa to ts sze from a the ns aong ts path for ts duraton. We consder ow-compexty congeston-aware agorthms that assgn the fows to the avaabe paths n an onne fashon and wthout spttng. Specfcay, we propose a myopc agorthm that assgns every arrvng fow to an avaabe path wth the mnmum margna cost (.e., the path whch yeds the mnmum ncrease n the networ cost after assgnment) and prove that t asymptotcay mnmzes the tota networ cost. Extensve smuaton resuts are presented to verfy the performance of the myopc agorthm under a wde range of traffc condtons and under dfferent datacenter archtectures. Furthermore, we propose randomzed versons of our myopc agorthm whch have much ower compexty and emprcay show that they can st perform very we n symmetrc networ topooges. Index Terms Marov chans, Load baancng, Onne agorthms, Routng agorthms, Datacenter networ I. INTRODUCTION There has been a dramatc shft over the recent decades wth search, storage, and computng movng nto arge-scae datacenters. Today s datacenters can contan thousands of servers and typcay use a mut-ter swtch networ to provde connectvty among the servers. To mantan effcency and quaty of servce, t s essenta that the data fows among the servers are mapped to the avaabe paths n the networ propery n order to baance the oad and mnmze the cost (e.g., deay, congeston, etc.). For exampe when a arge fow s routed poory, coson wth the other fows can cause some ns to become congested, whe other ess utzed paths are avaabe. The datacenter networs rey on path mutpcty to provde scaabty, fexbty, and cost effcency. Consequenty, there The authors are wth the Department of Eectrca Engneerng, Coumba Unversty, New Yor, NY, 27 USA (e-ma:s.mehrnoosh@coumba.edu, jghader@ee.coumba.edu). The research supported by NSF grants CNS and CNS An earer verson of ths paper appeared n INFOCOM 26 conference []. has been much research on fow schedung agorthms that mae better use of the path mutpcty (e.g., [2] [6]) or desgnng new networs wth better topoogca features (e.g., FatTree [2], VL2 [7], hypercube [8], hypergrd [9], random graphs such as JeyFsh [], etc.). In ths paper, we consder a genera networ topoogy where each n s assocated wth a cost whch s a convex functon of the n utzaton (e.g., ths coud be a atency functon). The networ cost s defned as the sum of the n costs. Fows among the varous source-destnaton pars are generated dynamcay over tme where each fow s assocated wth a sze (rate) and a duraton. Once a fow s assgned to a path n the networ, t consumes resource (bandwdth) equa to ts sze (rate) from a the ns aong ts path for ts duraton. The man queston that we as s the foowng. Is t possbe to desgn a ow-compexty agorthm, that assgns the fows to the avaabe paths n an onne fashon and wthout spttng, so as to mnmze the average networ cost? In genera, mut fow routng n networs has been extensvey studed from both networng systems and theoretca perspectve, however mut fow routng consdered n ths paper has two ey dstngushng objectves: ) t does not aow fow spttng because spttng the fow s undesrabe due to TCP reorderng effect []. Resovng pacet reorderng requres modfcaton of protoco stac [2], whch mght be costy. Wthout spttng, many versons of mut fow routng n networs become hard combnatora probems [3], [4]. In fact, the statc verson of the probem consdered n ths paper (.e., gven a statc st of fows, assgnng fows to paths wthout spttng so as to baance the oad n the networ) s nown to be NP-hard, through ts connecton to the Partton probem [5]. 2) t aows dynamc routng because t consders the current utzaton of ns n the networ when mang the routng decsons for newy arrved fows une statc soutons where the mappng of fows to the paths s fxed and requres the nowedge of the traffc matrx. A. Reated Wor Semna soutons for fow routng n datacenters (e.g. [7], [6]) rey on Equa Cost Mut Path (ECMP) oad baancng whch statcay spts the traffc among avaabe shortest paths (va fow hashng). However, t s we nown [3] [6], [7] In the Partton probem, gven a set of numbers, we are ased to dvde them nto two subsets such that the maxmum of the sum of the numbers n the sets s mnmzed. Ths can be reformuated as the oad baancng n a smpe two-node networ wth two parae edges.

2 IEEE/ACM TRANSACTIONS ON NETWORKING 2 that ECMP can baance oad poory snce t may map arge ong-ved fows to the same path, thus causng sgnfcant oad mbaance. Further, ECMP s suted for symmetrc archtectures such as FatTree and performs poory n presence of asymmetry ether due to n faures [8] or n recenty proposed datacenter archtectures []. Theoretca performance of ECMP n Cos networs under a statc fow mode has been studed n [9]. There have been recent efforts to address the shortcomngs of ECMP. The proposed agorthms range from centrazed soutons (e.g., [3], [4]), where a centrazed scheduer maes routng decsons based on goba vew of the networ, to dstrbuted soutons (e.g., [6], [2]) where routng decsons are made n a dstrbuted manner by the swtches. There are aso host-based protocos based on Mut Path TCP (e.g., [5]) where the routng decsons are made by the endhost transport protoco rather than by the networ operator; however, they requre sgnfcant changes to Transport ayer whch mght not be feasbe n pubc coud patforms [2]. Authors n [2] nvestgated a more genera probem based on a Gbbs sampng technque and proposed a pausbe heurstc that requres re-routng and nterrupton of fows (whch s operatonay expensve). There are aso agorthms that aow fow spttng and try to resove the pacet reorderng effect n symmetrc networ topooges [2], [2], [22]. As expaned, deang wth pacet reorderng nvoves overhead and modfcaton of protoco stac. Our wor s aso reated to a arge body of terature on traffc engneerng and congeston contro. For brevty, we ony hghght the most reevant wor. The frst ne of wor, e.g. [23] [25], studes the probem of mnmzng the cost of carryng traffc n a statc mut-commodty fow mode and under a convex cost functon for the n rates. Gven the nowedge of the traffc matrx (commodtes) among the nodes, routng agorthms are proposed that teratvey update the fracton of traffc of each fow that shoud be sent on each outgong n n the networ. They rey on spttng fows among the east weghted paths where the weght of each n s defned by ts margna n cost. The second ne of wor s atomc and non-atomc congeston games n game theory [26] [29]. In the context of routng, payers are the commodtes, strategy sets are the set of drected source-destnaton paths for the commodtes, the edge cost c e (f e ) s a functon of the amount of congeston f e over edge e, and the path cost c p (f) s the sum of the cost of the ns aong the path p. A payer ncurs a cost c p (f)f p () for sendng f p () amount of traffc over the path p. In the atomc games, each payer must choose a snge path to route ts commodty, whe n non-atomc games, payer can dstrbute ts commodty fractonay over the set of paths. The two versons are fundamentay dfferent. Whe the atomc game n genera does not admt a Nash equbrum, the nonatomc game aways has a Nash equbrum (Wardrop equbrum) [3]. In Wardrop equbrum, a the paths used by a gven commodty have equa cost. Moreover, t s nown n non-atomc games that sefsh best response moves (sefsh routng) by the payers teratvey converge to the Wardrop equbrum, whch s a oca mnmum of a potenta functon (networ cost) fe e c e(x)dx. The thrd ne of wor s obvous routng [3] [33] n whch routes are computed to optmze the worst-case performance over the set of traffc matrces. Ths ensures that the computed routes are prepared for changes n traffc demands wthout the need to update the routes, however ths s a pessmstc pont of vew and may be far from optma n reatvey stabe perods of traffc or stabe networs [32]. Whe the proposed myopc agorthm n ths paper s remnscent of pror agorthms under fow spttng and non-atomc games (e.g. [23] [25], [28] [3]), the resuts n ths paper are not trvay drawn from these pror wor. Frst, une [23] [25], [28] [3] that rey on spttng fows n any granuarty and reroutng them contnuousy to fnd the optma routng, we do not aow fow spttng and mgratons. Second, une [23] [25], [28] [3] that consder a statc set of fows wth nown traffc demand, we are deang wth a dynamc verson of the probem when fows arrve and depart dynamcay over tme and the traffc demand s not nown. Such constrants arse n practce due to the varyng nature of the traffc over tme and space n datacenters as we as undesrabty of pacet reorderng n fow spttng. Our technca approach rees on a carefu anayss of the fud mts of the system under the myopc pocy (wthout fow spttng) and proof of convergence to an nvarant set whch s the set of optma fow assgnments n steady state. Under unspttabe fows, the fud mts are not contnuousy dfferentabe whch poses a sgnfcant technca chaenge. Intutvey, as the number of fows n the system grows, the dfference between the optma expected networ cost under unspttabe fow assgnment and that under spttabe fow assgnment shoud vansh n the performance rato. We rgorousy estabsh ths ntuton, and further, present determnstc and randomzed agorthms wth ow compexty whch perform very we n practce. Fnay, Software Defned Networng (SDN) has enabed networ contro wth qucer and more fexbe adaptaton to changes n the networ topoogy or the traffc pattern and can be everaged to mpement centrazed or hybrd agorthms n datacenters [2], [34] [36]. The weght construct n the agorthms proposed n ths paper can provde an approach to optmay accommodate dynamc varatons n datacenter networ traffc n centrazed contro patforms such as OpenFow [34]. B. Contrbutons The man contrbutons of ths paper can be summarzed as foows. Asymptotc optmaty of a myopc agorthm. We propose and anayze a smpe fow schedung agorthm to mnmze the average networ cost (the sum of convex functons of n utzatons). Specfcay, we propose a myopc agorthm that assgns every arrvng fow to an avaabe path wth the mnmum margna cost (.e., the path whch yeds the mnmum ncrease n the networ cost after assgnment). We prove that ths smpe myopc agorthm s asymptotcay optma n any networ topoogy, n the sense that the performance rato between the

3 IEEE/ACM TRANSACTIONS ON NETWORKING 3 average networ cost under the myopc agorthm and the optma cost approaches as the mean number of fows n the system ncreases. The myopc agorthm does not rey on fow spttng, hence pacets of the same fow w trave aong the same path wthout reorderng. Further, t does not requre mgraton/reroutng of the fows or the nowedge of the traffc pattern. A ow compexty randomzed agorthm. We aso propose randomzed versons of our myopc agorthm whch have much ower compexty. In the randomzed agorthm wth parameter 2, nstead of consderng a the avaabe paths upon arrvng of a fow, paths are chosen at random and then the fow s assgned to the path wth the mnmum margna cost among these paths. Smar to the myopc agorthm, randomzed versons do not rey on fow spttng, fow mgraton/reroutng, or the nowedge of the traffc pattern. We emprcay nvestgate the effect of parameter on the agorthm performance. Emprca evauaton of the agorthms. We evauate our myopc agorthm and ts randomzed versons under varous woroad and networ topooges. For the fow generaton, we consder two traffc modes: () Posson arrva of fows wth exponentay dstrbuted duratons, and () based on data from emprca studes of datacenter traffc. For the networ topoogy, we consder FatTree (a hghy structured topoogy), and JeyFsh (a random topoogy). Our emprca resuts show that the myopc agorthm n fact performs very we under a wde range of traffc condtons n both datacenter topooges. Further, the randomzed agorthms can perform very we by choosng the proper parameter (the number of randomy chosen paths), n partcuar n symmetrc networ topooges (e FatTree) sma vaues of w suffce. C. Notatons Gven a sequence of random varabes {X n }, X n X ndcates convergence n dstrbuton, and X n X ndcates the amost sure convergence. Gven a Marov process {X(t)}, X( ) denotes a random varabe whose dstrbuton s the same as the steady-state dstrbuton of X(t) (when t exsts). s the Eucdan norm n R n. d(x, S) = mn s S s x s the dstance of x from the set S. u.o.c. means unformy over compact sets. II. MODEL AND PROBLEM STATEMENT A. Datacenter Networ Mode We consder a datacenter (DC) consstng of a set of servers (host machnes) connected by a coecton of swtches and ns. Dependng on the DC networ topoogy, a or a subset of the swtches are drecty connected to servers; for exampe, n FatTree [2] (Fgure ) ony the edge (top-of-the-rac) swtches are connected to servers, whe n JeyFsh [] (Fgure 2) a the swtches have some ports connected to servers. Nevertheess, we can mode any genera DC networ topoogy (FatTree, JeyFsh, etc.) by a graph G(V, E) where Aggregaton Edge Core Fg. : FatTree connectng 6 servers (rectanges) usng 4-port swtches (crces). Fg. 2: JeyFsh (random graph) connectng 6 servers (rectanges) usng 4-port swtches (crces). V s the set of swtches and E s the set of communcaton ns. A path between two swtches s defned as a set of ns that connects the swtches and does not ntersect tsef. The paths between the same par of source-destnaton swtches may ntersect wth each other or wth other paths n DC. B. Traffc Mode Each server can generate a fow destned to some other server. We assume that each fow beongs to a set of fow types J. A fow of type j J s a trpe (a j, d j, s j ) where a j V s ts source swtch (.e., the swtch connected to the source server), d j V s ts destnaton swtch (.e., the swtch connected to ts destnaton server), and s j s ts sze (bandwdth requrement). Note that based on ths defnton, we ony need to fnd the routng of fows n the swtch networ G(V, E) snce the routng from the source server to the source swtch or from the destnaton swtch to the destnaton server s trva (foows the drect n from the server to the swtch). Further, two swtches can have more than one fow type wth dfferent szes. We assume that type-j fows are generated accordng to a Posson process wth rate λ j, and each fow remans n the system for an exponentay dstrbuted amount of tme wth mean /µ j. It s possbe to extend our resuts to a more genera mode of fow arrva and servce tme, e.g., when the arrva process s a renewa process and servce tme dstrbuton has ower bounded hazard rate, usng a smar approach as n [37]. We w aso report smuaton resuts n Secton V that show that our myopc agorthm ndeed performs very we under much more genera arrva and servce tme processes. For any j J, et R j denote the set of avaabe paths from a j to d j, then each type-j fow must be accommodated by usng ony one of the paths from R j (.e., the fow cannot be spt among mutpe paths). Note that R j coud be the set of a possbe paths from a j to d j or a subset of them as desred by the networ operator. We assume that R j s nonempty for each j J. Defne Y (j) (t) to be the number of type-j fows

4 IEEE/ACM TRANSACTIONS ON NETWORKING 4 routed aong the path R j at tme t. The networ state s defned as ( ) Y (t) = (t); R j, j J. () Y (j) The onne (Marov) schedung agorthm determnes the path where an arrvng fow at tme t s paced, as a functon of the current networ state Y (t). We aso defne X (j) (t) = R j Y (j) (t) whch s the tota number of type-j fows n the networ at tme t. Let Z (t) be the tota amount of traffc (congeston) over n E. Based on our notatons, Z (t) = j J : R j, s j Y (j) (t), (2) where by we mean that n beongs to path. We aso defne ρ j = λ j /µ j whch s the mean offered oad by type-j fows. Note that under any Marov schedung agorthm, the networ state {Y (t)} t s a contnuous-tme, rreducbe Marov chan. It s aso postve recurrent, because the tota number of type-j fows X (j) (t) n the system s a Marov chan ndependent of the schedung agorthm, and ts statonary dstrbuton s Posson wth mean ρ j. Therefore, the process {Y (t)} t has a unque statonary dstrbuton as t. C. Probem Formuaton For the purpose of oad baancng, the networ can attempt to optmze dfferent objectves [38] such as mnmzng the maxmum n congeston n the networ or mnmzng the sum of n costs where each n cost s a convex functon of the n congeston (e.g. ths coud be a n atency measure [39]). Under both objectves, the traffc needs to be dstrbuted and baanced among the feasbe paths n the networ, whch s essenta for mantanng ow end-to-end deay for dfferent fows. In ths paper, we use the atter objectve but by choosng proper cost functons, an optma souton to the ater objectve can be used to aso approxmate the former objectve as we see beow. We defne g(z ) to be the cost of n when ts congeston s Z. Our goa s to fnd a fow schedung agorthm that assgns each fow to a snge path n the networ so as to mnmze the mean networ cost n the ong run, specfcay, mnmze m E [F (Y (t))] t subject to: servng each fow usng one path, where, F (Y (t)) = E g(z (t)). We consder poynoma cost functons of the form g(x) = x+α, α >, (4) + α where α > s a constant. Thus g s ncreasng and strcty convex n x. As α, the optma souton to (3) approaches the optma souton of the optmzaton probem whose objectve s to mnmze the maxmum n congeston n the networ 2. 2 Here we have consdered dentca ns for smpcty but the anayss s easy extendabe to the case that g( ) s a functon of x/c where c s the n capacty, or the case that each n has a weght and the goa s to mnmze the weghted summaton of the n costs. (3) III. ALGORITHM DESCRIPTION In ths secton, we descrbe our myopc agorthm for fow assgnment where each fow s assgned to one path n the networ (no spttng) wthout nterruptng/mgratng the ongong fows n the networ. Reca that Y (t) = (Y (j) (t)) s the networ state, Y (j) (t) s the number of type-j fows on path R j, and Z (t) s the tota traffc on n gven by (2). Agorthm Myopc Fow Schedung Agorthm Suppose a type-j fow arrves at tme t when the system s n state Y(t). Then, : Compute the path margna costs (Y (t)), R j, n ether of the forms beow: Integra form: Dfferenta form: (Y (t)) = (j) (Y (t)), (5) (Y (t)) = δ (j) (Y (t)). (6) 2: Pace the fow on a path such that = arg mn R j Brea tes n (7) unformy at random. (Y (t)). (7) Frst, we defne two forms of n margna cost that measure the ncrease n the n cost f an arrvng type-j fow at tme t s routed usng a path that uses n. Defnton. (Ln margna cost) For each n and fowtype j, the n margna cost s defned n ether of the forms beow. Integra form: (j) (Y (t)) = g Dfferenta form: (Z (t) + s j ) ( ) g Z (t). (8) δ (j) (Y (t)) = s j g ( Z (t) ). (9) Based on the n margna costs, we can characterze the ncrease n the networ cost f an arrvng type-j fow at tme t s routed usng path R j. Specfcay, et Y (t + ) = Y (t) + e (j), where e (j) denotes a vector whose correspondng entty to path and fow type j s one, and ts other enttes are zero. Then F (Y (t)) s the networ cost before the type-j fow arrva, and F (Y (t + )) s the networ cost after assgnng the type-j fow to path. Then, t s easy to see that F (Y (t + )) F (Y (t)) = [ ( ) ( g Z (t) + s j g Z (t) )] = (j) (Y (t)). () Smary, based on the dfferenta margna costs, we have F (Y (t)) Y (j) (t) = s j g ( Z (t) ) = δ (j) (Y (t)). ()

5 IEEE/ACM TRANSACTIONS ON NETWORKING 5 Agorthm descrbes our myopc fow assgnment agorthm that paces the newy generated fow on a path that mnmzes the ncrease n the networ cost based on ether forms () or (). Upon arrva of a fow, Agorthm taes the correspondng feasbe paths and ther n congestons nto the account for computng the path margna costs (t) but t does not requre to now any nformaton about the other ns n the networ. The two forms (5) and (6) are essentay dentca n our asymptotc performance anayss n the next secton, however t seems sghty easer to wor wth the dfferenta form (6). Agorthm can be mpemented ether centray or n a dstrbuted manner usng a dstrbuted shortest path agorthm that uses the n margna costs, (j) (t) or δ (j) (t), as n weghts. Remar. Note that n Agorthm the fow s assgned to a path wth the mnmum path margna cost. The path wth the mnmum path margna cost s not necessary the same as the path wth the mnmum end-to-end congeston (sum of n congestons n the path). IV. PERFORMANCE ANALYSIS VIA FLUID LIMITS The system state {Y (t)} t s a stochastc process whch s not easy to anayze, therefore we anayze the fud mts of the system nstead. Fud mts can be nterpreted as the frst order approxmaton to the orgna process {Y (t)} t and provde vauabe quatatve nsght nto the operaton of Agorthm. In ths secton, we ntroduce the fud mts of the process {Y (t)} t and present our man resut regardng the convergence of Agorthm to the optma cost. We deberatey defer the rgorous cams and proofs about the fud mts to Secton VII and for now many focus on the convergence anayss to the optma cost, whch s the man contrbuton of ths paper. A. Informa Descrpton of Fud Lmt Process In order to obtan the fud mts, we scae the process n rate and space. Specfcay, consder a sequence of systems {Y r (t)} t ndexed by a sequence of postve numbers r, each governed by the same statstca aws as the orgna system wth the fow arrva rates rλ j, j J (therefore, a system wth a arger r woud experence heaver traffc), and nta state Y r () such that Y r ()/r y() as r for some fxed y(). The fud-scae process s defned as y r (t) = Y r (t)/r, t. We aso defne y r ( ) = Y r ( )/r, the random state of the fud-scae process n steady state. If the sequence of processes {y r (t)} t converges to a process {y(t)} t (unformy over compact tme ntervas, wth probabty as r ), the process {y(t)} t s caed the fud mt. Then, (t) s the fud mt number of type-j fows routed through path. Accordngy, we defne z r(t) = Zr (t)/r and x (j)r (t) = X (j)r (t)/r and ther correspondng mts as z (t) and x (j) (t) as r. The fud mts under Agorthm foow possby random trajectores, and mght not be contnuousy dfferentabe; nevertheess, they satsfy the foowng set of dfferenta equatons. We state the resut as the foowng emma whose proof can be found n Secton VII. Lemma. (Fud equatons) Any fud mt y(t) satsfes the foowng equatons. For any j J, and R j, d dt y(j) p (j) (t) = λ j p (j) (y(t)) µ j (t) (2a) (y(t)) = f / arg mn p (j) (y(t)), R j (y(t)) = s j g (z (t)). (y(t)) (2b) R j p (j) (y(t)) = (2c) (2d) Equaton (2a) s smpy an accountng dentty for (t) statng that, on the fud-scae, the number of type-j fows over path R j ncreases at rate λ j p (j) (y(t)), and decreases at rate µ j due to departures of type-j fows on path. p (j) (y(t)) s the fracton of type-j fow arrvas paced on path. (y(t)) s the fud-mt margna cost of routng type-j fows n path when the system s n state y(t). Equaton (2b) foows from (7) and states that the fows can ony be paced on the paths whch have the mnmum margna cost mn Rj (y(t)). It foows from (2a) and (2c) that the tota number of typej fows n the system,.e., x (j) (t) = R j (t), foows a determnstc trajectory descrbed by the foowng equaton, d dt x(j) (t) = λ j µ j x (j) (t), j J, (3) whch ceary mpes that x (j) (t) = ρ j + (x (j) () ρ j )e µjt j J. (4) Consequenty at steady state, x (j) ( ) = ρ j, j J, (5) whch means that, n steady state, there s a tota of ρ j type-j fows on the fud scae. B. Man Resut and Asymptotc Optmaty In ths secton, we state our man resut regardng the asymptotc optmaty of our myopc agorthm. Frst note that by (5), the vaues of y( ) are confned to a convex compact set Υ defned beow Υ {y = ( ) :, R j = ρ j, j J }. (6) Consder the probem of mnmzng the networ cost n steady state on the fud scae (the counterpart of optmzaton (3)), mn F (y) s. t. y Υ (7) Denote by Υ Υ the set of optma soutons to the optmzaton (7). The foowng proposton states that the fud mts of Agorthm ndeed converge to an optma souton of the optmzaton (7). Proposton. Consder the fud mts of the system under Agorthm wth nta condton y(), then as t d(y(t), Υ ). (8)

6 IEEE/ACM TRANSACTIONS ON NETWORKING 6 Convergence s unform over nta condtons chosen from a compact set. The theorem beow maes the connecton between the fud mts and the orgna optmzaton probem (3). It states the man resut of ths paper whch s the asymptotc optmaty of Agorthm. Theorem. Let Y r (t) and Yopt(t) r be respectvey the system trajectores under Agorthm and any optma agorthm for the optmzaton (3). Then n steady state, [ ] m r E F (Y r ( )) [ ] =. (9) E F (Yopt( )) r For exampe, one optma agorthm that soves (3) s the one that every tme a fow arrves or departs, t re-routes the exstng fows n the networ n order to mnmze the networ cost at a tmes. Of course ths requres sovng a compex combnatora probem every tme a fow arrves/departs and further t nterrupts/mgrates the exstng fows. Under any agorthm (ncudng our myopc agorthm and the optma one), the mean number of fows n the system n steady state s O(r). Thus by Theorem, Agorthm has roughy the same cost as the optma cost when the number of fows n the system s arge, but at much ower compexty and wth no mgratons/nterruptons. The rest of ths secton s devoted to the proof of Proposton. The proof of Theorem rees on Proposton and s provded n Secton VII. C. Proof of Proposton We frst characterze the set of optma soutons Υ usng KKT condtons n the emma beow. Lemma 2. Let Γ j = { R j : > } R j, j J. A vector y Υ ff y Υ and there exsts a vector η such that where ( ) defned n (2d). (y) = η j, Γ j, (2a) (y) η j, R j \ Γ j, (2b) Proof of Lemma 2. Consder the foowng optmzaton probem, mn F (y) (2a) s.t. ρ j, j J (2b) R j, j J, R j. (2c) Snce F (y) s an strcty ncreasng functon wth respect to, for a j J, R j, t s easy to chec that the optmzaton (7) has the same set of optma soutons as the optmzaton (2). Moreover, both optmzatons have the same optma vaue. Hence we can use the Lagrange mutpers η j and ν (j) to characterze the Lagrangan as foows. L(η, ν, y) =F (y) + η j (ρ j ) j J ; R j (22) ν (j). j J ; R j From KKT condtons [4], y Υ, f and ony f there exst vectors η and ν such that the foowng hods. Feasbty: y Υ, (23a) η j, ν (j) j J, R j, (23b) Compementary sacness: η j (ρ j Statonarty: ; R j ) =, j J, (24a) ν (j) =, j J, R j, (24b) L(η, ν, y) =. j J, R j. (25a) Note that (23a) mpes (24a). It foows from (25a) that F (y) = η j + ν (j), j J, R j. (26) Defne Γ j as n the statement of the emma. Note that Γ j s nonempty for a j J by (23a). Then combnng (24b) and F (y) (26), j J, and notng that = (y) by defnton, yeds (2a)-(2b). Next, we show that the set of optma soutons Υ s an nvarant set of the fud mts, usng the fud mt equatons (2a)-(2d), and Lemma 2. Lemma 3. Υ s an nvarant set for the fud mts,.e., startng from any nta condton y() Υ, y(t) Υ for a t. Proof of Lemma 3. Consder a type-j fow and et I (j) (t) = arg mn Rj (y(t)) be the set of paths wth the mnmum path margna cost. Note that I (j) (t) p(j) (t) =, t, by (2b), therefore d ( ) ( ) (t) = λ j (t) µ j. (27) dt I (j) (t) I (j) (t) Snce y() Υ, t foows from Lemma 2 that () = ρ j. Hence, Equaton (27) has a unque I (j) () y(j) souton for I (j) (t) y(j) (t) whch s (t) = ρ j, t. (28) I (j) (t) On the other hand, snce x (j) () = ρ j, by (4), x (j) (t) = (t) = ρ j, t. (29) R j

7 IEEE/ACM TRANSACTIONS ON NETWORKING 7 Equatons (28) and 29 mpy that, at any tme t, (t) = for / I (j) (t), and (t) for ( I (j) (t) ) such that I (j) (t) y(j) (t) = ρ j. Hence, y(t) = (t) Υ by usng η j (t) = mn Rj (y(t)) n Lemma 2. Next, we show that the fud mts ndeed converge to the nvarant set Υ startng from an nta condton n Υ. Lemma 4. (Convergence to the nvarant set) Consder the fud mts of the system under Agorthm wth nta condton y() Υ, then d(y(t), Υ ). (3) Aso convergence s unform over the set of nta condtons Υ. Proof of Lemma 4. Startng from y() Υ, (4) mpes that x (j) (t) = (t) = ρ j j J, (3) R j at any tme t. To show convergence of y(t) to the set Υ, we use a Lyapunov argument. Specfcay, we choose F (.) as the Lyapunov functon and show that (d/dt)f (y(t)) < f y(t) / Υ. Let η j (y(t)) = mn Rj (y(t)). Then F (y) R j d (t) dt (d/dt)f (y(t)) = j J = [ µ j ρj (y(t))p (j) (t) (y(t)) (t) ] j J R j R j (a) = [ µ j ρj η j (y(t)) (y(t)) (t) ] (32) j J R j (b) < [ µ j ρj η j (y(t)) η j (y(t)) (t) ] (c) =. j J R j Equaty (a) foows from the fact that p (j) (t) = f (t) > η j (t), and I (j) (t) p(j) (t) =, t, by (2b) and (2c). Inequaty (b) foows from the fact that y(t) / Υ, so by Lemma 2, there exsts an R j such that (t) > but (y(t)) > η j (y(t)). Equaty (c) hods because of (3). Now we are ready to compete the proof of Proposton,.e., to show that startng from any nta condton n a compact set, unform convergence to the nvarant set Υ hods. Proof of Proposton. Frst note that (d/dt)f (y(t)) (as gven by (32)) s a contnuous functon wth respect to y(t) = ( (t) ). Ths s because the path margna costs (y(t)) are contnuous functons of y(t) and so s ther mnmum η j (y(t)) = mn Rj (y(t)). Next, note that by Lemma 4, for any ɛ >, and a Υ, there exsts an ɛ 2 > such that f F (a) F (Υ ) ɛ then, (d/dt)f (y(t)) y(t)=a ɛ 2 (33) By the contnuty of (d/dt)f (y(t)) n y(t), there exsts a δ > such that y(t) a δ mpes, (d/dt)f (y(t)) (d/dt)f (a) ɛ 2 /2 (34) Combnng (33) and (34), for a y(t) such that y(t) a δ, (d/dt)f (y(t)) ɛ 2 /2. By (4), for any δ >, we can fnd t δ arge enough such that for a t > t δ, y(t) a δ for some a Υ. Puttng everythng together, for any ɛ >, there exsts ɛ 2 > such that f F (y(t)) F (Υ ) ɛ then (d/dt)f (y(t)) ɛ 2 /2 <. Appyng Lyapunov argument wth F (.) as Lyapunov functon competes the proof of Proposton. V. SIMULATION RESULTS In ths secton, we provde smuaton resuts and evauate the performance of Agorthm under a wde range of traffc condtons n the foowng datacenter archtectures: FatTree whch conssts of a coecton of edge, aggregaton, and core swtches and offers equa ength path between the edge swtches. Fgure shows a FatTree wth 6 servers and 8 4-port edge swtches. For smuatons, we consder a FatTree wth 28 servers and 32 8-port edge swtches. JeyFsh whch s a random graph n whch each swtch has ports out of whch r ports are used for connecton to other swtches and the remanng r ports are used for connecton to servers. Fgure 2 shows a JeyFsh wth 4-port swtches, and = 4, r = 2 for a the swtches. For smuatons, we consder a JeyFsh constructed usng 2 8-port swtches and servers. Each 8-port swtch s connected to 5 servers and 3 remanng ns are randomy connected to other swtches (ths corresponds to = 8, r = 3 for a the swtches). For the 28-server FatTree, when source and destnaton swtches are ocated n dfferent (same) racs, our myopc agorthm consders 6 (4) equa ength canddate paths. For the case of d-reguar random graphs (where each node has d edges), the number of paths between 2 swtches can be very arge whch coud sgnfcanty ncrease the computatona compexty of the agorthm. To reduce the computaton overhead, we can negect the ong paths snce such paths w naturay have arge margna costs and w not be used by Agorthm. In our smuatons, for the case of JeyFsh, we consder (at most) the frst 2 shortest paths (n terms of the number of ns) for each pars of swtches. Our ratonae for seectng these archtectures stems from the fact that they are on two opposng sdes of the spectrum of topooges: whe FatTree s a hghy structured topoogy, JeyFsh s a random topoogy; hence they shoud provde a good estmate for the robustness of Agorthm to dfferent networ topooges and possbe n faures. We generate the fows under two dfferent traffc modes to whch we refer to as exponenta mode and emprca mode: Exponenta mode: Fows are generated per Posson processes and exponentay dstrbuted duratons. The parameters of duraton dstrbuton s chosen unformy at random from.5 to.5 for dfferent fows to smuate a more dynamc range of fow duratons. The fow szes are chosen accordng to a og-norma dstrbuton.

8 IEEE/ACM TRANSACTIONS ON NETWORKING 8 Normazed Networ Cost.2 CVX Ag Normazed Networ Cost ECMP Ag. CVX Normazed Networ Cost ECMP Ag. CVX Tme (second) (a) Convergence of networ cost Traffc Intensty (b) Exponenta traffc mode Traffc Intensty (c) Emprca traffc mode Fg. 3: Expermenta Resuts for FatTree. (a): Convergence of the networ cost under Agorthm, normazed wth the owerbound on the optma souton (CVX), to. The scang parameter r s here. (b) and (c): Performance rato of Agorthm and ECMP n FatTree, normazed wth the ower-bound (CVX) for exponenta and emprca traffc modes. Emprca mode: Fows are generated based on recent emprca studes on characterzaton of datacenter traffc. As suggested by these studes, we consder ognorma nter-arrva tmes [4], servce tmes based on the emprca resut n [], and og-norma fow szes [4]. Partcuary, the most perods of congeston tend to be short ved, namey, more than 9% of the fows that are more than second ong, are no onger than 2 seconds []. In both modes, the fow szes are og-norma wth mean.2 and standard devaton.4. Ths generates fow szes rangng from % to 4% of n capacty wth hgh probabty to capture the nature of fow szes n terms of mce and eephant fows. Furthermore, we consder a random traffc pattern,.e., source and destnaton of fows are chosen unformy at random. The n cost parameter α s chosen to be n these smuatons. Under both modes, to change the traffc ntensty, we eep the other parameters fxed and scae the arrva rates (wth parameter r). We report the smuaton resuts n terms of the performance rato between Agorthm and a benchmar agorthm (smar to (9)). Snce the optma agorthm (e.g. the one that every tme a fow arrves or departs, t re-routes the exstng fows n the networ n order to mnmze the networ cost at a tmes) s hard to mpement (and even unnown), nstead we use a convex reaxaton method to fnd a ower-bound on the optma cost at each tme. We note that, for FatTree topoogy, equa spttng of every fow among ts canddate paths s optma. For JeyFsh topoogy, every tme a fow arrves or departs, we use CVX [42], to mnmze F (Y (t)), by reaxng the combnatora constrants,.e., aowng spttng of fows among mutpe paths and re-routng the exstng fows. We compare the networ cost under Agorthm and tradtona ECMP (whch statcay assgns fows to the shortest paths (n number of ns) va fow hashng.), normazed by the owerbound on the optma souton (to whch we refer to as CVX n the pots). A. Expermenta Resuts for FatTree Fgure 3a shows that the aggregate cost under Agorthm ndeed converges to the optma souton (normazed cost rato goes to ) whch verfes Theorem. Fgures 3b and 3c show the cost performance under Agorthm and ECMP, normazed by the CVX ower-bound, under the exponenta and the emprca traffc modes respectvey. The traffc ntensty s measured n terms of the rato between the steady state offered oad and the bsecton bandwdth. For FatTree, the bsecton bandwdth depends on the number of core swtches and ther number of ports. As we can see, our myopc agorthm s very cose to the ower-bound on the optma vaue (CVX) for ght, medum, and hgh traffc ntenstes. As t s shown, the performance mproves at hgher traffc ntenstes whch correspond to arger vaues of r n Theorem. They aso suggest that Theorem hods under more genera arrva and servce tme processes. In ths smuatons, Agorthm gave a performance mprovement rangng form 5% to more than %, compared to ECMP, dependng on the traffc ntensty, under the emprca traffc mode. The standard devaton (SD) of performance rato for 3 dfferent runs ranges from.4 to. for Agorthm, and from.3 to.3 for ECMP as traffc ntensty grows. B. Expermenta Resuts for JeyFsh Fgure 4a shows that the aggregate cost under Agorthm ndeed converges to the optma souton whch agan verfes Theorem. Fgures 4b and 4c compare the performance of Agorthm and ECMP, normazed wth the ower-boud on the optma souton (CVX), under both the exponenta and emprca traffc modes. As before, the traffc ntensty s measured by the rato between the steady state offered oad and the bsecton bandwdth. To determne the bsecton bandwdth, we have used the bounds reported n [43], [44] for reguar random graphs. Agan we see that our myopc agorthm performs very we n a ght, medum, and hgh traffcs. In JeyFsh, Agorthm yeds performance gans rangng from 6% to 7%, compared to ECMP, under the emprca traffc mode. Correspondng SD for 3 dfferent

9 IEEE/ACM TRANSACTIONS ON NETWORKING 9 Normazed Networ Cost CVX Ag. Normazed Networ Cost ECMP Ag. CVX Normazed Networ Cost ECMP Ag. CVX Tme (second) (a) Convergence of networ cost Traffc Intensty (b) Exponenta traffc mode Traffc Intensty (c) Emprca traffc mode Fg. 4: Expermenta Resuts for JeyFsh. (a): Convergence of the networ cost under Agorthm n JeyFsh, normazed wth the ower-bound on the optma souton (CVX), to. The scang parameter r s here. (b) and (c): Performance rato of Agorthm and ECMP n JeyFsh, normazed wth the ower-bound (CVX) for exponenta and emprca traffc modes. runs ranges from.4 to. for Agorthm, and from. to.5 for ECMP as traffc ntensty grows. VI. RANDOMIZED MYOPIC ALGORITHMS Agorthm needs to consder a the avaabe paths for an arrvng fow and fnds the shortest path based on the (ntegra (5) or dfferenta (6)) margna cost of paths. In ths secton, we descrbe and emprcay evauate randomzed versons of our myopc agorthm whch have ess compexty than Agorthm, whe can effectvey provde a arge fracton of the performance gan obtaned by Agorthm. Our approach s motvated by the terature on randomzed oad baancng for schedung jobs n servers, where a wdey used dea s that, nstead of consderng a the servers and assgnng the arrvng job to the east-oaded server, servers are frst chosen at random (for some 2) and then the job s assgned to the east-oaded server among them. Ths dea was orgnay proposed n [45], where t was shown that havng = 2 eads to exponenta mprovement n the expected tme a job spends n the system over = whch s bascay the totay random assgnment. In our settng, a counterpart of ths approach can be used for schedung of fows n paths as foows. Fx, when a fow s generated, the agorthm chooses paths at random out of the avaabe paths for the fow, then cacuates the margna costs of these paths accordng to the ntegra or the dfferenta form formuas, and assgns the fow to the path wth the mnmum path margna cost among these paths. See Agorthm 2 for the fu descrpton. We notce that ECMP n structured topooges e FatTree, where a canddate paths for an arrvng fow have the same number of ns (same ength), s bascay the random assgnment of fows to the paths whch s dentca to settng = n Agorthm 2. Next, we emprcay evauate the performance of Agorthm 2 for dfferent vaues of. We present the resuts for two dfferent topooges and two traffc mode as n Secton V. For JeyFsh, we consder (at most) the frst 2 shortest paths (n terms of the number of ns) for each pars of swtches to be consstent wth Secton V. Agorthm 2 Randomzed Myopc Agorthm wth Parameter Suppose a type-j fow arrves at tme t when the system s n state Y(t). Then, : Choose paths from the set R j, unformy at random, et R () j denotes ths subset of paths. 2: Compute the path margna costs (Y (t)), R () j, n ether of the forms beow: Integra form: Dfferenta form: (Y (t)) = (j) (Y (t)), (35) (Y (t)) = δ (j) (Y (t)). (36) 3: Pace the fow on a path such that = arg mn R () j Brea tes n (37) unformy at random. A. Expermenta Resuts for FatTree (Y (t)). (37) Fgures 5 and 6 show the cost performance under Agorthm 2 wth dfferent vaues of, normazed by the cost of Agorthm, under the exponenta and the emprca traffc modes respectvey. Note that Agorthm 2 wth = 6 s equvaent to Agorthm, as there are at most 6 avaabe paths for an arrvng fow n the FatTree topoogy we descrbed n Secton V. Error bars n a pots correspond to standard devaton of normazed mean networ cost computed from resuts of 3 runs. In these two pots, we can see that the maxmum mprovement n networ cost we get by ncreasng happens at = 2 compared wth random assgnment of fows, =. Furthermore, as we ncrease vaue of we get smaer mprovement n performance. For nstance, normazed cost mproves about.4 by ncreasng from to 2, whe the mprovement from = 2 to = 4 s about., for traffc ntensty equa to.3 under exponenta mode (Fgure 5). Ths behavor s seen n

10 IEEE/ACM TRANSACTIONS ON NETWORKING Normazed Networ Cost Traffc Intensty = =2 =4 =8 =2 Ag. Fg. 5: Performance of Agorthm 2 wth dffrent vaues of, n FatTree under the exponenta traffc mode, normazed wth the Agorthm. Normazed Networ Cost Traffc Intensty = =2 =4 =8 =2 Ag. Fg. 7: Performance of Agorthm 2 wth dfferent vaues of n JeyFsh under the exponenta traffc mode, normazed wth the Agorthm. Normazed Networ Cost Traffc Intensty = =2 =4 =8 =2 Ag. Fg. 6: Performance of Agorthm 2 wth dfferent vaues of, n FatTree under the emprca traffc mode, normazed wth the Agorthm. Normazed Networ Cost Traffc Intensty = =2 =4 =8 =2 Ag. Fg. 8: Performance of Agorthm 2 wth dfferent vaues of n JeyFsh under the emprca traffc mode, normazed wth the Agorthm. both fgures, and s more profound for hgher traffc ntensty. B. Expermenta Resuts for JeyFsh Fgures 7 and 8 show the networ cost under Agorthm 2 wth dfferent vaues of, normazed by the cost of Agorthm, under the exponenta and the emprca traffc modes respectvey. Note that Agorthm 2 wth = 2 s equvaent to Agorthm, as there are at most 2 avaabe paths consdered between any two swtches n the JeyFsh topoogy we descrbed n Secton V. In these fgures, we observe the same behavor as what dscussed for FatTree: the performance mprovement obtaned by ncreasng by one s arger for smaer. Aso, comparng Fgures 7 and 8 wth Fgures 4b and 4c, n order for Agorthm 2 to beat ECMP whch ony consders shortest paths (n the terms of the number of ns) we need to choose 2. We aso note that n JeyFsh, for sma (e.g., =, 2), the normazed cost under the randomzed agorthm ncreases as traffc ntensty grows, une the resuts for FatTree. Ths can be justfed by notng that the symmetrc structure of FatTree aows random assgnment of fows to baance the oad better as traffc ntensty ncreases (hgher fow arrva rates) because the number of fow-to-path assgnment decsons ncreases. However, n JeyFsh the structure s asymmetrc and ong paths are used more frequenty by the randomzed agorthm as traffc ntensty ncreases. As a resut, the convexty of the n cost functon, and the fact that the networ cost s the summaton of a ns costs, w cause a arger networ cost n hgher traffc ntenstes. Based on the smuatons, we concude that to get a reasonaby good performance, we need smaer vaues of n FatTree compared to JeyFsh. Ths can be attrbuted to the fact that a the canddate paths for a fow n the FatTree topoogy have the same number of ns, whe n the JeyFsh topoogy, paths can be very dfferent n terms of ther number of ns. So seecton of paths competey at random, as used n Agorthm 2, mght ead to usng ong paths whch contrbute more to the networ cost. Thus, unform sampng seems more sutabe for symmetrc topooges e FatTree. We postpone the exact anayss of the randomzed myopc pocy to a future wor. VII. FORMAL PROOFS OF FLUID LIMITS AND THEOREM A. Proof of Fud Lmts We prove the exstence of fud mts under Agorthm and derve the correspondng fud equatons (2a)-(2d). Arguments n ths secton are qute standard [37], [46], [47].

11 IEEE/ACM TRANSACTIONS ON NETWORKING Reca that Y r (t) s the system state wth the fow arrva rate rλ j, j J, and nta state Y r (). The fud-scae process s y r (t) = Y r (t)/r, t [, ). Smary, z r(t) = Zr (t)/r and x (j)r (t) = X (j)r (t)/r are defned. We assume that y r () y() as r for some fxed y(). We frst show that, under Agorthm, the mt of the process {y r (t)} t exsts aong a subsequence of r as we show next. The process Y r (t) can be constructed as foows Y (j) r (t) =Y (j) Π d,j( r () + Π a,j ( t t P (j) (Y r (s))rλ j ds) (38) µ j Y (j) r (s)ds), j J, Rj where Π a,j (.) and Πd,j (.) are ndependent unt-rate Posson processes, and P (j) (Y r (t)) s the probabty of assgnng a type-j fow to path when the system state s Y r (t). Note that by the Functona Strong Law of Large Numbers [48], amost surey, r Πa,j(rt) t, u.o.c.; r Πd,j(rt) t, u.o.c. (39) where u.o.c. means unformy over compact tme ntervas. Defne the fud-scae arrva and departure processes as a r,j(t) = r Πa,j( t d r,j(t) = r Πd,j( P (j) t (Y r (s))rλ j ds), (4) µ j Y (j) r (s)ds). Lemma 5. (Convergence to fud mt sampe paths) If y r () y(), then amost surey, every subsequence (y rn, a rn, d rn ) has a further subsequence (y rn, a rn, d rn ) such that (y rn, a rn, d rn ) (y, a, d). The sampe paths y, a, d are Lpschtz contnuous and the convergence s u.o.c. Proof of Lemma 5. The proof s standard and foows from the fact that a r,j (.) and dr,j (.) are asymptotcay Lpschtz contnuous (see e.g., [37], [46], [49] for smar arguments), namey, there exsts a constant C > such that for t t 2 <, m sup(a r,j(t 2 ) a r,j(t )) C(t 2 t ), (4) r and smary for d r,j (.). More precsey, for arrva process (.), we argue that, a r,j m sup(a r,j(t 2 ) a r,j(t )) r = m sup r r Πa,j( t2 (a) ( t 2 m sup r r Πa,j rλ j ds ) t t P (j) (Y r (s))rλ j ds) = m sup( r r Πa,j(rλ j (t 2 t ))) where nequaty (a) foows from the fact that P (j) (Y r (s)). Usng (39), we obtan (4). The argument s smar for d r,j (.), notng that (yr (.)) s unformy bounded over any fnte tme nterva for arge r. So the mt (y, a, d) exsts aong the subsequence. Proof of Lemma. It foows from (38), (4), (39), and the exstence of the fud mts (Lemma 5), that (t) = () + a (j) (t) d (j) (t), where d (j) (t) = t y(j) (s)µ j ds, and R j a (j) (t) = λ j t, a (j) (t) s nondecreasng. The fud equatons (2a) and (2c) are the dffrenta form of these equatons (the fud sampe paths are Lpschtz contnuous so the dervatves exst amost everywhere), where p (j) (t) := da (j) λ j dt (t). (42) For any type j, and for (y(t)) defned n (2d), et w j (y(t)) = mn R j (y(t)). Consder any reguar tme t and a path / arg mn Rj (y(t)). By the contnuty of (y(t)), there must exst a sma tme nterva (t, t 2 ) contanng t such that (y(τ)) > w j (τ) τ (t, t 2 ). Consequenty, for a r arge enough aong the subsequence, (y r (τ)) > w j (y r (τ)) τ (t, t 2 ). Mutpyng both sdes by r α, t foows that (Y r (τ)) > w j (Y r (τ)), τ (t, t 2 ). Hence P (j), for a r arge enough aong the subsequence. Therefore a (j) (Y r (τ)) =, τ (t, t 2 ), and a r(j) (t, t 2 ) = (t, t 2 ) = whch shows that (d/dt)a (j) (t, t 2 ). Ths estabshes (2b). B. Proof of Theorem We frst show that (t) = at t F (y r ( )) = F, (43) where F = F (Υ ) s the optma cost. By Proposton and the contnuty of F ( ), for any fud sampe path y(t) wth nta condton y(), we can choose t ɛ arge enough such that gven any sma ɛ >, F (y(t ɛ )) F ɛ. Wth probabty, every subsequence y rn has a further subsequence y rn such that y rn (t) y(t) u.o.c. (see Lemma 5), hence, by the contnuous mappng theorem [48], we aso have F (y rn (t)) F (y(t)), u.o.c. For any ɛ 2 >, for r n arge enough, we can choose an ɛ 3 > such that, unformy over a nta states y rn () such that y rn () y() ɛ 3, P{ F (y rn (tɛ ) F (y(t ɛ )) < ɛ } > ɛ 2 (44) Ths cam s true, snce otherwse for a sequence of nta states y rn () y() we have P{ F (y rn (tɛ ) F (y(t ɛ )) < ɛ } ɛ 2, whch s mpossbe because, amost surey, we can choose a subsequence of r n aong whch unform convergence

12 IEEE/ACM TRANSACTIONS ON NETWORKING 2 F (y rn (t)) F (y(t)), wth nta condton y() hods. Hence, P{ F (y rn (tɛ )) F < 2ɛ } P{ F (y rn (tɛ ) F (y(t ɛ )) + F (y(t ɛ )) F < 2ɛ } P{ F (y rn (tɛ ) F (y(t ɛ )) < ɛ } > ɛ 2 whch n partcuar mpes Y (j) R j F (y rn ( )) = F, because ɛ and ɛ 2 can be made arbtrary sma. Hence, we have shown that every sequence F (y rn ( )) has a further subsequence F (y rn ( )) that converges to the same mt F (the unque optma cost). Therefore n vew of Theorem 2.6 of [48], we can concude that F (y r ( )) = F. Next, we show (9). Under any agorthm (ncudng Agorthm and the optma one), r ( )/r = X (j) r ( )/r, where X (j)r ( ) has Posson dstrbuton wth mean rρ j, and X (j)r ( ), j J, are ndependent. Let, s = max j J s j <. The traffc over each n s ceary bounded as Z r /r < s j X (j)r ( )/r = sx r ( )/r, where X r ( ) has Posson dstrbuton wth mean r j ρ j. Hence, F (y r ( )) s stochastcay domnated by E g ( sxr ( )/r ), and g s poynoma. It then foows that the sequence of random varabes {F (y r ( ))} (and aso {y r ( )}) are unformy ntegrabe under any agorthm. Then, n vew of (43), by Theorem 3.5 of [48], under our Agorthm. [ ] E F (Y r ( )/r) F. (45) Now consder any optma agorthm for the optmzaton (3). It hods that [ ] [ ] [ ] F (E yopt( ) r ) E F (yopt( )) r E F (y r ( )), where the frst nequaty s by Jensen s nequaty, and the second foows from defnton of optmaty. Tang the mt as r, t foows by an squeeze argument that [ ] E F (Yopt( )/r) r F. (46) Fnay, (45) and (46) w mpy (9) n vew of the poynoma structure of F. VIII. CONCLUDING REMARKS Ths paper presents a smpe myopc agorthm that dynamcay adjusts the n weghts as a functon of the n congestons and paces any newy generated fow on a east weght path n the networ, wth no spttng/mgraton of exstng fows. We demonstrate both theoretcay and expermentay that ths myopc agorthm has a good oad baancng performance. In partcuar, we prove that the agorthm asymptotcay mnmzes a networ cost and estabsh the reatonshp between the networ cost and the correspondng weght construct. Athough our theoretca resut s an asymptotc resut, our expermenta resuts show that the agorthm n fact performs very we under a wde range of traffc condtons and dfferent datacenter networs. Whe the agorthm has ow compexty, the rea mpementaton depends on how fast the weght updates and east weght paths can be computed n practca datacenters (e.g., based on SDN). One possbe way to mprove the computaton tmescae s to perform the computaton perodcay or ony for ong fows, whe usng the prevousy computed east weght paths for short fows or between the perodc updates. Another possbty s to use the randomzed versons of our myopc agorthm wth an optmzed parameter whch ony taes a sma random subset of avaabe paths nto account and fnds the shortest path among them. Whe ths agorthm has much ower compexty, t performs very we n structured topooges such as FatTree for sma. We eave theoretca anayss of the randomzed versons as an open probem for future wor. Fnay, we woud e to note that our myopc agorthm and ts randomzed versons can be drecty apped to schedung fowets nstead of schedung fows, whch can gve hgher rate/granuarty of fows [6], [36]. REFERENCES [] M. Shafee and J. Ghader, A smpe congeston-aware agorthm for oad baancng n datacenter networs, n Computer Communcatons, IEEE INFOCOM 26-The 35th Annua IEEE Internatona Conference on. IEEE, 26, pp. 9. [2] M. A-Fares, A. Loussas, and A. Vahdat, A scaabe, commodty data center networ archtecture, ACM SIGCOMM Computer Communcaton Revew, vo. 38, no. 4, pp , 28. [3] T. Benson, A. Anand, A. Aea, and M. Zhang, McroTE: Fne graned traffc engneerng for data centers, n Proceedngs of the 7th Conference on Emergng Networng Experments and Technooges. ACM, 2, p. 8. [4] M. A-Fares, S. Radharshnan, B. Raghavan, N. Huang, and A. Vahdat, Hedera: Dynamc fow schedung for data center networs. n NSDI, vo., 2, pp [5] C. Racu, S. Barre, C. Punte, A. Greenhagh, D. Wsch, and M. Handey, Improvng datacenter performance and robustness wth mutpath TCP, ACM SIGCOMM Computer Communcaton Revew, vo. 4, no. 4, pp , 2. [6] S. Kandua, D. Katab, S. Snha, and A. Berger, Dynamc oad baancng wthout pacet reorderng, ACM SIGCOMM Computer Communcaton Revew, vo. 37, no. 2, pp. 5 62, 27. [7] A. Greenberg, J. R. Hamton, N. Jan, S. Kandua, C. Km, P. Lahr, D. A. Matz, P. Pate, and S. Sengupta, VL2: A scaabe and fexbe data center networ, n ACM SIGCOMM Computer Communcaton Revew, vo. 39, no. 4, 29, pp [8] C. Guo, G. Lu, D. L, H. Wu, X. Zhang, Y. Sh, C. Tan, Y. Zhang, and S. Lu, BCube: A hgh performance, server-centrc networ archtecture for moduar data centers, ACM SIGCOMM Computer Communcaton Revew, vo. 39, no. 4, pp , 29. [9] M. Bradonjć, I. Sanee, and I. Wdjaja, Scang of capacty and reabty n data center networs, ACM SIGMETRICS Performance Evauaton Revew, vo. 42, no. 2, pp , 24. [] A. Snga, C.-Y. Hong, L. Popa, and P. B. Godfrey, Jeyfsh: Networng data centers randomy. n NSDI, vo. 2, 22, pp [] S. Kandua, S. Sengupta, A. Greenberg, P. Pate, and R. Chaen, The nature of data center traffc: Measurements & anayss, n Proceedngs of the 9th ACM SIGCOMM Conference On Internet Measurement Conference, 29, pp [2] A. Dxt, P. Praash, Y. C. Hu, and R. R. Kompea, On the mpact of pacet sprayng n data center networs, n Proceedngs of IEEE, INFOCOM, 23, pp

13 IEEE/ACM TRANSACTIONS ON NETWORKING 3 [3] S. Even, A. Ita, and A. Shamr, On the compexty of tme tabe and mut-commodty fow probems, n 6th Annua Symposum on Foundaton of Computer Scence. IEEE, 975, pp [4] G. M. Gusewte and P. M. Pardaos, Mnmum concave-cost networ fow probems: Appcatons, compexty, and agorthms, Annas of Operatons Research, vo. 25, no., pp , 99. [5] Y. Dntz, N. Garg, and M. X. Goemans, On the snge-source unspttabe fow probem, n Foundatons of Computer Scence, 998. Proceedngs. 39th Annua Symposum on. IEEE, 998, pp [6] R. Nranjan Mysore, A. Pambors, N. Farrngton, N. Huang, P. Mr, S. Radharshnan, V. Subramanya, and A. Vahdat, Portand: A scaabe faut-toerant ayer 2 data center networ fabrc, n ACM SIGCOMM Computer Communcaton Revew, vo. 39, no. 4, 29, pp [7] J. Cao, R. Xa, P. Yang, C. Guo, G. Lu, L. Yuan, Y. Zheng, H. Wu, Y. Xong, and D. Matz, Per-pacet oad-baanced, ow-atency routng for cos-based data center networs, n Proceedngs of the 9th ACM Conference on Emergng Networng Experments and Technooges. ACM, 23, pp [8] P. G, N. Jan, and N. Nagappan, Understandng networ faures n data centers: Measurement, anayss, and mpcatons, n ACM SIGCOMM Computer Communcaton Revew, vo. 4, no. 4, 2, pp [9] M. Chesa, G. Knder, and M. Schapra, Traffc engneerng wth equacost-mutpath: An agorthmc perspectve, IEEE/ACM Transactons on Networng, vo. 25, no. 2, pp , 27. [2] S. Sen, D. Shue, S. Ihm, and M. J. Freedman, Scaabe, optma fow routng n datacenters va oca n baancng, n Proceedngs of the 9th ACM Conference on Emergng Networng Experments and Technooges, 23, pp [2] J. W. Jang, T. Lan, S. Ha, M. Chen, and M. Chang, Jont VM pacement and routng for data center traffc engneerng, n Proceedngs of IEEE, INFOCOM, 22, pp [22] K. He, E. Rozner, K. Agarwa, W. Feter, J. Carter, and A. Aea, Presto: Edge-based oad baancng for fast datacenter networs, n Proceedngs of the 25 ACM Conference on Speca Interest Group on Data Communcaton, pp [23] R. Gaager, A mnmum deay routng agorthm usng dstrbuted computaton, IEEE transactons on communcatons, vo. 25, no., pp , 977. [24] N. Mchae and A. Tang, Hao: Hop-by-hop adaptve n-state optma routng, IEEE/ACM Transactons on Networng (TON), vo. 23, no. 6, pp , 25. [25] D. Xu, M. Chang, and J. Rexford, Ln-state routng wth hop-byhop forwardng can acheve optma traffc engneerng, IEEE/ACM Transactons on networng, vo. 9, no. 6, pp , 2. [26] R. W. Rosentha, A cass of games possessng pure-strategy nash equbra, Internatona Journa of Game Theory, vo. 2, no., pp , 973. [27] N. Nsan, T. Roughgarden, E. Tardos, and V. V. Vazran, Agorthmc game theory. Cambrdge Unversty Press Cambrdge, 27, vo.. [28] T. Roughgarden, Sefsh routng and the prce of anarchy. MIT press Cambrdge, 25, vo. 74. [29] P. Key, L. Massoué, and D. Towsey, Path seecton and mutpath congeston contro, n INFOCOM th IEEE Internatona Conference on Computer Communcatons. IEEE. IEEE, 27, pp [3] J. G. Wardrop, Road paper. some theoretca aspects of road traffc research. Proceedngs of the nsttuton of cv engneers, vo., no. 3, pp , 952. [3] D. Appegate and E. Cohen, Mang ntra-doman routng robust to changng and uncertan traffc demands: Understandng fundamenta tradeoffs, n Proceedngs of the 23 conference on Appcatons, technooges, archtectures, and protocos for computer communcatons. ACM, 23, pp [32] H. Wang, H. Xe, L. Qu, Y. R. Yang, Y. Zhang, and A. Greenberg, Cope: traffc engneerng n dynamc networs, n ACM SIGCOMM Computer Communcaton Revew, vo. 36, no. 4. ACM, 26, pp. 99. [33] M. Benows, M. Korzenows, and H. Räce, A practca agorthm for constructng obvous routng schemes, n Proceedngs of the ffteenth annua ACM symposum on Parae agorthms and archtectures. ACM, 23, pp [34] N. McKeown, T. Anderson, H. Baarshnan, G. Paruar, L. Peterson, J. Rexford, S. Shener, and J. Turner, OpenFow: Enabng nnovaton n campus networs, ACM SIGCOMM Computer Communcaton Revew, vo. 38, no. 2, pp , 28. [35] M. Casado, M. J. Freedman, J. Pettt, J. Luo, N. Gude, N. McKeown, and S. Shener, Rethnng enterprse networ contro, IEEE/ACM Transactons on Networng (TON), vo. 7, no. 4, pp , 29. [36] M. Azadeh, T. Edsa, S. Dharmapurar, R. Vadyanathan, K. Chu, A. Fngerhut, F. Matus, R. Pan, N. Yadav, G. Varghese et a., CONGA: Dstrbuted Congeston-Aware Load Baancng for Datacenters, n Proceedngs of the 24 ACM conference on SIGCOMM, 24, pp [37] A. L. Stoyar, An nfnte server system wth genera pacng constrants, Operatons Research, vo. 6, no. 5, pp. 2 27, 23. [38] M. Chesa, G. Knder, and M. Schapra, Traffc engneerng wth equacost-mutpath: An agorthmc perspectve, n Proceedngs of IEEE, INFOCOM, 24, pp [39] B. Fortz and M. Thorup, Internet traffc engneerng by optmzng OSPF weghts, n Proceedng of 9th annua jont conference of the IEEE computer and communcatons socetes. INFOCOM 2, vo. 2, pp [4] S. Boyd and L. Vandenberghe, Convex optmzaton. Cambrdge unversty press, 24. [4] D. Ersoz, M. S. Yousf, and C. R. Das, Characterzng networ traffc n a custer-based, mut-ter data center, n ICDCS 7. 27th Internatona Conference on Dstrbuted Computng Systems, 27. IEEE, pp [42] M. Grant and S. Boyd, CVX: Matab software for dscpned convex programmng, verson 2., Mar. 24. [43] J. Díaz, M. J. Serna, and N. C. Wormad, Bounds on the bsecton wdth for random d-reguar graphs, Theoretca Computer Scence, vo. 382, no. 2, pp. 2 3, 27. [44] B. Boobás, Random graphs. Sprnger, 998. [45] M. Mtzenmacher, The power of two choces n randomzed oad baancng, IEEE Transactons on Parae and Dstrbuted Systems, vo. 2, no., pp. 94 4, 2. [46] A. L. Stoyar and Y. Zhong, Asymptotc optmaty of a greedy randomzed agorthm n a arge-scae servce system wth genera pacng constrants, Queueng Systems, vo. 79, no. 2, pp. 7 43, 25. [47] J. Ghader, Y. Zhong, and R. Srant, Asymptotc optmaty of BestFt for stochastc bn pacng, ACM SIGMETRICS Performance Evauaton Revew, vo. 42, no. 2, pp , 24. [48] P. Bngsey, Convergence of probabty measures, 2nd ed. John Wey & Sons, 999. [49] S. N. Ether and T. G. Kurtz, Marov processes: Characterzaton and convergence. John Wey & Sons, 29, vo Mehrnoosh Shafee joned M.Sc./ Ph.D. program of the Department of Eectrca engneerng at Coumba n August 24. She s nterested n the anayss and desgn of resource aocaton agorthms for arge-scae dstrbuted systems. She dd her B.Sc. n EE department of Sharf Unversty of Technoogy, Tehran, Iran. Javad Ghader joned the Department of Eectrca Engneerng at Coumba Unversty n Juy 24. Hs research nterests ncude networ agorthms and networ contro and optmzaton. Dr. Ghader receved hs B.Sc. from the Unversty of Tehran, Iran, n 26, hs M.Sc. from the Unversty of Wateroo, Canada, n 28, and hs Ph.D. from the Unversty of Inos at Urbana-Champagn (UIUC), n 23, a n Eectrca and Computer Engneerng. He spent a one-year Smons Postdoctora Feowshp at the Unversty of Texas at Austn before jonng Coumba. He s the recpent of the Mac Van Vaenburg Graduate Research Award at UIUC, and Best Student Paper Fnast at the 23 Amercan Contro Conference, and Best Paper Award at CoNEXT 26, and NSF CAREER award n 27.