Load Balancing Without Regret in the Bulletin Board Model

Transcription

1 Nonam manuscript No will b insrtd by th ditor Load Balancing Without Rgrt in th Bulltin Board Modl Robrt Klinbrg Gorgios Piliouras Éva Tardos Rcivd: dat / Accptd: dat Abstract W analyz th prformanc of protocols for load balancing in distributd systms basd on no-rgrt algorithms from onlin larning thory Ths protocols trat load balancing as a rpatd gam and apply algorithms whos avrag prformanc ovr tim is guarantd to match or xcd th avrag prformanc of th bst stratgy in hindsight Our approach capturs two major aspcts of distributd systms First, in our stting of atomic load balancing, vry singl procss can hav a significant impact on th prformanc and bhavior of th systm Furthrmor, although in distributd systms participants can qury th currnt stat of th systm thy cannot rliably prdict th ffct of thir actions on it W addrss this issu by considring load balancing gams in th bulltin board modl, whr play- Robrt Klinbrg was supportd by NSF awards CCF , IIS , and AF , a Microsoft Rsarch Nw Faculty Fllowship, and an Alfrd P Sloan Foundation Fllowship Gorgios Piliouras was supportd by NSF grants CCF , AF , AFOSR grant FA and ONR grant N Éva Tardos was supportd by NSF grants CCF , AF , CCF , ONR grant N , and a Yahoo! Rsarch Allianc Grant Robrt Klinbrg Éva Tardos Dpartmnt of Computr Scinc Cornll Univrsity, Ithaca, NY {rdk, va}@cscornlldu Gorgios Piliouras School of Elctrical and Computr Enginring Gorgia Institut of Tchnology, Altanta, GA Dpartmnt of Economics Johns Hopkins Univrsity, Baltimor, MD gorgiospiliouras@gmailcom rs can find out th dlay on all machins, but do not hav information on what thir xprincd dlay would hav bn if thy had slctd anothr machin W show that undr ths mor ralistic assumptions, if all playrs us th wll-known multiplicativ wights algorithm, thn th quality of th rsulting solution is xponntially bttr than th worst corrlatd quilibrium, and almost as good as that of th worst Nash Ths tightr bounds ar drivd from analyzing th dynamics of a multi-agnt larning systm Kywords Larning Thory Gam Thory Pric of Anarchy 1 Introduction Gam thory provids a framwork that hlps us undrstand nvironmnts whr participants intract by slfishly making dcisions, and achiv a global outcom without xplicit coordination by a singl global dsignr Modling various problms from routing, ntwork dsign, and schduling as gams playd by slfish agnts has ld to many intrsting rsults Much of this litratur uss Nash quilibrium as th solution concpt, i, dfins Nash quilibrium as th outcom in a comptitiv gam Howvr, Nash quilibrium has svral drawbacks that call into qustion its plausibility as a prdiction of a gam s outcom First, th solution concpt tlls us nothing about th dynamics by which playrs can b xpctd to rach an quilibrium In most gams, natural gam play tnds not to convrg to Nash quilibria In fact, th problm of computing Nash quilibria in many gams turns out to b computationally hard it was rcntly shown to b PPAD-complt [9, 7] If computing quilibria is computationally hard, it sms unrasonabl to assum that

2 2 Robrt Klinbrg t al playrs will find such a solution Furthr, most gams hav many quilibria, hnc finding on would also involv coordination among th playrs to agr on on of th possibl outcoms To ovrcom ths drawbacks, rsarchrs hav considrd altrnativ solution concpts basd on th longrun avrag outcom of slf-adapting agnts who ract to ach othr s stratgis in rpatd play of th gam Adopting a paradigm known as sophisticatd larning in th conomics litratur [17], w assum in this papr that all playrs us no-rgrt stratgis No-rgrt algorithms hav th proprty that in any rpatd gam play, thir avrag loss pr tim stp approachs that of th singl bst stratgy with hindsight or bttr ovr tim Rgrt minimization can b don via simpl and fficint stratgis, and yt th no-rgrt proprty is analogous to th notion of quilibrium g, s th survy of Blum and Mansour [5] Outcom distributions rachd by such no-rgrt stratgis hav bn wll studid, s [6, 17, 29] If all playrs play stratgis using no-rgrt algorithms, it rsults in an mpirical distribution of play convrging to a wakr st of quilibria: th coars wak corrlatd quilibria[29, 26] Fostr and Vohra [13] introducd th strongr notion of intrnal rgrt, such that no-intrnal-rgrt algorithms convrg to th st of corrlatd quilibria Our intrst in this papr is th quality of outcoms whn playrs play simpl rgrt-minimizing stratgis without accss to full information about th gam Much of th work in algorithmic gam thory concrns th pric of anarchy, dfind as th ratio of th worst Nash quilibrium to th bst outcom, with rspct to som global quality masur of th solutions Blum t al in [3] wr th first to considr th quality of outcom rachd whn playrs using no-rgrt larning stratgis Thy considr congstion gams in th Wardrop nonatomic stting of infinitsimal agnts, and hnc th action of a singl playr dosn t hav significant impact on th systm In this stting thy xtnd th pric of anarchy rsults known for Nash quilibria, to outcoms rachd by no-rgrt larning Furthrmor, Blum at al [4] dfind th pric of total anarchy as th ratio of th worst outcom that can b rachd by rgrt minimizing playrs 1 to th bst outcom Blum t al [3,4] show that in som classs of gams rgrtminimizing playrs xhibit bhavior or global quality that is clos to that of a Nash quilibrium Mor rcntly, Roughgardn [27] showd that for a wid class of gams including congstion gams that satisfy a natural smoothnss condition bounds on th pric of anarchy automatically xtnd to th total pric of anarchy, whn th global quality is dfind to b th avrag cost 1 which is quivalnt to th worst corrlatd quilibrium Unfortunatly, in many othr classs of gams and undr natural global quality masurs, th pric of total anarchy can b significantly wors than th pric of anarchy, which suggsts that gnric no-rgrt larning is not ffctiv for ths gams In fact, th simplst such gam is a load balancing gam in which thr ar n jobs and n machins Each job slcts a machin on which to run, and valuats th outcom as th load of its machin, i th numbr of jobs on th machin it slctd For this class of gams th quality of th worst corrlatd quilibrium is Θ n [4], whras a squnc of paprs [22, 8, 23] shows that th worst Nash quilibrium is th symmtric fully mixd quilibrium, which has xponntially bttr quality, namly Θlog n/ log log n In distributd systms w nd to considr a furthr sourc of difficulty Traditional larning thory assums that aftr playing a round of a gam, ach playr can discovr th cost of ach possibl stratgy thy could hav usd givn th actions of thir opponnts This is a rasonabl assumption in gams with infinitsimally small playrs, whn actions of a singl playr hav ssntially no ffct on th systm It is also rasonabl whn th undrlying gam is wll-dfind and common knowldg amongst all playrs In distributd systms, howvr, this assumption is rathr unnatural Indd, diffrnt subsystms only nd to shar som common functionality, whras thir innr workings can vary widly and vn b updatd samlssly, and vry singl procss can hav significant impact on th bhavior of th systm Dspit th pssimistic thortical prdictions mntiond abov and th vn mor dmanding stting of applid systms, simpl and intuitiv adaptiv procdurs sm to work rasonably wll in practic In this work and in a rlatd papr [20], w analyz modls of such ntwork dynamics: w xplor th quality of outcoms rachd by som concrt larning stratgis W focus on th multiplicativ wights larning algorithm also known as Hdg [16], which is arguably th simplst and most intuitiv no-rgrt algorithm In [20] w study th dynamics of ths algorithms in atomic congstion gams in th traditional full information stting W show that in almost all such gams, th multiplicativ-wights larning algorithm rsults in convrgnc to pur quilibria In th gam of [20], ach procss consums non-ngligibl systm rsourcs and as a rsult can hav a significant impact on th prformanc and bhavior of th whol systm Moving away from th assumption of infinitsimally small usrs as in [3,11] adds a significant layr of complxity to th analysis of th systm How much closr can w gt to ralistic modls of applid systms bfor w bcom

3 Load Balancing Without Rgrt in th Bulltin Board Modl 3 ovrwhlmd by th complxity of th mrging dynamics [28]? In this papr, w tak a significant additional stp towards modling distributd systms, by moving away from th standard full information stting W considr load balancing in th so-calld bulltin board modl similar to th ons in [2,25] In this modl playrs can find out th dlay on all machins, but do not hav information on what thir xprincd dlay would hav bn if thy had slctd anothr machin Namly, playrs can qury th currnt stat of th systm but cannot rliably prdict th ffct of thir actions on it Although this chang in th playrs information might appar bnign at first glanc, it significantly altrs th systm bhavior Most importantly, th systm bcoms symmtric bcaus all playrs obsrv th sam fdback signal and rspond to it using idntical algorithms Thus, at any point in tim th playrs will sampl thir stratgis from idntical distributions, and our analysis only nds to focus on how this distribution volvs ovr tim This is quit diffrnt from our analysis of th full information stting in [20], which focusd on th symmtry-braking that invitably occurs whn atomic playrs us th Hdg algorithm in that stting Th symmtric stting that w study hr allows for a significantly simplr analysis, incorporating tchniqus that ar standard in th analysis of multiplicativ-wight algorithms in larning thory such as th us of KL-divrgnc as a potntial function as wll as som nw tchniqus spcific to our stting such as th martingal argumnt usd to analyz th random walk in Lmma 3 Anothr bnfit of this analysis, in addition to its simplicity, is th considrably bttr dpndnc of th convrgnc tim on th numbr of playrs and congstibl rsourcs W focus on load balancing gams of n playrs of qual wights 1/n and n machins dgs Th quivalnc btwn numbr of playrs and numbrs of machins is not critical for our rsults Howvr, w choos thm bcaus th prformanc gap btwn worst cas corrlatd and worst cas Nash quilibria is spcially punctuatd undr such assumptions g whn all machins xhibit idntity cost functions c x = x Furthrmor, w allow any cost functions c x which ar twic continuously diffrntiabl with boundd drivativs Our Rsults and Tchniqus W show that a natural and simpl multiplicativ-wights algorithm achivs xponntial improvmnt ovr th worst corrlatd quilibrium, for our class of load-balancing gams Our main rsult is that using th Hdg algorithm [16] in th bulltin board modl, th xpctd makspan of th outcom is boundd by Olog n, xponntially bttr than th known lowr bounds for gnric no-rgrt algorithms W also show that Hdg continus to satisfy th no-rgrt proprty vn in th bulltin board modl W utiliz KL-divrgnc to xprss th distanc btwn th mixd stratgy mployd by a playr at tim t and hr projctd stratgy at th symmtric Nash quilibrium of th non-atomic vrsion of th gam W show that whn this distanc is larg nough, thn it has a tndncy to shrink As a rsult w can prdict th volution of th systm by analyzing a random walk, that has ngativ drift only whn w ar far away from th origin, an analysis that is of indpndnt intrst Prior work Th thory of larning in gams has a long history; s [17] for an xtnsiv xposition of th litratur in this fild, which has primarily focusd on analyzing th convrgnc bhavior of various classs of larning procsss and rlating this bhavior to Nash quilibrium, corrlatd quilibrium, and thir rfinmnts S [5] for a mor rcnt survy Th rlationship btwn rgrt minimization, calibratd forcasting, and corrlatd quilibrium has bn studid by [14, 13], and th connction btwn ths topics and th pric of anarchy was first mad in [3,4] Whras ths paprs us rgrt bounds to stablish static quilibrium proprtis of th limiting distribution of play, our work rquirs dirctly analyzing th dynamics of th stochastic procss inducd by ths algorithms Thr has bn considrabl rsarch in algorithmic gam thory on undrstanding th bhavior of adaptiv procdurs in load-balancing gams and othr congstion gams, including bst-rspons dynamics [18], rplication protocols [10] and sampling procdurs [15] Ths simpl distributd protocols ar wll motivatd, but thy lack dsirabl larning-thortic proprtis such as th no-rgrt proprty An xcption is [11], which analyzs a continuous-tim procss in non-atomic congstion gams that can b rgardd as th continuum limit of th multiplicativ-wights larning procss studid hr Th shift from atomic to non-atomic congstion gams liminats th distinction btwn th solution quality of corrlatd, mixd Nash, and pur Nash quilibria, thus liminating th motivating qustion in our work whil also vading most of th tchnical difficultis w addrss in analyzing th discrttim procss in atomic congstion gams In th contxt of atomic congstion gams, Roughgardn [27] has rcntly shown that for a wid class of gams, including congstion gams that satisfy a natural smoothnss condition, bounds on th pric of anarchy automatically xtnd to th total pric of anarchy, whn th global quality is dfind to b th avrag cost

4 4 Robrt Klinbrg t al In [20] w introducd th study of th multiplicativ wights larning algorithm in atomic congstion gams Our stting was th standard full information on, whr all playrs hav accss to an accurat modl of th undrlying gam W show that in almost all such gams, th multiplicativ-wights larning algorithm rsults in convrgnc to pur quilibria As discussd arlir, shifting from th full information stting to th mor ralistic bulltin board modl invalidats th rsults of [20]; in particular this shift falsifis th prdiction of convrgnc to pur quilibria and ncssitats an analysis of th dynamics using compltly diffrnt tools 2 Prliminaris A stratgic gam is a tripl N; S i i N ; u i i N whr N is th st of playrs and for vry playr i N, S i is th st of pur stratgis or actions of playr i, and th utility function u i is a ral valud function dfind on S = i N S i For vry stratgy profil s S, u i s rprsnts th payoff positiv utility to playr i For any stratgy profil s S and any stratgy s i of playr i w us s i, s i to dnot th stratgy profil that w driv by substituting th i-th coordinat of th stratgy profil s with s i A stratgy profil s is a Nash quilibrium if u i s u i s i, s i for vry s i and vry i N Analogously, a Nash ɛ-quilibrium is dfind as a stratgy profil s such that u i s u i s i, s i ɛ for vry s i, and vry i N Ths notions ar xtndd to randomizd or mixd stratgis by using th xpctd payoff Extrnal rgrt of an onlin algorithm somtims rfrrd to as mrly rgrt is dfind as th maximum ovr all input instancs of th xpctd diffrnc in payoff btwn th algorithm s actions and th bst action If this diffrnc grows sublinarly with tim, thn th algorithm is said to xhibit no xtrnal rgrt, or simply no rgrt Similarly, an algorithm is said to xhibit ɛ-rgrt, if its avrag prformanc may b at most ɛ wors than that of th bst fixd action in hindsight, as tim gos to infinity Such larning algorithms can still in practic offr no-rgrt guarants by itrativly rducing thir ɛ prformanc gap to zro No-rgrt algorithms ar closly rlatd to a notion of corrlatd quilibrium W say that a probability distribution π ovr th stratgy profils S is a coars wak corrlatd quilibrium[29] if for all playrs i and stratgis s i S i, th xpctd payoff of playr i playing s i is no bttr than th xpctd payoff from th distribution, i, s S u isπs s S u is i, s i πs Not that a mixd Nash quilibrium is a coars corrlatd quilibrium whr th probability distribution π is a product distribution ovr th stratgy sts S i i, it is not corrlatd It is wll known that th long-run avrag outcom of rpatd play using no-xtrnal rgrt algorithm convrgs to th st of coars corrlatd quilibria 3 Systm Analysis In this sction w study th prformanc of larning algorithms in load-balancing gams, i congstion gams on paralll links using th bulltin board modl in which playrs assss dg costs according to th actual cost incurrd on that dg, and not th hypothtical cost if th playr had usd it W dmonstrat that using th Hdg algorithm in th bulltin board modl th procss rmains clos to th symmtric fully mixd quilibrium of th non-atomic vrsion of th gam As a rsult, its prformanc is xponntially bttr than th worst corrlatd quilibrium of th gam In this sction w first prsnt th dfinition of th gams w will b focusing on sction 31 Nxt, w introduc th multiplicativ updats algorithm and th bulltin board modl in sction 32, whr w prov that th no-rgrt proprty prsists in th bulltin board modl Th main part of th analysis is in sction 33, whil w dfr a fw tchnical lmmas to sction Dfining th gam and th social cost Th congstion gam w considr in this sction is an atomic congstion gam with a st of n playrs, ach having wight w i = 1/n, and n dgs with cost functions c x In ach priod t = 1, 2,, ach playr chooss on dg W dfin f t to b th total amount of flow on dg in priod t, i f t = j/n whr j is th numbr of playrs choosing in priod t W mak th following standing assumptions: for th dg, th function c x is twic continuously diffrntiabl, satisfis c 0 = 0 and c 1 1, and for som positiv constants A, B it satisfis c x A and 0 c x B for all x [0, 1] In sction 34, lmma 4 provs that ths hypothss imply th following inqualitis for all x [0, 1]: Ax c x B + 1x 1 As a masur of social cost, w adopt th maximum dg cost, max c [f t] Intrprting playrs as jobs and dgs as machins, this intrprtation of th social cost is quivalnt to th makspan Th inquality Ax c x B + 1x implis that for any flow vctor f th social cost max c f lis btwn A f and B + 1 f In particular, th social optimum

5 Load Balancing Without Rgrt in th Bulltin Board Modl 5 is Θ1/n As w hav mntiond in th introduction, vn for th xtrmly simpl cas in which c x = x for all, x i, a load-balancing gam in which playrs schdul n jobs on n machins, and th cost xprincd by playr i is proportional to th numbr of jobs on its machin th corrlatd quilibria of th gam can b xponntially wors than any Nash quilibrium 32 Th larning algorithm and th bulltin board modl To dfin th larning algorithm usd by ach playr, w lt ε b a small positiv numbr w ll nd to hav ε 1/n 3 for th analysis and w introduc th following notations c [t] = c f t, c [1 : t] = t c [r] r=1 Zt = E xp εc [1 : t 1] In priod t, ach playr sampls a random dg with probability P, t = xp εc [1 : t 1], 2 Zt i, to obtain P, t from P, t 1 w multiply it by xp εc [t 1] and thn rnormaliz all probabilitis so that thy sum to 1 At th first tim stp, th algorithm sampls an dg uniformly at random This algorithm for spcifying a mixd stratgy in priod t is a vrsion of th Hdg algorithm [16], modifid so that playrs assss dg costs according to th actual cost c [t 1] incurrd on that dg, and not th hypothtical cost c f t n if th playr had usd it for playrs that do not us th dg in this itration This modl is usually rfrrd to as th bulltin board modl Using th wll-known fact that Hdg itslf is a no-rgrt larning algorithm 2 first w prov that th bulltin board variant of Hdg is also a no-rgrt larning algorithm Proposition 1 Th bulltin board variant of Hdg in any load-balancing gam with non-dcrasing cost functions rtains th ɛ-rgrt proprty Proof Hdg is known to hav th ɛ-rgrt proprty vn in sttings whn th cost functions of th dgs can vary with tim[16] For th proof, lt us considr such a stting, whr th actual cost/latncy of ach dg at priod t is c t x t, whr x t is th load of th 2 providd that ε convrgs to zro at an appropriat rat dpnding on t, g εt = O1/n 3 t dg in qustion at priod t Naturally, all cost functions c t ar non-dcrasing functions of x Now, w will dfin a nw cost function C t as follows: { Cx t c t = x if x x t c t x t othrwis Lt us xamin what this nw cost function xprsss Undr ths cost functions, th latncy of any dg obsrvd at tim t is actually th worst possibl and any furthr incras on th load of any dg would hav no ffct on its latncy If this optimistic viw of th cost of th dgs wr actually tru, thn th algorithm w hav proposd would prform xactly as th Hdg algorithm Hdg is known to hav th no-rgrt proprty, hnc, th xpctd prformanc of th algorithm as t gos to infinity is roughly as good as that of th bst dg/stratgy in hindsight undr this modifid costs C Howvr, th actual cost of any stratgy undr th ral cost functions c, whn taking into account th ffct of th dviating playr, would b at last as bad as that undr th optimistic costs C As a rsult th prformanc of our algorithm is also of ɛ-rgrt in rgards to th bst stratgy in hindsight undr th tru cost valuations Although th proposition abov in its currnt form will suffic for our purposs, it can b straightforwardly xtndd to any no-rgrt algorithm and all congstion gams with non-dcrasing cost functions 33 Main thorms Th main rsult of this sction is th following bound on th distribution P t dtrmind by th Hdg algorithm 2 Thorm 1 If all playrs sampl thir stratgis at tim t using th distribution P t dtrmind by th Hdg algorithm 2, thn thr xist positiv constants α, β 0 such that for all tims t and all β > β 0 it holds with probability at last 1 xp αβ that max P, t < 2β/n Combining this thorm with Chrnoff bounds lads to a pric-of-anarchy typ rsult th long-run avrag social cost xcds th social optimum by a factor of at most Olog n Mor prcisly: Corollary 1 In th stting of Thorm 1, thr xist constants c 1, c 2 such that for all t, with probability at last 1 1/n c1, th flow f t sampld by th playrs satisfis max c f t c 2 log n n

6 6 Robrt Klinbrg t al Th proof of Thorm 1 rsts on analyzing a stochastic procss KLt dfind as th KL-divrgnc btwn th Nash quilibrium and P t Lt Q b th symmtric Nash quilibrium of th non-atomic congstion gam whr all playrs play th sam stratgy with dg st E and cost functions c E KL-divrgnc btwn P and Q is dfind as KLt = j E Qj log Qj/P j, t KL-divrgnc masurs th distanc 3 btwn th distributions Qj and P j, t It is zro if thy ar qual and positiv othrwis W will show that whn this distanc is larg nough, thn it has a tndncy to shrink Lmma 2 This rducs th analysis of KLt to xploring th bhavior of a kind of random walks, which fac ngativ drift only whn thy ar far away from th origin Lmma 3 provids this analysis Thorm 1 will follow from proving an xponntial tail bound for KLt Thorm 2 Thr xist positiv constants α, β 0 such that PrKLt > β/n < αβ for all β > β 0 W nxt sktch th proof of this tail bound In all of th following argumnts, log dnots th natural logarithm function W ll nd th following tchnical lmma Lmma 1 log Zt + 1 log Zt xp ε 1 P, tc [t] Proof W will us th fact that if 0 y 1, thn xp εy 1 + yxp ε 1; this can b vrifid by chcking that th lft sid is a convx function, th right sid is a linar function, and th lft and right sids ar qual whn y is an ndpoint of th intrval [0, 1] Zt + 1 Zt εc[1:t 1] εc[t] = Zt εc[1:t 1] [1 + c [t] ε 1] Zt = 1 + ε 1 P, tc [t] Th lmma follows by taking th logarithm of both sids and using th idntity log1 + y y W dnot th diffrnc KLt + 1 KLt as t 3 although it is not a tru distanc mtric sinc it is not symmtric Lmma 2 Th stochastic procss KLt satisfis E[ t P t] ACε/nKLt + Cε/n 2 3 In particular, KLt drifts to th lft at a rat of Ωε/n 2 whnvr it is gratr than 2/An Proof A simpl calculation using quation 2 using Lmma 1 justifis th bound P, t log εc [t] 1 ε P, tc [t] P, t + 1 E Taking a wightd avrag of th abov inqualitis, wightd by Q, w obtain t = P, t Q log P, t + 1 ε Qc [t] 1 xp ε P, tc [t] Now, using c [t] to dnot E[c [t] P t] and using c [ ft] to dnot c P, t, w may tak th conditional xpctation of both sids and apply th idntity 1 xp ε > ε 1 2 ε2 to obtain: E [ t P t] ε [Q P, t] c [t] + ε2 2 P, t c [t] εq P t c[ ft] + εq P t c[t] c[ ft] + ε2 2 Nxt, w will dnot th usual convx potntial function x c 0 y dy as Φx As a rsult, w hav for th first trm abov that εq P t ΦP t ε [ΦQ ΦP t] Aε P t Q 2 2, whr th last inquality uss th fact th Q minimizs Φ, combind with our assumption that c y A for all y It is not hard to prov that for som constant C, th additional inqualitis P t Q 2 2 C KLt, 4 n εq P t c[t] c[ ft] ε2 Cε/n 2 5 hold Lmmas 7 and 8 in sction 34, implying that th stochastic procss KLt satisfis E[ t P t] ACε/nKLt + Cε/n 2, 6 as claimd Nxt w giv th submartingal argumnt to show that th fact that KLt has ngativ drift whn its larg implis that th probability of KLt > β/n is xponntially small in β as claimd by Thorm 2

7 Load Balancing Without Rgrt in th Bulltin Board Modl 7 Lmma 3 Lt Y t t 0 b a random walk satisfying th following for som constant M 1: boundd diffrncs: Y t+1 Y t 1; ngativ drift: EY t+1 Y t Y t 1/M whnvr Y t M Thn thr xist constants α, λ 0 such that for all λ > λ 0 and t 0, w hav PrY t > λm < αλ Proof For t N, r R +, lt Er, t dnot th vnt that Y t > M + r + 1 For 0 s t lt Er, t, s dnot th vnt 4 Er, t, s = {Y s 1 M} {Y s, Y s+1,, Y t 1 > M} {Y t > M + r + 1} Not that th vnts Er, t, s s = 0, 1,, t ar disjoint and thir union is Er, t For s t r, w hav that PrEr, t, s = 0 Our uppr bound on PrEr, t will b stablishd by proving sparat uppr bounds for th rst of th probabilitis PrEr, t, s To this nd, for a spcifid valu of s, dfin a random variabl q = min{i s Y i M} and a stochastic procss { MY i + i if s i q Z i = MY q + q if i > q Our ngativ drift assumption for th stochastic procss Y i i 0 implis that th procss Z i i 0 is a suprmartingal: E [Z i Z s,, Z i 1 ] Z i 1 Also, th bound Z i+1 Z i M + 1 holds with probability 1 Applying Azuma s suprmartingal inquality, for vry γ > 0 w hav γ 2 Pr Z t Z s > γ < xp 2M t s If vnt Er, t, s occurs, thn w hav Z t = MY t + t > MM + r t = M 2 + Mr + M + t Z s = MY s + s MY s s M 2 + M + s Z t Z s > Mr + t s Thrfor, PrEr, t, s < xp [Mr + t s]2 2M t s 4 For s = 0, Er, t, 0 translats to {Y 0, Y 1,, Y t 1 > M} {Y t > M + r + 1} Summing ovr s, w obtain PrEr, t < t PrEr, t, s s=0 t r 1 s=0 xp [Mr + t s]2 2M t s Lt k = rm +1/2, and brak up th sum into trms in which t s k and thos in which t s > k PrEr, t < k u= r +1 + [Mr + u]2 xp 2M u u=k+1 [Mr + u]2 xp 2M u k < xp M 2 r 2 2M k u=0 u 2 + xp 2M u u=k+1 < k + 1 xp r2 8k x + xp k 2M dx r 2 k + 1 xp 4rM M xp [ 1+ rm M ] rm M /4 2 r 4M+1 For r > 1/4 th last lin implis that PrEr, t < [ 1 + 3rM ] r/8m By stting r = λm M 1 > 1/4 and c = M, w obtain PrY t > λm < [ 1 + 3λM M 1M + 1 2] λ 8 +c, which shows that th lmma holds whnvr α < 1/8 and λ 0 is a sufficintly larg constant dpnding on α and M Proof of Thorms 1 and 2: Lt Y t = KLt/ε W can show that for all t 0, Y t+1 Y t 1 Indd, sinc KLt + 1 KLt = j E Qj log P j, t/p j, t + 1 and ach of th trms of th form P j, t/p j, t + 1 lis in th [ ε, ε ] intrval, Y t satisfis th proprty of boundd diffrnc of Lmma 3 W apply Lmma 3 with M = A+Cn 2 /AC Morovr, th inquality E[ t P t] ACε/nKLt + Cε/n 2 implis that thr xist positiv constants α, β 0 such that PrKLt > β/n < αβ for all β > β 0

8 8 Robrt Klinbrg t al This provs Thorm 2 Th bound on max P, t in Thorm 1 now follows by combining th KL-divrgnc bound in Thorm 2 with Lmma 5 blow, which bounds th infinity-norms of two distributions P, Q in trms of thir corrsponding KL-divrgnc 34 Tchnical lmmas Th following tchnical lmmas complt th analysis th prformanc of Hdg: Lmma 4 Lt c x b a function in C 2 [0, 1] satisfying c 0 = 0, c 1 1; for all x [0, 1], c x A; for all x [0, 1], 0 c x B Thn Ax c x B + 1x for all x [0, 1] Proof For all x w hav c x = x 0 c y dy x 0 A dy, which stablishs that Ax c x To stablish th uppr bound on c x, w first us th man valu thorm to dduc that thr xists som x [0, 1] such that c x = c 1 c If thr xists y [0, 1] such that c y > B + 1, thn a scond application of th man valu thorm would imply th xistnc of z [0, 1] satisfying c z = c y c x y x > B, contradicting our hypothsis about c Hnc c y B + 1 for all y [0, 1] Now, for all x [0, 1], c x = x 0 c y dy x 0 B+1 dy, which stablishs that c x B + 1x Lmma 5 If P, Q ar two probability distributions on a finit st S, satisfying P 2 Q, thn KLQ; P P 16 Proof Lt s 0 b a point at which P s 0 = P Lt a = Qs 0, b = P s 0 Thn KLQ; P = Qs 0 log Qs 0 P s 0 + s s 0 Qs log Qs P s = a log a b + 1 a s s 0 Qs 1 a [ log P s Qs Sinc s s 0 Qs/1 a = 1, th sum on th right sid can b intrprtd as a wightd avrag of valu of th convx function logx at th points P s/qs ] Using Jnsn s inquality, w s that this is gratr than or qual to logx valuatd at th point x = Qs P s 1 a Qs = P s 1 a = 1 b 1 a s s 0 s s 0 Hnc w hav drivd th first lin of th following sris of bounds a 1 a KLQ; P a log + 1 a log b 1 b = b a x a x1 x dx Th intgrand is a strictly incrasing function of x for 0 < x < 1, so ltting c = a + b/2 w hav b a x a b x1 x dx x a c x1 x dx b c c a c1 c = 1 b a 2 4 c1 c b a2 4b Th assumption that P 2 Q implis a b/2, and th lmma follows immdiatly Lmma 6 In a non-atomic load balancing gam 5 with n dgs whos cost functions satisfy th conditions of Lmma 4, th Nash quilibrium Q satisfis for vry dg : A B + 1n Q B + 1 An Proof Sinc Q is a Nash quilibrium, thr xists 6 a z > 0 such that c Q = z for all Sinc thr is som 0 such that Q 0 1/n, w hav z B +1/n Now for any dg, th rlations AQ c Q and c Q = z B + 1/n togthr imply that Q B + 1/An Similarly, th xistnc of an dg 1 such that Q 1 1/n implis that z A/n from which it follows that Q A/B + 1n for all Lmma 7 For any distributions P, Q on an n-lmnt st S, if C/n Qs 1/2 for all s S, thn P Q 2 2 C KLQ; P n 5 i a non-atomic paralll-links congstion gam 6 Sinc c 0 = 0 for all, in th symmtric Nash Q of th non-atomic congstion gam w hav that Q, c Q > 0 for all Othrwis, playrs could dcras thir xpctd latncy by utilizing th mpty rsourc

9 Load Balancing Without Rgrt in th Bulltin Board Modl 9 Proof Lt xs = P s Qs W hav KLQ; P = s = s Qs log P s Qs Qs log 1 + xs Qs Using th idntity log1 + x x x 2, valid for 1/2 x 1, w obtain KLQ; P s s [ ] xs Qs Qs xs2 Qs 2 xs 2 Qs n C x 2 2, from which th lmma follows immdiatly Lmma 8 Lt P b any probability distribution on dgs and lt f = f E b th random flow vctor obtaind by ltting n playrs ach sampl an dg in E according to P and snd 1/n units of flow on that dg Lt c, c dnot th vctors c = Ec f, c = c Ef = c P, rspctivly Thr is a constant C such that εq P c c ε2 Cε n 2 Proof Lt us fix our attntion on on dg and lt x 0 = P Taylor s thorm with rmaindr nsurs that for all x [0, 1], Q B+1/An, and P 1 P /n P /n Q/n B + 1/An 2 Hnc th dot product is boundd abov by B 2 B+1 An B+1 An = B B An Rcalling that ε 1/n 2, w s that th inquality in th statmnt of th lmma is satisfid by stting C = B 2 B+1 A 4 Summary Givn that onlin larning is quit thoroughly undrstood in th stting of a singl larnr [6], it is rathr natural to hop for a thorough undrstanding of systms consisting of multipl larnrs, but th charactrization of such systms in xisting work is far from thorough Svral rcnt paprs hav pursud this dirction in th contxt of no-rgrt larning [3,4,27], but thir findings hav bn limitd to gams in which th norgrt proprty by itslf suffics to stablish bounds on th ovrall systm prformanc Our work stablishs that in many cass of intrst and spcifically in sttings closly rlatd to th rality of distributd systms this optimistic viw dos not matrializ Two systms consisting of no-rgrt larnrs can xhibit hug prformanc diffrncs Nvrthlss, our rsult is in ssnc a positiv rsult It shows that natural candidats g Hdg of no-rgrt algorithms prform wll An intrsting dirction for futur rsarch is th qustion of how much w can xtnd th family of allowabl no-rgrt algorithms whil still allowing for strong provabl prformanc bounds on th ovrall systm bhavior c x 0 x x 0 c x c x 0 c x 0 x x 0 + B 2 x x 0 2 Rfrncs This holds, sinc 0 c y B for all y Plugging th random variabl f into this bound, w find that c c c + c x 0 Ef x 0 + B 2 Ef x c c B 2 Varf If z i i = 1, 2,, n dnots a collction of indpndnt Brnoulli random variabls with Prz i = 1 = P, thn th random variabl f has th sam distribution as 1 n n i=1 z i, so its varianc is Varf = 1 n 2 n Varz i = P 1 P n To bound th dot product Q P c c from abov, w first not that whn Q < P w hav Q P c c 0 Th rmaining trms of th dot product according to lmma 6 satisfy Q P 1 R Aumann Subjctivity and corrlation in randomizd stratgis Journal of Mathmatical Economics 1:67-96, B Awrbuch, R Khandkar, S Rao Distributd algorithms for multicommodity flow problms via approximat stpst dscnt framwork In Procdings of 18th Annual Symposium on Discrt Algorithms SODA: , A Blum, E Evn-Dar, and K Ligtt, Routing without rgrt: on convrgnc to Nash quilibria of rgrt-minimizing algorithms in routing gams In Procdings of th twntyfifth annual ACM symposium on Principls of distributd computing, ACM Prss, 45-52, A Blum, M Hajiaghayi, K Ligtt, and A Roth, Rgrt Minimization and th Pric of Total anarchy In Procdings of th 40th annual ACM symposium on Thory of computing STOC, pags , A Blum and Y Mansour, Larning, Rgrt Minimization and Equilibria, in Algorithmic Gam Thory N Nisan, T Roughgardn, E Tardos and V Vazirani ds, Cambridg Univrsity Prss, N Csa-Bianchi and G Lugosi Prdiction, larning, and gams Cambridg Univrsity Prss, 2006

10 10 Robrt Klinbrg t al 7 X Chn and X Dng Sttling th Complxity of Two-Playr Nash Equilibrium In Procdings of th 47th Annual IEEE Symposium on Foundations of Computr Scinc FOCS, pp , A Czumaj and B Vöcking Tight Bounds for Worst-Cas Equilibria In Procdings of th 13th Annual ACMSIAM Symposium on Discrt Algorithms SODA, pp , C Daskalakis, P Goldbrg, and C Papadimitriou Th Complxity of Computing a Nash Equilibrium In Procdings of th 38th annual ACM symposium on Thory of computing STOC, S Fischr, H Räck, and B Vöcking Fast convrgnc to Wardrop quilibria by adaptiv sampling mthods In Procdings of th 38th annual ACM symposium on Thory of computing STOC, pp , S Fischr and B Vöcking On th volution of slfish routing In Procdings of th 12th Europan Symposium on Algorithms ESA, pp , D Fostr and S M Kakad Dtrministic Calibration and Nash Equilibrium In Procdings of th 17th Annual Confrnc on Larning Thory Confrnc on Larning Thory COLT, pp 33-48, D Fostr, and R Vohra Calibratd larning and corrlatd quilibrium Gams and Economic Bhavior 21:4055, D Fostr and R Vohra Rgrt in th on-lin dcision problm Gams and Economic Bhavior 29: 7-35, D Fotakis, and A Kaporis, P and Spirakis Atomic Congstion Gams: Fast, Myopic andconcurrnt Thory of Computing Systms 471:38-59, Y Frund, and R Schapir Adaptiv gam playing using multiplicativ wights Gams and Economic Bhavior, 29:79-103, D Fudnbrg, and D Lvin Th Thory of Larning in Gams MIT Prss M Gomans, V Mirrokni, and A Vtta, Sink Equilibria and Convrgnc In Procding of th 46th Annual IEEE Symposium on th Foundations of Computr Scinc FOCS, pp , S Hart, and A Mas-Colll Stochastic Uncoupld Dynamics and Nash Equilibrium Gams and Economic Bhavior 57, , R Klinbrg and G Piliouras and E Tardos Multiplicativ updats outprform gnric no-rgrt larning in congstion gams In Procdings of th 41th ACM Symposium on Thory of Computing STOC, R Klinbrg, G Piliouras, and E Tardos Load balancing without rgrt in th billboard modl In Procdings of th 28th Symposium on Principls of Distributd Computing PODC, pags 56-62, E Koutsoupias and C H Papadimitriou Worst-cas quilibria In Procdings of th 16th Annual Symposium on Thortical Aspcts of Computr Scinc, pp , M Mavronicolas and P Spirakis Th pric of slfish routing In Procdings of th thirty-third annual ACM symposium on Thory of computing STOC, pp , V Mirrokni and A Vtta Convrgnc Issus in Comptitiv Gams RANDOM-APPROX, M Mitznmachr How usful is old information? In Procdings of th 16th Annual ACM SIGACT- SIGOPS Symposium on Principls of Distributd Computing PODC, pags 83-91, G Piliouras A larning thortic approach to gam thory PhD Thsis, Cornll T Roughgardn Intrinsic Robustnss of th Pric of Anarchy In Procdings of th 41th ACM Symposium on Thory of Computing STOC, A Tang, J Wang, S Low Countr-Intuitiv Bhaviors in Ntworks undr End-to-nd Control IEEE /ACM Transactions on Ntworking, 142: , April H P Young, Stratgic Larning and Its Limits Arn Ryd Mmorial Lcturs Oxford, UK: Oxford Univrsity Prss, 2004