Minimizing Queue Length Regret Under Adversarial Network Models

Transcription

1 Mnmzng Queue Length Regret Under Adversaral Network Models QINGKAI LIANG and EYTAN MODIANO, MIT LIDS Stochastc models have been domnant n network optmzaton theory for over two decades, due to ther analytcal tractablty. However, these models fal to capture non-statonary or even adversaral network dynamcs whch are of ncreasng mportance for modelng the behavor of networks under malcous attacks or characterzng short-term transent behavor. In ths paper, we focus on mnmzng queue length regret under adversaral network models, whch measures the fntetme queue length dfference between a causal polcy and an oracle that knows the future. Two adversaral network models are developed to characterze the adversary s behavor. We provde lower bounds on queue length regret under these adversary models and analyze the performance of two control polces (.e., the MaxWeght polcy and the Trackng Algorthm). We further characterze the stablty regon under adversaral network models, and show that both the MaxWeght polcy and the Trackng Algorthm are throughput-optmal even n adversaral settngs. 11 ACM Reference Format: Qngka Lang and Eytan Modano Mnmzng Queue Length Regret Under Adversaral Network Models. Proc. ACM Meas. Anal. Comput. Syst. 2, 1, Artcle 11 (March 2018), 34 pages. 1 INTRODUCTION 1.1 Background and Motvaton Stochastc network models have been domnant n network optmzaton theory for over two decades, due to ther analytcal tractablty. For example, t s often assumed n wreless networks that the varaton of traffc patterns and the evoluton of channel capacty follow some statonary stochastc process, such as the..d. model and the ergodc Markov model. Many mportant network control polces (e.g., MaxWeght polcy [16]) have been derved to optmze network performance (e.g., throughput) under those stochastc network dynamcs. However, non-statonary or even adversaral dynamcs have been of ncreasng mportance n recent years. For example, modern communcaton networks frequently suffer from Dstrbuted Denal-of-Servce (DDoS) attacks or jammng attacks [17], where traffc njectons and channel condtons are controlled by some malcous entty n order to degrade network performance. As a result, t s mportant to develop effcent control polces that optmze network performance even n adversaral settngs. However, extendng the tradtonal stochastc network optmzaton framework to the adversaral settng s non-trval because Ths work was supported by NSF Grant CNS and by DARPA I2O and Raytheon BBN Technologes under Contract No. HROO l l-l 5-C Authors address: Qngka Lang; Eytan Modano, MIT LIDS. Permsson to make dgtal or hard copes of all or part of ths work for personal or classroom use s granted wthout fee provded that copes are not made or dstrbuted for proft or commercal advantage and that copes bear ths notce and the full ctaton on the frst page. Copyrghts for components of ths work owned by others than the author(s) must be honored. Abstractng wth credt s permtted. To copy otherwse, or republsh, to post on servers or to redstrbute to lsts, requres pror specfc permsson and/or a fee. Request permssons from permssons@acm.org Copyrght held by the owner/author(s). Publcaton rghts lcensed to the Assocaton for Computng Machnery /2018/3-ART11 $ Proceedngs of the ACM on Measurement and Analyss of Computng Systems, Vol. 2, No. 1, Artcle 11. Publcaton date: March 2018.

2 many mportant notons and analytcal tools developed for stochastc networks cannot be appled n adversaral settngs. For example, the tradtonal stochastc network optmzaton focuses on long-term network performance whle n an adversaral envronment the network may not have any steady state or well-defned long-term tme averages. Thus, typcal steadystate analyss and many equlbrum-based notons such as the network throughput regon cannot be used n networks wth adversaral dynamcs, and t s mportant to understand transent network performance wthn a fnte tme horzon n a non-statonary/adversaral envronment. In ths paper, we nvestgate effcent network control polces that can optmze network performance (.e., queue length) wthn a fnte tme horzon n an adversaral envronment. 1.2 Man Results We develop general adversaral network models and propose a new fnte-tme performance metrc, referred to as queue length regret (the formal defnton s gven n Secton 2.3): R π T = Q π (T ) Q * (T ), where Qπ (T ) s the total queue length acheved by control polcy π after a fnte tme horzon T, and Q* (T ) s the mnmum queue length acheved by some oracle that has perfect knowledge about the future. We frst prove that t s mpossble to acheve low queue length regret f the adversary s unconstraned. In partcular, there exst some adversaral network dynamcs such that the queue length regret grows at least lnearly wth the tme horzon T under any causal control polcy. Ths mpossblty result motvates us to study constraned adversaral dynamcs. We then study two adversaral network models where the network dynamcs are constraned to some admssble set. In partcular, we frst consder the (W, ε)-constraned adversary model, where the total arrvals are less than (1 ε) tmes of the total servces wthn any wndow of W slots. Although ths wndow-based model s relatvely lmted, t s wdely used by exstng works (e.g., [2 4, 9, 12]) due to ts analytcal tractablty and serves as a foundaton for understandng more generalzed adversary models. Observng the lmtaton of (W, ε)-constraned model. we then propose a more generalzed V T -constraned model, where the total queue length generated by the oracle durng ts sample path s upper bounded by V T. By varyng the values V T, the proposed V T -constraned adversary model can cover a wde range of adversaral settngs: from a strctly constraned adversary to a fully unconstraned adversary. Under the above two adversary models, we develop lower bounds on queue length regret. It s shown that no causal polcy can acheve sublnear queue length regret f W or V T grows lnearly wth T. We also analyze the queue length regret of two control algorthms: the MaxWeght polcy [16] and the Trackng Algorthm [3, 4] under the two adversaral models. In partcular, both the MaxWeght polcy and the Trackng Algorthm acheve sublnear queue length regret whenever W or V T grows sublnearly wth T, yet the theoretcal regret bound under the Trackng Algorthm s better than that under the MaxWeght polcy. The Trackng Algorthm s also asymptotcally regret-optmal under the (W, ε)-constraned adversary model. We summarze these results n Table 1. Fnally, based on the above analytcal results and the observaton that sublnear queue length regret s equvalent rate stablty, we characterze the stablty regon under adversaral network models, and show that both the MaxWeght polcy and the Trackng Algorthm are throughput-optmal even n adversaral settngs. 2

3 Table 1. Queue Length Regret Bounds (W, ε)-constraned V T -Constraned Adversary Adversary Lower Bound Ω(W ) Ω(V T ) O( T W ) f ε = 0 MaxWeght O(W/ε 3 O(V 1/3 ) f ε > 0 T T 2/3 ) Trackng Alg. O(W ) O( T V T ) 1.3 Related Work The study of adversaral network models dates back more than two decades ago. Rene Cruz [7] provded the frst concrete example of networks wth adversaral dynamcs, whch were later generalzed by Borodn et al. [5] under the Adversaral Queung Theory (AQT) framework. In AQT, n each tme slot, the adversary njects a set of packets at some of the nodes. In order to avod trvally overloadng the system, the AQT framework mposes a strngent wndow constrants: the maxmum traffc njected n every lnk over any wndow of W tme slots should not exceed the amount of traffc that the lnk can serve durng that nterval. Andrews et al. [1] ntroduced a more generalzed adversary model known as the Leaky Bucket (LB) model that dffers from AQT by allowng some traffc burst durng any tme nterval. The AQT model and the LB model have gven rse to a large number of results snce ther ntroducton, most of whch are about network stablty under several smple schedulng polces such as FIFO (see [6] for a revew of these results). However, the AQT and the LB models assume that only packet njectons are adversaral whle the underlyng network topology and lnk states reman fxed. Such a statc network model does not capture many adversaral envronments, such as wreless networks under jammng attacks where the adversary can control the channel states. Andrews and Zhang [3, 4] extended the AQT model to sngle-hop dynamc wreless networks, where both packets njectons and lnk states are controlled by an adversary, and prove the stablty of the MaxWeght algorthm n ths context. Jung et al. [2, 9] further extended the results of [3, 4] to mult-hop dynamc networks. Our wndow-based (W, ε)-constraned model s nspred by and smlar to the adversaral models used n [2 4, 9]. Moreover, we also develop a new V T -constraned model that relaxes the wndow constrants and generalzes the exstng wndow-based (W, ε)-constraned models. Recently, Paschos and Tassulas [14] consdered the problem of stablzng queues under a mxture of stochastc and adversaral traffc njectons, but ther results s lmted to a very specfc servce provsonng model and only traffc njectons are adversaral. Whle the above-mentoned works focused on network stablty, Neely [12] nvestgated the unversal network utlty maxmzaton problem where network utlty needs to be maxmzed subject to stablty constrants under adversaral network dynamcs. Algorthm (tme-average) performance s measured wth respect to a so-called W -slot look-ahead polcy. Such a polcy has perfect knowledge about network dynamcs over the next W slots but t s requred that under ths polcy the total arrvals to each queue should not exceed the total amount of servce offered to that queue durng every wndow of W slots. As a result, t s smlar to our (W, ε)-constraned model where strngent wndow constrants have to be enforced. In ths paper, we not only consders the (W, ε)-constraned model but also develop a more general V T -constraned model that gets rd of the wndow constrants. 3

4 In addton, Shakkotta et al. [8] also used the noton of queue regret n the multarmed bandt problem. However, ther analyss s ntended for stochastc envronments and cannot be carred over to adversaral envronments. In summary, our paper expands prevous work n a number of fundamental ways. Frst, we develop queue length regret lower bounds under both the (W, ε)-constraned and the V T -constraned models. As far as we know, none of the exstng works (e.g., [2 4, 9, 12]) provde lower bounds on queue length regret (or queue length), even under the restrctve (W, ε)-constraned model where strngent wndow constrants are mposed. Note that our lower bounds on queue length regret reveal fundamental lmts of the system. For example, our lower bound under the V T -constraned model reveals that f V T = Ω(T ), then no causal polcy can stablze the network even f there exsts some stablzng non-causal polcy. Moreover, these lower bounds are also crtcal to establshng the optmalty of the Trackng algorthm under the (W, ε)-constraned model. Second, we provde analyss under the new V T -constraned adversary model whch generalzes the adversaral network dynamcs models used by exstng works. As far as we know, exstng works (e.g., [2 4, 9, 12]) all use the (W, ε)-constraned adversary model or smlar wndows-based varants due to ts analytcal tractablty. In ths paper, we propose a new V T -constraned adversary model whch gets rd of the wndow constrans and covers the full spectrum of network dynamcs. Due to the lack of wndow-based structure, the analyss carred out n exstng works cannot be appled to the V T -constraned model. In ths paper, we develop queue length regret upper bounds for the MaxWeght polcy and the Trackng algorthm under the V T -constraned model by usng a new traffc sheddng technque, whch converts a general V T -constraned adversary to a (W, ε)-constraned adversary and then optmzes the regret bounds by carefully choosng the amount of traffc to shed. Such a proof technque may be used to adapt any queue length regret bounds derved under the (W, ε)-constraned model to that under the V T -constraned model. Fnally, to the best of our knowledge, ths s the frst paper that characterzes the throughput regon under arbtrary (and possbly adversaral) network dynamcs, whch provdes a necessary and suffcent condton on network dynamcs such that the network s stable. The characterzaton of the throughput regon s based on our analyss under the new V T -constraned model and the equvalence between sublnear queue length regret and rate stablty. 1.4 Organzaton of Ths Paper We frst ntroduce the system model and relevant performance metrcs n Secton 2. We study the (W, ε)-constraned and V T -constraned adversary models n Sectons 3 and 4, respectvely. In Secton 5, we characterze the stablty regon under adversaral network models. Fnally, smulaton results and conclusons are gven n Secton 6 and 7, respectvely. 2 SYSTEM MODEL 2.1 Asymptotc Notatons Let f and g be two functons defned on some subset of real numbers. Then f(x) = f(x) f(x) O(g(x)) f lm sup x g(x) <. Smlarly, f(x) = Ω(g(x)) f lm nf x g(x) > 0. Also, f(x) = Θ(g(x)) f f(x) = O(g(x)) and f(x) = Ω(g(x)). In addton, f(x) = o(g(x)) f = 0, and n ths case we say that f(x) s sublnear n g(x). lm x f(x) g(x) 4

5 2.2 Network Model Consder a network wth N queues (the set of all queues are denoted by N = {1,, N}). Tme s slotted wth a fnte horzon T = {0,, T 1}. Let ω t denote the network event that occurs n slot t, whch ndcates the current network parameters, such as a vector of condtons for each lnk, a vector of exogenous arrvals to each node, or other relevant nformaton about the current network lnks and exogenous arrvals. At the begnnng of each tme slot t, the network operator observes the current network event ω t and chooses a control acton α t from some acton space D ωt that can depend on ω t. The network event ω t and the control acton α t together produce the servce vector b(α t, ω t ) b(t) = (b 1 (t),, b N (t)) and the arrval vector a(α t, ω t ) a(t) = (a 1 (t),, a N (t)). Note that a (t) ncludes both the exogenous arrvals from outsde the network to queue, and the endogenous arrvals from other queues (.e., routed packets from other queues to queue ). Thus, the above network model accounts for both sngle-hop and mult-hop networks, and the control acton α t may correspond to, for example, jont routng, rate allocaton and schedulng decsons n a mult-hop network. Let Q(t) = (Q 1 (t),, Q N (t)) be the queue length vector at the begnnng of slot t (before the arrvals n that slot). The queung dynamcs are Q (t + 1) = [Q (t) + a (t) b (t)] +, N, t T, where [x] + = max{x, 0}. T 1 We assume that the sequence of network events {ω t } t=0 are generate accordng to an arbtrary process (possbly non-statonary or even adversaral), except for the followng boundedness assumptons. Under any network event and any control acton, the arrvals and the servce rates n each slot are bounded by constants that are ndependent of the tme horzon T : 0 a (t) A, 0 b (t) B, t T, N. For smplcty, we assume B A such that both arrvals and servces are upper bounded by B n each slot. A polcy π generates a sequence of control actons ( α π 0,, α π T 1) wthn the tme horzon. In each slot t, the queue length vector, the controlled arrval vector and the servce rate vector under polcy π s denoted by Q π (t), a π (t) and b π (t), respectvely. A causal polcy s one that generates the current control acton α t only based on the knowledge up untl the current slot t. In contrast, a non-causal polcy may generate the current control acton α t based on knowledge of the future. Example: An example of the above network model s the power control problem n wreless downlnk systems wth N lnks. In each slot t, the controller observes the current network events ω t = ( a(t), s(t) ), where a(t) and s(t) correspond to the vector of exogenous arrvals and the vector of channel capactes n slot t, respectvely. Then the controller takes a control acton α t as a power allocaton vector α t = (α (1) t,, α (N) t ), subject to an nstantaneous power constrant α t D ωt, where α () t s the power allocated to lnk n slot t. The constrant set D ωt could be, for example, the set of power allocaton vectors that satsfy the peak power constrant α() t α peak. The servce rate for each lnk s determned by the rate-power functon b (α t, ω t ). For example, one possble form of the rate-power functon s b (α t, ω t ) = s (t) log 1 + SINR (α t, ω t ), 5

6 where SINR (α t, ω t ) s the sgnal-to-nterference-plus-nose rato over lnk when power vector α t s allocated under network event ω t. 2.3 Performance Metrcs Our objectve s to fnd a causal control polcy that keeps the total queue length as small as possble. Note that a network wth adversaral dynamcs may not have any steady state or well-defned tme averages. Hence, t s crucal to understand the transent behavor of the network, and the tradtonal equlbrum-based performance metrcs may not be approprate n an adversaral settng. As a result, we ntroduce the noton of queue length regret to measure the fnte-tme performance acheved by a causal control polcy. Defnton 2.1 (Queue Length Regret). Gven the tme horzon T, the queue length regret acheved by a causal polcy π under a sequence of network events ω 0,, ω T 1 s defned to be R π T {ω 0,, ω T 1 } = Q π (T ) Q * (T ), (1) N N where Q* (T ) s the mnmum total queue length generated by the optmal non-causal polcy that knows the the entre sequence of network events {ω 0,, ω T 1 } n advance. The worst-case queue length regret acheved by polcy π s R π T = sup R π T {ω 0,, ω T 1 }. ω 0,,ω T 1 In ths setup, a polcy π s chosen and then the adversary selects the sequence of network events {ω 0,, ω T 1 } that maxmze the regret. Intutvely, the noton of queue length regret captures the worst-case queue length dfference between a causal polcy and an deal T -slot lookahead non-causal polcy. Ths metrc also measures the prce of causalty,.e., the cost of not knowng the future. A desrable frst order characterstc of a good polcy π s that t acheves sublnear regret R π T = o(t ) such that Rπ T /T 0 as the tme horzon T. In other words, the tme-average queue growth rate asymptotcally approaches the one acheved by the optmal non-causal polcy. We wll also demonstrate the equvalence between sublnear queue length regret and rate stablty n Secton 5. In addton to beng sublnear, the queue length regret should also have a low growth rate. A lower growth rate of regret mples that the polcy has a better learnng ablty and can adapt to the adversaral envronment faster. We defne the mnmax queue length regret as the mnmal queue length regret that can be acheved over the space of causal polces. A polcy s sad to be asymptotcally regret-optmal f t acheves the mnmax regret up to a constant multplcatve factor; ths mples that, n terms of growth rate of regret, the performance of the polcy s the best possble. Fnally, note that the growth rate of regret s strongly related to the noton of convergence tme (see [13] for more detals). Unfortunately, the followng theorem shows that n general no causal polcy can acheve sublnear queue length regret for any sequence of network events. Theorem 2.2. For any causal polcy π, there exsts a sequence of network events ω 0,, ω T 1 such that the queue length regret R π T {ω 0,, ω T 1 } T/2. Proof. We prove ths theorem by constructng a sequence of network events ω 0,, ω T 1 such that the lower bound s attaned. Consder the power control example mentoned n 6

7 Secton 2.2 wth N = 2 lnks. The constrant on power allocaton s α (1) t + α (2) t 1 for each t T, and the rate-power functon s b (t) = α () t s (t). Wthout loss generalty, assume that the tme horzon T s an even number. The exogenous arrvals and channel capactes n the frst T/2 slots s a 1 (t) = a 2 (t) = 2, s 1 (t) = s 2 (t) = 2, t = 0,, T/2 1. Under the power allocaton constrant, the total number of packets that can be cleared n the frst T/2 slots s at most T. For any causal polcy π, let n 1 and n 2 be the number of packets cleared over lnk 1 and 2 durng the frst T/2 slots, respectvely. Then t s clear that n 1 + n 2 T, whch mples that mn{n 1, n 2 } T/2. Defne * = arg mn =1,2 n (tes are broken arbtrarly). Then the queue length over lnk * after T/2 slots s Q π *(T/2) = T n * = T mn{n 1, n 2 } T/2. In the next T/2 slots, the adversary can set a *(t) = 0, s *(t) = 0, t = T/2,, T 1. For the other lnk (ts ndex s denoted by ), the adversary can set a (t) = 0, s (t) = 2, t = T/2,, T 1. Snce there s no capacty to clear any packet over lnk * n the last T/2 slots, we have Q π *(T ) = Qπ *(T/2) T/2, whch mples that Qπ (T ) T/2. On the other hand, the optmal non-causal polcy can choose the followng sequence of power allocaton vectors { α (* ) t, α ( ) (1, 0), t = 0,, T/2 1, t = (0, 1), t = T/2,, T 1, such that Q* (T ) = 0, whch mples that the queue length regret acheved by polcy π s at least T/2. Ths completes the proof. Remark: Note that the above constructon requres the value of T. We can elmnate the dependence on the tme horzon T by usng the standard Doublng Trck (see Secton n [15]). The detals about the doublng trck are gven n Appendx A.1. Theorem 2.2 shows that sublnear queue length regret s not achevable f the adversary has unconstraned power n determnng the network dynamcs. As a result, n the followng two sectons, we develop two adversary models where the sequence of network events (.e. network dynamcs) that the adversary can select s constraned to some admssble set. In Secton 3, we consder the (W, ε)-constraned adversary model that s an extenson of the wdely-known yet very strngent model used n Adversaral Queung Theory. Next n Secton 4, we develop a more relaxed adversary model called the V T -constraned adversary. Lower bounds on queue length regret and the performance of some commonly-used algorthms are analyzed under the two adversary models. 7

8 3 (W, ε)-constrained ADVERSARY MODEL In ths secton, we nvestgate the (W, ε)-constraned adversary model whch s an extenson of the classcal Adversaral Queung Theory (AQT) [5] (smlar to the models used n[2 4, 9, 12]). It has strngent constrants on the set of admssble network dynamcs that the adversary can set, yet s analytcally tractable, whch facltates our subsequent nvestgaton of a more relaxed adversary model n Secton 4. We frst gve the defnton of (W, ε)-constraned network dynamcs. Defnton 3.1 ((W, ε)-constraned Dynamcs). Gven a wndow sze W [1, T ] and a load factor ε [0, 1], a sequence of network events ω 0,, ω T 1 s (W, ε)-constraned f there exsts a (possbly non-causal) polcy π such that for any t = 0, W, 2W, a π (τ) (1 ε) b π (τ), N. (2) Any network satsfyng the above s called a (W, ε)-constraned network. In other words, the tme horzon s dvded nto frames of sze W slots, and t s requred that there exsts a (possbly non-causal) polcy such that durng every frame the total amount of arrvals to each queue s less than or equal to 1 ε tmes of the total amount of servces offered to that queue. Denote by A T (W, ε) the set of all sequences of network events {ω 0,, ω T 1 } that are (W, ε)-constraned. Then the (W, ε)-constraned adversary can only select the sequence of network events from the constraned set A T (W, ε). In ths context, the worst-case queue length regret acheved by a causal polcy π s defned to be R π T = sup R π T {ω 0,,ω T 1 } A T (W,ε) {ω 0,, ω T 1 } where R π T ( ) s gven n (1). In the followng, we frst provde a lower bound on queue length regret under the (W, ε)- constraned adversary model (Secton 3.1), and then analyze the worst-case queue length regret acheved by several common control polces (Secton 3.2). Note that throughout ths secton we manly focus on the dependence of queue length regret on W, ε and T whle treatng the number of users N a constant. 3.1 Lower Bound on Queue Length Regret The followng theorem provdes a lower bound on queue length regret under the (W, ε)- constraned adversary model. Theorem 3.2. For any causal polcy π, there exsts a sequence { of network } events {ω 0,, ω T 1 } A T (W, ε) such that R π T {ω 0,, ω T 1 } max (1 2ε)W/2, 0. Proof. For any gven causal polcy, we construct a sequence of network events such that the lower bound s attaned. The constructon s smlar to the one used n the proof of Theorem 2.2. The dfference s that the constructed sequence of network events are also (W, ε)-constraned here. See Appendx A.2 for the detaled proof. Remarks: Note that f ε s some small (ε < 1/2) constant ndependent of the wndow sze W, then the above lower bound s of order Ω(W ). If the wndow sze W s comparable wth the tme horzon T,.e., W = Θ(T ), no causal polcy can acheve sublnear (worst-case) queue length regret under the (W, ε)-constraned adversary model. On the other hand, f, 8

9 W = o(t ), there mght exst some causal polcy that attans sublnear queue length regret, whch we nvestgate n the next secton. In partcular, we show that the above regret lower bound can be asymptotcally attaned by some causal polcy and thus the mnmax queue length regret n (W, ε)-constraned networks s Θ(W ). 3.2 Algorthm Performance n (W, ε)-constraned Networks In ths secton, we analyze the worst-case queue length regret acheved by two network control algorthms under the (W, ε)-constraned adversary model. The frst s the famous MaxWeght polcy [16] that was proved to be throughput-optmal n stochastc networks. The second s a generalzed verson of the Trackng Algorthm [3, 4] that was orgnally proposed n Adversaral Queung Theory MaxWeght. In each slot t, the MaxWeght algorthm smply observes the current network event ω t and chooses the control acton as follows: = arg max Q (t) b (α t, ω t ) a (α t, ω t ). (3) α MW t α t D ωt The soluton to (3) depends on the partcular network model. For example, n a sngle-hop wreless network wth prmary nterference, the soluton to (3) just corresponds to the one that serves the queue wth the largest product of queue length and servce rate; n a nput-queued swtch wth crossbar constrants, solvng (3) s equvalent to solvng the Maxmum Weght Matchng problem [10]. The followng theorem gves the performance of the MaxWeght polcy n (W, ε)-constraned networks. Theorem 3.3. Under the (W, ε)-constraned adversary model, the worst-case queue length regret acheved by the MaxWeght algorthm s O( T W ) for any ε 0. Moreover, n the specal case where ε > 0 and a mn > 0, a better queue length regret bound W of O ε 3 a mn can be acheved by the MaxWeght algorthm, where a mn s the mnmum arrval to each queue n each slot. Proof. The proof s based on the Lyapunov drft analyss. However, nstead of consderng the one-slot drft as n the tradtonal stochastc analyss, we fnd upper bounds on the W -slot drft and make sample-path arguments. See Appendx A.3 for detals. There are several mportant observatons about Theorem 3.3. Frst, sublnear worst-case queue length regret could be acheved by the MaxWeght polcy under the (W, ε)-constraned adversary model as long as W = o(t ). Notcng that sublnear regret cannot be acheved by any causal polcy f W = Ω(T ) (Theorem 3.2), we have the followng corollary. Corollary 3.4. Under the (W, ε)-constraned adversary model, sublnear worst-case queue length regret s achevable f and only f W = o(t ). Second, the O( T W ) queue length regret bound could be much larger than the lower bound n Theorem 3.2 when W s sgnfcantly smaller than T. Thrd, f a mn > 0 and the system s n the sub-crtcal regme (ε > 0), then the performance W bound of the MaxWeght polcy s O ε 3 a mn, whch could be sgnfcantly better than the O( T W ) bound when W s much smaller than T and ε, a mn s not too small. Ths s analogous to the performance of the MaxWeght polcy n stochastc networks: strong 9

10 stablty 1 can be acheved f the system s strctly nsde the stablty regon (sub-crtcally loaded) whle only rate stablty can be acheved f the system s on the boundary of the stablty regon (crtcally-loaded) [11]. Fnally, t should be noted that n order to derve a better regret bound n the sub-crtcal regme (ε > 0), we requre an addtonal assumpton that a mn > 0 and the obtaned bound s nversely proportonal to the value of a mn. It s stll unknown whether a better regret bound could be obtaned n the sub-crtcal regme wthout such an assumpton. We conjecture that the regret bound of O(W/ε 3 ) may hold for MaxWeght even wthout the assumpton that a mn > 0, snce no evdence shows that there s any dscontnuty at a mn = 0. On other hand, f the assumpton that a mn > 0 s not satsfed, then the O( T W ) bound can be appled, whch s suffcent to ensure that all of the subsequent results about MaxWeght (e.g., Corollary 3.4) hold true Trackng Algorthm. The orgnal Trackng Algorthm was proposed n [3, 4] to solve a schedulng problem under an adversary model smlar to the (W, ε)-constraned adversary. However, t only works for a very specfc network model: () the network has to be sngle-hop where the arrval vector s ndependent of the control acton, and () the control acton has to satsfy the prmary nterference constrants,.e., only one lnk ncdent on the same node can be actvated n each slot. Next, we extend the orgnal Trackng Algorthm to accommodate the general network model consdered n ths paper. Let Ω be the set of all possble network events that could happen n each slot. In order for the Trackng Algorthm to work, the cardnalty of Ω has to be fnte (otherwse t could be dscretzed nto a fnte set as n [3]). For example, n a sngle-hop network, suppose each network event ω t corresponds to a couple (a(t), s(t)) where a(t) s a vector of exogenous packet arrvals n slot t and s(t) a vector of lnk states n slot t. For any lnk and tme t, assume that 0 a (t) B and a (t) s an nteger, and each lnk only has a fnte number of S states. Then Ω = (SB) N. We mantan a vrtual queue for each physcal queue and each type of network event ω Ω. In partcular, let q,ω (t) be the vrtual queue length n slot t assocated wth lnk and network event w, whch corresponds to the debts the Trackng Algorthm owned to the optmal (non-causal) polcy over lnk under network event ω. Here, the debts correspond to the queue length dfference between the Trackng Algorthm and the optmal polcy, and the goal of the Trackng Algorthm s to track the queue length trajectory under the optmal polcy. In addton, the optmal polcy corresponds to any sequence of control actons that satsfes the wndow constrants (2). Note that the optmal sequence of actons cannot be calculated onlne. Instead, t s calculated at the end of every wndow of W slots and the debt owned durng ths wndow wll be updated at the begnnng of the next wndow. In each slot, the Trackng Algorthm just pcks the control acton that clears as much debt as possble. The detaled algorthm descrpton s shown n Algorthm 1. The vrtual debt queues are updated at two tmes. Frst, the vrtual debt queues are updated n each slot t, after we observe the network event ω t and an acton α t s taken: q,ωt (t + 1) = [ q,ωt (t) + a (ω t, α t ) b (ω t, α t ) ] +, (4) 1 A queung system s strongly stable f Q (t) B for some constant B as t. A system s rate-stable f Q (t)/t 0 as t. 10

11 and the cleared debt n slot t by acton α t s defned to be q,ωt (α t ) = q,ωt (t) q,ωt (t + 1). T A In step 3, the Trackng Algorthm just pcks the acton αt that maxmzes the total cleared debt n slot t. Note that durng the above procedure, only the vrtual queues assocated wth network event ω t are updated whle the vrtual queues assocated wth any other types of network events reman unchanged. Second, the vrtual debt queues are also updated every W slots (at the end of each wndow), n order to add the debt owned to the optmal actons durng the past wndow. Such a procedure s shown n steps 5-6. The optmal sequence of control actons {α τ * } t W +1 durng the past wndow [t W + 1, t] s frst calculated, and then the correspondng debts are added to each vrtual queue. In partcular, the debts owned to the optmal actons n the past wndow for each vrtual queue q,ω s τ B ω b (ω, α τ * ) a (ω, α τ * ), where B ω s the set of slots durng the past wndow when network event ω happens. Algorthm 1 Trackng Algorthm (TA) 1: Intalze q,ω (0) = 0 for any N and ω Ω. 2: for t = 0,, T 1 do T A 3: Choose the control acton α that clears as much total debt as possble: t T A αt = arg max α t D ωt q,ωt (α t ), and update vrtual debt queues accordng to (4). 4: f mod (t, W ) = W 1 then 5: Compute the sequence of optmal control actons {α τ * } t W +1 n the past wndow [t W + 1, t], whch s any soluton that satsfes t t a (ω τ, α τ * ) (1 ε) b (ω τ, α τ * ), N. W +1 6: For each N and ω Ω, update q,ω (t + 1) = q,ω (t) + τ B ω W +1 where B ω = {τ t W + 1 τ t, ω τ = ω}. 7: end f 8: end for b (ω, α τ * ) a (ω, α τ * ), The followng theorem gves the worst-case queue length regret acheved by the Trackng Algorthm under the (W, ε)-constraned adversary model. Theorem 3.5. Assume that the sze of the network event space Ω s fnte. Then under the (W, ε)-constraned adversary model, the worst-case queue length regret acheved by the Trackng Algorthm s O(W ) for any ε 0. Proof. We frst present an upper bound on the vrtual queue length, whch s gven n Lemma 3.6. Ths lemma shows that the queue length dfference between the Trackng Algorthm and the optmal polcy s at most O(W ). 11

12 Lemma 3.6. For any t T and any type of network event ω Ω, we have q,ω (t) NBW. The proof to Lemma 3.6 s presented n Appendx A.4. Intutvely, snce the Trackng Algorthm clears as much debt as possble n each slot, t can emulate the behavor of the optmal polcy n the past, and the O(W ) gap s due to the delayed debt updated. Now we prove Theorem 3.5. Let Q(t), a(t) and b(t) be the physcal queue length vector, the arrval vector and the servce vector n slot t under the Trackng Algorthm. Also let Q * (t), a * (t) and b * (t) be the queue length vector, the arrval vector and the servce vector n slot t under the optmal polcy. Wthout loss of generalty, let T = RW for some postve nteger R. For each N, let τ be the last tme t when Q (t) = 0. Assume that τ s contaned n frame r and let t = (r + 1)W (.e., the begnnng of frame r + 1). Clearly we have t τ W and thus Then t follows that T 1 Q (T ) = Q (t ) + a (t) b (t) t 1 Q (t ) Q (τ ) + a (t) W B. (5) t=τ t=t T 1 W B + a (t) + a * (t) a * (t) b (t) t=t T 1 W B + a (t) + b * (t) a * (t) b (t) t=t = W B + ω W t T ω [ ] b * (t) a * (t) b (t) a (t). Here, the frst nequalty s due to (5) and the second nequalty s due to Equaton (2). The last equalty regroups tme slots accordng to the type of network event that occurred n each slot, where we defne T ω = {t t t T 1, ω t = ω}, ω W. Note that by the defnton of the vrtual queue and Lemma 3.6, we have for any N [ ] b * (t) a * (t) b (t) a (t) t T ω q,ω (T ) q,ω (t ) NBW. Then t follows that Q (T ) W B + Ω NBW. Note that the above nequalty holds for all sequence of networks events and that Q * (T ) 0. Then we can conclude that the worst-case queue length regret acheved by the Trackng Algorthm s R T = Q (T ) Q * (T ) NW B + Ω N 2 BW = O(W ). Ths completes the proof to Theorem

13 There are several mportant observatons about Theorem 3.5. Frst, sublnear worst-case queue length regret could be acheved by the Trackng Algorthm n (W, ε)-constraned networks as long as W = o(t ). Moreover, the queue length regret bound of the Trackng Algorthm s better than that of the MaxWeght polcy, n terms of ther dependence on W, ε and T. Second, the Trackng Algorthm s asymptotcally regret-optmal under the (W, ε)-constraned adversary model (f ε s a small constant), snce the queue length regret acheved by the Trackng Algorthm has the same order as the lower bound n Theorem 3.2. Ths also mples that Θ(W ) s also the growth rate of the mnmax queue length regret n (W, ε)-constraned networks. Corollary 3.7. Suppose ε s a small constant ndependent of T and W. Then under the (W, ε)-constraned adversary model, the mnmax queue length regret s Θ(W ). Thrd, the Trackng Algorthm needs to mantan a vrtual queue for each type of network events whle the sze of the network event space Ω may be exponental n the number of users N. As a result, the Trackng Algorthm may not be a practcal algorthm. The purpose of presentng the Trackng Algorthm s to demonstrate that the lower bound n Theorem 3.2 could be asymptotcally acheved by a causal polcy. Note that Andrews and Zhang [3, 4] proposed methods to get rd of the exponental dependence on N, at the expense of much more nvolved algorthms. Ther methods may be adapted to our scenaro to acheve a better dependence on N, but t s left for future work snce the focus of ths paper s the scalng of queue length regret wth T whle N s treated as a constant. 4 V T -CONSTRAINED ADVERSARY MODEL The aforementoned (W, ε)-constraned model s relatvely restrctve, where the strngent constrants (2) have to be satsfed for every wndow of W slots. In ths secton, we consder a general adversary model where the wndow constrants (2) are relaxed. The new adversary model s parameterzed by the nherent varaton n the sequence of network events, whch s measured as follows. Gven a sequence of network events ω 0,, ω T 1 and a (possbly non-causal) polcy, we defne V π {ω 0,, ω T 1 } = max t T Q π (t). The above functon measures the peak queue length acheved by polcy π durng ts sample path. We further defne V {ω 0,, ω T 1 } = mn V π {ω 0,, ω T 1 }, π.e., the mnmum peak queue length that could be acheved by any (possbly non-causal) polcy under the sequence of network events ω 0,, ω T 1. Note that V ( ) only depends on {ω 0,, ω T 1 } and measures the nherent varatons n the sequence of network events. Now we defne the noton of V T -constraned dynamcs where the value of V ( ) s constraned by some budget V T. Defnton 4.1 (V T -Constraned Dynamcs). Gven some V T [0, NT B], a sequence of network events ω 0,, ω T 1 s V T -constraned f V {ω 0,, ω T 1 } V T. 13

14 Any network satsfyng the above s called a V T -constraned network. Intutvely, the V T -constraned model requres that the total queue length generated by the oracle durng ts sample path should be upper bounded by V T. Denote by V T the set of all possble sequences of network events that are V T -constraned. A V T -constraned adversary can only select the sequence of network events from the set V T. In ths context, the worst-case queue length regret acheved by a causal polcy π s defned as R π T = sup R π T {ω 0,, ω T 1 }. {ω 0,,ω T 1 } V T where R π T ( ) s gven n (1). Note that we restrct the range of V T to [0, NT B] snce the peak queue length wthn T slots s at most NT B. Any larger value of V T has the same effect as V T = NT B. Note also that the larger V T s, the more varatons the network could have. By varyng the value of V T from 0 to NT B, the above V T -constraned adversary model covers the full spectrum of network dynamcs. If V T = 0, then the arrvals should be less than or equal to the servces for each queue n every slot, and network dynamcs s strngently constraned. If V T = NT B (whch corresponds to the maxmum total queue length that could be buld up durng T slots), the network dynamcs s completely unconstraned. In the followng, we wll frst provde a lower bound on queue length regret under the V T -constraned adversary model n Secton 4.1 and then analyze the regret acheved by the MaxWeght polcy and the Trackng Algorthm n Secton 4.2. Some mportant dscussons are provded n Secton Lower Bound on Queue Length Regret The followng theorem provdes a lower bound on the queue length regret under the V T - constraned adversary model. Theorem 4.2. For any causal polcy π, there exsts a sequence of network events {ω 0,, ω T 1 } V T such that R π T {ω 0,, ω T 1 } cv T, where c s some constant ndependent of T and V T. Proof. For any gven causal polcy, we construct a sequence of network events such that the lower bound s attaned. The constructon s smlar to the one used n the proof of Theorem 2.2. The dfference s that the constructed sequence of network events are V T -constraned here. See Appendx A.5 for the detaled proof. If V T = Ω(T ), then no causal polcy can acheve sublnear queue length regret under the V T -constraned adversary model. On the other hand, f V T = o(t ), there mght exst some causal polcy that attans sublnear queue length regret, whch we nvestgate n Secton Algorthm Performance n V T -Constraned Networks In ths secton, we analyze the queue length regret acheved by two algorthms n V T - constraned networks: the MaxWeght polcy and the Trackng Algorthm. In partcular, we show that both algorthms acheves sublnear regret f V T = o(t ) MaxWeght. The MaxWeght polcy dscussed n Secton 3.2 can be drectly appled n V T -constraned networks. The followng theorem gves the worst-case queue length regret acheved by the MaxWeght polcy under the V T -constraned adversary model. 14

15 Theorem 4.3. Under the V T -constraned adversary model, the worst-case queue length regret acheved by the MaxWeght polcy s O T 2/3 V 1/3 T. Proof. We consder a new system that s obtaned by sheddng a certan amount of traffc from the orgnal system such that the new system s (W, 0)-constraned for some wndows sze W that s to be selected later. By the defnton of V T, there exsts some (possbly non-causal) polcy π * such that max t T Q π* (t) = V T. (6) Denote by ã π* (t) and a π* (t) the arrval vector n the new system and n the orgnal system n slot t under π *, respectvely. Also let X T be the total amount of shed traffc wthn the tme horzon,.e., T 1 X T = t=0 a π* (t) T 1 a π* (t). Now we dvde the tme horzon nto frames of sze W slots. Wthout loss of generalty, assume that W dvdes T. Then the total number of frames s T/W. Denote by t r = (r 1)W the begnnng of frame r. In order to make the new system (W, 0)-constraned, we can shed traffc n each frame r such that π* Q (t r+1 ) = 0, where Q π* (t) s the queue length vector n slot t n the new system under polcy π *. Note that Qπ* (t r+1 ) V T by equaton (6), and thus at most V T arrvals need to be shed n frame r to ensure π* Q (t r+1 ) = 0. Therefore, n order to make the new system (W, 0)-constraned, at most V T T/W arrvals need to be shed durng the entre tme horzon,.e., t=0 X T V T T/W. (7) Let Q(t) and Q(t) be queue length vector n slot t f the MaxWeght algorthm s appled to the orgnal system and the new system, respectvely. Then t s clear that Q (T ) X T + Q (T ). (8) Snce the new system s (W, 0)-constraned, by the proof of Theorem 3.3, we have Q (T ) c 1 W T (9) for some constant c 1 > 0. Combng (7), (8) and (9) we have Q (T ) V T T/W + c 1 W T. Choosng W = c 2 V 2/3 T T 1/3 T for some constant c 2 > 0, we have Q (T ) ( 1 + c 1 c2 )T 2/3 V 1/3 T = O(T 2/3 V 1/3 T ), c 2 whch mples that the worst-case queue length regret acheved by the MaxWeght polcy under the V T -constraned adversary model s O(T 2/3 V 1/3 T ). There are several observatons about Theorem 4.3. Frst, the MaxWeght polcy acheves sublnear queue length regret under the V T -constraned adversary model whenever V T = o(t ). Notce that sublnear regret cannot be acheved by any causal polcy f V T = Ω(T ) (Theorem 4.2). We have the followng corollary. 15

16 Corollary 4.4. Under the V T -constraned adversary model, sublnear worst-case queue length regret s achevable f and only f V T = o(t ). Second, the MaxWeght polcy does not attan the Ω(V T ) lower bound n Theorem 4.2, especally when V T s sgnfcantly smaller than T Trackng Algorthm. The Trackng Algorthm ntroduced under the (W, ε)-constraned adversary model requres that the wndow constrants (2) be satsfed for some wndow sze W. However, there mght be no wndow structure under the V T -constraned adversary model and thus the Trackng Algorthm cannot be drectly appled n V T -constraned networks. We slghtly modfy the Trackng Algorthm of Secton 3.2 by settng W = T V T NB. Note that V T [0, NT B], whch guarantees that W T. Moreover, step 6 of the orgnal Trackng Algorthm s tweaked to fnd a sequence of control actons {α τ * } t W +1 that satsfes the followng constrants: t t a (ω τ, α τ * ) b (ω τ, α τ * ) + V T, N. (10) W +1 W +1 Note that by the defnton of V T -constraned networks, there always exsts a feasble soluton satsfyng the above constrants. Under the above settng, the worst-case queue length regret acheved by the Trackng Algorthm under the V T -constraned adversary model s gven n the followng theorem 2. Theorem 4.5. Under the V T -constraned adversary model, the worst-case queue length regret acheved by the Trackng Algorthm s O( V T T ). Proof. It can be easly verfed that Lemma 3.6 stll holds n V T -constraned networks. Then followng the smlar lne of argument as n the proof to Theorem 3.5, we have T 1 Q (T ) W B + a (t) + a * (t) a * (t) b (t) t=t T 1 W B + a (t) + b * (t) a * VT T (t) b (t) + W t=t = W B + Ω NBW + VT T W, T V T NB, we have where the second nequalty s due to (10). Choosng W = T VT Q (T ) (1 + B) NB + ( Ω + 1) T V T NB, whch mples that the worst-case queue length regret acheved by the Trackng Algorthm under the V T -constraned adversary model s O( T V T ). There are several mportant observatons about Theorem 4.5. Frst, smlar to MaxWeght, the Trackng Algorthm also guarantees sublnear queue length regret whenever V T = o(t ). Second, the Trackng Algorthm has a better regret bound than that under the MaxWeght polcy when V T s sgnfcantly smaller than T. Thrd, the regret bound of the Trackng 2 As s dscussed n Secton 3.2.2, the set of possble network events should be fnte n order for the Trackng Algorthm to work. 16

17 Algorthm does not attan the Ω(V T ) regret lower bound n Theorem 4.2. Thus, fndng a causal polcy that can close the gap remans an open problem. 4.3 Dscussons Senstvty of Trackng Algorthm to Parameters. Note that the above Trackng Algorthm requres V T as a parameter. Unfortunately, n practce, t s mpossble to know the precse value of V T for a gven network n advance. To allevate ths ssue, we can search for the correct value of V T. Note that the range for V T s [0, NBT ]. Then one may perform bnary search to fnd the correct value of V T by runnng the Trackng algorthm wth dfferent values of V T over multple epsodes wthn the tme horzon (e.g., f the tme horzon s T = 10 5 slots, then one epsode could be 10 3 slots). Smlar technques can be appled f the Trackng Algorthm s used n (W, ε)-constraned networks where the values W and ε are requred as nput parameters Relatonshp Between Adversary Models. The V T -constraned adversary model generalzes the (W, ε)-constraned adversary model: any sequence of network events that are (W, ε)-constraned must also be V T -constraned wth V T = O(W ) due to the wndow structure. The analyss n the V T -constraned adversary model also gves a more general condton for sublnear queue length regret Queue Length Regret vs. Queue Length. Note that under the (W, ε)-constraned model, the optmal queue length s bounded by 0 Q* (T ) O(W ); under the V T - constraned model, the optmal queue length s bounded by 0 Q* (T ) O(V T ). As a result, the queue length regret bounds we derved under the two constraned model are also legtmate bounds for total queue length. Whle queue length regret and queue length may be quanttatvely smlar under the two constraned models, t does not mply that queue length regret s equvalent to queue length n general. For example, f we replace queue length regret by queue length, the mpossblty result n Theorem 2.2 would become meanngless snce the adversary can always trvally overload the system to obtan Ω(T ) queue length. By comparson, under the noton of queue length regret, Theorem 2.2 establshes that the gap between a causal polcy and the optmal polcy could be very large, and thus t mpossble for any causal polcy to stablze the network even f the system s not overloaded. Such an mpossblty result s the prmary motvaton for mposng constrants on the best achevable queue length performance as n Sectons 3 and 4. 5 STABILITY REGION IN ADVERSARIAL NETWORKS In ths secton, we characterze the stablty regon under adversaral network models. We frst gve the defnton of rate stablty. Defnton 5.1 (Rate Stablty). A network s rate-stable under a control polcy π f lm Qπ (T ) = 0. (11) T T Intutvely, rate stablty requres that the long-term arrval rate s less than or equal to the long-term servce rate. The followng observaton drectly follows from the defnton of queue length regret. Observaton 1. In any V T -constraned network wth V T = o(t ), rate stablty s equvalent to sublnear queue length regret. Proof. See Appendx A.6. 17

18 Combng the above observaton wth Theorems 4.3 and 4.5, we have the followng corollary. Corollary 5.2. In any V T -constraned network wth V T = o(t ), both the MaxWeght polcy and the Trackng Algorthm acheve rate stablty. The noton of stablty regon descrbes a necessary and suffcent condton such that rate stablty could be acheved. In partcular, n a sngle-hop stochastc network, t was shown n [11] that the stablty regon can be descrbed by the exstence of a causal polcy π such that T 1 T 1 lm a (t) lm b π (t),. T T T T t=0 However, such condtons cannot be appled n adversaral settngs snce the above lmts may not exst. Thus, we need a new characterzaton of the stablty regon for networks wth arbtrary (possbly adversaral) dynamcs. as s gven n the followng theorem. Theorem 5.3. The network s rate-stable under some causal polcy f and only f V {ω 0,, ω T 1 } = o(t ) as T, for a gven sequence of network events ω 0, ω 1,. t=0 Proof. See Appendx A.7. In other words, the stablty regon can be descrbed as the set of sequences of network events {ω 0, ω 1, } such that v {ω 0,, ω T 1 } = o(t ). We say that a control polcy s throughput-optmal f t acheves rate stablty whenever the sequence of network events ω 0, ω 1, s nsde the stablty regon. Combnng Theorem 5.3 and Corollary 5.2, we conclude the followng. Corollary 5.4. Both the MaxWeght polcy and the Trackng Algorthm are throughputoptmal n networks wth arbtrary (possbly adversaral) dynamcs. 6 SIMULATIONS In ths secton, we emprcally valdate the theoretcal bounds derved n ths paper and compare the performance of the MaxWeght polcy and the Trackng Algorthm n dfferent scenaros. In our smulatons, we consder a one-hop wreless network wth a sngle base staton and N users. In each slot, the base staton observes the current channel rate for each user and selects one of the users to serve. The packet arrval process and the evoluton of channel rates are arbtrary and possbly adversaral. In the followng, we nvestgate two specfc scenaros. 6.1 Scenaro I: Network wth Adaptve Adversary Frst, we consder a scenaro where the channel rate vector n each slot s controlled by an adaptve adversary. There are N = 2 users n the network, and tme s dvded nto frames of W slots. In the frst W/2 slots of each frame, the arrvals to each user are 2 packets/slot and the channel rate for each user s also 2 packets/slot. In the remanng slots of each frame, there are no arrvals to both users whle the channel rate s zero for the user wth a longer queue and 2 packets/slot for the other user. If the two users have the same queue length, tes are broken randomly. Such a scenaro s smlar to the one that we use to prove the regret lower bound under the (W, ε)-constraned adversary model (see the proof of Theorems 3.2), and t has been shown that ths s a (W, 0)-constraned adversary. 18