Delay Reduction via Lagrange Multipliers in Stochastic Network Optimization

Transcription

1 IEEE TRANS. ON AUTOMATIC CONTROL, OL. 56, ISSUE 4, PP , APRIL Delay Reducion via Lagrange Mulipliers in Sochasic Nework Opimizaion Longbo Huang, Michael J. Neely Absrac In his paper, we consider he problem of reducing nework delay in sochasic nework uiliy opimizaion problems. We sar by sudying he recenly proposed quadraic Lyapunov funcion based algorihms (QLA, also known as he MaxWeigh algorihm). We show ha for every sochasic problem, here is a corresponding deerminisic problem, whose dual opimal soluion exponenially aracs he nework backlog process under QLA. In paricular, he probabiliy ha he backlog vecor under QLA deviaes from he aracor is exponenially decreasing in heir Euclidean disance. This is he firs such resul for he class of algorihms buil upon quadraic Lyapunov funcions. The resul quanifies he nework graviy role of Lagrange Mulipliers in nework scheduling. I no only helps o explain how QLA achieves he desired performance bu also suggess ha one can roughly subrac ou a Lagrange muliplier from he sysem induced by QLA. Based on his finding, we develop a family of Fas Quadraic Lyapunov based Algorihms (FQLA), which use virual placeholder bis and virual conrol processes for decision making. We prove ha FQLA achieves an O(1/ ), O(log( )] 2 ) ] performance-delay radeoff for problems wih discree acion ses, and achieves a square-roo radeoff for coninuous problems. The performance of FQLA is similar o he opimal radeoffs achieved in prior work by Neely (2007) via drif-seering mehods, and shows ha QLA can also be used o approach such performance. Index Terms Queueing, Dynamic Conrol, Lyapunov analysis, Sochasic Opimizaion I. INTRODUCTION In his paper, we consider he problem of reducing nework delay in he following general framework of he sochasic nework uiliy opimizaion problem. We are given a ime sloed sochasic nework. The nework sae, such as he nework channel condiion, is ime varying according o some probabiliy law. A nework conroller performs some acion based on he observed nework sae a every ime slo. The chosen acion incurs a cos (since cos minimizaion is mahemaically equivalen o uiliy maximizaion, below we will use cos and uiliy inerchangeably), bu also serves some amoun of raffic and possibly generaes new raffic for he nework. This raffic causes congesion, and hus leads o backlogs a nodes in he nework. The goal of he conroller is o minimize is ime average cos subjec o he consrain ha he ime average oal backlog in he nework is finie. Longbo Huang ( longbohu@usc.edu) and Michael J. Neely (web: hp://www-rcf.usc.edu/ mjneely) are wih he Deparmen of Elecrical Engineering, Universiy of Souhern California, Los Angeles, CA 90089, USA. This work was presened in par a he 7h Inl. Symposium on Modeling and Opimizaion in Mobile, Ad Hoc, and Wireless Neworks (WiOp), Seoul, June, This maerial is suppored in par by one or more of he following: he DARPA IT-MANET program gran W911NF , he NSF gran OCE , he NSF Career gran CCF This seing is very general, and many exising works fall ino his caegory. Furher, many echniques have been used o sudy his problem (see 1] for a survey). In his paper, we focus on algorihms ha are buil upon quadraic Lyapunov funcions (called QLA in he following), e.g., 2], 3], 4], 5], 6], 7]. These QLA algorihms are easy o implemen, greedy in naure, and are parameerized by a scalar conrol variable. I has been shown ha when he nework sae is i.i.d., QLA algorihms can achieve a ime average uiliy ha is wihin O(1/ ) o he opimal. Therefore, as grows large, he ime average uiliy can be pushed arbirarily close o he opimal. However, such close-o-opimal uiliy is usually a he expense of large nework delay. In fac, in 3], 4], 7], i is shown ha an O( ) nework delay is incurred when an O(1/ ) close-o-opimal uiliy is achieved. Two recen papers 8] and 9], which show ha i is possible o achieve wihin O(1/ ) of opimal uiliy wih only O(log( )) delay, use a more sophisicaed algorihm design approach based on exponenial Lyapunov funcions. Therefore, i seems ha hough being simple in implemenaion, QLA algorihms have undesired delay performance. However, we noe ha he delay resuls of QLA are usually given in erms of long erm upper bounds of he average nework backlog e.g., 7]. Thus hey do no examine he possibiliy ha he acual backlog vecor (or is ime average) converges o some fixed value. Work in 10] considers drif properies owards an invarian backlog vecor, derived in he special case of a one-hop downlink sysem and when he problem exhibis a unique opimal Lagrange muliplier. An upper bound on he long erm deviaion of he acual backlog and he Lagrange muliplier vecor is obained. While his suggess Lagrange mulipliers are graviaional aracors, he bounds here do no show ha he he acual backlog is very unlikely o deviae significanly from he aracor. In his paper, we focus on obaining sronger probabiliy resuls of he seady sae backlog process behavior under QLA. We firs show ha under QLA, even hough he backlog can grow linearly in, i ypically says close o an aracor, which is he dual opimal soluion of a deerminisic opimizaion problem. In paricular, he probabiliy ha he backlog vecor deviaes from he aracor is exponenially decreasing in disance. I significanly ighens he aracor analysis in 10]. I also implies ha a large amoun of he daa is kep in he nework simply for mainaining he backlog a he righ level. Therefore, we can replace hese daa wih some fake daa (denoed as place-holder bis 11]) wihou heavily affecing QLA s performance. Based on his finding, we propose he Fas Quadraic Lyapunov based Algorihms (FQLA),

2 IEEE TRANS. ON AUTOMATIC CONTROL, OL. 56, ISSUE 4, PP , APRIL which can be inuiively viewed as subracing ou a Lagrange muliplier from he sysem induced by QLA. We show ha when he nework sae is i.i.d., FQLA is able o achieve wihin O(1/ ) of opimal uiliy wih an O(log( )] 2 ) delay guaranee for problems wih discree acion ses, and achieve an O(1/ ), O(log( )] 2 )] radeoff for problems wih a se of coninuous acion opions. The developmen of FQLA also provides addiional insighs ino QLA and he role of Lagrange mulipliers in sochasic nework opimizaion. FQLA is closely relaed o he TOCA algorihm in 8], which obains he same logarihmic and square-roo radeoffs for he energy-delay problem (up o a log( ) difference) via drif seering echniques. However, we noe ha FQLA differs from TOCA in he following: Firs, TOCA is consruced based on exponenial Lyapunov funcions; while FQLA uses simpler quadraic Lyapunov funcions. Second, FQLA is designed o mimic QLA and is hus relaed o dual subgradien algorihms; whereas TOCA is designed o ensure he primal consrains are saisfied. Third, FQLA requires an arbirary small bu nonzero fracion of packe droppings, hence can no be applied o problems where packe dropping is no allowed. We now summarize he main conribuions of his paper: (1) This paper proves ha in seady sae, he backlog vecor process under QLA is exponenially araced o an aracor, which is he dual opimal soluion of a deerminisic opimizaion problem. This is he firs such resul for he class of widely used QLA (MaxWeigh) algorihms. This exponenial aracion resul quanifies he nework graviy role of Lagrange mulipliers in nework scheduling and helps o explain how QLA achieves he desired performance. I is also he firs heoreical resul in he lieraure ha explains he recen delay improvemen resul observed in 12], where by using LIFO ogeher wih QLA, one achieves a significan (around 90%) nework delay reducion. (2) This paper proposes he Fas Quadraic Lyapunov based Algorihms (FQLA) o subrac ou a Lagrange muliplier from he nework under QLA for delay reducion. FQLA is usually easy o implemen, and can achieve an O(1/ ), O(log( )] 2 ) ] performance-delay radeoff for general sochasic nework opimizaion problems wih a discree se of acion opions as well as a square-roo radeoff for coninuous problems. The paper is organized as follows: In Secion II, we se up our noaions. In Secion III, we sae our nework model. We hen review he QLA algorihm and define he deerminisic problem in Secion I. In Secion, we show ha he backlog process under QLA always says close o an aracor. In Secion I, we propose he FQLA algorihm. Secion II provides simulaion resuls. II. NOTATIONS R (R + or R ): he se of (nonnegaive or non-posiive) real numbers R n (or R n +): he se of n dimensional column vecors, wih each elemen being in R (or R + ) bold symbols a and a T : column vecor and is ranspose a b: vecor a is enrywise no less han vecor b a b : he Euclidean disance of a and b III. SYSTEM MODEL In his secion, we specify he general nework model we use. We consider a nework conroller ha operaes a nework wih he goal of minimizing he ime average cos, subjec o he queue sabiliy consrain. The nework is assumed o operae in sloed ime, i.e., {0, 1, 2,...}. We assume here are r 1 queues in he nework. A. Nework Sae We assume here are a oal of M differen random nework saes, and define S = {s 1, s 2,..., s M } as he se of possible saes. Each paricular sae s i indicaes he curren nework parameers, such as a vecor of channel condiions for each link, or a collecion of oher relevan informaion abou he curren nework channels and arrivals. Le S() denoe he nework sae a ime. We assume ha S() is i.i.d. every ime slo, and le p si denoe is probabiliy of being in sae s i, i.e., p si = P r{s() = s i }. We assume he nework conroller can observe S() a he beginning of every slo, bu he p si probabiliies are no necessarily known. Noe ha if S() conains muliple componens, e.g., if S() is a vecor of channel saes of he nework, he componens can be correlaed o each oher. See Secion III-D for an example. B. The Cos, Traffic, and Service A each ime, afer observing S() = s i, he conroller chooses an acion x() from a se X (si), i.e., x() = x (si) for some x (si) X (si). The se X (si) is called he feasible acion se for nework sae s i and is assumed o be ime-invarian and compac for all s i S. The cos, raffic, and service generaed by he chosen acion x() = x (si) are as follows: (a) The chosen acion has an associaed cos given by he cos funcion f() = f(s i, x (si) ) : X (si) R + (or X (si) R in reward maximizaion problems); (b) The amoun of raffic generaed by he acion o queue j is deermined by he raffic funcion A j () = A j (s i, x (si) ) : X (si) R +, in unis of packes; (c) The amoun of service allocaed o queue j is given by he rae funcion µ j () = µ j (s i, x (si) ) : X (si) R +, in unis of packes; Noe ha A j () includes boh he exogenous arrivals from ouside he nework o queue j, and he endogenous arrivals from oher queues, i.e., he ransmied packes from oher queues, o queue j (See Secion III-C and III-D for furher explanaions). We assume he funcions f(s i, ), µ j (s i, ) and A j (s i, ) are ime-invarian, heir magniudes are uniformly upper bounded by some consan δ max (0, ) for all s i, j, and hey are known o he nework operaor. We also assume ha here exiss a se of acions {x (si)k } k=1,2,...,,...,m wih x (si)k X (si) such ha { s i p si k ϑ(si) k A j (s i, x (si)k ) µ j (s i, x (si)k )] } ɛ for some ɛ > 0 for all j, wih k ϑ(si) k = 1 and ϑ (si) k 0 for all s i and k. Tha is, he consrains are feasible wih ɛ slackness. Thus, here exiss a saionary randomized policy ha sabilizes all queues (where

3 IEEE TRANS. ON AUTOMATIC CONTROL, OL. 56, ISSUE 4, PP , APRIL ϑ (si) k represens he probabiliy of choosing acion x (si)k when S() = s i ). In he following, we use: A() = (A 1 (),..., A r ()) T, µ() = (µ 1 (),..., µ r ()) T, (1) o denoe he arrival and service vecors a ime. I is easy o see from above ha if we define: hen A() µ() B for all. B = rδ max, (2) C. Queueing, Average Cos, and he Sochasic Problem Le q() = (q 1 (),..., q r ()) T R r +, = 0, 1, 2,... be he queue backlog vecor process of he nework, in unis of packes. We assume he following queueing dynamics: q j ( + 1) = max q j () µ j (), 0 ] + A j () j, (3) and q(0) = 0. By using (3), we assume ha when a queue does no have enough packes o send, null packes are ransmied. In his paper, we adop he following noion of queue sabiliy: E { r j=1 q j } lim sup j=1 r E { q j (τ) } <. (4) Noe ha his crierion is no resricive. I can be shown ha our resuls remain he same under a wide class of oher sabiliy crierions used in he lieraure. For more deails, see 13] and 14]. We also use f π av o denoe he ime average cos induced by an acion-seeking policy π, defined as: f π av lim sup E { f π (τ) }, (5) where fav(τ) π is he cos incurred a ime τ by policy π. We call an acion-seeking policy feasible if a every ime slo i only chooses acions from he feasible acion se X (S()). We hen call a feasible acion-seeking policy under which (4) holds a sable policy, and use fav o denoe he opimal ime average cos over all sable policies. In every slo, he nework conroller observes he curren nework sae and chooses a conrol acion, wih he goal of minimizing ime average cos subjec o nework sabiliy. This goal can be mahemaically saed as: (P1) min : fav π, s.. (4). In he following, we will refer o (P1) as he sochasic problem. This sochasic problem framework can be used o model many nework problems, e.g., he energy minimizaion problem 3] and he access poin pricing problem 5]. We noe ha a similar nework model wih sochasic penalies is reaed in 15] using a fluid model and a primal-dual approach ha achieves opimaliy in a limiing sense. The framework is also reaed in 7] using a quadraic Lyapunov based algorihm (QLA) ha provides an explici O(1/ ), O( )] performance-delay radeoff when he nework sae is i.i.d.. D. An Example of he Model Here we provide an example o illusrae our model. Consider he 2-queue nework in Fig.1. In every slo, he nework operaor decides wheher or no o allocae one uni power o serve packes a each queue, so as o suppor all arriving raffic, i.e., mainain queue sabiliy, wih minimum energy expendiure. The number of arrival packes R(), is i.i.d. over slos, being eiher 2 or 0 wih probabiliies 5/8 and 3/8 respecively. Each channel sae CH 1 () or CH 2 () can be eiher G=good or B=bad. However, he wo channels are correlaed, so ha (CH 1 (), CH 2 ()) can only be in he channel se CH = {(B, B), (B, G), (G, G)}. We assume (CH 1 (), CH 2 ()) is i.i.d. over slos and akes any value in CH wih probabiliy 1 3. When a link s channel sae is good, one uni of power can serve 2 packes over he link, oherwise i can only serve one. We assume powers can be allocaed o boh channels wihou affecing each oher. Fig. 1. '"#$%()#$% *"#$%('&#$% *&#$%!"#$%!&#$% +,"#$% +,&#$% A 2-queue sysem In his case, he nework sae S() is a riple (R(), CH 1 (), CH 2 ()) and is i.i.d.. There are six possible nework saes. A each sae s i, he acion x (si) is a pair (x 1, x 2 ), wih x i being he amoun of energy spen a queue i, and (x 1, x 2 ) X (si) = {0/1, 0/1}. The cos funcion is f(s i, x (si) ) = x 1 + x 2, s i. The nework saes, he raffic funcions, and he service rae funcions are summarized in Fig. 2. Noe here A 1 () = R() is par of S() and is independen of x (si) ; while A 2 () = µ 1 () hence depends on x (si). Also noe ha A 2 () equals µ 1 () insead of minµ 1 (), q 1 ()] due o our idle fill assumpion in Secion III-C. S() R() CH 1() CH 2() A 1() A 2() µ 1() µ 2() s 1 0 B B 0 x 1 x 1 x 2 s 2 0 B G 0 x 1 x 1 2x 2 s 3 0 G G 0 2x 1 2x 1 2x 2 s 4 2 B B 2 x 1 x 1 x 2 s 5 2 B G 2 x 1 x 1 2x 2 s 6 2 G G 2 2x 1 2x 1 2x 2 Fig. 2. Nework sae, Traffic, and Rae funcions I. QLA AND THE DETERMINISTIC PROBLEM In his secion, we firs review he quadraic Lyapunov funcions based algorihms (he QLA algorihm) 7] for solving he sochasic problem. Then we define he deerminisic problem and is dual problem. We hen also discuss some properies of he dual funcion. The dual problem and he properies of he dual funcion will be used laer for our analysis of he seady sae backlog behavior under QLA. A. The QLA algorihm To solve he sochasic problem using QLA, we firs define a quadraic Lyapunov funcion L(q()) = 1 r 2 j=1 q2 j (). We hen define he one-slo condiional Lyapunov drif: (q()) = E { L(q(+1)) L(q()) q() }, where he expecaion is aken over he random nework sae S() and he possible random acions. From (3), we obain he following: (q()) B 2 E { r q j () µ j () A j () ] q() }. j=1

4 IEEE TRANS. ON AUTOMATIC CONTROL, OL. 56, ISSUE 4, PP , APRIL Now add o boh sides he erm E { f() q() }, where 1 is a scalar conrol variable, we obain: (q()) + E { f() q() } { B 2 E f() (6) + r j=1 q j () µ j () A j () ] } q(). The QLA algorihm is hen obained by choosing an acion x a every ime o minimize he righ hand side of (6) given q(). Specifically, he QLA algorihm works as follows: 1 QLA: A every ime slo, observe he curren nework sae S() and he backlog q(). If S() = s i, choose x (si) X (si) ha solves he following: r max : f(s i, x) + q j () µ j (s i, x) A j (s i, x) ] (7) s.. x X (si). j=1 Depending on he problem srucure, (7) can usually be decomposed ino separae pars ha are easier o solve, e.g., 3], 5]. Also, i can be shown, as in 7] ha, f QLA av = f av + O(1/ ), q QLA = O( ), (8) where fav QLA and q QLA are he expeced average cos and he expeced average nework backlog under QLA, respecively. B. The Deerminisic Problem Consider he deerminisic problem as follows: min : F(x) s i p si f(s i, x (si) ) (9) s.. A j (x) s i p si A j (s i, x (si) ) B j (x) s i p si µ j (s i, x (si) ) j x (si) X (si) i = 1, 2,..., M, where p si corresponds o he probabiliy of S() = s i and x = (x (s1),..., x (s M ) ) T. The dual problem of (9) can be obained as follows (0 R r has all enries being 0): max : g(γ), s.., γ 0, (10) where g(γ) is called he dual funcion and is defined as: { g(γ) = inf f(s i, x (si) ) (11) p si x (s i ) X (s i ) s i + j γ j Aj (s i, x (si) ) µ j (s i, x (si) ) ]}. Here γ = (γ 1,..., γ r ) T is he Lagrange muliplier of (9). I is well known ha g(γ) in (11) is concave in he vecor γ, and hence he problem (10) can usually be solved efficienly, paricularly when cos funcions and rae funcions are separable over differen nework componens. I is also 1 We assume wihou loss of generaliy ha an opimal soluion of (7) exiss. This condiion can easily be saisfied if all f(s i, ), µ j (s i, ) and A j (s i, ) funcions are coninuous. well known ha in many siuaions, he opimal value of (10) is he same as he opimal value of (9) and in his case we say ha here is no dualiy gap 16]. We noe ha he deerminisic problem (9) is no necessarily convex as he ses X (si) are no necessarily convex, and he funcions f(s i, ), A j (s i, ) and µ j (s i, ) are no necessarily convex. Therefore, here may be a dualiy gap beween he deerminisic problem (9) and is dual (10). Furhermore, solving he deerminisic problem (9) may no solve he sochasic problem. This is so since a every nework sae, he sochasic problem may require ime sharing over more han one acion, bu he soluion o he deerminisic problem gives only a fixed operaing poin per nework sae. However, one can show ha he dual problem (10) gives he exac value of fav, where fav is he opimal ime average cos for he sochasic problem, even if (9) is non-convex. For a given γ, le x γ = (x (s1) γ, x (s2) γ,..., x (s M ) γ ) T wih x (si) γ X (si), i, be a minimizer of he righ-hand side of g(γ). The erm G γ = (G γ,1, G γ,2,..., G γ,r ) T wih: G γ,j = A j (x γ ) B j (x γ ) (12) = p si µj (s i, x (si) γ ) + A j (s i, x (si) γ ) ], s i is hen called he subgradien of g( ) a γ 16]. I is well known ha for any oher ˆγ R r, we have: (ˆγ γ) T G γ g(ˆγ) g(γ). (13) Using G γ B, we noe ha (13) also implies: g(ˆγ) g(γ) B ˆγ γ ˆγ, γ R r (14) We are now ready o sudy he seady sae behavior of q() under QLA. To simplify noaions and highligh he scaling effec of he scalar in QLA, we use g 0 (γ) and γ 0 o denoe he dual objecive funcion and an opimal soluion of (10) when = 1; and use g(γ) and γ (also called he opimal Lagrange muliplier) for heir counerpars wih general 1. To simplify analysis, we assume he following hroughou: Assumpion 1: γ = (γ 1,..., γ r )T is unique 1. Noe ha Assumpion 1 is no very resricive. In fac, i holds in many nework uiliy opimizaion problems, e.g., 10]. In many cases, we also have γ 0. Moreover, for he assumpion o hold for all 1, i suffices o have jus γ 0 being unique. This is shown in he following lemma. Lemma 1: γ = γ 0. Proof: From (11) we see ha: { g(γ)/ = inf f(s i, x (si) ) p si x (s i ) X (s i ) s i + j ˆγ j Aj (s i, x (si) ) µ j (s i, x (si) ) ]}, where ˆγ j = γj. The righ hand side is exacly g 0(ˆγ), and so is maximized a ˆγ = γ 0. Thus g(γ) is maximized a γ 0.

5 IEEE TRANS. ON AUTOMATIC CONTROL, OL. 56, ISSUE 4, PP , APRIL BACKLOG ECTOR BEHAIOR UNDER QLA In his secion we sudy he backlog vecor behavior under QLA of he sochasic problem. We firs look a he case when g 0 (γ) is locally polyhedral. We show ha q() is mosly wihin O(log( )) disance from γ in his case, even when S() evolves according o a more general ime homogeneous Markovian process. We hen consider he case when g 0 (γ) is locally smooh, and show ha q() is mosly wihin O( log( )) disance from γ. As we will see, hese wo resuls also explain how QLA funcions. The choices of hese wo ypes of g 0 (γ) funcions are made based on heir pracical generaliy. See Secion -C for furher discussion. A. When g 0 () is locally polyhedral In his secion, we sudy he backlog vecor behavior under QLA for he case where g 0 (γ) is locally polyhedral wih parameers ɛ, L, i.e., here exis ɛ, L > 0, such ha for all γ 0 wih γ γ 0 < ɛ, he dual funcion g 0 (γ) saisfies: g 0 (γ 0) g 0 (γ) + L γ 0 γ. (15) We will show ha in his case, even if S() is a general ime homogeneous Markovian process, he backlog vecor will mosly be wihin O(log( )) disance o γ. Hence he same is also rue when S() is i.i.d.. To sar, we assume for his subsecion ha S() evolves according o a ime homogeneous Markovian process. Now we define he following noaions. Given 0, define T si ( 0, k) o be he se of slos a which S(τ) = s i for τ 0, 0 +k 1]. For a given ν > 0, define he convergen inerval T ν 17] for he S() process o be he smalles number of slos such ha for any 0, regardless of pas hisory, we have: M p s i E{ T si ( 0, T ν ) H( 0 ) } ν, (16) T ν here T si ( 0, T ν ) is he cardinaliy of T si ( 0, T ν ), and H( 0 ) = {S(τ)} 0 1 denoes he nework sae hisory up o ime 0. For any ν > 0, such a T ν mus exis for any saionary ergodic processes wih finie sae space, hus T ν exiss for S() in paricular. When S() is i.i.d. every slo, we have T ν = 1 for all ν 0, as E { T si ( 0, 1) H( 0 ) } = p si. Inuiively, T ν represens he ime needed for he process o reach is near seady sae. The following heorem summarizes he main resuls. Recall ha B is defined in (2) as he upper bound of he magniude change of q() in a slo, which is a funcion of he nework size r and δ max. Theorem 1: If g 0 (γ) is locally polyhedral wih consans ɛ, L > 0, independen of, hen under QLA, (a) There exis consans ν > 0, D η > 0, all independen of, such ha D = D(ν), η = η(ν), and whenever q() γ D, we have: E { q( + T ν ) γ q() } q() γ η. (17) In paricular, he consans ν, D and η ha saisfy (17) can be chosen as follows: Choose ν as any consan such ha 0 < ν < L/B. Then choose η as any value such ha 0 < η < T ν (L Bν). Finally, choose D as: 2 (T 2 D = max ν + T ν )B 2 η 2 ] 2T ν (L η T ν Bν), η. (18) (b) For given consans ν, D, η in (a), here exis some consans c, β > 0, independen of, such ha: P(D, m) c e β m, (19) where P(D, m) is defined as: P(D, m) lim sup P r{ q(τ) γ > D + m}. (20) Noe ha if m = log( ) β, by (19) we have P(D, m) c. Also if a seady sae disribuion of q() γ exiss under QLA, e.g., when q j () only akes ineger values for all j, in which case q() is a discree ime Markov chain wih counably infinie saes and he limi of 1 1 P r{ q(τ) γ > D + m} exiss as, hen one can replace P(D, m) wih he seady sae probabiliy ha q() deviaes from γ by an amoun of D + m, i.e., P r{ q() γ > D + m}. Therefore Theorem 1 can be viewed as showing ha when (15) is saisfied, for a large, he backlog q() under QLA will mosly be wihin O(log( )) disance from γ. This implies ha he average backlog will roughly be j γ j, which is ypically Θ( ) by Lemma 1. However, his fac will also allow us o build FQLA upon QLA o subrac ou roughly j γ j daa from he nework and reduce nework delay. Theorem 1 also highlighs a deep connecion beween he seady sae behavior of he nework backlog q() and he srucure of g 0 (γ). We also noe ha (15) is no very resricive. In fac, if g 0 (γ) is polyhedral (e.g., X (si) is finie for all s i ), wih a unique opimal soluion γ 0 0, hen (15) can usually be saisfied (see Secion II for an example). To prove he heorem, we need he following lemma. Lemma 2: For any ν > 0, under QLA, we have for all, E { q( + T ν ) γ 2 q() } (21) q() γ 2 + (T 2 ν + T ν )B 2 2T ν ( g(γ ) g(q()) ) + 2T ν νb γ q(). Proof: See Appendix A. We now use Lemma 2 o prove Theorem 1. Proof: (Theorem 1) Par (a): We firs show ha if (15) holds for g 0 (γ) wih L, hen i also holds for g(γ) wih he same L. To his end, suppose (15) holds for g 0 (γ) for all γ saisfying γ γ 0 < ɛ. Then for any γ 0 such ha γ γ < ɛ, we have γ/ γ 0 < ɛ, hence: g 0 (γ 0) g 0 (γ/ ) + L γ 0 γ/. Muliplying boh sides by, we ge: g 0 (γ 0) g 0 (γ/ ) + L γ 0 γ/. Now using γ = γ 0 and g(γ) = g 0 (γ/ ), we have for all γ γ < ɛ : g(γ ) g(γ) + L γ γ. (22) 2 I can be seen from (14) ha B L. Thus T νb > η.

6 IEEE TRANS. ON AUTOMATIC CONTROL, OL. 56, ISSUE 4, PP , APRIL Since g(γ) is concave, we see ha (22) indeed holds for all γ 0. Now for a given η > 0, if: (T 2 ν + T ν )B 2 2T ν ( g(γ ) g(q()) ) (23) +2T ν νb γ q() η 2 2η γ q(), hen by (21), we have: E { q( + T ν ) γ 2 q() } ( q() γ η) 2, which hen by Jensen s inequaliy implies: (E { q( + T ν ) γ q() } ) 2 ( q() γ η) 2. Thus (17) follows whenever (23) holds and q() γ η. I suffices o choose D and η such ha D η and ha (23) holds whenever q() γ D. Now noe ha (23) can be rewrien as he following inequaly: g(γ ) g(q()) + (Bν + η T ν ) γ q() + Y, (24) where Y = (T 2 ν +Tν)B2 η 2 2T ν. Choose any ν > 0 independen of such ha Bν < L and choose η (0, T ν (L Bν)). By (22), if: L q() γ (Bν + η T ν ) γ q() + Y (25) hen (24) holds. Now choose D as defined in (18), we see ha if q() γ D, hen (25) holds, which implies (24), and equivalenly (23). We also have D η, hence (17) holds. Par (b): Now we show ha (17) implies (19). Choose consans ν, D and η ha are independen of in (a). Denoe Y () = q() γ, we see hen whenever Y () D, we have E { Y ( + T ν ) Y () q() } η. I is also easy o see ha Y ( + T ν ) Y () T ν B, as B is defined in (2) as he upper bound of he magniude change of q() in a slo. Define Ỹ () = max Y () D, 0 ]. We see ha whenever Ỹ () T ν B, we have: E { Ỹ ( + T ν ) Ỹ () q()} (26) = E { Y ( + T ν ) Y () q() } η. Now define a Lyapunov funcion of Ỹ () o be L(Ỹ ()) = e wỹ () wih some w > 0, and define he T ν -slo condiional drif o be: Tν (Ỹ ()) E{ L(Ỹ ( + T ν)) L(Ỹ ()) q()} = E { e wỹ (+Tν) e wỹ () q() }. (27) I is shown in Appendix B ha by choosing w =, we have for all Ỹ () 0: η T 2 ν B2 +T νbη/3 Tν (Ỹ ()) e2wtνb wη 2 ewỹ (). (28) Taking expecaion on boh sides, we have: E { e wỹ (+Tν) e wỹ ()} e 2wTνB wη 2 E{ e wỹ ()}. (29) Now summing (29) over { 0, 0 + T ν,..., 0 + (N 1)T ν } for some 0 {0, 1,..., T ν 1}, we have: E { e wỹ (0+NTν) e wỹ (0)} Ne 2wTνB N 1 j=0 wη 2 E{ e wỹ (0+jTν)}. Rearranging he erms, we have: N 1 j=0 wη 2 E{ e wỹ (0+jTν)} Ne 2wTνB + E { e wỹ (0)}. Summing he above over 0 {0, 1,..., T ν 1}, we obain: NT ν 1 =0 wη 2 E{ e wỹ ()} T ν 1 NT ν e 2wTνB + E { e wỹ (0)}. Dividing boh sides wih NT ν, we obain: 1 NT ν NT ν 1 =0 0=0 wη 2 E{ e wỹ ()} e 2wTνB (30) + 1 T ν 1 E { e wỹ (0)}. NT ν 0=0 Taking he limsup as N goes o infiniy, we obain: lim sup wη 2 E{ e wỹ (τ)} e 2wTνB. (31) Using he fac ha E { e wỹ (τ)} e wm P r{ỹ (τ) > m}, lim sup Plug in w = wη 2 ewm P r{ỹ (τ) > m} e2wtνb. (32) η T 2 ν B2 +T νbη/3 and use he definiion of Ỹ (): P(D, m) 2e2wTνB e wm (33) wη = 2(T 2 ν B 2 + T ν Bη/3)e η 2 2η Tν B+η/3 e ηm T ν 2B2 +Tν Bη/3, where P(D, m) is defined in (20). Therefore (19) holds wih: 2η Tν B+η/3 c = 2(T ν 2 B 2 + T ν Bη/3)e η 2, β = η T 2 ν B 2 + T ν Bη/3. (34) I is easy o see ha c and β are boh independen of. Noe from (30) and (31) ha Theorem 1 indeed holds for any finie q(0). We will laer use his fac o prove he performance of FQLA. The following heorem is a special case of Theorem 1 and gives a more direc illusraion of Theorem 1. Recall ha P(D, m) is defined in (20). Define: P (r) (D, m) (35) lim sup P r{ j, q j (τ) γ j > D + m}. Theorem 2: If he condiion in Theorem 1 holds and S() is i.i.d., hen under QLA, for any c > 0: P(D 1, ck 1 log( )) c 1 c, (36) P (r) (D 1, ck 1 log( )) c 1 c, (37)

7 IEEE TRANS. ON AUTOMATIC CONTROL, OL. 56, ISSUE 4, PP , APRIL where D 1 = 2B2 L + L 4, K 1 = B2 +BL/6 L/2 and c 1 = 8(B 2 L +BL/6)e B+L/6 L. 2 Proof: Firs we noe ha when S() is i.i.d., we have T ν = 1 for ν = 0. Now choose ν = 0, T ν = 1 and η = L/2, hen we see from (18) ha 2B 2 L 2 /4 D = max, L L 2 ] 2B2 L + L 4. Now by (34) we see ha (19) holds wih c = c 1 and β = L/2 B 2 +BL/6. Thus by aking D 1 = 2B2 L + L 4, we have: P(D 1, ck 1 log( )) c e ck1β log( ) = c 1e c log( ), where he las sep follows since β K 1 = 1. Thus (36) follows. Equaion (37) follows from (36) by using he fac ha for any consan ζ, he evens E 1 = { j, q j (τ) γ j > ζ} and E 2 = { q(τ) γ > ζ} saisfy E 1 E 2. Thus: P r{ j, q j (τ) γ j > ζ} P r{ q(τ) γ > ζ}. Theorem 2 can be viewed as showing ha for a large, he probabiliy for q j () o deviae from he j h componen of γ is exponenially decreasing in he disance. Thus i rarely deviaes from γ j by more han Θ(log( )) disance. B. When g 0 () is locally smooh In his secion, we consider he backlog behavior under QLA, for he case where he dual funcion g 0 (γ) is locally smooh a γ 0. Specifically, we say ha he funcion g 0 (γ) is locally smooh a γ 0 wih parameers ε, L > 0 if for all γ 0 such ha γ γ 0 < ε, we have: g 0 (γ 0) g 0 (γ) + L γ γ 0 2, (38) This condiion conains he case when g 0 (γ) is wice differeniable wih g(γ 0) = 0 and a T 2 g(γ)a 2L a 2, a for any γ wih γ 0 γ < ε. Such a case usually occurs when he ses X (si), i = 1,..., M are convex, hus a coninuous se of acions are available. Noice ha (38) is a looser condiion han (15) in he neighborhood of γ 0. As we will see, such srucural difference of g 0 (γ) in he neighborhood of γ 0 grealy affecs he behavior of backlogs under QLA. Theorem 3: If g 0 (γ) is locally smooh a γ 0 wih parameers ε, L > 0, independen of, hen under QLA wih a sufficienly large, we have: (a) There exiss D = Θ( ) such ha whenever q() γ D, we have: E { q( + 1) γ q() } q() γ 1. (39) (b) P(D, m) c e β m, where P(D, m) is defined in (20), c = Θ( ) and β = Θ(1/ ). Theorem 3 can be viewed as showing ha, when g 0 (γ) is locally smooh a γ 0, he backlog vecor will mosly be wihin O( log( )) disance from γ. This conrass wih Theorem 1, which shows ha he backlog will mosly be wihin O(log( )) disance from γ. Inuiively, his is due o he fac ha under local smoohness, he drif owards γ is smaller as q() ges closer o γ, hence a Θ( ) disance is needed o guaranee a drif of size Θ(1/ ); whereas under (15), any nonzero Θ(1) deviaion from γ roughly generaes a drif of size Θ(1) owards γ, ensuring he backlog says wihin O(log( )) disance from γ. To prove Theorem 3, we need he following corollary of Lemma 2. Corollary 1: If S() is i.i.d., hen under QLA, E { q( + 1) γ 2 q() } q() γ 2 + 2B 2 2 ( g(γ ) g(q()) ). Proof: When S() is i.i.d., we have T ν = 1 for ν = 0. Proof: (Theorem 3) Par (a): We firs see ha for any γ wih γ γ < ε, we have γ/ γ 0 < ε. Therefore, g 0 (γ 0) g 0 (γ/ ) + L γ/ γ 0 2. (40) Muliply boh sides wih, we ge: g(γ ) g(γ) + L γ γ 2. (41) Similar as in he proof of Theorem 1 and by Corollary 1, we see ha for (39) o hold, we need q() γ 1 and: 2B 2 2 ( g(γ ) g(q()) ) 1 2 q() γ, which can be rewrien as: g(γ ) g(q()) ) + 1 q() γ + 2B2 1. (42) 2 By (41), we see ha for (42) o hold, we only need: L q() γ 2 1 q() γ + B 2. (43) I is easy o see ha (43) holds whenever: q() γ B2 L + + 4B2 L = 2L/ 2L Denoe D = + +4B 2 L 2L. We see now when is large, (39) holds for any q() wih D q() γ < ε. Now since g(γ) is concave, i is easy o show ha (42) holds for all q() γ D. Hence (39) holds for all q() γ D, proving Par (a). Par (b): By an argumen ha is similar as in he proof of Theorem 1, we see ha Par (b) follows wih: β 3 = and c = 2( B 2 + B /3)e 6 3B +1. C. Discussion of he choices of g 0 (γ) 3 B 2 +B Noe ha in our analysis, we have focused only on he dual funcion g 0 (γ) being eiher locally polyhedral or locally smooh. These choices are made based on heir pracical generaliy. To be more precise, assume wihou loss of generaliy ha here is only one nework sae and he se of feasible acions is a compac subse of R n. In pracice, his acion se is usually finie due o digiizaion. Thus we see from he definiion of g 0 (γ) ha an acion, if chosen given a Lagrange muliplier γ, will remain he chosen acion for a range of Lagrange mulipliers around γ. Hence g 0 (γ) is polyhedral in his case. Now if he granulariy of he acion ses becomes finer and finer, we can expec he dual funcion g 0 (γ) o be smooher and smooher, in he sense ha moving from one acion o anoher close-by acion does no affec he value of

8 IEEE TRANS. ON AUTOMATIC CONTROL, OL. 56, ISSUE 4, PP , APRIL g 0 (γ) by much. Evenually if he granulariy is fine enough hen he acion se can be viewed as convex. Now if he opimal nework performance is achieved a some acion no a he boundary of he acion se, hen we see ha in a small neighborhood of γ, we will usually have a locally smooh g 0 (γ) funcion. Furher noe ha in boh cases, he srucure of g 0 (γ) is independen of. Hence he condiions in Theorem 1 and 3 can ypically be saisfied in pracice. D. Implicaions of Theorem 1 and 3 Consider he following simple problem: an operaor operaes a single queue and ries o suppor a Bernoulli arrival, i.e., eiher 1 or 0 packe arrives every slo, wih rae λ = 0.5 (he rae may be unknown o he operaor) wih minimum energy expendiure. The channel is ime-invarian. The rae-power curve over he channel is given by: µ() = log(1 + P ()), where P () is he allocaed power a ime. Thus o obain a rae of µ(), we need P () = e µ() 1. In every ime slo, he operaor decides how much power o allocae and serves he queue a he corresponding rae, wih he goal of minimizing he ime average power consumpion subjec o queue sabiliy. Le Φ denoe he ime average energy expendiure incurred by he opimal policy. I is no difficul o see ha Φ = e Now we look a he deerminisic problem: min : (e µ 1), s.. : 0.5 µ { I is easy o obain g(γ) = inf µ (e µ 1) + γ(0.5 µ) }. Hence by he KKT condiions 16] one obains ha γ = e 0.5 and he opimal policy is o serve he queue a he consan rae µ = 0.5. Suppose now QLA is applied o he problem. Then a slo, if q() = q, QLA chooses he power o achieve he rae µ() such ha (a] + = maxa, 0]): µ() arg min{ (e µ 1) + q(0.5 µ)} = log( q )] +. (44) which incurs an insananous power consumpion of P () q() 1. In his case, i can be shown ha Theorem 3 applies. Thus for mos of he ime q() γ, γ + ], i.e., q() e 0.5, e ]. Hence i is almos always he case ha: log(e ) µ() log(e ), which implies: µ() Thus by a similar argumen as in 8], one can show ha P Φ + O(1/ ), where P is he average power consumpion. Now consider he case when we can only choose o operae a µ {0, 1 4, 3 4, 1}, wih he corresponding power consumpions being: P {0, e 1 4 1, e 3 4 1, e 1}. One can similarly obain Φ = 1 2 (e e 1 4 ) and γ = 2 (e 3 4 e 1 4 ). In his case, Φ is achieved by ime sharing he wo raes { 1 4, 3 4 } wih equal porion of ime. I can also be shown ha Theorem 1 applies in his case. Thus we see ha under QLA, q() is mosly wihin log( ) disance o γ. Hence by (44), we see ha QLA almos always chooses beween he wo raes { 1 4, 3 4 }, and uses hem wih almos equal frequencies. Hence QLA is also able o achieve P = Φ + O(1/ ) in his case. The above argumen can be generalized o many sochasic nework opimizaion problems. Thus, we see ha Theorem 1 and 3 no only provide us wih probabilisic deviaion bounds of q() from γ, bu also help o explain why QLA is able o achieve he desired uiliy performance: under QLA, q() always says close o γ, hence he chosen acion is always close o he se of opimal acions. E. Discussion of Scalabiliy of Theorem 1 and 3 We noe ha hough our resuls hold for many general mulihop neworks, he decaying exponens in Theorem 1 and 3 will usually depend on he nework size r, e.g., B is a funcion of r. Hence he aracion may be looser as he nework size r increases. erifying wheher he exponens in Theorem 1 and 3 are opimal wih respec o r will be an ineresing fuure research opic. I. THE FQLA ALGORITHM In his secion, we propose a family of Fas Quadraic Lyapunov based Algorihms (FQLA) for general sochasic nework opimizaion problems. We firs provide an example o illusrae he idea of FQLA. We hen describe FQLA wih known γ, called FQLA-Ideal, and sudy is performance. Afer ha, we describe he more general FQLA wihou such knowledge, called FQLA-General. For breviy, we only describe FQLA for he case when g 0 (γ) is locally polyhedral. FQLA for he oher case is discussed in 18]. A. FQLA: a Single Queue Example To illusrae he idea of FQLA, we firs look a an example. Figure 3 shows a slo sample backlog process under QLA. 3 We see ha afer roughly 1500 slos, q() always says very close o γ, which is a Θ( ) scalar in his case. To reduce delay, we can firs find W (0, γ ) such ha: under QLA, here exiss a ime 0 so ha q( 0 ) W and once q() W, i remains so for all ime (he solid line in Fig. 3 shows one for hese 10 4 slos). We hen place W fake bis (called place-holder bis 11]) in he queue a ime 0, i.e., iniialize q(0) = W, and run QLA. I is easy o show ha he uiliy performance of QLA will remain he same wih his change, and he average backlog is now reduced by W. However, such a W may require W = γ Θ( ), hus he average backlog may sill be Θ( ).! * q() Sar here Number of place holder bis W Fig !2 W()!W q() maxw()!w, 0] +! max!4! Lef: A sample backlog process; Righ: Example of W () and q(). FQLA insead finds a W such ha in seady sae, he backlog process under QLA rarely goes below i, and places W place-holder bis in he queue a ime 0. FQLA hen uses an auxiliary process W (), called he virual backlog process, o keep rack of he backlog process ha should have 3 This sample backlog process is one sample backlog process of queue 1 of he sysem considered in Secion II, under QLA wih = 50.

9 IEEE TRANS. ON AUTOMATIC CONTROL, OL. 56, ISSUE 4, PP , APRIL been generaed if QLA is used. Specifically, FQLA iniializes W (0) = W. Then a every slo, QLA is run using W () as he queue size, and W () is updaed according o QLA. Wih W () and W, FQLA works as follows: A ime, if W () W, FQLA performs QLA s acion (obained based on S() and W ()); else if W () < W, FQLA carefully modifies QLA s acion so as o mainain q() maxw () W, 0] for all (see Fig.3 for an example). Similar as above, his roughly reduces he average backlog by W. The difference is ha now we can show ha W = maxγ log( )]2, 0] mees he requiremen. Thus i is possible o bring he average backlog down o O(log( )] 2 ). Also, since W () can be viewed as a backlog process generaed by QLA, i rarely goes below W in seady sae. Hence FQLA is almos always he same as QLA, hus is able o achieve an O(1/ ) close-o-opimal uiliy performance. B. The FQLA-Ideal Algorihm In his secion, we presen he FQLA-Ideal algorihm. We assume he value γ = (γ 1,..., γ r )T is known a-priori. FQLA-Ideal: (I) Deermining place-holder bis: For each j, define: W j = max γ j log( )] 2, 0 ], (45) as he number of place-holder bis of queue j. (II) Place-holder-bi based acion: Iniialize q j (0) = 0, and W j (0) = W j, j. For 1, observe he nework sae S(), solve (7) wih W () in place of q(). Perform he chosen acion wih he following modificaion: Le A() and µ() be he arrival and service rae vecors generaed by he acion. For each queue j, do (Idle fill if needed): a) If W j () W j : admi A j () arrivals, serve µ j () daa, i.e., updae he backlog by: q j ( + 1) = max q j () µ j (), 0 ] + A j (). b) If W j () < W j : admi Ãj() = max A j () W j + W j (), 0 ] arrivals, serve µ j () daa, i.e., updae he backlog by: q j ( + 1) = max q j () µ j (), 0 ] + Ãj(). c) Updae W j () by: W j ( + 1) = max W j () µ j (), 0 ] + A j (). From above we see ha FQLA-Ideal is he same as QLA based on W () when W j () W j for all j. When W j () < W j for some queue j, FQLA-Ideal admis roughly he excessive packes afer W j () is brough back o be above W j for he queue. Thus for problems where QLA admis an easy implemenaion, e.g., 3], 5], i is also easy o implemen FQLA. However, we also noice wo differen feaures of FQLA: (1) By (45), W j can be 0. However, when is large, his happens only when γ0j = γ j = 0 according o Lemma 1. In his case W j = γ j = 0, and queue j indeed needs zero place-holder bis. (2) Packes may be dropped in Sep II-(b) upon heir arrivals, or afer hey are admied ino he nework in a mulihop problem. Such packe dropping is naural in many flow conrol problems and does no change he naure of hese problems. In oher problems where such opion is no available, he packe dropping opion is inroduced o achieve desired delay performance, and i can be shown ha he fracion of packes dropped can be made arbirarily small. Noe ha packe dropping here is o compensae for he deviaion from he desired Lagrange muliplier, hus is differen from ha in 19], where packe dropping is used for drif seering. C. Performance of FQLA-Ideal Here we look a he performance of FQLA-Ideal. We firs have he following lemma ha shows he relaionship beween q() and W () under FQLA-Ideal. We will use i laer o prove he delay bound of FQLA. Noe ha he lemma also holds for FQLA-General described laer, as FQLA-Ideal/General differ only in he way of deermining W = (W 1,..., W r ) T. Lemma 3: Under FQLA-Ideal/General, we have j, : max W j () W j, 0 ] q j () max W j () W j, 0 ] +δ max (46) where δ max is defined in Secion III-B o be he upper bound of he number of arriving or deparing packes of a queue. Proof: See Appendix C. The following heorem summarizes he main performance resuls of FQLA-Ideal. Recall ha for a given policy π, f π av denoes is average cos defined in (5) and f π () denoes he cos induced by π a ime. Theorem 4: If he condiion in Theorem 1 holds and a seady sae disribuion exiss for he backlog process generaed by QLA, hen wih a sufficienly large, we have under FQLA-Ideal ha, q = O(log( )] 2 ), (47) fav F I = fav + O(1/ ), (48) P drop = O(1/ c0 log( ) ), (49) where c 0 = Θ(1), q is he ime average backlog, fav F I is he expeced ime average cos of FQLA-Ideal, fav is he opimal ime average cos and P drop is he ime average fracion of packes ha are dropped in Sep-II (b). Proof: Since a seady sae disribuion exiss for he backlog process generaed by QLA, we see ha P(D, m) in (20) represens he seady sae probabiliy of he even ha he backlog vecor deviaes from γ by disance D +m. Now since W () can be viewed as a backlog process generaed by QLA, wih W (0) = W insead of 0, we see from he proof of Theorem 1 ha Theorem 1 and 2 hold for W (), and by 7], QLA based on W () achieves an average cos of fav + O(1/ ). Hence by Theorem 2, here exis consans D 1, K 1, c 1 = Θ(1) so ha: P (r) (D 1, ck 1 log( )) c 1. By c he definiion of P (r) (D 1, ck 1 log( )), his implies ha in seady sae: P r{w j () > γ j + D 1 + m} c 1e m K 1. Now le: Q j () = maxw j () γ j D 1, 0]. We see ha P r{q j () > m} c 1e m K 1, m 0. We hus have Q j = O(1), where Q j is he ime average value of Q j (). Now i is easy o see by (45) and (46) ha q j () Q j () + log( )] 2 + D 1 + δ max for all. Thus (47) follows since for a large : q j Q j + log( )] 2 + D 1 + δ max = Θ(log( )] 2 ).

10 IEEE TRANS. ON AUTOMATIC CONTROL, OL. 56, ISSUE 4, PP , APRIL Now consider he average cos. To save space, we use FI for FQLA-Ideal. From above, we see ha QLA based on W () achieves an expeced average cos of f av + O(1/ ). Thus i suffices o show ha FQLA-Ideal performs almos he same as QLA based on W (). Firs we have for all 1 ha: f F I (τ) = f F I (τ)1 E(τ) + f F I (τ)1 Ec (τ). Here 1 E(τ) is he indicaor funcion of he even E(τ), E(τ) is he even ha FQLA-Ideal performs he same acion as QLA a ime τ, and 1 E c (τ) = 1 1 E(τ). Taking expecaion on boh sides and using he fac ha when FQLA-Ideal akes he same acion as QLA, f F I (τ) = f QLA (τ), we have: E { f F I (τ) } E { f QLA } (τ)1 E(τ) E { δ max 1 Ec (τ)}. Taking he limi as goes o infiniy on boh sides and using f QLA (τ)1 E(τ) f QLA (τ), we ge: fav F I fav QLA + δ max lim = f QLA av 1 + δ max lim 1 E { } 1 E c (τ) P r{e c (τ)}. (50) However, E c (τ) is included in he even ha here exiss a j such ha W j (τ) < W j. Therefore by (37) in Theorem 2, for a large such ha 1 2 log( )]2 D 1 and log( ) 8K 1, lim P r{e c (τ)} P (r) (D 1, log( )] 2 D 1 ) Using his fac in (50), we obain: = O(c 1/ 1 2K 1 log( ) ) = O(1/ 4 ). (51) f F I av = f QLA av + O(δ max / 4 ) = f av + O(1/ ), where he las equaliy holds since fav QLA = fav + O(1/ ). This proves (48). (49) follows since packes are dropped a ime τ only if E c (τ) happens, hus by (51), he fracion of ime when packe dropping happens is O(1/ c0 log( ) ) wih c 0 = 1 2K 1 = Θ(1), and each ime no more han rb packes can be dropped. D. The FQLA-General algorihm Now we describe he FQLA algorihm wihou any a-priori knowledge of γ, called FQLA-General. FQLA-General firs runs he sysem for a long enough ime T, such ha he sysem eners is seady sae. Then i chooses a sample of he queue vecor value o esimae γ and uses ha o decide W. FQLA-General: (I) Deermining place-holder bis: a) Choose a large ime T (See Secion I-E for he size of T ) and iniialize W (0) = 0. Run he QLA algorihm wih parameer, a every ime slo, updae W () according o he QLA algorihm and obain W (T ). b) For each queue j, define: W j = max W j (T ) log( )] 2, 0 ], (52) as he number of place-holder bis. (II) Place-holder-bi based acion: same as FQLA-Ideal. The performance of FQLA-General is summarized as follows: Theorem 5: Assume he condiions in Theorem 4 hold and he sysem is in seady sae a ime T, hen under FQLA-General wih a sufficienly large, wih probabiliy 1 O( 1 ): (a) q = O(log( )] 2 ), (b) f F G 4 av = fav + O(1/ ), and (c) P drop = O(1/ c0 log( ) ), where c 0 = Θ(1) and fav F G is he expeced ime average cos of FQLA-General. Proof: We will show ha wih probabiliy of 1 O( 1 ), 4 W j is close o maxγ j log( )]2, 0]. The res can hen be proven similarly as in he proof of Theorem 4. For each queue j, define: v + j = γ j log( )]2, v j = max γ j 1 2 log( )]2, 0 ]. Noe ha v j is defined wih a max ] operaor. This is due o he fac ha γ j can be zero. As in (51), we see ha by Theorem 2, here exiss D 1 = Θ(1), K 1 = Θ(1) such ha if is such ha 1 4 log2 ( ) D 1 and log( ) 16K 1, hen: P r { j, W j (T ) / v j, v+ j ]} P (r) (D 1, 1 2 log( )]2 D 1 ) = O(1/ 4 ). Thus P r { W j (T ) v j, v+ j ] j} = 1 O(1/ 4 ), implying: P r { W j ˆv j, ˆv+ j ] j} = 1 O(1/ 4 ). where ˆv + j = max γ j 1 2 log( )]2, 0 ] and ˆv j = max γ j 3 2 log( )]2, 0 ]. Hence for a large, wih probabiliy 1 O( 1 ): if γ 4 j > 0, we have γ j 3 2 log( )]2 W j γ j 1 2 log( )]2 ; else if γ j = 0, we have W j = γ j. The res of he proof is similar o he proof of Theorem 4. E. Pracical Issues From Lemma 1 we see ha he magniude of γ can be Θ( ). This means ha T in FQLA-General may need o be Ω( ), which is no very desirable when is large. We can insead use he following heurisic mehod o accelerae he process of deermining W: For every queue j, guess a very large W j. Then sar wih his W and run he QLA algorihm for some T 1, say slos. Observe he resuling backlog process. Modify he guess for each queue j using a bisecion algorihm unil a proper W is found, i.e. when running QLA from W, we observe flucuaions of W j () around W j insead of a nearly consan increase or decrease for all j. Then le W j = maxw j log( )] 2, 0]. To furher reduce he error probabiliy, one can repea Sep-I (a) muliple imes and use he average value as W (T ).

11 IEEE TRANS. ON AUTOMATIC CONTROL, OL. 56, ISSUE 4, PP , APRIL II. SIMULATION In his secion we provide simulaion resuls for he FQLA algorihms. For simpliciy, we only consider he case where g 0 (γ) is locally polyhedral. We consider a five queue sysem similar o he example in Secion III-D. In his case r = 5. The sysem is shown in Fig. 4. The goal is o perform power allocaion a each node so as o suppor he arrival wih minimum energy expendiure. Fig. 4. R() CH1() CH2() CH() CH4() CH5() q1 q2 q3 q4 q5 A five queue sysem In his example, he random nework sae S() is he vecor (R(), CH i (), i = 1,.., 5). Similar as in Secion III-D, we have: A() = (R(), µ 1 (), µ 2 (), µ 3 (), µ 4 ()) T and µ() = (µ 1 (), µ 2 (), µ 3 (), µ 4 (), µ 5 ()) T, i.e., A 1 () = R(), A i () = µ i 1 () for i 2, where µ i () is he service rae obained by queue i a ime. R() is 0 or 2 wih probabiliies 3 8 and 5 8, respecively. CH i() can be Good or Bad wih equal probabiliies for 1 i 5. When he channel is good, one uni of power can serve wo packes; oherwise i can serve only one. We assume CH i () are all independen and all channels can be acivaed a he same ime wihou affecing ohers. I can be verified ha γ = (5, 4, 3, 2, )T is unique. In his example, he backlog vecor process evolves as a Markov chain wih counably many saes. Thus here exiss a saionary disribuion for he backlog vecor under QLA. repea Sep-I 100 imes and use heir average as W (T ). The up-lef plo in Fig. 5 shows ha he average queue sizes under boh FQLAs are always close o he value 5log( )] 2 (r = 5). The up-righ plo shows ha he percenage of packes dropped decreases rapidly and ges below 10 4 when 500 under boh FQLAs. These plos show ha in pracice, may no have o be very large for Theorem 4 and 5 o hold. The boom plo shows a sample (W 1 (), W 2 ()) process for a slo inerval under FQLA-Ideal wih = 1000, considering only he firs wo queues of Fig. 4. We see ha (W 1 (), W 2 ()) always remains close o (γ 1, γ 2 ) = (5, 4 ), and W 1() W 1 = 4952, W 2 () W 2 = For all values, he average power expendiure is very close o 3.75, which is he opimal energy expendiure, and he average of j W j() is very close o 15 (plos omied for breviy). Ineresingly, he aracion phenomenon in he boom plo of Fig. 5 was also observed in a recen paper 12], which implemened he QLA algorihm in a 40-node wireless sensor nework esbed. I has also been shown in 12] ha by using QLA plus LIFO, one can reduce he delay experienced by all bu a small fracion of he nework raffic by more han 90%. While his fac can no be explained by any previous resuls on QLA, i can easily be explained using Theorem 1 and 3 as follows: Consider a node j. Under LIFO, new packes enering Node j are placed o he fron of he buffer. We also know ha q j I = γ j log( )]2, γ j + log( )]2 ] for mos of he ime. Thus mos packes ener and leave Node j when q j I. Hence for mos packes, Node j is a queue wih on average no more han 2log( )] 2 packes. Hence mos packes only need o wai for on average no more han Θ(log( )] 2 ) packes before geing served !1 250 FQLA!I FQLA!G rlog 2 () 10!2 FQLA!I FQLA!G III. LAGRANGE MULTIPLIER: SHADOW PRICE AND NETWORK GRAITY W 2 () !3 10!4 10! (W 1 (), W 2 ()) (5000, 4000) W 1 () Fig. 5. FQLA-Ideal performance: Up-Lef - Average queue size; Up-Righ - Percenage of packes dropped; Boom - Sample (W 1 (), W 2 ()) process for 10000, ] and = 1000 under FQLA-Ideal. We simulae FQLA-Ideal and FQLA-General wih = 50, 100, 200, 500, 1000 and We run each case for slos. For FQLA-General, we use T = 50 in Sep-I and 4 I is well known ha Lagrange Mulipliers can play he role of shadow prices o regulae flows in many flow-based problems wih differen objecives, e.g., 20]. This imporan feaure has enabled he developmen of many disribued algorihms in resource allocaion problems, e.g., 21]. However, a problem of his ype ypically requires daa ransmissions o be represened as flows. Thus in a nework ha is discree in naure, e.g., ime sloed or packeized ransmission, a rae allocaion soluion obained by solving such a flow-based problem does no immediaely specify a scheduling policy. Recenly, several Lyapunov algorihms have been proposed o solve uiliy opimizaion problems under discree nework seings. In hese algorihms, backlog vecors ac as he graviy of he nework and allow opimal scheduling o be buil upon hem. I is also revealed in 17] ha QLA is closely relaed o he dual subgradien mehod and backlogs play he same role as Lagrange mulipliers in a ime invarian nework. Now we see by Theorem 1 and 3 ha he backlogs indeed play he same role as Lagrange mulipliers even under a more general sochasic nework. 4 This secion appeared in he WiOp 2009 paper. However, i is no included in he IEEE TAC version due o space consideraion.