Stochastic Network Optimization with Non-Convex Utilities and Costs

Transcription

1 PROC. INFORMATION THEORY AND APPLICATIONS WORKSHOP (ITA), SAN DIEGO, FEB Sochasic Nework Opiizaion wih Non-Convex Uiliies and Coss Michael J. Neely Absrac This work considers non-convex opiizaion of ie averages of nework aribues in a general sochasic nework. This includes axiizing a non-concave uiliy funcion of he ie average hroughpu vecor in a ie-varying wireless syse, subjec o nework sabiliy and o an addiional collecion of ie average penaly consrains. We develop a siple algorih ha ees all desired sabiliy and penaly consrains, and, subjec o a convergence assupion, yields a ie average vecor ha is a local opiu of he desired uiliy funcion. We also consider algorihs ha yield local near opial soluions, where he disance o a local opiu can be ade as sall as desired wih a corresponding radeoff in average delay. Our soluion uses Lyapunov opiizaion wih a cobinaion of sochasic dual and prial-dual echniques. We also discuss he relaive advanages and disadvanages of hese echniques. Index Ters Queueing analysis, opporunisic scheduling, flow conrol, wireless neworks I. INTRODUCTION We consider a queueing nework ha operaes in discree ie {0, 1, 2,...}. Le Q() = (Q 1 (),..., Q K ()) be he vecor of queue backlogs on slo. The arrival and service of he queues are deerined by a rando even ω() and a nework conrol acion α(), chosen in reacion o his even fro an absrac se of possible acions A ω() ha possibly depends on ω(). The α() and ω() values also affec wo aribue vecors x() = (x 1 (),..., x M ()), y() = (y 0 (), y 1 (),..., y L ()) according o general (possibly non-convex, disconinuous) funcions ˆx ( ), ŷ l ( ) for {1,..., M} and l {0, 1,..., L}: x () = ˆx (α(), ω()), y l () = ŷ l (α(), ω()) Define x = (x 1,..., x M ) as he liiing ie average expecaion of he aribue vecor x() under a paricular policy (eporarily assued o exis): x= 1 1 li E {x(τ)} Define ie average expecaions y l siilarly. The goal is o deerine an algorih for choosing acions α() A ω() Michael J. Neely is wih he Elecrical Engineering deparen a he Universiy of Souhern California, Los Angeles, CA. (web: hp://wwwrcf.usc.edu/ jneely). This aerial is suppored in par by one or ore of he following: he DARPA IT-MANET progra gran W911NF , he NSF Career gran CCF , and coninuing hrough paricipaion in he Nework Science Collaboraive Technology Alliance sponsored by he U.S. Ary Research Laboraory. Fig. 1. Uiliy(x) Aribue x (such as hroughpu) An exaple non-concave uiliy for hroughpu axiizaion. over ie o solve he following general sochasic nework opiizaion proble: Miniize: y 0 f(x) (1) Subjec o: y l g l (x) 0 l {1,..., L} (2) x X (3) All queues Q k () are ean rae sable (4) α() A ω() (5) where g l (x) are coninuous and convex funcions of x R M for each l {1,..., L}, f(x) is a coninuous bu possibly non-convex funcion of x R M, and X is a convex subse of R M. The definiion of ean rae sable will be given in Secion II. This non-convex proble has applicaions in nework uiliy axiizaion and in oher areas of nework science. For exaple, suppose we desire o axiize a non-concave uiliy funcion of hroughpu in a wireless nework. An exaple sigoidal-ype uiliy funcion is shown in Fig. 1, which has a fla area near zero, illusraing ha we earn sall uiliy unless hroughpu crosses a hreshold. Consrained opiizaion of a su of such uiliy funcions is a difficul proble ha can be shown, in soe cases, o be a leas as difficul as cobinaorial bin-packing. Hence, we do no aep o find a global soluion. Such non-convex probles are sudied for non-sochasic (saic) neworks in [1], where difficulies are discussed and a heurisic algorih is derived ha has opialiy properies in a lii of a large nuber of users. Furher reaen of non-convex opiizaion in a saic conex, including efficien heurisics, are provided in [2]. In his paper, we consider non-convex opiizaion for sochasic neworks via he general proble (1)-(5). We show ha, under soe convergence assupions, we can find a local opiu. In he special case when f(x) is convex, we provide a sronger resul ha shows he global soluion can

2 PROC. INFORMATION THEORY AND APPLICATIONS WORKSHOP (ITA), SAN DIEGO, FEB be approached o wihin any desired accuracy. Our echniques exend our prior drif-plus-penaly or dual-based approach o sochasic nework opiizaion in [3], which uses Lyapunov opiizaion, virual queues, and auxiliary variables. These echniques alone would allow one o iniize (1) in he case f(x) is convex, subjec o consrains (2), (4), (5). We inroduce an absrac se consrain (3), which we believe is new even in he convex case (we recenly also used his in a universal scheduling conex in [4]). However, if hese dualbased echniques were eployed for a non-convex funcion f(x), he algorih would approach a global opiu of he ie average of f(x()), which is no necessarily even a local opiu of f(x). In his paper, we achieve local opialiy by cobining hese echniques wih a prial-dual based approach siilar o ha of [5][6][7]. I is known ha, in he convex case, he dual approach is robus o syse changes [3][8][4] and provides direc bounds on uiliy, delay, and convergence ie [3][9][10][11]. Unforunaely, such properies are less clear wih he prialdual approach, paricularly for he non-convex case, and he proof we provide in his paper for he non-convex case does no specify he ie required for convergence. This illusraes soe of he challenges involved wih non-convex opiizaion, even if we only wan o find a local opiu. We parially address his issue in he special case of axiizing nonconvex uiliy funcions ha are enrywise non-decreasing. In his case, we briefly describe an alernaive 3-phase algorih ha explores he aribue space and ies all sochasic decisions direcly o he resul of a saic non-convex opiizaion. The advanage here is ha he sochasic phases are convex and have srong convergence guaranees, while he non-convex opiizaion is isolaed o a saic proble for which any saic solver can be eployed. The ehod of Lyapunov drif o solve queue sabiliy probles (i.e., probles ha involve only he consrain (4)) was pioneered by Tassiulas and Ephreides in [12][13], where i was shown ha greedy acions ha iniize a drif expression () every slo guaranee sabiliy whenever possible. In [9][3][10][11], we exended his resul o provide join sabiliy and perforance opiizaion (or penaly iniizaion ), using a greedy acion o iniize a drif-plus-penaly eric () V penaly() every slo, where V is a non-negaive paraeer ha affecs a perforance-delay radeoff. Choosing V = 0 reduces o he original Tassiulas-Ephreides approach, while increasing V ainains nework sabiliy while pushing he ie average penaly of he proble o wihin O(1/V ) of opial, wih a radeoff in average queue backlog ha is O(V ). This ehod can be applied o uli-hop neworks wih general consrains, and is known o be relaed o dual subgradien algorihs when applied o saic convex progras [3]. A relaed ehod, also of he dual ype, is presened for a wireless downlink in [14], alhough he fluid odel arguens here do no prove he [O(1/V ), O(V )] perforance-delay properies. A differen prial-dual approach is considered for he special case of uiliy-opial opporunisic scheduling in a wireless downlink in [5][6]. This work assues all users always have daa o send, and hence does no consider randoly arriving raffic or queue sabiliy issues. I also uses an infinie horizon ie-average o infor scheduling decisions, which, unlike he dual ehod, akes i difficul o apply o syses wih paraeers ha igh change. Join queue sabiliy and perforance opiizaion is reaed for uli-hop neworks wih his prial-dual approach in [7]. This work proves ha he uiliy of a relaed fluid nework is opial, and conjecures ha he uiliy of he acual nework will also be close o opial. Specifically, he infinie horizon ie average of [5][6] is replaced by an exponenially weighed average in [7], and i is conjecured ha uiliy approaches opialiy when he exponenial decay paraeer is scaled (see Secion 4.9 in [7]). In he nex secion we describe he syse odel and he firs approach o he non-convex sochasic opiizaion. Secion III analyzes he perforance of he algorih using he drif-plus-penaly echnique wih a penaly ha involves a parial derivaive (incorporaing he prial-dual approach). We also copare agains he perforance of a pure dual-based algorih in he special case of convex probles. Secion IV develops our second approach o he non-convex proble, for he case of opiizing non-concave uiliy funcions ha are enrywise non-decreasing. A. Model Assupions II. SOLVING THE PROBLEM The queueing dynaics for k {1,..., K} are given by: Q k ( 1) = ax[q k () b k (), 0] a k () (6) where b() = (b 1 (),..., b K ()), a() = (a 1 (),..., a K ()) are non-negaive service and arrival vecors, deerined as arbirary funcions of α() and ω(): a k () = â k (α(), ω()), b k () = ˆb k (α(), ω()) The srucure of a k () being ouside he ax[, 0] operaor in (6) also allows reaen of uli-hop neworks, where a k () can be a su of exogenous and endogenous arrivals [3][15]. We assue he rando even process ω() is i.i.d. over slos, wih soe (possibly unknown) disribuion. Second oens of he arrival, service, and aribue coponens are assued o be finie under all possible conrol acions. Specifically, for all (possibly randoized) choices of α A ω(), ade based on knowledge of ω(), here is a finie consan σ 2 > 0 such ha: E { â k (α(), ω()) 2} σ 2, E {ˆbk (α(), ω()) 2} σ 2 E {ˆx (α(), ω()) 2} σ 2, E { ŷ l (α(), ω()) 2} σ 2 These second oen bounds also iply he firs oens of nework aribues lie wihin a bounded region, even if he acual realizaions of hese aribues are unbounded. The se X in (3) is assued o be a copac and convex subse of R M. We assue he funcions g l (x) are convex (no necessarily differeniable) for all l {1,..., L}, and ha hey are Lipschiz coninuous, in he sense ha for any x = (x 1,..., x M ), γ = (γ 1,..., γ M ) we have: g l (x) g l (γ) ν l, x γ (7)

3 PROC. INFORMATION THEORY AND APPLICATIONS WORKSHOP (ITA), SAN DIEGO, FEB where ν l, are finie non-negaive consans. The funcion f(x) is possibly non-convex, bu is assued o be bounded, coninuously differeniable, and o have bounded parial derivaives, so ha: f(x) f(γ) β x γ (8) for soe finie consans β 0. Le f in, f ax be finie consans such ha for all we have f in f(x()) f ax. B. Queue Sabiliy We firs define wo ypes of queue sabiliy. Le Q() be a discree ie process defined over slos {0, 1, 2,...}, assued o ake (possibly negaive) real values. Negaive queue processes will be useful o ee cerain consrains. Definiion 1: Q() is ean rae sable if: E { Q() } li = 0 Definiion 2: Q() is srongly sable if: 1 1 li sup E { Q(τ) } < Under ild echnical assupions i can be shown ha srong sabiliy iplies ean rae sabiliy, and ha srong sabiliy is equivalen o he exisence of a saionary disribuion in cases when Q() is defined over a counably infinie sae Markov chain. C. Auxiliary Variables To enable he absrac se consrain (3) o be e and o allow opiizaion over he convex bu possibly non-linear funcions g l (x), we inroduce a vecor of auxiliary variables γ() = (γ 1 (),..., γ M ()). This is an addiional collecion of decision variables ha us be chosen every slo, subjec only o he consrain: γ() X We ransfor he proble (1)-(4) ino: Miniize: y 0 f(x) (9) Subjec o: y l g l (γ) 0 l {1,..., L} (10) where noaion g l (γ) is defined: γ = x {1,..., M} (11) All queues Q k () are ean rae sable (12) α() A ω(), γ() X (13) g l (γ) = 1 1 li E {g l (γ(τ))} The oivaion for his new proble is he observaion ha, assuing we (eporarily) resric o algorihs ha have well defined liis, he new proble is equivalen o (1)-(5). Indeed, suppose ha α () is a policy ha ees all consrains of he proble (1)-(5), and ha yields a cos eric in (1) of y 0 f(x ), where y l and x represen ie averages associaed wih his policy. Then his policy also saisfies he consrains of proble (9)-(13), and yields he sae cos value in (9), if we define consan auxiliary variable decisions γ () = x for all (hese saisfy he required consrains (13) because of (3)). I follows ha he infiu cos eric of he proble (1)-(5) over all policies ha ee he consrains is greaer han or equal o he infiu associaed wih proble (9)-(13). Conversely, by Jensen s inequaliy, i can be shown ha any algorih ha solves he consrains in he proble (9)-(13) us also saisfy he consrains of he proble (1)-(5), and produces a cos (1) ha is he sae. For sipliciy of exposiion, we have chosen o wrie he sochasic nework opiizaion probles (1)-(5) and (9)-(13) wih noaion ha iplicily assues liis are well defined. However, a soluion need no have well defined liis, and we can ore precisely define he proble using a li sup, as done in fuure secions (for exaple, he consrain (2) is wrien ore generally as (16)). D. Virual Queues To ensure he inequaliy consrains (10) and equaliy consrains (11) are e, we inroduce virual queues Z l () and H () for l {1,..., L}, {1,..., M}, wih updae equaion: Z l ( 1) = ax[z l () y l () g l (γ()), 0] (14) H ( 1) = H () γ () x () (15) The Z l () queues are always non-negaive. The H () queues have a differen srucure because hey enforce an equaliy consrain, and can possibly be negaive. Lea 1: If E {Z l (0)} < and E { H (0) } < for all l {1,..., L} and {1,..., M}, and if Z l () and H () are ean rae sable, hen: li sup [y l () g l (x())] 0 l {1,..., L} (16) li γ () x () = 0 {1,..., M} (17) li dis (x(), X ) = 0 (18) where for each > 0, x() and γ() are defined: x() = 1 1 E {x(τ)}, γ() = 1 1 E {γ(τ)} (19) where y l () is defined siilarly, and where dis (x(), X ) is he disance beween he vecor x() and he se X, being zero if and only if x() is in he (closed) se X. The proxiiy of he ie averages o heir required values for all ie > 0 is provided as a funcion of he virual queue sizes in (20) and (21) of he proof. Proof: (Lea 1) Fix > 0. Noe by suing (15) over τ {0,..., 1} we have: 1 1 H () H (0) = γ (τ) x (τ) Dividing boh sides by and aking expecaions yields: E {H ()} E {H (0)} = γ () x () (20)

4 PROC. INFORMATION THEORY AND APPLICATIONS WORKSHOP (ITA), SAN DIEGO, FEB The above holds for all > 0. Assuing ha H () is ean rae sable, we can ake a lii of (20) as o yield (17). 1 Now noe ha γ() X for all, and hence (because X is convex), γ() X for all > 0. This ogeher wih (17) iplies (18). Finally, fro (14) we have for all slos τ: Z l (τ 1) Z l (τ) y l (τ) g l (γ(τ)) Fixing > 0 and suing over τ {0,..., 1} yields: 1 1 Z l () Z l (0) y l (τ) g l (γ(τ)) Taking expecaions, dividing by and using Jensen s inequaliy yields: E {Z l ()} E {Z l(0)} y l () g l (γ()) y l () g l (x()) ν l, x () γ () (21) where he las inequaliy is due o (7). Assuing Z l () is ean rae sable, we can ake he li sup of boh sides of (21) and use (17) o yield (16). E. Lyapunov Opiizaion Define Θ() =[Q(); Z(); H(); x av ()] as he collecive queue vecor ogeher wih he curren running ie-average of aribue vecor x(τ): x av () = 1 1 x(τ) (22) Define he following non-negaive funcion, called a Lyapunov funcion: L(Θ()) = 1 Q k () 2 1 Z l () 2 1 H () 2 (23) Define he condiional Lyapunov drif (Θ()) by: 2 (Θ()) =E {L(Θ( 1)) L(Θ()) Θ()} (24) Lea 2: Under any algorih for choosing α() A ω() and γ() X for all, we have for all and all possible Θ(): where (Θ()) RHS(α(), γ(), Θ()) (25) RHS(α(), γ(), Θ()) = { B Q k ()E â k (α(), ω()) ˆb } k (α(), ω()) Θ() Z l ()E {ŷ l (α(), ω()) g l (γ()) Θ()} H ()E {γ () ˆx (α(), ω()) Θ()} 1 Noe ha if E { H () } / 0 hen E {H ()} / 0. 2 Sricly speaking, correc noaion should be (Θ(), ), as he drif ay be non-saionary, alhough we use he sipler noaion (Θ()) as a foral represenaion of he righ hand side of (24). where B is a finie consan ha saisfies he following for all and all possible Θ(): B E { b k () 2 a k () 2 Θ() } E { (y l () g l (γ())) 2 Θ() } E { (γ () x ()) 2 Θ() } (26) Such a finie consan B exiss by he syse boundedness assupions. Proof: The proof follows by squaring he equaions (6), (14), (15) and is oied for breviy (see [3]). Our algorih seeks o ake conrol acions α() A ω(), γ() X every slo in reacion o he observed ω() and o he curren queue saes and x av () values in Θ(), o iniize: RHS(α(), γ(), Θ()) V E {ŷ 0 (α(), ω()) Θ()} V f(x av ()) E {ˆx (α(), ω()) Θ()} (27) where x av () is defined in (22), and where we define x av (0) as an iniial saple ˆx(α( 1), ω( 1)) under any decision α( 1) A ω( 1) aken a soe preliinary slo. The paraeer V is chosen as a non-negaive consan ha will affec proxiiy o he opial soluion, wih a corresponding radeoff in average queue congesion, as in [3]. In he special case when here is no non-convex funcion, so ha f(x) = 0, we do no require he ie averages x av (). In his case he above algorih reduces o one ha is siilar o our work in [3], wih he excepion ha we are considering an absrac se consrain X and he convex funcions g l (x) are no necessarily onoonic. In he case when f(x) 0, our algorih includes a derivaive uliplied by a running ie average, as in he prial-dual algorihs ha do no consider queue sabiliy in [5][6]. This prial-dual coponen is also siilar o fluid lii works in [7][16], which do consider queue sabiliy, wih he excepion ha our algorih includes auxiliary variables and does no use an exponenial weighed average. 3 An advanage of our approach is ha i is copaible wih non-convex objecives, ensures all convex consrains are saisfied as, and provides explici bounds on consrain violaion for all > 0. The algorih is specified as follows: Every slo, observe ω(), Θ() (where Θ() includes all curren queue saes and he curren x av ()), and perfor he following: (Auxiliary Variables) Choose γ() X o iniize: Z l ()g l (γ()) H ()γ () 3 Using an exponenial weighed average would be advanagous for robusness o syse changes, bu i is no clear if he algorih would converge properly in his case.

5 PROC. INFORMATION THEORY AND APPLICATIONS WORKSHOP (ITA), SAN DIEGO, FEB (Choosing α()) Choose α() A ω() o iniize: Q k ()[â k (α(), ω()) ˆb k (α(), ω())] Z l ()ŷ l (α(), ω()) V ŷ 0 (α(), ω()) H ()ˆx (α(), ω()) V f(x av ()) ˆx (α(), ω()) (Queue Updaes) Updae he virual queues Z l () and H () according o (14), (15), and updae he acual queues Q k () via (6). Also updae x av () by (22). III. PERFORMANCE ANALYSIS A. Opialiy via ω-only Policies We say ha he proble (1)-(5) is feasible if here is an algorih for choosing α() A ω() over ie so ha all consrains (2)-(5) are saisfied. We assue hroughou ha he proble is feasible. Define y op 0 f op as he infiu value of (1) over all feasible policies. I urns ou ha his infiu can be characerized by ω-only policies. Specifically, we say ha a policy α () is ω-only if for all i chooses α () A ω() as a saionary and randoized funcion of he observed ω(), i.i.d. over all slos for which he sae ω() value is observed. By he law of large nubers, all ie averages resuling fro an ω-only policy are well defined and are equal o heir expecaions over one slo. In paricular, we have for all : y l = E {ŷ l (α (), ω())}, x = E {ˆx(α (), ω())} For a given ɛ > 0, define he se Y ɛ as he se of all vecors (y, x) = (y 0, y 1,..., y L, x 1,..., x M ) ha can be achieved as ie averages by soe ω-only policy α () A ω() ha saisfies: y l g l (x) ɛ l {1,..., L} dis(x, X ) ɛ { E â k (α (), ω()) ˆb } k (α (), ω()) ɛ The se Y ɛ is bounded by he boundedness assupions, and can be shown o be convex and o saisfy Y ɛ1 Y ɛ2 whenever ɛ 1 ɛ 2. Define Cl(Y ɛ ) as he closure of Y ɛ, and define Y as he lii of Cl(Y ɛ ) as ɛ 0: Y = i=1 Cl(Y 1/i ) I can be shown ha Y is non-epy whenever he proble (1)-(5) is feasible, and is bounded, closed, and convex. Theore 1: (Opialiy over he se Y) The se Y represens he se of all vecors (y, x) ha can be achieved as a lii poin of a rajecory (y(), x()) generaed by a feasible policy ipleened over {0, 1, 2,...}. Furher, he infiu value y op 0 f op is he soluion o he following proble: Miniize: y 0 f(x) Subjec o: (y 0, y 1,..., y L, x 1,..., x M ) Y Proof: Oied for breviy (see, for exaple, [9][10]). Theore 2: For any vecor (y, x ) Y and for any ɛ > 0, here is an ω-only policy α () such ha: E {ŷ l (α (), ω())} y l ɛ l {0, 1,..., L} (28) E {ˆx (α (), ω())} x ɛ {1,..., M} (29) E {ŷ l (α (), ω())} g l (E {ˆx(α (), ω())}) ɛ (30) dis (E {ˆx(α (), ω())}, X ) ɛ (31) } E {â k (α (), ω())} E {ˆbk (α (), ω()) ɛ k (32) Proof: The proof follows by observing ha any poin in Y can be achieved arbirarily closely by an ω-only policy ha saisfies he required consrains o wihin ɛ. B. The Perforance Theore For a given consan C 0, we say ha an algorih is a C-approxiaion if every slo, given he exising Θ() on slo, i chooses α() A ω(), γ() X o achieve a value in (27) ha is wihin C of he infiu over all possible decisions given Θ(). A 0-approxiaion achieves he exac infiu and is given by he algorih in Secion II. Using C > 0 allows for approxiae ipleenaions. Theore 3: Suppose V 0, ω() is i.i.d. over slos, he proble (1)-(5) is feasible, and any C-approxiae algorih is used every slo. For sipliciy, assue ha all queues are iniially zero, so ha Θ(0) = 0. Then: (a) All queues are ean rae sable, and hence all required consrains are saisfied. In paricular, he equaliies and inequaliies (16)-(18) are ensured. Bounds on expeced queue sizes (and hence, consrain violaions) for all > 0 are provided in inequaliy (35) of he proof. (b) For all > 0 and for any (y, x ) Y (including he one ha opiizes he infiu cos), we have: y 0 () 1 1 x (τ) f(x av (τ)) E y0 1 1 x E f(x av (τ)) B C V where B is defined by (26). (c) If all ie averages converge, so ha here are consan vecors x, y such ha x av () x wih probabiliy 1, x() x, and y () y, hen he achieved liiing poin is a near local opiu, in he sense ha for any (y, x ) Y (including any local or global opiu): x f(x) y 0 y x 0 f(x) B C V The resul of par (c) can be viewed as a near-local opiu resul because of he fudge-facor (B C)/V, which can be ade arbirarily sall as he V paraeer is increased. Indeed, he above resul shows ha he achieved cos, as easured by a linearized version of he f(x) funcion abou our achieved x vecor, is no ore han (B C)/V above he cos associaed wih any oher (y, x ) Y, when easured agains he sae linearized funcion. In paricular, suppose we consider oving fro our achieved (y, x) vecor owards any oher (y, x )

6 PROC. INFORMATION THEORY AND APPLICATIONS WORKSHOP (ITA), SAN DIEGO, FEB Y by aking a sall sep ɛ(y y, x x) in his direcion (for soe sall value ɛ > 0). If ɛ is sall, hen he cos differenial cos (ɛ) incurred saisfies: cos (ɛ) ɛ (y 0 y 0 ) (x x ) f(x) (B C) V where he inequaliy follows by par (c) of he above heore. I follows ha such a sep canno decrease cos by ore han approxiaely ɛ(bc)/v (where he approxiaion is due o he linearizaion), which can be ade arbirarily sall wih a suiably large choice of V. Hence, his is a near local opiu. The value of V affecs he coefficien in he decay of E { H () } /, E {Z l ()} /, E {Q k ()} / derived in (35) of he proof, which also affecs he decay in consrain violaion via (20), (21). I can be shown ha if a ild Slaer-ype condiion holds for he consrains (2)-(5) of he proble, hen all queues are srongly sable and have average backlog ha is O(V ) (see [3][10][9] for siilar resuls). Proof: (Theore 3 par (a)) By siply adding he sae hing o boh sides of (25) in Lea 2, we have: (Θ()) V E {y 0 () Θ()} V f(x av ()) E {x () Θ()} B Q k ()E {a k () b k () Θ()} Z l ()E {y l () g l (γ()) Θ()} H ()E {γ () x () Θ()} V E {y 0 () Θ()} V f(x av ()) E {x () Θ()} Because, given he exising Θ() on slo, our C- approxiaion iniizes he righ-hand-side of he above expression o wihin C of he infiu over all policies for choosing α() A ω(), γ() X, we have: (Θ()) V E {y 0 () Θ()} V f(x av ()) E {x () Θ()} B C Q k ()E {a k() b k() Θ()} Z l ()E {yl () g l (γ ()) Θ()} H ()E {γ() x () Θ()} V E {y 0() Θ()} V f(x av ()) E {x () Θ()} where α () A ω(), γ () X are any alernaive conrol acions, and for all k {1,..., K}, l {0, 1,..., L}: a k() =â k (α (), ω()), b k() =ˆb k (α (), ω()) y l () =ŷ l (α (), ω()) For any (y, x ) Y and any ɛ > 0, Theore 2 ensures here is an ω-only algorih α () ha saisfies (28)-(32). For sipliciy, assue ha (28)-(32) hold for ɛ = 0. 4 Now plug his algorih ino he righ hand side of he above drif expression, and noe ha, because i is ω-only, i akes decisions independen of backlog (so ha he condiional expecaion given Θ() is he sae as he uncondiional expecaion). Furher, plug decisions γ () = E {ˆx(α (), ω())} = x (his saisfies γ() X as required because of (31) wih ɛ = 0 and noing ha X is closed). This siplifies he righ-hand-side of he above drif bound o: (Θ()) V E {y 0 () Θ()} V f(x av ()) E {x () Θ()} B C V y 0 V x f(x av ()) (33) Because of he boundedness assupions on he se X, he firs and second oens of he aribue vecors, and he funcion f(x) and is derivaives, he above drif bound siplifies o: (Θ()) D (34) where D is any finie consan ha saisfies for all, Θ(): D B C V [y0 E {y 0 () Θ()}] V β [x E {x () Θ()}] Taking an expecaion of boh sides and using he law of ieraed expecaions yields: E {L(Θ( 1))} E {L(Θ())} D The above holds for all. Suing over τ {0,..., 1} and using he fac ha L(Θ(0)) = 0 yields: E {L(Θ())} D By definiion of L(Θ()) in (23) we have: E { Q k () 2}, E { Z l () 2}, E { H () 2} 2D Because E { X } 2 E { X 2} for any rando variable X, we have: E {Q k ()} 2, E {Z l ()} 2, E { H () } 2 2D Dividing by 2 and aking square roos shows ha for all > 0: E {Q k ()}, E {Z l()}, E { H () } 2D (35) Taking a lii as proves ean rae sabiliy of all queues. 4 The sae resul (33) can be derived wihou his assupion by aking a lii as ɛ 0.

7 PROC. INFORMATION THEORY AND APPLICATIONS WORKSHOP (ITA), SAN DIEGO, FEB Proof: (Theore 3 pars (b) and (c)) We sar wih he drif bound in (33) derived in he proof of par (a), which holds for all 0. Consider his inequaliy for soe slo τ. Taking expecaions of boh sides and using he law of ieraed expecaions and he definiion of (Θ(τ)) in (24), we have: E {L(Θ(τ 1))} E {L(Θ(τ))} V E {y 0 (τ)} V x (τ) f(x av (τ)) E V x B C V y0 E f(x av (τ)) (36) Fix any slo > 0. Suing he above over τ {0,..., 1} and using elescoping sus yields: 1 E {L(Θ())} E {L(Θ(0))} V E {y 0 (τ)} 1 V x (τ) f(x av (τ)) E 1 (B C V y0) V x E f(x av (τ)) Dividing by V and using he fac ha L(Θ(0)) = 0 and L(Θ()) 0 yields: y 0 () 1 1 x (τ) f(x av (τ)) E B C y0 1 1 x E f(x av (τ)) V This proves par (b). Par (c) is an iediae consequence, noing boundedness and coninuiy of derivaives of f(x). C. Variable V for Local Opialiy As in [17], we can use an algorih ha gradually increases he value of V while ainaining ean rae sabiliy. This eliinaes he fudge facor (BC)/V in Theore 3 par (c). A disadvanage is ha i precludes srong sabiliy of queues even when a Slaer condiion is saisfied. Theore 4: Suppose ha we use V () insead of V in he drif expression (27), and we use any C-approxiaion, i.e., any decisions ha coe wihin C of iniizing (27) under his variable V () seing on all slos. Suppose ω() is i.i.d. over slos, he proble (1)-(5) is feasible, and all queues are iniially epy. Furher assue ha we use he following increasing V () funcion: V () =V 0 (1 ) d where V 0 > 0 and d saisfies 0 < d < 1. Then: (a) All queues are ean rae sable as before, and hence he equaliies and inequaliies in (16)-(18) are saisfied. (b) If all ie averages converge, so ha here are vecors x, y such ha x av () x wih probabiliy 1, x() x, and y () y, hen he achieved poin is a local opiu, in he sense ha for any (y, x ) Y (including any local or global opiu): y 0 x f(x) y 0 x f(x) Tha is, he achieved vecor (y, x ) globally opiizes he linearized cos funcion. In paricular, if we sar a our achieved (y, x) and ove in he direcion of any oher (y, x ) Y by aking a sall sep ɛ(y y, x x), hen: cos (ɛ) li ɛ 0 ɛ = (y 0 y 0 ) (x x ) f(x) 0 Thus, such a sall sep canno iprove cos. While he heore holds for any V 0 > 0 and 0 < d < 1, hese values affec he anner in which he uiliy and virual queue values converge, as shown in he proof. Proof: (Theore 4) The proof of par (a) is siilar o [17] and is oied for breviy. Here we prove par (b). Following he proof of Theore 3 alos exacly, we find ha (36) ranslaes o he following for any slo τ 0: E {L(Θ(τ 1))} E {L(Θ(τ))} V (τ)e {y 0 (τ)} V (τ)x (τ) f(x av (τ)) E V (τ)x B C V (τ)y0 E f(x av (τ)) Dividing everyhing by V (τ) yields: E {L(Θ(τ 1))} E {L(Θ(τ))} E {y 0 (τ)} V (τ) V (τ) x (τ) f(x av (τ)) E B C x V (τ) y 0 E f(x av (τ)) Fix > 0, and su he above over τ {0,..., 1}. Collecing ers yields: E {L(Θ())} V ( 1) 1 1 τ=1 1 E {y 0 (τ)} 1 [ E {L(Θ(τ))} B C V (τ) y 0 1 V (τ 1) 1 V (τ) x (τ) f(x av (τ)) E 1 x E f(x av (τ)) Because V (τ) is non-decreasing, we have for all τ: 1 V (τ 1) 1 V (τ) 0 ]

8 PROC. INFORMATION THEORY AND APPLICATIONS WORKSHOP (ITA), SAN DIEGO, FEB Using his, dividing everyhing by, and using non-negaiviy of L( ) yields for all > 0: y 0 () 1 1 x (τ) f(x av (τ)) E 1 1 (B C) y0 1 1 x E f(x av (τ)) V (τ) However, we have: 1 1 (B C) (B C) 1 1 = ( 1) d = O( d ) 0 V (τ) V 0 Thus, aking liis of he above drif bound and using he assupion ha ie averages converge yields he resul. D. Convex Probles and he Pure Dual Approach A significan liiaion of he prial-dual coponen in he above algorihs is ha he running ie average x av () in (22) precludes adapaion when syse condiions change. An addiional liiaion is ha our heore requires vecors x av (), x(), y() o converge, bu lacks a proof of convergence and a easure of how long such convergence would ake. 5 For coparison, here we show ha hese liiaions can be solved by he drif-plus-penaly algorih wihou using parial derivaives, via he pure dual-based approach, when f(x) is convex. These advanages are exploied in an alernaive algorih for he non-convex proble in Secion IV. The analysis in his subsecion is siilar o [3][17], wih he excepion ha we include he absrac se consrain X. Assue here ha he funcion f(x) is coninuous and convex (possibly non-differeniable). Define finie bounds f in and f ax o saisfy for all and all (possibly randoized) decisions α() A ω() : f in f(e {ˆx(α(), ω())}) f ax The following algorih is oivaed by seeking o solve a odified version of he proble (9)-(13), wih he cos eric (9) changed o iniizing y 0 f(γ), which can be shown o sill be equivalen o he original proble (1)-(5) by Jensen s inequaliy and he fac ha f(γ) is convex. Using he sae virual queues and he sae Lyapunov funcion L(Θ()) as before (wih he excepion ha Θ() no longer includes x av () inforaion), we have by adding he sae hing o boh sides of he drif inequaliy of Lea 2: (Θ()) V E {y 0 () f(γ()) Θ()} B Q k ()E {a k () b k () Θ()} Z l ()E {y l () g l (γ()) Θ()} H ()E {γ () x () Θ()} V E {y 0 () f(γ()) Θ()} (37) 5 Under an addiional Slaer condiion, he policy is a Markov chain wih all queues srongly sable, and so convergence would follow if we had a counably infinie sae space. Our policy is now o observe Θ() and ω() for each slo, and o ake acions α() A ω(), γ() X o iniize he righ hand side of he above drif bound. For a given consan C 0, we define a C-approxiaion for his algorih o be one ha, every slo and given Θ(), chooses α() A ω(), γ() X o coe wihin C of he infiu of he righ hand side of (37). A 0-approxiaion would ake decisions every slo as follows: (Auxiliary Variables) Choose γ() X o iniize: V f(γ()) H ()γ () Z l ()g l (γ()) (Choosing α()) Choose α() A ω() o iniize: Q k ()[â k (α(), ω()) ˆb k (α(), ω())] Z l ()ŷ l (α(), ω()) H ()ˆx (α(), ω()) V ŷ 0 (α(), ω()) (Queue Updae) Updae virual queues Z l (), H () by (14), (15), and updae acual queues Q k () by (6). Theore 5: ( Pure Dual Perforance) Suppose proble (1)-(5) is feasible, ω() is i.i.d. over slos, and he funcion f(x) is convex. For sipliciy, assue all iniial queue values are 0. If we use any C-approxiaion on each slo, hen: (a) All queues are ean rae sable, and hence all required consrains are saisfied. In paricular, he equaliies and inequaliies (16)-(18) are ensured. Furher, bounds on he expeced queue size for all > 0 are provided in inequaliy (39) of he proof. (b) For all slos > 0, he achieved ie average expeced cos saisfies: y 0 () f(x()) y op 0 f op B C V β E { H () } where we recall ha x() is defined in (19), and y 0 () is defined siilarly. The er E { H () } / decays o zero a leas as fas as O(1/ ), as shown in (39) of he proof. (c) Suppose we use any C-approxiaion applied o a variable-v version of (37), wih V () = V 0 (1 ) d for any consans V 0 > 0 and 0 < d < 1. Then his ainains ean rae sabiliy of all queues, ensures all consrains (16)-(18), and yields exac cos opialiy: li [y 0() f(x())] = y op 0 f op We noe ha if an addiional Slaer condiion is saisfied, hen under he fixed V algorih all queues can be shown o be srongly sable wih averages O(V ) (see relaed resuls in [3][10][9]). In any flow conrol probles, he queues can be shown o be deerinisically bounded by a consan of size O(V ), even wihou he Slaer condiion [10][4]. In he variable-v scenario, queues will no be srongly sable even if he Slaer condiion is saisfied, and hence he cos of achieving exac opialiy is having infinie average backlog and delay.

9 PROC. INFORMATION THEORY AND APPLICATIONS WORKSHOP (ITA), SAN DIEGO, FEB Proof: (Theore 5) Because every slo τ he C- approxiaion coes wihin C of he infiu in he righ hand side of (37), we have: (Θ()) V E {y 0 () f(γ()) Θ()} B C Q k ()E {a k() b k() Θ()} Z l ()E {yl () g l (γ ()) Θ()} H ()E {γ() x () Θ()} V E {y 0() f(γ ()) Θ()} where a k (), b k (), y l (), γ () are associaed wih any alernaive decisions α () A ω(), γ () X. Fix (y op, x op ) Y as he opial soluion in Theore 1, yielding cos f(x op ) = f op. Plug he policy of (28)-(32), and for sipliciy again assue ɛ = 0. Also plug γ () = x op. Because hese decisions are independen of Θ(), we have: (Θ()) V E {y 0 () f(γ()) Θ()} B C V y op 0 V f op (38) Thus, (Θ()) ˆD, where ˆD is defined: ˆD =B C V (y op 0 y in 0 ) V (f op f in ) where we define y0 in as a lower bound on E {y 0 () Θ()} over all possible α(). This has he sae srucure as (34), and hence we obain for all > 0 (copare wih (35)): E {Q k ()}, E {Z l()}, E { H () } 2 ˆD (39) Taking a lii of (39) as shows ha all queues are ean rae sable, proving par (a). To prove (b), ake expecaions of boh sides of (38) and use ieraed expecaions o obain: E {L(Θ( 1))} E {L(Θ())} V E {y 0 () f(γ())} B C V (y op 0 f op ) Fix > 0, su he above over τ {0,..., 1}, and divide by o yield: E {L(Θ())} E {L(Θ(0))} V y 0 () V f(γ()) B C V (y op 0 f op ) where we have used Jensen s inequaliy in he convex funcion f(γ). Noing ha L(Θ(0)) = 0, L(Θ()) 0, and dividing by V, we have: y 0 () f(γ()) B C y op 0 f op (40) V However, noe fro (8) ha: f(x()) f(γ()) β x () γ () f(γ()) β E { H () } / (41) where (41) follows fro (20). Using (41) and (40) proves he resul of par (b). Par (c) is siilar o Theore 4. Reark 1: While Theores 3-5 assue ω() is i.i.d. over slos, his assupion is no crucial, and he sae policies can be shown o yield siilar resuls by using a T -slo Lyapunov drif arguen, under he assupion ha ω() is ergodic wih a decaying eory propery, so ha averages over T slos are close o he seady sae average [18][19][20][3]. Reark 2: We can reove auxiliary variables γ() and se H () = 0 for all if: (i) We use he prial-dual ehod, here is no absrac se consrain (3), and g l (x) = 0 for all l, or (ii) We use he pure dual ehod, here is no absrac se consrain (3), and g l (x) = f(x) = 0 for all l. IV. AN ALTERNATIVE SEARCH METHOD Here we provide an alernaive algorih for he non-convex proble in he special case of uiliy axiizaion wih enrywise non-decreasing uiliy funcions. I involves hree phases: Two phases are convex sochasic nework opiizaions for which we have sronger perforance guaranees as in Secion III-D. All non-convexiy is isolaed o he reaining phase ha involves a deerinisic non-convex proble for which any available solver can be eployed. To begin, le f(x) = φ(x), where φ(x) is a uiliy funcion o be axiized. Suppose ha φ(x) is possibly non-concave, bu is enrywise non-decreasing. For sipliciy, assue ŷ l ( ) = 0 for all l, so ha he se Y consiss of vecors (0, x). Le θ 1,..., θ N be a collecion of N differen search direcion vecors, being non-negaive M-diensional vecors. Assue for sipliciy ha for each θ n, he se Y conains a poin x such ha x = ηθ n for soe non-negaive scalar η (his is rivially rue if Y conains he origin). Define η n as he larges value of η for which his is rue, so ha x n = η n θ n is he larges value of x ha we can push in his direcion. Our new goal is o find a feasible policy ha yields a uiliy φ(x) ha is a local axiu over he se C defined: C =Conv({x 1,..., x N }) where Conv( ) denoes he convex hull. Because Y is convex, any poin in his convex hull is achievable. The value of each x n depends on he value of η n, which is unknown if he sochasics of he nework are unknown. However, suppose we know a bound on η n, so ha 0 η n ηn ax. If his is no a rue bound, hen we siply redefine x n= in[η n, ηn ax ]θ n. Our algorih has 3 phases: Phase 1: Deerine he vecors {x 1,..., x N } by solving N differen sochasic nework opiizaion probles, each wih auxiliary variables η n (): Maxiize: Subjec o: η n x = η n θ n x X g l (x) 0 l {1,..., L} All queues Q k () are ean rae sable α() A ω(), 0 η n () η ax n

10 PROC. INFORMATION THEORY AND APPLICATIONS WORKSHOP (ITA), SAN DIEGO, FEB Each of hese is a siple convex sochasic nework opiizaion proble, and we can run he pure dual approach o provide accurae esiaes η n of he opial η n value for each n. Noe ha his can be done using N differen virual neworks in parallel, reusing he sae ω() evens ha are observed. The equaliy consrains can eiher be reaed as addiional aribues h() = x() η n ()θ n wih an absrac se consrain h {0}, or can siply be reaed using virual queues of he ype (15). Phase 2: Define esiaes x n = η n θ n. Use any nonsochasic opiizer o find a local iniu of he following saic opiizaion: Maxize: Subjec o: φ(x) N n=1 p n x n = x p n 0 n {1,..., N} N n=1 p n = 1 This ype of proble can be solved, for exaple, via ehods in [21], including Newon-ype ehods ha do no resric o sall sep sizes. Phase 3: Given he opial x fro phase 2, run an algorih o solve he following sochasic nework opiizaion proble: Maxiize: Subjec o: η x X x = ηx g l (x) 0 l {1,..., L} All queues Q k () are ean rae sable α() A ω(), 0 η n () 2 This is again a basic convex sochasic nework opiizaion. I seeks o axiize he ie average aribue x in he direcion of he x vecor obained in phase 2 (noicing ha he resuling opiu igh push even pas he se C). Here we lii our search o doubling x, wih he inen of obaining a vecor x wih uiliy φ(x) a leas as good as φ(x ). The x vecor used in his phase igh be periodically changed as a resul of periodically running he oher wo phases in he background. V. CONCLUSIONS We have provided wo approaches o consrained opiizaion of non-convex funcions of ie average aribues in a sochasic nework. The firs approach cobines dual-based and prial-dual operaions. I ensures all desired (convex) consrains are saisfied, and, provided ha he algorih converges, i also provides eiher a local opiu or nearlocal opiu, depending on wheher we use a fixed V or variable V algorih. While he ehod provides guaranees on he expeced queue backlogs a any ie, i does no provide such guaranees for he uiliy, and i is no clear how uch ie is required for convergence o he local in. I also requires an infinie horizon ie average ha is no robus o syse changes. This is in conras o he pure dual approach o convex probles ha provide explici backlog and uiliy bounds for all ie, and which are known o adap o syse changes [4][8]. The second approach uses 3 phases, wo of which are sochasic and convex, while he reaining phase is purely deerinisic bu non-convex. These algorihs significanly exend our abiliy o opiize dynaic neworks. REFERENCES [1] J.W. Lee, R. R. Mazudar, and N. B. Shroff. Non-convex opiizaion and rae conrol for uli-class services in he inerne. IEEE/ACM Trans. on Neworking, vol. 13, no. 4, pp , Aug [2] M. Chiang. Nonconvex opiizaion of counicaion syses. Advances in Mechanics and Maheaics, Special volue on Srang s 70h Birhday, Springer, vol. 3, [3] L. Georgiadis, M. J. Neely, and L. Tassiulas. Resource allocaion and cross-layer conrol in wireless neworks. Foundaions and Trends in Neworking, vol. 1, no. 1, pp , [4] M. J. Neely. Universal scheduling for neworks wih arbirary raffic, channels, and obiliy. ArXiv echnical repor, arxiv: v1, Jan [5] R. Agrawal and V. Subraanian. Opialiy of cerain channel aware scheduling policies. Proc. 40h Annual Alleron Conference on Counicaion, Conrol, and Copuing, Monicello, IL, Oc [6] H. Kushner and P. Whiing. Asypoic properies of proporionalfair sharing algorihs. Proc. of 40h Annual Alleron Conf. on Counicaion, Conrol, and Copuing, [7] A. Solyar. Maxiizing queueing nework uiliy subjec o sabiliy: Greedy prial-dual algorih. Queueing Syses, vol. 50, pp , [8] M. J. Neely and R. Urgaonkar. Cross layer adapive conrol for wireless esh neworks. Ad Hoc Neworks (Elsevier), vol. 5, no. 6, pp , Augus [9] M. J. Neely. Dynaic Power Allocaion and Rouing for Saellie and Wireless Neworks wih Tie Varying Channels. PhD hesis, Massachuses Insiue of Technology, LIDS, [10] M. J. Neely. Energy opial conrol for ie varying wireless neworks. IEEE Transacions on Inforaion Theory, vol. 52, no. 7, pp , July [11] M. J. Neely, E. Modiano, and C. Li. Fairness and opial sochasic conrol for heerogeneous neworks. Proc. IEEE INFOCOM, March [12] L. Tassiulas and A. Ephreides. Sabiliy properies of consrained queueing syses and scheduling policies for axiu hroughpu in ulihop radio neworks. IEEE Transacaions on Auoaic Conrol, vol. 37, no. 12, pp , Dec [13] L. Tassiulas and A. Ephreides. Dynaic server allocaion o parallel queues wih randoly varying conneciviy. IEEE Transacions on Inforaion Theory, vol. 39, pp , March [14] A. Eryilaz and R. Srikan. Fair resource allocaion in wireless neworks using queue-lengh-based scheduling and congesion conrol. Proc. IEEE INFOCOM, March [15] M. J. Neely and R. Urgaonkar. Opporunis, backpressure, and sochasic opiizaion wih he wireless broadcas advanage. Asiloar Conference on Signals, Syses, and Copuers, Pacific Grove, CA, Oc [16] A. Solyar. Greedy prial-dual algorih for dynaic resource allocaion in coplex neworks. Queueing Syses, vol. 54, pp , [17] M. J. Neely. Max weigh learning algorihs wih applicaion o scheduling in unknown environens. arxiv: v1, Feb [18] L. Tassiulas. Scheduling and perforance liis of neworks wih consanly changing opology. IEEE Trans. on Inf. Theory, May [19] M. J. Neely, E. Modiano, and C. E Rohrs. Dynaic power allocaion and rouing for ie varying wireless neworks. IEEE Journal on Seleced Areas in Counicaions, vol. 23, no. 1, pp , January [20] R. Urgaonkar and M. J. Neely. Opporunisic scheduling wih reliabiliy guaranees in cogniive radio neworks. IEEE Transacions on Mobile Copuing, vol. 8, no. 6, pp , June [21] D. P. Bersekas. Nonlinear Prograing. Ahena Scienific, Belon, MA, 1995.