Variability Aware Network Utility Maximization

Transcription

1 aably Awae Newok ly Maxmzaon nay Joseph and Gusavo de ecana Depamen of Eleccal and Compue Engneeng, he nvesy of exas a Ausn axv:378v3 [cssy] 3 Ap 0 Absac Newok ly Maxmzaon NM povdes he key concepual famewok o sudy esouce allocaon amongs a collecon of uses/enes acoss dscplnes as dvese as economcs, law and engneeng In newok engneeng, hs famewok has been paculaly nsghfull owads undesandng how Inene poocols allocae bandwdh, and movaed dvese eseach on dsbued mechansms o maxmze newok uly whle ncopoang new elevan consans, on enegy/powe, soage, sably, ec, fo sysems angng fom communcaon newoks o he sma-gd Howeve when he avalable esouces and/o uses ules vay ove me, a use s allocaons wll end o vay, whch n un may have a demenal mpac on he uses uly o qualy of expeence hs pape noduces a genealzed NM famewok whch explcly ncopoaes he demenal mpac of empoal vaably n a use s allocaed ewads I explcly ncopoaes adeoffs amongs he mean and vaably n uses allocaons We popose an onlne algohm o ealze vaance-sensve NM, whch, unde saonay egodc assumpons, s shown o be asympocally opmal, e, acheves a me-aveage equal o ha of an offlne algohm wh knowledge of he fuue vaably n he sysem hs subsanally exends wok on NM o an nesng class of elevan poblems whee uses/enes ae sensve o empoal vaably n he sevce o allocaed ewads I INODCION Newok ly Maxmzaon NM povdes he key concepual famewok o sudy fa esouce allocaon among a collecon of uses/enes acoss dscplnes as dvese as economcs, law and engneeng In newok engneeng hs famewok has ecenly seved as a paculaly nsghfull seng n whch o sudy evese engnee how he Inene s congeson conol poocols allocae bandwdh, how o devse schedules fo weless sysems wh me vayng channel capaces, and movaed he developmen of dsbued mechansms o maxmze newok uly n dvese sengs ncludng communcaon newoks and he sma gd, whle ncopoang new elevan consans, on enegy, soage, powe conol, sably, ec Howeve when he avalable esouces and/o uses ules vay ove me, allocaons amongs uses wll end o vay, whch n un may have a demenal mpac on he uses uly o peceved sevce qualy Indeed empoal vaably n uly, sevce, esouces o assocaed pces ae paculaly poblemac when humans ae he evenual ecpens of he allocaons Humans ypcally vew empoal vaably negavely, as sgn of an unelable sevce, newok o make nsably, o as a sevce whch when vewed hough human s cognve and behavoal esponses can, and wll, anslae o a degaded Qualy of hs eseach was suppoed n pa by Inel and Csco unde he AWN pogam, and by he NSF unde Gan CNS Expeence QoE Fo example empoal vaably n vdeo qualy has been shown o lead o hyseess effecs n humans qualy judgmens can can subsanally degade a use s QoE hs n un can lead uses o make decsons, eg, change povde, ac upon peceved make nsables, ec, whch can have seous mplcaons on busneses and engneeed sysems, o economc makes hs pape noduces a genealzed NM famewok whch explcly ncopoaes he demenal mpac of empoal vaably n a use s allocaed ewads We use he em ewads as a poxy epesenng he esulng uly of, o any ohe quany assocaed wh, allocaons o uses/enes n a sysem Ou goal s o explcly ackle he ask of ncopoang adeoffs amongs he mean and vaably n uses ewads hus, fo example, n a vaance-sensve NM seng, may make sense o educe a use s mean ewad so as o educe s vaably As wll be dscussed n he sequel hee ae many ways n whch empoal vaaons can be accouned fo, and whch, n fac, pesen dsnc echncal challenges In hs pape we shall ake a smple elegan appoach o he poblem whch seves o addess sysems whee adeoffs amongs he mean and vaably ove me need o be made ahe han sysems whee he mean o age s known, o whee he ssue a hand s he cumulave vaance a he end of a gven eg, nvesmen peod o bee descbe he chaacescs of he poblem we noduce some pelmnay noaon We shall consde a newok shaed by a se N of uses o ohe enes whee N = N denoes he numbe of uses n he sysem houghou he pape, we dsngush beween andom vaables and andom funcons and he ealzaons by usng uppe case lees fo he fome and lowe case fo he lae We use bold lees o denoe vecos, eg, a = a : N We le a : denoe he fne lengh sequence a : Fo a funcon on, denoes s devave hus f epesens he ewad allocaed o use a me, hen = : N s he veco of ewads o uses N a me and : epesens he ewads allocaed ove me =,, slos o he same uses We assume ha ewad allocaons ae subjec o me vayng newok consans, c fo =,,, whee c : N coesponds o convex funcon, hus mplcly defnng a convex se of feasble ewad allocaons o fomally capue he mpac of he me-vayng esouces on uses QoE consde he followng offlne convex opmzaon

2 poblem OP: a :, = }{{} Poxy fo use s QoE subjec o c 0 {,,}, max : N whee fo each N, a : = mn {,,}, N, = τ τ= We efe o hs as an offlne opmzaon because mevayng me consans c : ae assumed o be known, and allow funcons, makng he opmzaon N poblem convex Noe ha he fs em n a use s poxy QoE = capues he degee o whch QoE nceases n hs/he allocaed ewads a any me, wheeas he second em ypcally nceasng n a would penalzes empoal vaably n ewad allocaon Hence, hs geneal fomulaon allows us o adeoff beween mean and vaably assocaed wh he ewad allocaons by appopaely choosng he funcons, N A Man esul and conbuons he man conbuon of hs pape s n devsng an onlne algohm, fo Adapve aably-awae esouce A allocaon, whch ealzes vaance-sensve NM nde saonay egodc assumpons on he me-vayng consans, we show A s asympocally opmal, e, acheves a pefomance equal o ha of he offlne opmzaon OP noduced eale hs s a song opmaly esul, whch a fs sgh may be supsng due o he dependency of a n he objecve of OP on ewad allocaons ove me and he me vayng naue of he consans c he key dea explos he chaacescs of he poblem, by keepng onlne esmaes fo he elevan quanes assocaed wh uses allocaons, eg, he mean, vaance, and mean QoE, whch ove me ae shown o convege, and whch evenually enable he onlne polcy o poduce allocaons coespondng o he opmal saonay polcy Povng hs esul s somewha challengng as eques, showng ha he esmaes based on allocaons poduced by ou onlne polcy, A, whch self depends on he esmaed quanes, wll convege o he desed values o ou knowledge hs s he fs aemp o genealze he NM famewok n hs decon We wll conas ou poblem fomulaon and appoach o some of he pas wok n he leaue addessng vaance mnmzaon, sk-sensve conol and ohe MDP based famewoks he elaed wok below B elaed Wok Newok ly Maxmzaon NM povdes he key concepual famewok o sudy how o allocae ewads faly amongs a collecon of uses/enes [?] povdes an ovevew of NM Bu all he wok on NM ncludng seveal majo exensons fo eg, [?], [?], [?] ec have gnoed he mpac of vaably n ewad allocaon on he qualy of expeence of uses Addng a vaance em n he objecve funcon, would ake hngs ou of he geneal dynamc pogammng seng, see eg [?] Indeed, ncludng vaance n he uly/cos o uses a each me, sgnfes he oveall cos s no decomposable, e, can no be wen as a sum of coss each dependen only on he allocaon a ha me hs makes sensvy o vaably challengng Fo nsance, [?] dscusses mnmum vaance conolle fo lnea sysems Secon 53 whee he objecve s he mnmzaon of he sum of second momens of he oupu vaable Sum of second momens s consdeed nsead of he vaance, whch allows he cumulave cos o be epesened sum of he coss ncued ove me Noe howeve, ha mnmzaon of second momens does no decly addess vaably unless he mean s zeo he vaance of he cumulave cos s ncopoaed n he objecve fo poblems n sk sensve opmal conol see [?] o capue he sk assocaed wh a polcy Noe howeve, ha he vaance s of he cumulave cos ahe han of he vaably as seen by a use ove me o summaze, o ou knowledge hee ae no pevously poposed woks on NM ha addesses he negave mpac of vaably he algohm poposed hee falls no he class of sochasc fxed pon algohms see [?] Ou algohm s also elaed o he algohms poposed n [?] and [?] alhough hese woks also gnoe vaably C Oganzaon of he pape In Secon II, we dscuss he sysem model and assumpons We sudy he opmaly condons fo OP n Secon III We noduce OPSA n I and sudy s opmaly condons We sa Secon by fomally noducng ou onlne algohm A hen do a convegence analyss of A n Subsecon -A, and conclude he secon by esablshng he asympoc opmaly of A n Subsecon -B We conclude he pape n Secon I he poofs of some of he nemedae esuls used n he pape ae dscussed n an appendx gven a he end of he pape II SYSEM MODEL We consde a sloed sysem whee slos ae ndexed by {0,,}, and he sysem seves a fxed se of uses N and le N = N Le + = {b : b 0} A sequence b n a Eucldean space s sad o convege o a se A f lm nf b a = 0, a A whee denoes he Eucldean nom assocaed wh he space Fo a funcon on, denoes s devave We use I as he ndcao funcon, e, fo any se A, we le I {a A} = f a A, and zeo ohewse We assume ha he ewad allocaon N + n slo s consaned o sasfy he followng nequaly c 0,

3 3 whee c s pcked fom a abaly lage fne se C of eal valued maps on N + We make he followng assumpons on hese consans: Assumpons C-C4 me vayng consans C hee s a consan mn 0 such ha fo any c C, c 0 fo such ha = mn fo each N C hee s a consan max > 0 such ha fo any c C and N + sasfyng c 0, we have max fo each N C3 Each funcon c C s convex and dffeenable on an open se conanng [ mn, max ] C4 Fo any c C and such ha = mn fo each N, c < 0 o c 0 f c s an affne funcon C5 Le C be a saonay egodc pocess, and le πc : c C denoe he saonay dsbuon assocaed We le C π denoe a andom consan wh dsbuon πc : c C We could allow he consans mn and max o be use dependen Bu, we avod ha fo noaonal smplcy he condon C4 s mposed o ensue ha he consan se s nce when used as a feasble se fo an opmzaon poblem OP see fo eg Lemma Nex we dscuss he assumpons on he funcons, Fo each N, we make he assumpons N and dscussed nex Assumpons and Le v max = max mn : s defned and wce connuously dffeenable on an open se conanng [0,v max ] wh mn v [0,vmax] v = d mn, > 0 and mn v [0,vmax] v < 0 Fuhe, we assume ha fo any wo elemens x and x n any Eucldean space d wh x x, and α 0, wh ᾱ = α, we have αx +ᾱx x < α +ᾱ whee denoes he Eucldean nom assocaed wh he space Fo each N, le max v [0,vmax] v = d max, : s defned and dffeenable on an open se conanng [ mn, max ] Fuhe, we assume ha s concave and scly nceasng on [ mn, max ] Noe ha by pckng, fo each N, he funcons fom he followng se = {v +δ α : α [05,] wh δ > 0 f α }, we sasfy he equemens n Noe ha hs ncludes he deny funcon v = v Also, he funcon v = v +δ fo any abaly smallδ > 0 sasfes he condons n, We sasfy f we pck he funcons N fom followng class of scly concave nceasng funcons paamezed by α 0, [?] { logx f α =, α x = α x α ohewse, hese funcons ae commonly used o enfoce faness o oban allocaons ha ae α fa see [?] A lage α coesponds o a moe fa allocaon Noe ha we have o ensue ha 0 / [ mn, max ] o ensue ha funcon s well defned, and even f hs s no he case, we could use α +δ nsead of α fo an abaly small posve shfδ n he agumen o avod hs equemen We wll see lae ha A can be made moe effcen f s lnea fo some uses N We defne he followng subses of N : N l = { N : s lnea }, N n = { N : s no lnea } We focus on obanng an algohm fo ewad allocaon ha can be mplemened a a cenalzed coodnao ha has access oc a he begnnng of slo Fo nsance, n a cellula newok seng lke n WN, hs could be a basesaon ha esmaes he channel senghs of he uses n he newok o fnd c A Applcaons and scope of he model he pesence of me vayng consans c 0 allows us o apply he model o seveal neesng and useful sengs In pacula, hee we focus on a weless newok seng by dscussng hee cases WN, WN-E and WN-, and show ha he model can handle poblems nvolvng me vayng exogenous consans and me vayng uly funcons We sa by dscussng case WN whee he ewad n a slo s he ae allocaed o he use n ha slo Le P denoe a fne bu abaly lage se of posve vecos whee each veco coesponds o he peak ansmsson ae veco x { fo a slo seen by uses n a weless newok Le C = c p : c p = } N p,, p P Hee, fo any allocaon, /p s he facon of me he weless sysem needs o seve use n slo o delve daa a he ae of o use n a slo whee he use has peak daa ansmsson ae p hus, he consan c p 0 can be seen as a schedulng consan ha coesponds o he equemen ha he sum of he facons of me ha dffeen uses ae seved n a slo should be less han o equal o one me vayng exogenous consans: We can also allow fo me vayng exogenous consans on he weless sysem by appopaely defnng he se C Fo nsance, consde case WN-E whee a base saon n a cellula newok allocaes aes o uses some of whom ae seamng vdeos As poned above QoE of uses vewng vdeo conen s sensve o empoal vaably n qualy Bu, whle allocang aes o hese uses, we also need o accoun fo he me vayng esouces equemens of he voce and daa affc handled by he { basesaon We can deal wh hs consan by defnng C } = c p,f : c p,f = N p f, p P,f F, whee F s a fne se of eal numbes n [0,] whee each

4 4 elemen n he se coesponds o he facon of me n a slo ha s ulzed he voce and daa affc me vayng uly funcons: Fo he uses seamng vdeo conen dscussed n he case WN-E, s moe appopae o vew he peceved vdeo qualy of a use n a slo as he ewad fo ha use n ha slo Howeve, fo uses seamng vdeo conen, he dependence of peceved vdeo qualy n a sho duaon slo oughly a second long whch coesponds o a collecon of 0-30 fames on he compesson ae s me vayng hs s ypcally due o he possbly changng naue of he conen, eg, fom an acon o a slowe scene Hence, he uly funcon ha maps he ewad e, peceved vdeo qualy deved fom he allocaed esouce e, he ae s me vayng hs s he seng n he case WN-, and we can handle as follows Le q, w denoe he scly nceasng concave funcon ha, n slo, maps he peceved vdeo qualy o he ae w allocaed o use Fo each use, le Q be a fne se of such funcons Hence, we can vew WN- as a case ha has he followng se of consans: C = { c p,q : c p,q = q, p N p P,q Q N} Noe ha each elemen n C 3 s a convex funcon Fo WN and WN-E, we can vefy ha by choosng max = max p P max N p and an mn sasfyng 0 mn N mn p P mn N p, we sasfy C-C4 In WN-, f we assume ha each funcon q Q s dffeenable and convex wh q0 = 0 whch ae vey easonable assumpons on he dependence beween qualy and compesson ae, hen we can vefy ha by choosng mn = 0 and max = max p P max N max q Q qp, we sasfy C-C4 aably awae ae adapaon fo vdeo: he above fomulaon s applcable o he poblem of fndng opmal jon vdeo ae adapaon ha maxmzes he sum QoE of uses seamng vdeos ulzng esouces of a shaed newok Gven he pedcons fo explosve gowh of vdeo affc n he nea fuue see [?], hs s among one of he mpoan newokng poblems oday Fo a use vewng a vdeo seam, vaaons n vdeo qualy ove me has a demenal mpac on he use s QoE, see eg, [?], [?], [?] Indeed [?] even pons ou ha vaaons n qualy can esul n a QoE ha s wose han ha of a consan qualy vdeo wh lowe aveage qualy Fuhemoe, [?] poposed and evaluaed a mec fo QoE whch oughly coesponds o he choces = and v = v +δ n he model descbed above fo a vey small δ > 0 III OPIMAL AIANCE-SENSIIE OFFLINE POLICY In hs secon, we sudy OP, he offlne fomulaon fo opmal jon ewad allocaon noduced n Secon I In he offlne seng, we assume hac :, e, he ealzaon of he pocess C :, s known We denoe he objecve funcon of OP by φ, e, φ : = a :, N = and N and ae funcons sasfyng N and especvely, and a : = τ= τ Hence he opmzaon = poblem OP can be ewen as: max φ : : 3 subjec o c 0 {,,}, 4 mn {,,}, N, 5 whee c C s a convex funcon fo each he nex esul asses ha OP s a convex opmzaon poblem sasfyng Slae s condon Secon 53, [?] and ha has a unque soluon Lemma OP s a convex opmzaon poblem sasfyng Slae s condon wh a unque soluon Poof: Snce we made he assumpons and, he convexy of he objecve of OP s easy o esablsh once we pove he convexy of he funcon a fo each N sng and he defnon of a, we can show ha a s a convex funcon fo each N he deals ae gven nex Fo wo dffeen qualy vecos : and, any N, α 0, and ᾱ = α, we have : ha a α : +ᾱ : = a α +ᾱ : = α +ᾱ = α τ+ᾱ τ τ= = α τ +ᾱ = τ= τ τ= sng, we have ha a α : +ᾱ : α +ᾱ = α = τ= τ τ = τ= a : +ᾱ a :

5 5 hus, a s a convex funcon sng he above agumens and concavy of and a, we conclude ha OP s a convex opmzaon poblem Noe ha, fom snce we have a sc nequaly, he nequaly above s a sc one unless = + τ τ τ= τ= hus, fo he nequaly no o be a sc one, we eque ha a : = a : Fuhe, Slae s condon s sasfed and manly follows fom he assumpon C4 Now, fo any N, and a ae no necessaly scly concave Bu, we can sll show ha he objecve s scly concave as follows Le : and be wo opmal soluons o OP hen, : fom he concavy of he objecve, α +ᾱ : : s also an opmal soluon fo any α 0, and ᾱ = α Due o concavy of and convexy of a, hs s only possble f fo each N and, α +ᾱ = α + ᾱ, and a α : +ᾱ = : α a : +ᾱ a : Fom above dscusson, a α : +ᾱ : s equal o α a : + ᾱ a : fo each N only f a : = a : fo each N, and = + τ= τ τ= τ fo each N and Snce fo each N, a : = a :, due o opmaly of : and, we have ha : a : N = = a : N = = N = + τ τ a : τ= τ= Snce s a scly nceasng funcon fo each N, he above equaon mples ha and hus, τ = τ= τ, τ= =, N Fom he above dscusson, we can conclude ha OP has a unque soluon We le denoe he opmal soluon o OP : Snce OP s a convex opmzaon poblem sasfyng Slae s condon Lemma, Kaush-Kuhn-ucke KK condons [?]gven nex ae necessay and suffcen fo opmaly Le m = = KK-OP: s an opmal soluon o OP f and only f s : feasble, and hee exs non-negave consans µ and : γ : N such ha fo all N and {,,}, : we have a : m µ c, + γ = 0, 6 µ c = 0, 7 γ mn = 0, 8 Hee c c, denoes, and we have used he fac ha fo any N and τ {,,} a τ : = τ τ τ= Fom 6, we see ha he opmal ewad allocaon n any me slo depends on he ene allocaon : only hough he followng fou quanes assocaed wh : : me aveage ewadm, evaluaed a he N vaance seen by he especve uses So, f a gene evealed hese quanes, he opmal allocaon fo each slo, can be deemned by solvng an opmzaon ha only eques he knowledge of c assocaed wh cuen slo and no c : We explo hs key dea whle fomulang he onlne algohm A poposed n Secon I A ELAED POBLEM: OPSA In hs secon, we noduce and sudy anohe opmzaon poblem OPSA closely elaed o OP he fomulaon OP manly nvolves me aveages of vaous quanes assocaed wh Insead, he fomulaon of OPSA s based on he expeced value of he coespondng quanes evaluaed usng he saonay dsbuon of C ecall ha see C5 C s a saonay egodc pocess wh saonay dsbuon πc : c C, e, fo c C, πc s he pobably of he even c = c Snce C s fne, we assume ha πc > 0 fo each c C whou any loss of genealy Le c c C be a veco epesenng he ewad allocaon c N o he uses fo each c C Alhough we ae abusng he noaon noduced eale whee denoed he he allocaon o he uses n slo, one can dffeenae beween he funcons based on he conex n whch hey ae beng dscussed Now, le φ π cc C = πc c N c C a π c c C, a π c c C = πc c c c Cπc c C

6 6 he opmzaon poblem OPSA gven below: max c c C φ π cc C, subjec o cc, c C, c mn, c C he nex esul gves few useful popees of OPSA Lemma a OPSA s a convex opmzaon poblem sasfyng Slae s condon b OPSA has a unque soluon Poof: he poof s smla o ha of Lemma and s easy o esablsh once we pove he convexy of he funcon a π sng Lemma a, we can conclude ha KK condons ae necessay and suffcen fo opmaly fo OPSA Le π c : c C denoe he opmal soluon KK-OPSA: hee exs consans µ π c : c C and γ π c N : c C ae such ha πc π c a π π c c C π c πc π c c C µ π cc π c+γ π c = 0, 9 µ π cc π c = 0, 0 γ π c π c mn = 0, whee c c denoes, and we used followng esul: fo any c 0 C, N, a π c c C = πc 0 c 0 c c 0 c Cπc ADAPIE AIANCE AWAE EWAD ALLOCAION In hs secon, we pesen ou onlne algohm A o solve OP, and esablsh s asympoc opmaly he ewad allocaons fo A ae obaned by solvng OPAm, v, c gven below: max whee N v m +h 0 v subjec o c 0, mn N, 3 h 0 v = v v v N Noe ha OPAm, v, c s closely elaed o OP- ONLINE dscussed n Subsecon I-A Also, noe ha h 0 e,v does no depend on he allocaon and hus can be gnoed whle solvng he opmzaon poblem Bu, modfes he objecve funcon and hus he opmal value of he objecve funcon o ensue cean nce popees fo he paal devaves of lae see Lemma 3 b Le m,v,c denoe he opmal soluon o OPAm,v,c Also, le H be gven by: H = [ mn, max ] N [0,v max ] N, whee denoes coss poduc opeao fo ses Nex, we descbe he algohm A n deal A consss of hee seps, A0-A, gven nex: Adapve aance awae ewad allocaon A A0: Inalze: Le m0, v0 H In each slo + fo 0, cay ou he followng seps: A: he ewad allocaon n slo s gven by m,ê, v,c + and wll be denoed by + when he dependence on he vaables s clea fom conex A: In slo, updae m as follows: fo all N, m + = m + + m, 4 and updae v as follows: fo all N l, v + = v 0, and fo all N n, v + = v + + m v 5 We see ha he updae equaons 4-5 oughly ensue ha he paamees m and v N n keep ack of mean ewad and vaance n ewad especvely assocaed wh he ewad allocaon unde A Also, noe ha we do no have o keep ack of he esmaes of vaance n ewad seen by uses wh lnea We le θ = m, v fo each he updae equaons 4-5 ensue ha θ says n he se H Fo any m,v,c H, we have v > 0 see assumpon Hence, OPAm, v, c s a convex opmzaon poblem wh a unque soluon Fuhe, usng assumpon C4, we can show ha sasfes Slae s condon Hence, he opmal soluon fo OPAm, v, c sasfes KK condons gven below KK-OPAm, v, c: hee exs non-negave consans µ and γ : N such ha fo all N v m +γ µ c = 0, 6 µ c = 0, 7 γ mn = 0 8 Le hm,v,c denoe he opmal value of he objecve funcon of OPAm,v,c, e, h s a funcon defned on an open neval he obvous one ha can be obaned fom

7 7 he domans of he funcons, conanng H as N gven below hm,v,c = v N + v v m, whee sands fo m,v,c In he nex esul, we esablsh connuy and dffeenably popees of m,v,c also denoed by n he esul and hm, v, c especvely, vewng hem as funcons of m,v Lemma 3 Fo any c C, and θ = m,v H a θ,c s a connuous funcon of θ b Fo each N, hθ,c m hθ,c v = m v, = v v m c E[ θ,c π ] s a connuous funcon of θ d Fo each N, E[hθ,C π ] m E[hθ,C π ] v = E[ θ,c π ] m v, = v [ v E θ,cπ m ] Poof Skech: Poofs of pas a and b manly ely on some fundamenal esuls on peubaon analyss of opmzaon poblems fom [?] and [?] Pa a can be poved usng heoem n [?] he esul n pa b can be shown usng heoem 4 n [?] hs heoem ells us ha f cean condons ae me, hen we can evaluae he paal devave of he opmal value of a paamec opmzaon poblem wh espec o any paamee by jus evaluang he paal devave of he objecve of he opmzaon poblem, and hen subsung he opmal soluon Fo nsance, by usng he heoem, we can evaluae he paal devave of he opmal value hθ,c wh espec o m as follows We fs evaluae he paal devave of he objecve funcon of OPA θ,c: m h 0 v+ N v m = m v Now, on subsung n he above expesson, we oban he fs esul n pa b he ohe esuls can be obaned smlaly Pas c and d can shown usng pas a and b especvely, and Bounded Convegence heoem see [?] Fom pa b of he above esul, we see ha he updae equaons 4-5 ensue ha θ moves n a decon ha nceases h hs s n pa due o he caeful choce of he funcon h 0 whch s ndependen of vaables beng opmzed appeang n he objecve funcon of OPA Nex, we fnd elaonshps beween he opmal soluon π c : c C of OPSA and OPA owads ha end, le m π = c C πcπ c and vπ = a π π c c C fo each N Nex, le H = {m,v H : m,v sasfes 9 0}, whee he condons 9-0 ae gven below: E[ m,v,cπ ] = m N, 9 a m,v,c π = v N n 0 Pa a of he nex esul povdes a fxed pon lke elaonshp fo he opmal soluon o OPSA usng he opmal soluon funcon of OPA, and pa b s a useful consequence of pa a A poof fo he esul s gven n Appendx A Lemma 4 m π,v π sasfes a m π,v π,c = π c fo each c C, and b m π,v π H he nex esul ells us ha we can oban he opmal soluon o OPSA fom any elemen n H by usng he opmal soluon funcon Fuhe, gves us vey useful unqueness esuls fo he componens of he elemens n H A poof fo he esul s gven n Appendx B Lemma 5 Suppose m,v H hen, a m,v,c c C s an opmal soluon o OPSA Suppose ha m,v H hen, b m,v,c = m,v,c fo each c C, and c m = m fo each N, and v = v fo each N n d m = m π fo each N, and v = v π fo each N n ll now, we focused only on he opmzaon poblem OPA assocaed wh A In he nex subsecon, we sudy he evoluon of θ unde A A Convegence Analyss In hs subsecon, we focus on esablshng some popees elaed o he convegence of he sequence θ ha ae key o poof of he man opmaly esul heoem owads ha end, we sudy he he dffeenal equaon dθ = ḡθ, d whee ḡθ s a funcon akng values n 3N defned as follows: fo θ = m,v H, le ḡθ = E[θ,C π ] m, ḡθ N+ = I { N n} E [ θ,cπ m ] v he movaon fo sudyng he above dffeenal equaon should be paly clea by compang he HS of wh he updae equaons n 4-5 n A

8 8 Now we sudy n lgh of he above esul and oban a convegence esul fo he dffeenal equaon, whch ells us ha fo any nal condon, θ evolvng accodng o conveges o he se H gven by H = {θ = m π,v : θ,c = θ π,c c C, v = } v π N n We can vefy ha H H usng 9-0 A poof fo he nex esul s dscussed n Appendx C Lemma 6 Suppose θ evolves accodng o hen, θ conveges o H as ends o nfny fo any θ0 H Now, due o he above esul, we have a key convegence esul fo he dffeenal equaon whch s closely elaed o he updae equaons 4-5 of A Nex, we use hs esul o oban a convegence esul fo θ We do so by vewng 4-5 as a sochasc appoxmaon updae equaon, and usng a esul fom [?] ha helps us o elae he dffenal equaon Lemma 7 If θ 0 H, hen he sequence θ geneaed by A conveges almos suely o he se H Poof Skech: We can pove he esul by vewng 4-5 as a sochasc appoxmaon updae equaon he poof manly uses Lemma 6 and heoem of Chape 6 fom [?] ha gves suffcen condons fo convegence of a sochasc appoxmaon scheme We had poned ou ha ou man nees s n he convegence popees of m, v he nex N esul uses Lemma 7 o esablsh he desed convegence popey A poof fo he esul s gven n Appendx D Lemma 8 If θ 0 H, hen he sequence θ geneaed by A sasfes: a Fo each N, lm m = m π, and b lm θ,c = θ π,c, and c Fo each N n, lm v = v π Nex, we use Lemma 8 and saonay o esablsh cean popees assocaed wh he me aveages of he ewad allocaons unde he onlne scheme A Fo bevy, n he followng esul, we le denoe m, v,c A poof fo he esul s gven n Appendx E Lemma 9 Fo almos all sample pahs, a Fo each N, lm τ = lm m b Fo each N, lm τ= a : = lm v he nex esul esablshes he asympoc opmaly of A, e, f we un A fo long enough peod, he dffeence n pefomance of A and he opmal fne hozon polcy becomes neglgble heoem Fo almos all sample pahs he followng wo saemens hold: a Feasbly: he allocaon : assocaed wh A sasfes 4 and 5, and fo each N b Opmaly: A s asympocally opmal, e, lm φ : φ : = 0 Poof: Snce he allocaon : assocaed wh A sasfes and 3 n each me slo, also sasfes 4 and 5 hus, pa a s ue o pove pa b, consde any ealzaon of c : Le µ : and γ : N : be he sequences of non negave eal numbes sasfyng 6, 7 and 8 fo he ealzaon Hence, fom he non-negavy of hese numbes, and feasbly of, we have : whee ϕ : φ : ϕ : = N = a + = : µ c = N γ mn Snce ϕ s a dffeenable concave funcon, we have see [?] B Asympoc Opmaly of A ϕ : ϕ : + ϕ : : :,

9 9 whee denoes he do poduc Hence, we have φ : ϕ : a : N = µ c + + = = N γ mn = N a : µ c, + γ τ τ= Fom Lemma 9 a-c and he connuy of he funcons nvolved, we can conclude ha he followng em appeang above can be made as small as desed by choosng lage enough and hen choosng a lage enough : a : τ τ= + v m Also, max Hence, akng lms n, lm φ : φ : 0 holds fo almos all sample pahs Fom opmaly of :, φ : φ : Fom he above wo nequales, he esul follows I CONCLSIONS he wo man conbuons of hs wok ae summazed below: We popose a novel famewok fo ewad allocaon o uses who ae sensve o empoal vaably n he ewad allocaon he fomulaon allows adeoffs beween mean and vaably assocaed wh he ewad allocaon of he uses by appopaely choosng he funcons, N We poposed an asympocally opmal onlne algohm A o solve poblems fallng n hs famewok APPENDIX A POOF OF LEMMA 4 Fo each c C, by choosng m π,v π,c = π c, µ = µ c πc and γ = γπ c πc fo all N, we can vefy ha m π,e π,v π,c along wh µ and γ : N sasfy 6-8 usng he fac ha π c : c C, µ π c : c C and γ πc N : c C sasfy 9- Pa b follows fom he defnons of m π and v π APPENDIX B POOF OF LEMMA 5 Now, snce µ : and γ : N : sasfy 6, 7 Fo each c C, m,v,c s an opmal soluon and 8, we have φ o OPA and hus, hee exs lke hose n KK- OPA gven n 6-8 non-negave consansµ : c and γ c : N such ha fo all N, a : N = c v c m + +γ µ cc c = 0, µ cc c = 0, = N γ c mn = 0, a : whee we used c nsead of m,v,c fo bevy Fo each N l, due o lneay we have ha τ s a consan, and hence s ndependen of s agumen hus, τ= + v m we have v = a C π Fuhe, noe ha m,v H and hence sasfes 9-0 sng hese agumens, we can ewe he above equaons as follows: fo all c C c a π C π c E[ C π ]+γ µ cc c = 0, µ cc c = 0, γ c mn = 0, Now fo each c C, mulply he above equaons wh πc and one obans KK-OPSA 9- wh πcµ c : c C and πcγ c N : c C as Lagange mulples Fom Lemma a, OPSA sasfes Slae s condon and hence KK condons ae suffcen fo opmaly of OPSA hus, we have ha m,v,c c C s an opmal soluon o OPSA hs poves pa a Now suppose ha m,v,m,v H, and suppose ha fo some c 0 C and N, m,v,c 0 m,v,c 0 hus, usng hs ogehe wh pa a, we

10 0 have ha m,v,c c C and m,v,c c C ae wo dsnc soluons o OPSA Howeve, hs conadcs fac ha OPSA has a unque soluon see Lemma b hus, b has o hold Now suppose ha m,v,m,v H and ha c does no hold hen, we can conclude ha aleas one of he condons gven n pa c does no hold Fo nsance, suppose ha v j v j fo some j N n hs along wh he fac ha m,v,m,v H and hus hey sasfy 0 mples ha a m,v,c π a m,v,c π hus, we can conclude ha fo some c 0 C and N, m,v,c 0 m,v,c 0 We can each he same concluson f any of he condons gven n c ae volaed Bu, he concluson conadcs pa b hus, c has o hold Pa d follows fom pa c and Lemma 4 pa b APPENDIX C POING LEMMA 6 We le θ π = m π,e π,v π, and θ = m,v, and consde he Lyapunov funcon θ = E[hθ π,c π ] E[hθ,C π ] hen d θ d = θ dθ d = θḡθ+zθ = θḡθ, whee he las sep follows fom Lemma?? Le θ = θḡθ hen fom Lemma 3 d and Lemma??, we have ha fo any θ H, v E[ θ,cπ ] m θ = N + N n v [ v E θ,cπ m ] he expesson above s he negave of a sum of posve weghed squaes Hence, θ 0 θ H 3 { Now, le H = θ H : } θ = 0 Snce s a connuously dffeenable funcon on he compac se H sasfyng 3, we can use LaSalle s heoem see heoem 44 n [?] o conclude ha θ conveges o he lages nvaan se n H Le H π denoe he se In he emanng pa of he poof, we pove ha H π H fom whch he man clam follows Nong ha θ = 0 fo any θ Hπ, and usng he expesson fo gven above, we can show ha E[ θ,c π ] = m θ H π 4 Also, fo any θ H π, θ = 0, and hence usng he fac ha mn v [0,vmax] v < 0, we have ha [ v = E θ,cπ m ] N n Fom he above concluson and 4, we can conclude ha fo any θ H π, we have θ H Snce H H, we have ha fo H π H Now, snce θ conveges o H π, we can conclude ha θ conveges o H and he esul follows APPENDIX D POOF FO LEMMA 8 Fo any m,v H, m = m π and fom Lemma 7, θ conveges o H Hence a holds o show b, pck some c C, and noe ha θ,c s a unfomly connuous funcon of θ on H unfom connuy follows fom he connuy of θ,c poved n Lemma 3 a, and compacness of H Hence, fo any ǫ > 0, we can fnd a δ > 0 such ha fo any θ H, d θ,c, θ,c < ǫ fo any θ H such ha d θ,θ < δ Hee d denoes he Eucldean dsance mec fo 3N In pacula, fo any θ H, d θ,c, θ,c < ǫ fo any θ H such ha d θ,θ < δ Fom he defnon of H, θ,c = c C θ π,c c C snce θ H hus, we have ha d θ,c, θ π,c < ǫ fo any θ H such ha d θ,θ < δ Fom Lemma 7, we have ha θ conveges o he se H Hence, fo a suffcenly lage, d θ,θ < δ fo some θ H, and hus d θ,c, θ π,c < ǫ hus, pa b holds Pas c and d can be poved usng a smla appoach as above by usng he followng facs: θ conveges o H ; v = v π fo any m,v H ; and Fo each N, s unfomly connuous on [0,v max ] APPENDIX E POOF OF LEMMA 9 Consde any ealzaonc ofc Fo anyc C, usng Lemma 8 b and he egodcy of C, we have lm = I c=c θ,c = θ π,c lm = I c=c = πc θ π,c Snce, θ,c = c C I c =c θ,c and C s a fne se, we can use he above esul o conclude ha lm θ,c = lm I c=c θ,c = = c C lm = c C = = c Cπc θ π,c = c Cπc π c = m π I c=c θ,c hs poves pa a sng he egodcy of C, pas b can be poved usng a smla appoach as above by usng pa c of Lemma 8