High-Probability Regret Bounds for Bandit Online Linear Optimization

Transcription

1 High-Probabiliy Regre Bouds for Badi Olie Liear Opimizaio Peer L. Barle UC Berkeley Varsha Dai Uiversiy of Chicago Aleader Rakhli UC Berkeley homas P. Hayes I Chicago hayes@i-c.org Ambuj ewari I Chicago ewari@i-c.org Sham M. Kakade I Chicago sham@i-c.org Absrac We prese a modificaio of he algorihm of Dai e al. [8] for he olie liear opimizaio problem i he badi seig, which wih high probabiliy has regre a mos O agais a adapive adversary. his improves o he previous algorihm [8] whose regre is bouded i epecaio agais a oblivious adversary. We obai he same depedece o he dimesio 3/ as ha ehibied by Dai e al. he resuls of his paper res firmly o hose of [8] ad he remarkable echique of Auer e al. [] for obaiig highprobabiliy bouds via opimisic esimaes. his paper aswers a ope quesio: i elimiaes he gap bewee he high-probabiliy bouds obaied i he full-iformaio vs badi seigs. 1 Iroducio I he olie liear opimizaio problem, here is a fied decisio se D R ad he player or decisio maker makes a decisio a ime {1,..., }. Simulaeously, a adversary chooses a loss vecor L ad he player suffers loss L. he goal is o miimize regre which measures how much worse he player did as compared o ay fied decisio, eve oe chose wih complee kowledge of he sequece L 1,..., L, R L mi L. he adversary ca be oblivious o he player s moves i which case i chooses he eire sequece L 1,..., L i advace of he player s moves. A adapive adversary ca, however, choose L based o he player s moves 1,..., 1 up o ha poi. I he full iformaio versio of he problem, he loss vecor L is revealed o he player a he ed of roud. For his case, Kalai ad Vempala [1] gave a efficie algorihm assumig ha he offlie problem give L miimize L over D ca be solved efficiely. Noe ha he sadard PB, AR ad A graefully ackowledge he suppor of DARPA uder gra FA epers problem is a special case of his problem because we ca choose he se D o be {e 1,..., e }, he ui vecors formig he sadard basis of R. Kalai ad Vempala separaed he issue of he umber of available decisios from he dimesioaliy of he problem ad gave a algorihm wih epeced regre Opoly. I may impora cases, for eample he olie shores pah problem [15], he size of he decisio se ca be epoeial i he dimesioaliy. So, i is impora o desig algorihms ha have polyomial depedece o he dimesio. I he parial iformaio or badi versio of he problem, he oly feedback ha he player receives a he ed of roud is is ow loss L. he badi versio of he epers problem was cosidered by Auer e al. [] who gave a umber of algorihms for he problem. heir Ep3 algorihm achieves O epeced regre agais oblivious adversaries. However, due o he large variace of he esimaes kep by Ep3 i fails o ejoy a similar regre boud wih high probabiliy. o address his issue, he auhors used he idea of high cofidece upper bouds o derive he Ep3.P algorihm which achieves O regre wih high probabiliy. he regre of hese algorihms also has a D depedece o he umber D of available acios. Hece, hese cao be used direcly if D is large. Awerbuch ad Kleiberg [4] were he firs o cosider he geeral olie liear opimizaio problem i he badi seig. For oblivious adversaries, hey proved a regre boud of O poly /3. he case of a geeral adapive adversary was hadled by McMaha ad Blum [14] bu hey could oly prove a regre boud of O poly 3/4. Dai ad Hayes [7] laer showed ha McMaha ad Blum s algorihm acually ejoys a regre boud of O poly /3. However, he kow lower boud for he badi problem was he same as ha i he full iformaio case, amely Ω. herefore, i was a impora ope quesio if here is a algorihm wih a regre boud of Opoly for he badi olie liear opimizaio problem. A affirmaive aswer was recely give by Dai e al. [8]. heir algorihm has epeced regre a mos O poly agais a oblivious adversary. I was sill o kow if he same bouds could be achieved wih high probabiliy ad agais adapive adversaries as well. I his paper, we show how o do his by combiig Dai e al. s echiques wih hose of Auer e al. []. Like Ep3.P, our GEOMERICHEDGE.P algorihm

2 keeps biased esimaes of he losses of differe acios such ha, wih high probabiliy, he sums of hese esimaes are lower bouds because we use losses o gais o he acual ukow cumulaive losses Lemma 5. he badi versio of he olie shores pah problem has recely received a lo of aeio. I ca be used o model, for eample, rouig i ad hoc wireless eworks. If we wa o make our rouig algorihm secure agais adversarial aacks, i is ecessary o desig algorihms ha work agais adapive adversaries [3, 13]. herefore, obaiig low regre agais adapive adversaries is o oly a impora heoreical problem bu i also has pracical implicaios. he algorihm wih he bes regre guaraee so far is by György e al. [11]. here he auhors cosider a umber of feedback models. Our feedback model i his paper correspods o wha hey call he pah-badi model. For his model, hey give a efficie algorihm specially desiged for he badi olie shores pah problem ha achieves O poly /3 regre wih high probabiliy agais a adapive adversary where is he umber of edges i he graph. Our resuls imply ha i is acually possible o achieve O 3/ regre wih high probabiliy. However, sice our algorihm is o efficie, he ques for a efficie algorihm wih he same regre, eve for his special problem, is sill o. he key ools from probabiliy heory ha we use i our proofs are Bersei-ype iequaliies, such as Freedma s. hese provide sharper coceraio bouds for marigales i he presece of variace iformaio. here is a simple corollary of Freedma s iequaliy ha we hik is useful o jus i our seig bu more geerally. We sae i as Lemma i Secio 4. he prese work closes he gap bewee full iformaio ad badi olie opimizaio agais he adapive adversary i erms of he growh of regre wih. As we said above, our algorihm is o ecessarily efficie, because he decisio space migh eed o be discreized o a fie level. We meio ha a parallel work by Aberehy, Haza, ad Rakhli [1] provides a efficie algorihm for he seig; however, heir resul holds i epecaio oly agais a oblivious adversary. he prese paper ad [1] are addressig disparae aspecs of he problem ad eiher resul ca be cocluded from he oher. I remais a ope quesio wheher here eiss a efficie algorihm which ejoys high probabiliy bouds o he regre. Prelimiaries Le D [ 1, 1] deoe he decisio space. A each of ime seps, he evirome selecs a cos vecor L, ad simulaeously, he player decisio maker selecs D. he loss icurred by he decisio maker for his predicio is L. Le L mi : mi L be he loss of he bes sigle decisio i hidsigh. he goal of he decisio maker is o miimize he regre, R L L mi. We assume ha L [0, 1] for all D. We also assume ha he evirome is adapive, i.e., he cos vecor L seleced by he evirome a ime may deped arbirarily o he hisory L 1, 1,..., L 1, 1 oe ha wihou loss of geeraliy his depedece may be assumed o be deermiisic. We show ha eve agais such a powerful evirome, i is possible o esure ha R is small wih high probabiliy. As i [8], we will require a baryceric spaer for D. Recall ha a baryceric spaer for D is a se {y 1,..., y } D such ha every D ca be wrie as a liear combiaio of y i s wih coefficies i [ 1, 1]. A c-baryceric spaer is defied similarly where we allow coefficies o be i [ c, c]. For c > 1, c-baryceric spaers for D may be foud efficiely see [4]. However, for ease of eposiio we ll assume ha we have a acual baryceric spaer. Usig a c-baryceric spaer isead will oly affec he cosas. Fially, if he se D is oo large for eample if i is ifiie we ca replace i by a cover of size a mos 4 /, as he loss of he opimal decisio i his cover is wihi a addiive of he opimal loss i D; see [8][Lemma 3.1] for deails. Accordigly, afer doig his rasformaio if ecessary, we may assume ha D is fiie ad l D O l. Oly he logarihm of he cardialiy of he se will eer i our bouds. 3 Algorihm ad Mai Resul he algorihm preseed below is a modificaio of he algorihm i [8]. Noe ha he differece is i he way we updae weighs w, usig lower cofidece iervals. his idea of usig cofidece iervals is moivaed by he Ep3.P algorihm of Auer e al. []. Feedig i cofidece bouds, as opposed o ubiased esimaes of he losses, o he epoeial updaes is he crucial chage we make o he algorihm of Dai e al [8]. Lemma 5 below shows ha, wih high probabiliy, for ay D, L lower bouds L up o a addiive O erm. Our algorihm reduces o Ep3.P i he special case of he -armed badi problem whe D {e 1,..., e }. As we poi ou i he e secio, Auer e al. s proof ca be simplified by usig he simple corollary of Freedma s iequaliy [10] ha we sae as Lemma below. he mai resul of his paper is he followig guaraee o he algorihm. heorem 1 Le 4, ad δ 1 e. If we se γ 3/, δ δ D log, ad η 1, he agais + l1/δ ay adapive adversary wih probabiliy a leas 1 4δ, R O 3/ l/δ. he depedece o is opimal up o logarihmic facors. We ge he same depedece o as Dai e al. [8]. he lower boud kow for his problem is Ω [8]. Recely, O regre bouds have bee obaied for he sochasic versio of he problem [9]. his leads us o cojecure ha he lower boud is igh ad i remais a ope

3 Algorihm 3.1: GEOMERICHEDGE.PD, γ, η, δ D, w 1 : 1 W 1 : D for 1 o D, p w W + γ I{ spaer} Sample accordig o disribuio p Icur ad observe loss l : L C : E p [ ] L : l C 1 D, L : L C 1 l1/δ D, w +1 : w ep{ η L } W +1 w +1 quesio o close he gap for he depedece o bewee upper ad lower bouds. We also oe here ha alhough he aalysis we provide is for losses, esseially he same algorihm, wih a similar aalysis, works for gais. We jus have o make a few obvious chages o he algorihm: isead of subracig, we add he correcio erm o he gai esimaes ad replace η wih η i he epoeial updae. 4 Coceraio for Marigales I his secio we derive a coceraio iequaliy for marigale differece sequeces. I is a direc applicaio of Freedma s iequaliy. Lemma Suppose X 1,..., X is a marigale differece sequece wih X b. Le Var X Var X X 1,..., X 1. Le V Var X be he sum of codiioal variaces of X s. Furher, le σ V. he we have, for ay δ < 1/e ad 4, Prob { X > ma σ, b } l1/δ l1/δ log δ. Proof: Noe ha a crude upper boud o Var X is b. hus, σ b. We choose a discreizaio 0 α 1 < α 0 <... < α l such ha α i+1 α i for i 0 ad α l b. We will specify he choice of α 0 shorly. We he have, Prob X > ma{σ, α 0 } l1/δ l Prob X > ma{σ, α 0 } l1/δ & α j 1 < σ α j l Prob X > α j l1/δ & αj 1 < V α j l Prob X > α j l1/δ & V α j l 4α ep j l1/δ αj + 3 α j l1/δ b l ep α j l1/δ l1/δ α j + 3 b where he iequaliy follows from Freedma s iequaliy heorem 9. If we ow choose α 0 b l1/δ he α j b l1/δ for all j ad hece every erm i he above summaio is bouded by ep l1/δ 1+/3 < δ. Choosig l log esures ha α l b. hus we have Prob X > ma{σ, b l1/δ} l1/δ Prob X > ma{σ, α 0 } l1/δ l + 1δ log + 1δ log δ. his iequaliy says ha, roughly speakig, X is of he order of σ l1/δ which is a ceral limi heoremlike behavior ecep ha σ here is o fied bu is he acual sum of codiioal variaces, a radom quaiy. he overall cosa i fro of σ is 4. his ca be improved o by a slighly more careful aalysis. We already kow of wo isaces i he lieraure where Lemma ca be used o give shorer proofs of cerai probabilisic upper bouds. 1. he firs is i he proof of Ep3.P s regre boud iself. o show ha he esimaes are upper bouds o he acual losses of a acio, he auhors eplicily use he epoeial mome mehod i he proof of heir Lemma 6.1. Esseially he same lemma ca be proved by a direcio applicaio of he above lemma.. he oher isace is i Cesa-Biachi ad Geile s paper [5] o olie o bach coversios. Whe a olie algorihm is ru o i.i.d. daa wih a o-egaive ad bouded loss fucio, he codiioal variace of he loss a ime ca immediaely be bouded by he risk of he hypohesis a ime 1. he auhors use his fac alog wih a applicaio of Freedma s iequaliy o prove a sharp upper boud Proposiio i heir paper o he average risk of he hypoheses geeraed by he olie algorihm i erms of is acual cumulaive loss. he same resul ca be quickly derived by a applicaio of he above lemma.

4 5 Aalysis he remaider of he paper is devoed o he proof of heorem 1. We firs sae several resuls obaied i Dai e al [8] which will be impora i our proofs. Lemma 3 For ay D ad {1,..., }, i holds ha 1. L /γ. C 1 /γ. 3. p C E L C 1. We ow prove a boud o he perurbed esimaed coss, L, which are used o updae he disribuio. Lemma 4 For all D, L + l1/δ. Proof: For each D, L L + C 1 l1/δ γ + l1/δ γ + l1/δ usig Lemma 3 ad he choice of γ 3/. 5.1 High Cofidece Bouds Le E [ ] deoe E[ 1,..., 1 ]. Sice we are cosiderig adapive bu deermiisic adversaries, L is o radom give 1,..., 1. Observe ha E [ ] E p [ ] ad hus, E [ L ] L. However, he flucuaios of he radom variable L are very large. he followig lemma provides a boud o hese flucuaios. Lemma 5 Assume 4. Le δ δ D log. he wih probabiliy a leas 1 δ, simulaeously for all D, L L l1/δ Proof: Fi D. Le M M L L. he M is a marigale differece sequece. Usig Lemma 3, M γ Le V Var M ad le σ V. Usig Lemma, we have ha wih probabiliy a leas 1 δ log, L L + ma{σ, 1 + l1/δ } l1/δ 1 Now oe ha σ C 1 1 C 1 +, by he arihmeic mea-geomeric mea iequaliy. Subsiuig his io 1, we have L { L + ma 1 + l1/δ, C 1 + } l1/δ wih probabiliy a leas 1 δ log. Fially, akig a uio boud over all D ad rearragig usig he fac ha ma{a + b, c} a + ma{b, c} if a 0 gives he required resul. 5. Poeial Fucio Aalysis By Lemma 4 ad our choice of η 1 + l1/δ, we have η L 1. I he followig compuaio, we will use he facs ha e a 1 a + a wheever a 1. W +1 W w ep η L W w 1 η W L + η L 1 + η p L + γ L + p η L spaer sice by defiiio of p, w W Noe ha we have, p L p γ I{ spaer}. p L + p C 1 l1/δ p L l1/δ + where he las sep is by Lemma 3. Furher, sice b + c b + c for every b, c, apply-

5 ig he defiiio of L, we also have p η L η p L + C 1 l1/δ η p L + 4 C 1 l1/δ γ η η [ [ p L + 4 l1/δ ] p L + 4 l1/δ by successive applicaios of Lemma 3. Puig hese ogeher, we have W +1 W 1 + η p L l1/δ + + γ L spaer p C 1 + η p L l1/δ + 8η akig logs, usig he fac ha l1 +, ad summig over, we have W +1 l η [ p L W 1 + l1/δ γ + L spaer + η p L + 8η l1/δ ] he e hree lemmas will boud he hree summaios ha appear o he righ had side above. Lemma 6 Wih probabiliy a leas 1 δ, L p L + 1 l1/δ l1/δ γ + 1. ] Proof: Le us defie : E p p ad Y : l L. Noe ha E L E l ad herefore Y is a marigale differece sequece. We boud he codiioal variace of Y as follows. Var Y E Y E L l E L + E l by Cauchy-Schwarz E L + 1 sice l 1 C by Lemma 3 E C p by Jese s iequaliy + 1 by Lemma 3. Moreover, Y /γ + 1 by Lemma 3. Applyig Bersei s iequaliy for marigale differeces see Appedi o he sequece Y, we obai ha wih probabiliy a leas 1 δ, Y + 1 l1/δ l1/δ γ + 1, which is he desired boud. Lemma 7 Wih probabiliy a leas 1 δ, γ L γ + γ spaer 1 + l1/δ. Proof: Usig Lemma 5, wih probabiliy a leas 1 δ, we have, for all spaer, γ L γ L + γ 1 + l1/δ γ + γ 1 + l1/δ, because L, beig he loss of a eleme of he spaer, is bouded by 1. Summig over he elemes of he spaer, we ge he desired boud. Lemma 8 Wih probabiliy a leas 1 δ, p L + l1/δ. Proof: Firs we observe ha for 1, p L p L L L p L l C 1 C C 1 C 1

6 Summig over, p L C 1. Lemma 3 ells us ha, o he oe had, he summads C 1 are uiformly bouded by /γ, ad o he oher had, ha each oe has epecaio, eve codiioed o he previous oes. Applyig he Hoeffdig-Azuma iequaliy o he marigale differece sequece C 1 E p C 1 i follows ha, wih probabiliy a leas 1 δ, compleig he proof. C 1 + l1/δ, Subsiuig he bouds of Lemmas 6, 7 ad 8 io, we obai ha wih probabiliy a leas 1 3δ, l W +1 W 1 η [ L l1/δ l1/δ γ l1/δ + γ + γ 1 + l1/δ + η + η l1/δ + 8η l1/δ ] 3 O he oher had, usig Lemma 5, we have wih probabiliy a leas 1 δ, for all D, l W +1 W 1 η η L l D L η1 + l1/δ l D. 4 Combiig 3 wih 4, we have ha wih probabiliy a leas 1 4δ, for every D, L L l1/δ + 1 l D η l1/δ l1/δ γ l1/δ + γ + γ 1 + l1/δ + η + η l1/δ + 8η l1/δ Recall ha η 1 + l1/δ, γ 3/, δ δ/ D log, ad l D O l. Pluggig i hese values yields L L mi + O 3/ l/δ, compleig he proof of heorem 1. 6 Coclusios ad Ope Problems We preseed a algorihm ha achieves he desired regre boud of O wih high probabiliy. However, he ques for a efficie algorihm wih he same high-probabiliy guaraee, eve for he special case of badi olie shores pahs, is sill ope. Achievig similar resuls for geeral cove fucios is also a iriguig ope quesio. A Coceraio Iequaliies he followig iequaliies are well kow. heorem 9 is from [10]. Lemmas 10 ad 11 ca be foud, for isace, i [6], Appedi A. heorem 9 Freedma Suppose X 1,..., X is a marigale differece sequece, ad b is a uiform upper boud o he seps X i. Le V deoe he sum of codiioal variaces, V Var X i X 1,..., X i 1. i1 he, for every a, v > 0, Prob Xi a ad V v ep a. v + ab/3 Lemma 10 Bersei s iequaliy for marigales Le Y 1,..., Y be a marigale differece sequece. Suppose ha Y [a, b] ad E[Y X 1,..., X 1 ] v a.s.

7 for all {1,..., }. he for all δ > 0, Pr Y > v l1/δ + l1/δb a/3 δ Lemma 11 Hoeffdig-Azuma iequaliy Le Y 1,..., Y be a marigale differece sequece. Suppose ha Y c almos surely for all {1,..., }. he for all δ > 0, Pr Y > c l1/δ δ Refereces [1] Jacob Aberehy, Elad Haza, ad Aleader Rakhli. Compeig i he dark: A efficie algorihm for badi liear opimizaio. I Proceedigs of he 1s Aual Coferece o Learig heory COL, 008. o appear. [] Peer Auer, Nicolò Cesa-Biachi, Yoav Freud, ad Rober E. Schapire. he osochasic muliarmed badi problem. SIAM Joural o Compuig, 31:48 77, 003. [3] Baruch Awerbuch, David Holmer, Herb Rubes, ad Rober Kleiberg. Provably compeiive adapive rouig. I Proceedigs of he 31s IEEE INFOCOM, volume 1, pages , 005. [4] Baruch Awerbuch ad Rober Kleiberg. Adapive rouig wih ed-o-ed feedback: Disribued learig ad geomeric approaches. I Proceedigs of he 36h ACM Symposium o heory of Compuig SOC, 004. [5] Nicolò Cesa-Biachi ad Claudio Geile. Improved risk ail bouds for o-lie algorihms. IEEE rasacios o Iformaio heory, 541:386 39, Ja 008. [6] Nicolò Cesa-Biachi ad Gábor Lugosi. Predicio, Learig, ad Games. Cambridge Uiversiy Press, 006. [7] Varsha Dai ad homas P. Hayes. Robbig he badi: Less regre i olie geomeric opimizaio agais a adapive adversary. I Proceedigs of he 17h ACM-SIAM Symposium o Discree Algorihms SODA, 006. [8] Varsha Dai, homas P. Hayes, ad Sham M. Kakade. he price of badi iformaio for olie opimizaio. I J. C. Pla, D. Koller, Y. Siger, ad S. Roweis, ediors, Advaces i Neural Iformaio Processig Sysems 0 NIPS 007. MI Press, 008. [9] Varsha Dai, homas P. Hayes, ad Sham M. Kakade. Sochasic liear opimizaio uder badi feedback. I Proceedigs of he 1s Aual Coferece o Learig heory COL, 008. o appear. [10] David A. Freedma. O ail probabiliies for marigales. he Aals of Probabiliy, 31: , Feb [11] Adrás György, amás Lider, Gábor Lugosi, ad György Oucsák. he o-lie shores pah problem uder parial moiorig. Joural of Machie Learig Research, 8: , 007. [1] Adam Kalai ad Saosh Vempala. Efficie algorihms for olie decisio problems. Joural of Compuer ad Sysem Scieces, 713:91 307, 005. [13] Rober Kleiberg. Olie decisio problems wih large sraegy ses. PhD hesis, MI, 005. [14] H. Breda McMaha ad Avrim Blum. Olie geomeric opimizaio i he badi seig agais a adapive adversary. I Proceedigs of he 17h Aual Coferece o Learig heory COL, 004. [15] Eiji akimoo ad Mafred K. Warmuh. Pah kerels ad muliplicaive updaes. Joural of Machie Learig Research, 4: , 003.