A Stochastic View of Optimal Regret through Minimax Duality

Transcription

1 A Sochasic View of Opimal Regre hrough Minimax Dualiy Jacob Abernehy Compuer Science Division UC Berkeley Alekh Agarwal Compuer Science Division UC Berkeley Peer L. Barle Compuer Science Division Deparmen of Saisics UC Berkeley Alexander Rakhlin Deparmen of Saisics Universiy of Pennsylvania Absrac We sudy he regre of opimal sraegies for online convex opimizaion games. Using von Neumann s minimax heorem, we show ha he opimal regre in his adversarial seing is closely relaed o he behavior of he empirical minimizaion algorihm in a sochasic process seing: i is equal o he maximum, over join disribuions of he adversary s acion sequence, of he difference beween a sum of minimal expeced losses and he minimal empirical loss. We show ha he opimal regre has a naural geomeric inerpreaion, since i can be viewed as he gap in Jensen s inequaliy for a concave funcional he minimizer over he player s acions of expeced loss defined on a se of probabiliy disribuions. We use his expression o obain upper and lower bounds on he regre of an opimal sraegy for a variey of online learning problems. Our mehod provides upper bounds wihou he need o consruc a learning algorihm; he lower bounds provide explici opimal sraegies for he adversary. Inroducion Upon a review of he cenral resuls in adversarial online learning mos of which can be found in he recen book Cesa-Bianchi and Lugosi 7 one canno help bu noice frequen similariies beween he guaranees on performance of online algorihms and he analogous guaranees under sochasic assumpions. However, discerning an explici link has remained elusive. Vovk 2 noices his phenomenon: for some imporan problems, he adversarial bounds of online compeiive learning heory are only a iny amoun worse han he average-case bounds for some sochasic sraegies of Naure. In his paper, we aemp o build a bridge beween adversarial online learning and saisical learning. Using von Neumann s minimax heorem, we show ha he opimal regre of an algorihm for online convex opimizaion is exacly he difference beween a sum of minimal expeced losses and he minimal empirical loss, under an adversarial choice of a sochasic process generaing he daa. his leads o upper and lower bounds for he opimal regre ha exhibi several similariies o resuls from saisical learning. he online convex opimizaion game proceeds in rounds. A each of hese rounds, he player learner predics a vecor in some convex se, and he adversary responds wih a convex funcion which deermines he player s loss a he chosen poin. In order o emphasize he relaionship wih he sochasic seing, we denoe he player s choice as f F and he adversary s choice as z Z. Noe ha his differs, for insance, from he noaion in 2. Suppose F is a convex compac class of funcions, which consiues he se of Player s choices. he Adversary draws his choices from a closed compac se Z. We also define a coninuous bounded loss funcion l : Z F R and assume ha l is convex in he second argumen. Denoe by lf = {l, f : f F} he associaed loss class. Le P be he se of all probabiliy disribuions on Z. Denoe a sequence Z,..., Z by Z. We denoe a join disribuion on Z by a bold-face p and is condiional and marginal disribuions by p Z and p m, respecively. he online convex opimizaion ineracion is described as follows. Online Convex Opimizaion OCO Game A each ime sep = o, Player chooses f F Adversary chooses z Z Player observes z and suffers loss lz, f he objecive of he player is o minimize he regre lz, f inf lz, f. I urns ou ha many online learning scenarios can be realized as insances of OCO, including predicion wih exper advice, daa compression, sequenial invesmen, and forecasing wih side informaion see, for example, 7. Le us briefly menion previous work and ouline our conribuions. Our saring poin, heorem, is similar o Sec. 2.0 of 7, bu exends i o non-oblivious adversaries and arbirary losses. Dealing wih non-oblivious adversaries viz., hose whose acions depend on player s choices enails considering a nesed sequence of inf/ pairs, and much of he generaliy and difficuly comes from his seup. he use of minimax dualiy has a long hisory in decision heory and Bayesian analysis. In he case of binary sequence predicion,

2 we refer o 9, 8 and references herein. For he log loss, he minimax sraegy is known o have a closed form, and here is a large body of work on he subjec in differen communiies see 8, 7. he use of Rademacher averages in he conex of predicion firs appeared in 6 for he absolue loss. he aim of his paper is o provide a unifying framework, as well as ools for sudying minimax regre in a general seing: a non-oblivious adversary and convex loss funcions. One of he conribuions is he upper bound in erms of Rademacher averages heorem 8, which exends ha in 7 o non-linear losses. he lower bound of heorem 9 generalizes he asympoic bound for exper case e.g. 7. We provide an imporan geomeric viewpoin and show ha fas raes can be obained by sudying properies of he minimum expeced risk funcional. We show ha srong convexiy implies smoohness of his funcional, hus recovering known resuls on logarihmic regre. When he funcional is non-differeniable, an explici opimal sraegy of he adversary is o play he disribuion ha exhibis he nondiffereniabiliy. 2 Applying von Neumann s minimax heorem Define he value of he OCO game which we also call he minimax regre as R := inf f F inf z Z f F z Z inf f F z Z lz, f inf lz, f. he OCO game has a purely opimizaion flavor. However, applying von Neumann s minimax heorem shows ha is value is closely relaed o he behavior of he empirical minimizaion algorihm in a sochasic process seing. heorem Under he assumpions on F, Z, and l given in he previous secion, R = E inf E lz, f Z p f F inf lz, f, 2 where he remum is over all join disribuions p on Z and he expecaions are over he sequence of random variables {Z,..., Z } drawn according o p. he proof relies on he following version of von Neumann s minimax heorem; i appears as heorem 7. in 7. Proposiion 2 Le Mx, y denoe a bounded real-valued funcion on X Y, where X and Y are convex ses and X is compac. Suppose ha M, y is convex and coninuous for each fixed y Y and Mx, is concave for each x X. hen inf Mx, y = inf Mx, y. x X y Y y Y x X Proof: of heorem Consider he las opimizaion choice z in Eq.. If we insead draw z according o a probabiliy disribuion, and compue he expeced value of he quaniy in he parenheses in Eq., hen maximizing his expeced value over all disribuions on Z is equivalen o maximizing over z. Hence, R = inf inf f F z Z E Z p inf f F z Z f F p P lz, f inf lz, f. 3 In he las expression, i is undersood ha sums are over he sequence {z,..., z, Z }, ha is, he firs elemens are quanified in he rema, while he las Z is a random variable. We now apply Proposiion 2 o he las inf/ pair in 3, wih Mf, p = E Z p lz, f inf lz, f which is convex in f by assumpion and linear in p. Moreover, he se F is compac, and boh F and P are convex. We conclude ha R = inf inf f F z Z f F z Z E lz, f inf = inf inf f F z Z inf p P f F f F z Z p P + inf f F E lz, f E inf lz, f lz, f lz, f. Noe ha, in he las line, he maximizing disribuion p depends on he previous choices z, bu no on any of he f s. As we swap inf / from inside ou, z s are aken o be random variables and denoed by Z. Pulling he expecaion on he hird erm ouside and repeaing he process imes, we arrive a he saemen of he heorem. We refer o for more deails. We can hink of Eq. 2 as a game where he adversary goes firs. A every round he plays a disribuion and he player responds wih a funcion ha minimizes he condiional expecaion. We remark ha we can allow he player o choose f s non-deerminisically in he original OCO game. In ha case, he original infimum should be over disribuions on F. We hen do no need convexiy of l in f F s in order o apply von Neumann s heorem, and he resuling expression for he value of he game is he same. 3 Firs Seps he presen work focuses on analyzing he expression in Equaion 2 for a range of differen choices of Z and F, as,

3 well as for various assumpions made abou he loss funcion l. We are no only ineresed in upper- and lower-bounding he value of he game R, bu also in deermining he ypes of disribuions p ha maximize or almos maximize he expression in 2. o ha end, define p-regre as R p = E min E lz, f Z min lz, f f F for any join disribuion p of z,..., z Z. In his secion we will provide an array of analyical ools for working wih R p. 3. Regre for IID and Produc Disribuions I is naural o consider i.i.d. processes and produc disribuions p as candidaes for maximizing R p. Lemma 3 For any i.i.d. disribuion p, R p 0. Hence, R 0. Proof: For an i.i.d. disribuion Eq. 4 becomes R p = min E lz, f E min min E lz, f min E lz, f 4 lz, f = 0 where he inequaliy is due o he fac ha E min min E. Observe ha R p for an i.i.d. process is he difference beween he minimum expeced loss and he expecaion of he empirical loss of an empirical minimizer. Wih he goal of sudying various ypes of disribuions, we now define he following hierarchy: R i.i.d. := p=p R p; R indep. := p=p... p R p, where p, p,..., p are arbirary disribuions on Z. I is immediaely clear ha 0 R i.i.d R indep. R. 5 We will see ha, given paricular assumpions on F, Z and l, some of he gaps in he above hierarchy are significan, while ohers are no. Before coninuing, however, we need o develop some ools for analyzing he minimax regre. 3.2 ools for a General Analysis We now inroduce wo new objecs ha help o simplify he expression in 2 as well as derive properies of R p. Definiion 4 Given ses F, Z, we can define he minimum expeced loss funcional Φ as Φp := inf E Z p lz, f, where p is some disribuion on Z. Defining an inner produc h, p = hzdpz for a disribuion p, we observe ha Φp = inf l, f, z p. Definiion 5 For any Z,..., Z Z, we denoe ˆP = Z, he empirical disribuion. Wih his addiional noaion, we can rewrie 4 as R p = E Φp Z E Φ ˆP. 6 hus, he Adversary s ask is o induce a large deviaion beween he average sequence of condiional disribuions {p Z } and an empirical sample ˆP from hese condiionals, where he deviaion is defined by way of he funcional Φ. Lemma 6 he funcional Φ is concave on he space of disribuions over Z and R is concave wih respec o join disribuions on Z. he easy proof of his lemma is in he full version. I is indeed concaviy of Φ ha is key o undersanding he behavior of R. A hin of his can already be seen in he proof of Lemma 3, where he only inequaliy is due o he concaviy of he min. In he nex secion, we show how his descripion of regre can be inerpreed hrough a Bregman divergence in erms of Φ. 3.3 Divergences and he Gap in Jensen s Inequaliy We now show how o inerpre regre hrough he lens of Jensen s Inequaliy by providing ye anoher expression for i, now in erms of Bregman Divergences. We begin by revisiing he i.i.d. case p = p = p... p, for some disribuion p on Z. Equaion 6 simplifies o a very naural quaniy, R p = Φp E Φ ˆP. 7 Noice ha ˆP is a random quaniy, and in paricular ha E ˆP = p. As Φ is concave, wih an immediae applicaion of Jensen s Inequaliy we obain R p 0. For arbirary join disribuions p, we can similarly inerpre regre as a gap in Jensen s Inequaliy, albei wih some added complexiy. Definiion 7 If F is any convex differeniable funcional on he space of disribuions on Z, we define Bregman divergence wih respec o F as D F q, p = F q F p F p, q p. If F is non-differeniable, we can ake a paricular subgradien v p F p in place of F p. Noe ha he noion of subgradiens is well-defined even for infinie-dimensional convex funcions. Having chosen 2 a mapping p v p Here, we mean differeniable wih respec o he Fréche or Gâeaux derivaive. We refer he reader o 0 for precise definiions of funcional Bregman Divergences. 2 he assumpion of compacness of F, ogeher wih he characerizaion of he subgradien se in Secion 4.2, allow us, for insance, o define he mapping p v p by puing a uniform measure on he subgradien se and defining v p o be he expeced subgradien wih respec o i. In fac, he choice of he mapping is no imporan, as long as i does no depend on q.

4 F p, we define a generalized divergence wih respec o F and v p as D F q, p = F q F p v p, q p. hroughou he paper, we focus only on he divergence D Φ, and hus we omi Φ from he noaion for simpliciy. Given he definiion of divergence, i immediaely follows ha, for a random disribuion q, ΦE q E Φq = E Dq, E q since he linear erm disappears under he expecaion. his simple observaion is quie useful; noice we now have an even simpler expression for i.i.d. regre 7: Rp = E D ˆP, p. In oher words, he p -regre is equal o he expeced divergence beween he empirical disribuion and is expecaion. his will be a saring poin for obaining lower bounds for R. For general join disribuions p, le us rewrie he expression in 6 as E U E Φp Z E Φ ˆP, where we replaced he average wih a uniform disribuion on he rounds. Roughly speaking, he nex lemma says ha one can obain E Φ ˆP from E U E Φp Z hrough hree applicaions of Jensen s inequaliy, due o various expecaions being pulled inside or ouside of Φ. Lemma 8 Suppose p is an arbirary join disribuion. Denoe by p Z and p m he condiional and marginal disribuions, respecively. hen Rp = 0 + 2, 8 where 0 = D p m, pm, = E p Dp Z, p m, 2 = E p D ˆP, p m. Proof: he marginal disribuion saisfies E p Z = p m, and i is easy o see ha E ˆP = pm. Given his, we see ha Rp = E p Φp Z Φ ˆP { }}{ { = E p Φp Z Φp m } 0 {}}{ + Φp m Φ pm 2 {}}{ E p Φ ˆP Φ pm. his lemma sheds some ligh on he influence of an i.i.d. vs. produc vs. arbirary join disribuion on he regre. For produc disribuions, every condiional disribuion is idenical o is marginal disribuions, hus implying = 0. Furhermore, for any i.i.d. disribuion, each marginal disribuion is idenical o he average marginal, hus implying ha 0 = 0. Wih his in mind, i is emping o asser ha he larges regre is obained a an i.i.d. disribuion, since ransiions from i.i.d o produc, and from produc o arbirary disribuion, only subrac from he regre value. While appealing, his is unforunaely no he case: in many insances he final erm, 2, can be made larger wih a non-i.i.d. and even non-produc disribuion, even a he added cos of posiive 0 and erms, so ha R i.i.d. = or as a funcion of. In some cases, however, we show ha a lower bound on he regre can be obained wih an i.i.d. disribuion a a cos of only a consan facor. 4 Properies of Φ In saisical learning, he rae of decay of predicion error is known o depend on he curvaure of he loss: more curvaure leads o faser raes see, for example, 5, 6, 4, and slow e.g. Ω /2 raes occur when he loss is no sricly convex, or when he minimizer of he expeced loss is no unique 5, 7. here is a sriking parallel wih he behavior of he regre in online convex opimizaion; again he curvaure of he loss plays a cenral role. Roughly speaking, if l is srongly convex or exp-concave, second-order gradiendescen mehods ensure ha he regre grows no faser han log e.g. ; if l is linear, he regre can grow no faser han e.g. 22; inermediae raes can be achieved as well if he curvaure varies 3. he previous secion expresses regre as a sum of divergences under Φ, and ha suggess ha he curvaure of Φ should be an imporan facor in deermining he raes of regre. We shall see ha his is he case: curvaure of Φ leads o large regre, while flaness of Φ implies small regre. We will now show how properies of he loss funcion class deermine he curvaure of Φ. In laer secions we will show how such curvaure properies lead direcly o paricular raes for R. Firs, le us provide a fruiful geomeric picure, rooed in convex analysis. I allows us o see he funcion Φ, roughly speaking, as a mirror image of he funcion class. 4. Geomeric inerpreaion of Φ In general, he se Z is uncounable, so care mus be aken wih regard o various noions we are abou o inroduce. We refer he reader o Chaper 0 of 5 for he discussion of finie vs infinie-dimensional spaces in convex analysis. Since Z is compac by assumpion, we can discreize i o a fine enough level such ha he upper and lower bounds of his paper hold, as long as he resuls are non-asympoic. In he presen Secion, for simpliciy of exposiion, we will pose ha he se Z is finie wih cardinaliy d. his assumpion is required only for he geomeric inerpreaion; our proofs are correc as long as Z is compac.

5 Hence, disribuions over he se Z are associaed wih d- dimensional vecors. Furhermore, each f F is specified by is d values on he poins. We wrie l f R d for he loss vecor of f, l, f. Le us denoe he se of all such vecors by lf. We hen have Φp = inf E plz, f = l f, p = σ lf p, f F where σ S x = s S s, x is he por funcion for he se S. his funcion is one of he mos basic objecs of convex analysis see, for insance, 2. I is well-known ha σ S = σ cos ; in oher words, he por funcion does no change wih respec o aking convex hull see Proposiion 2.2., page 37, 2. o his end, le us denoe S = co lf R d. R d S Figure : Dual cone as he epigraph of he por funcion. Φ is he resricion o he simplex. I is known ha he por funcion is sublinear and is epigraph is a cone. o visualize he por funcion, consider he R d R space. Embed he se S R d in R d {}. hen consruc he conic hull of S {}. I urns ou ha he cone which is dual o he consruced conic hull is he epigraph of he por funcion σ S. he dual cone is he se of vecors which form obuse or righ angles wih all he vecors in he original cone. Hence, one can visualize he surface σ S as being a righ angles o he conic hull of S {}. Now, he funcion Φ is jus he resricion of σ S o he simplex see Figure. We can now deduce properies of Φ from properies of he loss class. 4.2 Differeniabiliy of Φ Lemma 9 he subdifferenial se of Φ is he se of expeced minimizers: Φp = {l f : f arg min E plz, f}. Hence, he funcional Φ is differeniable a a disribuion p iff arg min E p lz, f is unique. Proof: he saemen follows from Proposiion 2..5 in 2. In paricular, for Φ o be differeniable for all disribuions, he loss funcion class should no have a face exposed o he origin. his geomerical picure and is implicaions will be sudied furher in Secion 6. I is easy o verify ha sric convexiy of lz, f in f implies uniqueness of he minimizer for any p and, hence, differeniabiliy of Φ. Φ σ S 4.3 Flaness of Φ hrough curvaure of l In his secion we show ha curvaure in he loss funcion leads o flaness of Φ. We would indeed expec such a resul o hold since regre decaying faser han O /2 is known o occur in he case of curved losses e.g. 3, and decomposiion 6 suggess ha his should imply flaness of Φ. More precisely, we show ha if lf, z is srongly convex in f wih respec o some norm, hen Φ is srongly fla wih respec o he l norm on he space of disribuions. Before saing he main resul, we provide several definiions. Definiion 0 A convex funcion F is α-fla or α-smooh wih respec o a norm when F y F x F x, y x + α x y 2 9 for all x, y. We will say ha a concave funcion G is α-fla if G saisfies 9. Le us also recall he definiion of l or variaional norm on disribuions. Definiion For wo disribuions p, q on Z, we define p q = dpz dqz. Z heorem 2 Suppose lz, f is σ-srongly convex in f, ha is, l z, f + g lz, f + lz, g σ f g for any z Z and f, g F. Suppose furher ha l is L Lipschiz, ha is, lz, f lz, g L f g. Under hese condiions, he Φ-funcional is 2L2 σ -fla wih respec o. he proof uses he following lemma, which shows sabiliy of he minimizers. Is proof is in he full version. Lemma 3 Fix wo disribuions p, q. Le f p and f q be he funcions achieving he minimum in Φp and Φq, respecively. Under he condiions of heorem 2, f p f q 2L σ p q. Proof:of heorem 2 We have Φp Φq = E p lz, f p E q lz, f q = E p lz, f p E q lz, f p + E q lz, f p E q lz, f q. 0 Le us firs sudy he second erm in he expression above. As f p is he minimizer of E p lz, f, we have: So E p lz, f p lz, f q 0 E q lz, f p lz, f q E q lz, f p lz, f q E p lz, f p lz, f q = lz, f p lz, f q qz pzdz L f p f q pz qz dz.

6 Using Lemma 3, we ge: E q lz, f p lz, f q 2L2 σ p q 2. As for he firs erm in 0, E p lz, f p E q lz, f p = lz, f p pz qzdz z = l, f p, p q. 2 he fac ha l, f p is a subdifferenial of Φ a p is proved in. We conclude ha he erms in 0 are he firs and he second order erms in he expansion of Φ. Lemma 6 he p-regre can be upper-bounded as R p E D ˆP, P where P = ˆP + p Z. Proof: Consider he following difference: δ : = E Φ p Z E Φ ˆP = E Φ p Z E Φ P + E Φ P E Φ ˆP We remark ha we can arrive a above resuls by explicily considering he dual funcion Φ, proving srong convexiy of Φ wih respec o which follows from our assumpion on l, and hen concluding srong flaness of Φ wih respec o. his is indeed he main inuiion a he hear of our proof. 5 Upper Bounds on R In his secion, we exhibi wo general upper bounds on R ha hold for a wide class of OCO games. he firs bound, which holds when he funcional Φ is differeniable and no oo curved, is of he form R = Olog. he second, which holds for arbirary Φ, e.g. where he funcional may even have a non-differeniabiliy, is saed in erms of he Rademacher complexiy of he class F. Such Rademacher complexiy resuls imply a regre upper bound on he order of. An inriguing observaion is ha hese bounds are proved wihou acually exhibiing a sraegy for he Player, as is ypically done. his illusraes he power of he minimax dualiy approach: we can prove he exisence of an opimal algorihm, and deermine is performance, all wihou providing is consrucion. 5. Fas Raes: Exploiing he Curvaure For differeniable Φ wih bounded second derivaive, we can prove ha he regre grows no faser han logarihmically in. Of course, raes of log have been given previously, 20, 2. We build upon hese resuls in he presen work by showing ha logarihmic regre mus always arise when Φ saisfies a flaness condiion. heorem 4 Suppose he Φ funcional is differeniable and α-fla wih respec he norm on P. hen R 4α log. We immediaely obain he following corollary. Corollary 5 Suppose funcions lz, f are σ-srongly convex and L Lipschiz in f. hen R 8L2 σ log. Furhermore, as we show in Secion 7.3, he log bound is igh for quadraic funcions; here is an explici join disribuion for he adversary which aains his value. he proof of heorem 4 involves he following lemma. For he firs difference we use concaviy of Φ. he second difference can be wrien as a divergence because he linear erm vanishes in expecaion. Indeed, E Φ P, Z p Z = 0 because he gradien does no depend on Z, while E Z Z Z = p Z. Hence, δ is no more han E Φ ˆP + E D ˆP, P and so R p = E Φ p Z E Φ ˆP = E Φ p Z + δ E Φ p Z E Φ ˆP + E D ˆP, P. Before proceeding, noe ha we may inerpre P as he condiional expecaion of he uniform disribuion ˆP given Z,..., Z. he flaness of Φ will allow us o show ha P deviaes very slighly from ˆP in expecaion indeed, by no more han O. his is crucial for obaining fas raes: for general Φ which 2 may be non-differeniable, i is naural o expec D ˆP, P = Ω/. In his case, he regre would be bounded by O / = O, rendering he above lemma useless. Proof:of heorem 4 We have ha he divergence erms in Lemma 6 are bounded as D ˆP, P α Z 2 p Z 4α because he norm beween disribuions is bounded by 4: Z z p z Z 2 dz 4. z

7 5.2 General Upper Bounds We sar wih he definiion of Rademacher averages, one of he cenral noions of complexiy of a funcion class. Definiion 7 Denoe by Rad lf := E ɛ ɛ lf, Z he daa-dependen Rademacher averages of he class lf. Here, ɛ... ɛ are independen Rademacher random variables uniform on {±}. We will omi he subscrip and dependence on Z, for he sake of simpliciy. In saisical learning heory, Rademacher averages ofen provide he ighes guaranees on he performance of empirical risk minimizaion and oher mehods. he nex resul shows ha he Rademacher averages play a key role in online convex opimizaion as well, as he minimax regre is upper bounded by he wors-case over he sample Rademacher averages. In he nex secion, we will also show lower bounds in erms of Rademacher averages for cerain linear games, showing ha his noion of complexiy is fundamenal for OCO. heorem 8 R 2 Rad Z Z lf. Proof: Le p be an arbirary join disribuion. Le ˆf be an empirical minimizer over Z, a sequence-dependen funcion. hen R p = E E Φp Z E p Z lz, ˆf Φ ˆP lz s, ˆf, s= as he paricular choice of ˆf is subopimal. Replacing he ˆf by he remum over F, R p E E = E E E p Z lz, ˆf lz, ˆf E p Z lz, f lz, f E p Z lz, f lz, f lz, f lz, f, where we renamed each dummy variable Z as Z. Even hough Z and Z have he same condiional expecaion, we canno generally exchange hem keeping he disribuion of he whole quaniy inac. Indeed, he condiional disribuions for τ > will depend on Z and no on Z. he rick is o exchange hem one by one 3, saring from = and going backwards, inroducing an addiional remum. One 3 We hank Ambuj ewari for poining ou a misake in our original proof. We refer o 9 for a similar analysis. can view he sequence {Z } as being angen o {Z } see 8. o his end, for any fixed ɛ {, +}, E = E lz, f lz, f lz, f lz, f + ɛ lz, f lz, f because for he las sep, indeed, Z and Z can be exchanged. Since his holds for any ɛ, we can ake i o be a Rademacher random variable. hus, E lz, f lz, f + ɛ lz, f lz, f = E ɛ E Z,Z E Z E ɛ lz, f lz, f + ɛ lz, f lz, f lz, f lz, f + ɛ lz, f lz, f. Repeaing he process, we have ha R p is bounded by E ɛ ɛ lz Z,Z, f lz, f 2 E ɛ Z ɛ lz, f = 2 RadlF. Z Properies of Rademacher averages are well-known. For insance, he Rademacher averages of a funcion class coincide wih hose of is convex hull. Furhermore, if l is Lipschiz, he complexiy of lf can be upper bounded by he complexiy of F, muliplied by he Lipschiz consan. For example, we can immediaely conclude ha if he loss funcion is Lipschiz and he funcion class is a convex hull of a finie number M of funcions, he minimax value of he game is bounded by R C log M for some consan C. Similarly, a class wih VC-dimension d would have log M replaced by d. heorem 8 is, herefore, giving us he flexibiliy o upper bound he minimax value of OCO for very general classes of funcions. Finally, we remark ha mos known upper bounds on Rademacher averages do no depend on he underlying disribuion, as hey hold for he wors-case empirical measure see 6, p. 27. hus, he remum over he sequences migh no be a hinderance o using known bounds for RadlF.

8 5.3 Linear Losses: Primal-Dual Ball Game Le us examine he linear loss more closely. Of paricular ineres are linear games when F = B is a ball in some norm and Z = B, he wo norms being dual. For his case, heorem 8 gives an upper bound of R 2 E ɛ Z f ɛ Z = 2 E ɛ Z ɛ Z. 3 Fix Z... Z and observe ha he expeced norm is a convex funcion of Z. Hence, he remum over Z is achieved a he boundary of Z. he same saemen holds for all Z s. Le z,..., z be he sequence achieving he remum. Now ake a disribuion for round o be p z = 2 z + z and le p = p... p be he produc disribuion. I is easy o see ha R 2 E ɛ Z = 2E p E ɛ ɛ Z = 2E ɛ ɛ z ɛ Z = 2 E p RadF. 4 Also noe ha p has zero mean. I will be shown in Secion 7. ha he lower bound arising from his disribuion is E p E ɛ ɛ Z = E p RadF, which is only a facor of 2 away. hus, he adversary can play a produc disribuion ha arises from he maximizaion in 3 and achieve regre a mos a facor 2 from he opimum. 6 Ω bounds for non-differeniable Φ In his secion, we develop lower bounds on he minimax value R based on he geomeric view-poin described in Secion 4.. heorem 4 shows ha he regre is upper bounded by log for he case of srongly convex losses, and his upper bound is igh if he loss funcions are quadraic, as we show laer in he paper. hus, flaness of Φ implies low regre. Wha abou he converse? I urns ou ha if Φ is nondiffereniable has a poin of infinie curvaure, he regre is lower-bounded by, and his rae is achieved wih p = p, where p corresponds o a poin of non-differeniabiliy of Φ. he geomeric viewpoin is fruiful here: verices poins of non-differeniabiliy of Φ correspond o exposed faces in he loss class S = co lf, suggesing ha he lower bounds of Ω arise from having wo disinc minimizers of expeced error a sriking parallel o he analogous resuls for sochasic seings 5, 7. o be more precise, verices of σ S and Φ ranslae ino fla pars non-singleon exposed faces of Sco lf and he oher way around. Corresponding o an exposed face is a poring hyperplane. If lf is non-negaive, hen any exposed face facing he origin is pored by a hyperplane wih posiive co-ordinaes which can be normalized o ge a disribuion. So a non-singleon face exposed o he origin is equivalen o having a leas wo disinc minimizers f and g of E p lz, for some p, as discussed in Secion 4.2. Define he se of expeced minimizers under p as F := {f F : E p lz, f = inf E plz, f}. hus, lf F S p co lf. he lower bound we are abou o sae arises from flucuaions of he empirical process over he se F. o ease he presenaion, we will refer o he sample average lz, f as Ê lz, f. heorem 9 Suppose F S p is a non-singleon face of co lf, pored by p i.e. F >. Fix any f F and le Q lf be any subse conaining l, f. Define Q = {g l, f : g Q}, he shifed loss class. hen for > 0 F, R R p = E c Q lf E p lz, f Ê lz, f E G q, q Q where G q is he Gaussian process indexed by he cenered funcions in Q, and c is some absolue consan. Proof: Recalling ha E lz, f = inf g F E lz, g = Φp for all f F, we have R R p = Φp E p Φ ˆP = Φp E inf Φp E lz, f inf Ê lz, f = E E p lz, f Ê lz, f E Q lf E p lz, f Ê lz, f f:l f Q Now, fix any f F. he proof of heorem 2.2 in 4 reveals ha empirical flucuaions are lower bounded by he remum of he Gaussian process indexed by Q. o be precise, here exiss 0 F such ha for > 0 F wih probabiliy greaer han c, inf Ê lz, f lz, f E q Q G q c 2, f:l f Q for some absolue consans c, c 2. Rearranging and using he fac ha E lz, f E lz, f = 0 for f F, E lz, f Ê lz, f + Ê lz, f E lz, f f:l f Q c 2 E q Q G q

9 wih probabiliy a leas c. he remum is non-negaive because f Q and herefore E E lz, f Ê lz, f E q Q G q c c 2. f:l f Q Consider anoher example of an i.i.d. disribuion ha pus equal mass on wo poins ±z 0 on each round, wih z 0 =. I hen follows ha his i.i.d. disribuion achieves he regre equal o he lengh of he random walk E ɛ, which is known o be asympoically 2/π. We remark ha in he expers case, he lower bound on regre becomes log N, as he Gaussian process reduces o N independen Gaussian random variables. We discuss his and oher examples in he nex secion. 7 Lower Bounds for Special Cases We now provide lower bounds for paricular games. Some of he resuls of he secion are known: we show how he proofs follow from he general lower bounds developed in he previous secion. 7. Linear Loss: Primal-Dual Ball Game Here, we develop lower bounds for he case considered in Secion 5.3. As before, o prove a lower bound i is enough o ake an i.i.d. or produc disribuion. In paricular, he produc disribuion described afer Eq. 3 is of paricular ineres. o his end, choose p = p... p o be a produc of symmeric disribuions on he surface of he primal ball Z wih E p Z = 0. We conclude ha Φp = 0 and R is greaer han E Φ ˆP = E inf f Z = E Z by he definiion of dual norm. Now, because of symmery, E Z = E E ɛ ɛ Z. 5 We conclude ha R E RadF, he expeced Rademacher averages of he dual ball acing on he primal ball. his is wihin a facor of 2 of he upper bound 4 of Secion 5.3. Now, consider he paricular case of F = Z = B 2, he Euclidean ball. We will consider hree disribuions p. Suppose p is such ha p Z pus mass on he inersecion of B 2 and he subspace perpendicular o s= Z s and E Z Z = 0. hen E Z = by unraveling he sum from he end. In fac, his is shown o be he opimal value for his problem in 2. We conclude ha a non-produc disribuion achieves he opimal regre for his problem. Consider any symmeric i.i.d. disribuion on he surface of he ball Z. Noe ha for his case we sill have he lower bound of Eq. 5. Kinchine-Kahane inequaliy hen implies R 2 and he consan 2 is opimal see 3 in he absence of furher assumpions. We conclude ha for he Euclidean game, he bes sraegy of he adversary is a sequence of dependen disribuions, while produc and i.i.d. disribuions come wihin a muliplicaive consan close o from i. 7.2 Expers Seing he expers seing provides some of he easies examples for linear games. We sar wih a simplified game, where F = Z = N, he N-simplex. he Φ funcion for his case is easy o visualize. We hen presen he usual game, where he se Z = 0, N. In boh cases, we are ineresed in lower-bounding regre he simplified game Le us look a he game when only one exper can suffer a loss of per round, i.e. he space of acions Z conains N elemens e,..., e N. he probabiliy over hese choices of he adversary is an N-dimensional simplex, jus as he space of funcions F. For any p N, Φp = min E p Z f = min p f = min p i f N f i N and herefore he Φ has he shape of a pyramid wih is maximum a p = N and Φp = /N. he regre is lowerbounded by an i.i.d game wih his disribuion p a each round, i.e. R Φp E ΦU = N E min Z f = E max i N f N N n i where n i is he number of imes e i has been chosen ou of rounds. his is he expeced maximum deviaion from he mean of a mulinomial disribuion, i.e. /N minus he smalles proporion of balls in any bin afer balls have been disribued uniformly a random. o obain he lower bound on he maximum deviaion, le us urn o Secion 6. he convex hull of he negaive loss class co lf is he simplex iself. his is also he face pored by he uniform disribuion p. he lower bound of heorem 9 involves he Gaussian process indexed by a se Q. Le us ake f = N and F = {e,..., e N } {f }. We can verify ha E e i Z = Φp = N, he covariance of he process indexed by Q = lf is E e i Z N e j Z N = N for i j and he variance is E e 2 i Z N 2 = N N. Le {Y 2 i } N be he Gaussian random variables wih he aforemenioned covariance srucure. hen Y i Y j 2 = E Y i Y j 2 = 2 N. We can now consruc independen Gaussian random variables {X i } N wih he same disance by puing 2 N on he diagonal of he covariance marix. By,

10 Slepian s Lemma, 2 E i X i E i Y i, hus giving us he lower bound log N R c N for his problem, for some absolue consan c and large enough he general case In he more general game, any exper can suffer a 0/ loss. hus, p is a disribuion on 2 N losses Z. o lower bound he regre, choose a uniform disribuion on 2 N binary vecors as he i.i.d. choice for he adversary. We have Φ = 2 N min f N f E Z = /2. As for he oher erm, E Φ ˆP = E min f N f Z. hus, he regre is 2 ɛ i,, R E max i N where ɛ i, are Rademacher {±}-valued random variables. I is easy o show ha he expeced maximum is lower bounded by c log N/. his coincides wih a resul in 7, which shows ha he asympoic behavior is log N/ Quadraic Loss We consider he quadraic loss, lz, f = f z 2. his loss funcion is -srongly convex, and herefore we already have he Olog bound of Corollary 5. In his secion, we presen an almos maching lower bound using a paricular adversarial sraegy. he problem of quadraic loss was previously addressed in 20; we reprove heir lower bound in our framework, borrowing a number of ricks from ha work. Following Secion 6, i is emping o use an i.i.d. disribuion and compue he regre explicily. Unforunaely, his only leads o a consan lower bound, whereas we would hope o mach he upper bound of log. We can show his easily: le p := p be some i.i.d. disribuion, hen E Φ ˆP = E Z 2 E Z, Z 2 = E Z 2 E Z 2 = varz = Φp. hus Rp = Φp E Φ ˆP = Φp, where we see ha he las erm is independen of. Indeed, obaining log regre requires ha we look furher han i.i.d. o his end, define c := and c := c +c 2 for =,,..., 2. We consruc our disribuion p using his sequence as follows. Assume Z = F =, and for convenience le Z :s := s Z. Also, for his secion, we use a shorhand for he condiional expecaion, E := E Z,..., Z. Each condiional disribuion is chosen as { +cz : 2, for z = p Z = z Z,..., Z := c Z : 2, for z =. Noice ha his choice ensures ha E Z = c Z :, i.e. he condiional expecaion is idenical o he observed sample mean scaled by some shrinkage facor c. ha +cz: 2 0, follows from he saemen c which is proven by an easy inducion. We now recall a resul shown in 20: c = log log log + o. his crucial lemma leads direcly o he main resul of his secion. heorem 20 Wih p defined above, R p = c and herefore R p = log log log + o. he proof follows closely along he lines of 20 and can be found in he full version. Acknowledgemens We graefully acknowledge he por of he NSF under awards DMS and DMS and of DARPA under award FA Jacob is pored in par by a Yahoo! PhD Fellowship. References J. Abernehy, A. Agarwal, P. L. Barle, and A. Rakhlin. A sochasic view of opimal regre hrough minimax dualiy. CoRR, abs/ , hp://arxiv.org/abs/ J. Abernehy, P. L. Barle, A. Rakhlin, and A. ewari. Opimal sraegies and minimax lower bounds for online convex games. In COL, P. L. Barle, E. Hazan, and A. Rakhlin. Adapive online gradien descen. In NIPS, P. L. Barle, M. I. Jordan, and J. D. McAuliffe. Convexiy, classificaion, and risk bounds. Journal of he American Saisical Associaion, 0473:38 56, J. M. Borwein and A. S. Lewis. Convex Analysis and Nonlinear Opimizaion. Advanced Books in Mahemaics. Canadian Mahemaical Sociey, o appear. 6 N. Cesa-Bianchi and G. Lugosi. On predicion of individual sequences. Ann. Sa., 276: , N. Cesa-Bianchi and G. Lugosi. Predicion, Learning, and Games. Cambr. Univ. Press, V. de la Peña and E. Gine. Decoupling: From Dependence o Independence. Springer, Y. Freund. Predicing a binary sequence almos as well as he opimal biased coin. Inf. Compu., 822:73 94, B. A. Frigyik, S. Srivasava, and M. R. Gupa. Funcional bregman divergence and bayesian esimaion of disribuions. CoRR, abs/cs/0623, E. Hazan, A. Kalai, S. Kale, and A. Agarwal. Logarihmic regre algorihms for online convex opimizaion. In COL, pages , J.-B. Hiriar-Urruy and C. Lemaréchal. Fundamenals of Convex Analysis. Springer, R. Laala and K. Oleszkiewicz. On he bes consan in he Khinchin-Kahane inequaliy. Sudia Mah., 09:0 04, G. Lecué and S. Mendelson. Sharper lower bounds on he performance of he empirical risk minimizaion algorihm. Available a hp:// lecue/lm2.pdf. 5 W. S. Lee, P. L. Barle, and R. C. Williamson. he imporance of convexiy in learning wih squared loss. IEEE ransacions on Informaion heory, 445: , S. Mendelson. A few noes on saisical learning heory. In S. Mendelson and A. J. Smola, ediors, Advanced Lecures in Machine Learning, LNCS 2600, Machine Learning Summer School 2002, Canberra, Ausralia, February -22, pages 40. Springer, S. Mendelson. Lower bounds for he empirical minimizaion algorihm. IEEE ransacions on Informaion heory, o appear. 8 N. Merhav and M. Feder. Universal predicion. IEEE ransacions on Informaion heory, 44: , K. Sridharan and A. ewari. Convex games in banach spaces working ile, Unpublished. 20 E. akimoo and M. Warmuh. he minimax sraegy for gaussian densiy esimaion. In COL, pages Morgan Kaufmann, San Francisco, V. Vovk. Compeiive on-line linear regression. In NIPS 97: Proceedings of he 997 conference on Advances in neural informaion processing sysems 0, pages , Cambridge, MA, USA, 998. MI Press. 22 M. Zinkevich. Online convex programming and generalized infiniesimal gradien ascen. In ICML, pages , 2003.