Online Nonparametric Regression

Transcription

1 Olie Noparameric Regressio Alexader Rakhli Uiversiy of Pesylvaia Karhik Sridhara Uiversiy of Pesylvaia February 0, 204 Absrac We esablish opimal raes for olie regressio for arbirary classes of regressio fucios i erms of he sequeial eropy iroduced i 4. The opimal raes are show o exhibi a phase rasiio aalogous o he i.i.d./saisical learig case, sudied i 6. I he frequely ecouered siuaio whe sequeial eropy ad i.i.d. empirical eropy mach, our resuls poi o he ieresig pheomeo ha he raes for saisical learig wih squared loss ad olie oparameric regressio are he same. I addiio o a o-algorihmic sudy of miimax regre, we exhibi a geeric forecaser ha ejoys he esablished opimal raes. We also provide a recipe for desigig olie regressio algorihms ha ca be compuaioally efficie. We illusrae he echiques by derivig exisig ad ew forecasers for he problems of olie regressio wih fiie expers ad olie liear regressio. Iroducio Wihi he olie regressio framework, daa x, y,...,x, y,... arrive i a sream, ad we are asked wih sequeially predicig each ex respose y give he curre x ad he daa x i, y i observed hus far. Le i= ŷ deoe our predicio, ad le he qualiy of his forecas be evaluaed via square loss ŷ y 2. Wihi he field of ime series aalysis, i is assumed ha daa are geeraed accordig o some model. The parameers of he model ca he be esimaed from daa, leveragig he laws of probabiliy. Aleraively, i he compeiive approach, sudied wihi he field of olie learig, he aim is o develop a predicio mehod ha does o assume a geeraive process of he daa 7. The problem is he formulaed as ha of miimizig regre ŷ y 2 if f x y 2 wih respec o some bechmark class of fucios F. This class ecodes our prior belief abou he family of regressio fucios ha we expec o perform well o he sequece. Noably, a upper boud o regre is required o hold for all sequeces. I he pas wey years, progress i olie regressio for arbirary sequeces, sarig wih he paper of D. Foser 8, has bee almos exclusively o fiie dimesioal liear regressio a icomplee lis icludes 9,, 20, 4, 2, 3, 9. This is o be corased wih Saisics, where regressio has bee sudied for rich oparameric classes of fucios. Impora excepios o his limiaio i he olie regressio framework ad works ha parly moivaed he prese paper are he papers of Vovk 23, 2, 22. Vovk cosiders regressio wih large classes, such as subses of a Besov or Sobolev space, ad remarks ha here appears o be wo disic approaches o obaiig he upper bouds i olie compeiive regressio. The firs approach, which Vovk erms Defesive Forecasig, explois uiform covexiy of he space, while he secod a aggregaig echique such as he Expoeial Weighs Algorihm is based o he meric eropy of he space. Ieresigly, he wo seemigly differe approaches yield disic upper bouds, based o he respecive properies of he space. I paricular, Vovk asks wheher here is a uified view of hese echiques. The prese paper addresses hese quesios ad esablishes opimal performace for olie regressio. Sice mos work i olie learig is algorihmic, he boudaries of wha ca be proved are defied by he regre miimizaio algorihms oe ca fid. Oe of he mai algorihmic workhorses is he aggregaig procedure

2 meioed above. However, he difficuly i usig a aggregaig procedure beyod simple parameric classes e.g. subses of R d lies i he eed for a poiwise cover of he se of fucios. The same difficuly arises whe oe uses PAC-Bayesia bouds ha require a volumeric argume. Noably, his difficuly has bee overcome i saisical learig, where i has log bee recogized sice he work of Vapik ad Chervoekis ha i is sufficie o cosider a empirical cover of he class a poeially much smaller quaiy. Such empirical eropy is ecessarily fiie, ad is growh wih is oe of he key complexiy measures for i.i.d. learig. I paricular, he rece work of 6 shows ha he behavior of empirical eropy characerizes he opimal raes for i.i.d. learig wih square loss. To mimic his developme, i appears ha we eed o udersad empirical coverig umbers i he sequeial predicio framework. Sequeial aalogues of coverig umbers, combiaorial parameers, ad he Rademacher complexiy have bee recely iroduced i 5. These complexiy measures were show o boh upper ad lower boud miimax regre of olie learig wih absolue loss for arbirary classes of fucios. These raes, however, are o correc for he square loss case. Cosider, for isace, fiie-dimesioal regressio, where he behavior of miimax regre is kow o be logarihmic i ; he Rademacher rae, however, cao yield raes faser ha. A hi as o how o modify he aalysis for curved losses appears i he paper of 6 where he auhors derived raes for log-loss via a wo-level procedure: he se of desiies is firs pariioed io small balls of a criical radius γ; a miimax algorihm is employed o each of hese small balls; ad a overarchig aggregaig procedure combies hese algorihms. Regre wihi each small ball is upper bouded by classical Dudley eropy iegral wih respec o a poiwise meric defied up o he γ radius. The mai echical difficuly i his paper is o prove a similar saeme usig empirical sequeial coverig umbers. I paricular, our resuls imply he same phase rasiio as he oe exhibied i 5 for i.i.d. learig wih square loss. More precisely, uder he assumpio of he Oβ p behavior of sequeial eropy, he miimax regre ormalized by ime horizo decays as 2 2+p if p 0,2, ad as /p for p 2. We prove lower bouds ha mach up o a logarihmic facor, esablishig ha he phase rasiio is real. Eve more surprisigly, i follows ha, uder a mild assumpio ha sequeial Rademacher complexiy of F behaves similarly o is i.i.d. cousi, he raes of miimax regre i olie regressio wih arbirary sequeces mach, up o a logarihmic facor, hose i he i.i.d. seig of Saisical Learig. This pheomeo has bee oiced for some parameric classes by various auhors e.g. 5. The pheomeo is eve more srikig give he simple fac ha oe may cover he regre saeme, ha holds for all sequeces, io a i.i.d. guaraee. Thus, i paricular, we recover he resul of 6 hrough compleely differe echiques. Sice i may siuaios, oe obais opimal raes for i.i.d. learig from a regre saeme, he relaxaio framework of 3 provides a oolki for developig improper learig algorihms i he i.i.d. sceario. Afer characerizig miimax raes for olie regressio, we ur o he quesio of developig algorihms. We firs show ha a algorihm based o he Rademacher relaxaio is admissible see 3 ad yields he raes derived i a o-cosrucive maer i he firs par of he paper. This algorihm is o geerally compuaioally feasible, bu, i paricular, does achieve boh raes exhibied by Vovk 2 for Besov spaces. We show ha furher relaxaios i fiie dimesioal space lead o he famous Vovk-Azoury-Warmuh forecaser. For illusraio purposes, we also derive a predicio mehod for fiie class F. 2 Backgroud Le X be some se of covariaes, ad le F be a class of fucios X, = Y. We sudy he olie regressio sceario where o roud,...,, x X is revealed o he learer who subsequely makes a predicio ŷ R; Naure he reveals 2 y B,B. Isead of, we cosider a slighly modified oio of regre α ŷ y 2 if f x y 2 2 While we develop our resuls for square loss, similar saemes hold for much more geeral losses, as will be show i he full versio of his paper. 2 The assumpio of bouded resposes ca be removed by sadard rucaio argumes see e.g. 0. 2

3 for some α 0,. I is well-kow ha a upper boud o such a regre oio leads o he so-called opimisic raes which scale favorably wih he cumulaive loss L = if f x y 2 2, 8. More precisely, pose we show a upper boud of U /α +U 2 o regre i 2. The regre i is upper bouded by 4 L U + 2U + 4U 2 3 by cosiderig he case L 4U ad is coverse. Ulike mos previous approaches o he sudy of olie regressio, we do o sar from a algorihm, bu isead direcly work wih miimax regre. We will be able o exrac a o ecessarily efficie algorihm afer geig a hadle o he miimax value. Le us iroduce he oaio ha makes he miimax regre defiiio more cocise. We use o deoe a ierleaved applicaio of he operaors iside repeaed over =... rouds. Wih his oaio, he miimax regre of he olie regressio problem described earlier ca be wrie as V α = x if ŷ y α ŷ y 2 if f x y 2 where each x rages over X ad ŷ, y rage over B,B. The usual miimax regre oio is simply give whe α = 0 as V 0. As meioed above, i he i.i.d. sceario i is possible o employ a oio of a cover based o a sample, haks o he symmerizaio echique. I he olie predicio sceario, symmerizaio is more suble, ad ivolves he oio of a biary ree, he smalles eiy ha capures he sequeial aure of he problem. To his ed, le us sae a few defiiios. A Z -valued ree z of deph is a complee rooed biary ree wih odes labeled by elemes of Z. Equivalely, we hik of z as labelig fucios, where z is a cosa label for he roo, z 2, z 2 + Z are he labels for he lef ad righ childre of he roo, ad so forh. Hece, for ɛ = ɛ,...,ɛ ±, z ɛ = z ɛ,...,ɛ Z is he label of he ode o he -h level of he ree obaied by followig he pah ɛ. For a fucio g : Z R, g z is a R-valued ree wih labelig fucios g z for level or, i plai words, evaluaio of g o z. Nex, le us defie sequeial coverig umbers oe of he key complexiy measures of F. Defiiio 5. A se V of R-valued rees of deph forms a β-cover wih respec o he l q orm of a fucio class F R X o a give X -valued ree x of deph if 4 f F, ɛ ±, v V s.. f x ɛ v ɛ q β q. A β-cover i he l sese requires ha f x ɛ v ɛ β for all. The size of he smalles β-cover is deoed by N q β,f, x, ad N q β,f, = x logn q β,f, x. We will refer o x logn q β,f, x as sequeial eropy of F. I paricular, we will sudy he behavior of V α F whe sequeial eropy grows polyomially3 as he scale β decreases: logn 2 β,f, = β p, p > 0. 5 We also cosider he parameric p = 0 case whe sequeial coverig iself behaves as N 2 β,f, = β d 6 e.g. liear regressio i a bouded se i R d. We remark ha he l cover is ecessarily -depede, so he form we assume here is N β,f, = /β d. 7 3 I is sraighforward o allow cosas i his defiiio, ad we leave hese deails ou for he sake of simpliciy. 3

4 3 Mai Resuls We ow sae he mai resuls of his paper. They follow from he more geeral echical saemes of Lemmas 4, 5, 6 ad 7. We ormalize V α by i order o make he raes comparable o hose i saisical learig. Furher, hroughou he paper C,c refer o cosas ha may deped o B, p. Their values ca be foud i he proofs. Theorem. For a class F wih sequeial eropy growh logn 2 β,f, β p, For p > 2, he miimax regre 4 is bouded as For p 0,2, he miimax regre is bouded as For he parameric case 6, For fiie se F, V 0 C log F V 0 C d log Theorem 2. The upper bouds of Theorem are igh 5 : V 0 C /p V 0 C 2/2+p For p 2, for ay class F of uiformly bouded fucios wih a lower boud of β p o sequeial eropy growh, V 0 Ω /p For p 0,2, for ay class F of uiformly bouded fucios, here exiss a slighly modified class F wih he same sequeial eropy growh such ha V 0 Ω 2/2+p There exiss a class F wih he coverig umber as i 6, such ha V 0 Ωd log For he followig heorem, we assume ha L is kow a priori. Adapiviy o L ca be obaied hrough a doublig-ype argume 7. Theorem 3. Addiioally, he followig opimisic raes hold for regre : For p > 2, regre is upper bouded by C L /p log +C /p log For p 0,2, regre is upper bouded by C L log + C log. The boud gais a exra log facor for p = 2 For he parameric case 7, regre is upper bouded by C L d log +C d log where L = if f x y 2. Remark. The opimisic rae for p > 2 appears o be slower ha he hypohesized L 2/p + 2/p rae, ad we leave he quesio of obaiig his rae as fuure work. Remark 2. If we assume ha y s are draw from disribuios wih bouded mea ad subgaussia ails, he same upper bouds ca be show wih a exra log facor. Nex, we prove he hree heorems saed above. The proofs are of he plug-ad-play syle : he overarchig idea is ha he opimal raes ca be derived simply by assumig a appropriae corol of sequeial eropy, be i a parameric or a oparameric class. Proof of Theorem. We appeal o Eq. 3 i Lemma 4 below. Fix ad le z deoe he X R-valued ree. Defie he class G = g f : g f z = f x µ, f F. Observe ha he values of g f ouside of rage of z are 4 For p = 2, V 0 C log /2. 5 The Ω oaio presses logarihmic facors 4

5 immaerial. Also oe ha he coverig umber of G o z coicides wih he coverig umber of F o x. Now, Lemma 5 applied o his class G, ogeher wih η B, yields V 0 32B 2 logn 2 γ,f, + B if 4ρ + 2 γ logn 2 δ,f,dδ 8 ρ 0,γ We ow evaluae he above upper boud for he β p growh of sequeial eropy a scale β. I paricular, for he case p > 2, we may choose γ = maximum of he fucio ad ρ = /p. The N 2 B,F, = ad he firs erm disappears. We are lef wih B V 0 4 p B δ 2 p/2 4 p p 2 p /p p 2 2p + 2 = /p p 2 For he case p 0,2, Eq. 8 gives a upper boud We choose γ = /p+2 ad ρ = : 32B 2 γ p + B if ρ 0,γ 4ρ + 2 γ ρ ρ δ p/2 dδ 32B 2 p p+2 + 4B δ 2 p p+2 2 4B + 32B B 2 p 2 p For he case p = 2, we gai a exra facor of log sice he iegral of δ is he logarihm. For he parameric case 6, we choose γ = /2 ad ρ =. The Eq. 8 yields for > 8, V 0 6B 2 d log + 4B + 2 /2 p p+2 d log/δdδ 6B 2 d log + 4B + 2 d log. I he fiie case, logn 2 γ,f, log F for ay γ. We he have ake γ = 0 oe ca see ha his value is allowed for he paricular case of a fiie class; or, use a small eough value. The, V 0 32B 2 log F. Normalizig by yields he desired raes i he saeme of he heorem. Proof of Theorem 2. The firs wo lower bouds are proved i Lemma 9 ad 0. The lower boud for he parameric case follows from he i.i.d. lower boud i 6. Proof of Theorem 3. For opimisic raes, we sar wih he upper boud i 2 ad defie G as above. We he appeal o Lemma 6 ad obai γ V α α 6logN γ,f, z + α if 4ρ + 6logγ/ρ δlogn δ,f, zdδ. 0 ρ 0,γ For logn β,f, β p decay of eropy for p < 2, we ake ρ = B, γ =. The firs erm i 0 ca be ake o be zero, as we may ake oe fucio a scale γ =. The ifimum i 0 evaluaes o 4 + 6logB δ p dδ 4 + 6logB 2 p δ2 p 4 + 6logB 2 p. /B ρ /B For p = 2, we gai a exra log facor: 4 + 6logB 2. For p > 2, we ake ρ = p ad γ =. The ifimum i 0 evaluaes o 4 p + 6p log 2 p δ2 p 4 p 2 p p + 6p log 2 p p 2 For he parameric case 7, we ake γ = ad ρ = B. The 0 is upper bouded by 4 + 6logB /B dδlog/δdδ 4 + 4d logb. The fial opimisic raes are obaied by followig he boud i 3. p. 9 5

6 3. Offse Rademacher Complexiy ad he Chaiig Techique Le us recall he defiiio of sequeial Rademacher complexiy of a class F E ɛ f x ɛ x iroduced i 4, where he expecaio is over a sequece of idepede Rademacher radom variables ɛ = ɛ,...,ɛ ad he remum is over all X -valued rees of deph. While his complexiy boh upper- ad lowerbouds miimax regre for absolue loss, i fails o capure he possibly faser raes oe ca obai for regressio. We show below ha modified, or offse, versios of his complexiy do i fac give opimal raes. These complexiies have a exra quadraic erm beig subraced off. Iuiively, his variace erm exiguishes he -ype flucuaios above a cerai scale. Below his scale, complexiy is give by he Dudley-ype iegral. The opimal balace of he scale gives he correc raes. As ca be see from he proof of Theorem, he criical scale γ is rivial zero for a fiie case, he /2 for a parameric class, /p+2 for p 0,2, ad he becomes irreleva e.g. cosa a p > 2. Ideed, for p > 2, he rae is give purely by sequeial Rademacher complexiy, as curvaure of he loss does o help. I paricular, ca achieve hese raes for p > 2 by simply liearizig he square loss. The same pheomeo occurs i saisical learig wih i.i.d. daa 6. We remark ha 2 sudies bouds for esimaio wih squared loss for he empirical risk miimizaio procedure ad observes ha i is eough o oly cosider oe-sided esimaes raher ha coceraio saemes. The offse sequeial Rademacher complexiies are of his oe-sided aure. I Lemma 4 below, we provide a boud o miimax regre via offse sequeial Rademacher complexiies. Lemma 4. The miimax value V α of olie regressio wih resposes y i a bouded ierval B,B is upper bouded by V α E ɛ 4ɛ η ɛf x ɛ µ ɛ f x ɛ µ ɛ 2 αη ɛ 2,η 2 ad V 0 E ɛ 4Bɛ f x ɛ µ ɛ f x ɛ µ ɛ 2 3 where x rages over all X -valued rees, µ ad η over all B,B-valued rees of deph. Furhermore, V 0 E Bɛ f x ɛ µ ɛ f x ɛ µ ɛ 2 4 where µ rages over B/2,B/2-valued rees. We ow show ha offse Rademacher complexiies ca be upper bouded by sequeial eropies via he chaiig echique. Lemma 5 below is a aalogue of he Dudley-ype iegral boud E ɛ g x ɛ x g G if ρ 0, 4ρ + 2 ρ logn 2 δ,g, zdδ for sequeial Rademacher proved i 5. Crucially, he upper boud of Lemma 5 allows us o choose a criical scale γ. Lemma 5. Le η be a B,B-valued ree of deph. For ay Z -valued ree z ad a class G of fucios Z A, A ad ay γ 0, A, E 4ɛ η ɛg z ɛ g z ɛ 2 32B 2 logn 2 γ,g, z + B if 4ρ + 2 g G ρ 0,γ γ ρ logn 2 δ,g, zdδ 5 6

7 For opimisic raes, we ca ake advaage of a addiioal offse. This offse arises from he quadraic erm due o he α muliple of he loss of he algorihm. Lemma 6. Le η be a B,B-valued ree of deph. For ay Z -valued ree z ad a class G of fucios Z A, A, for ay γ 0, A, E 4ɛ η ɛg z ɛ g z ɛ 2 αη ɛ 2 α 6A 2 logn γ,g, z 6 g G + α if 4ρ + 6logγ/ρ ρ 0,γ γ ρ δlogn δ,g, zdδ The chaiig argumes of Lemmas 5 ad 6 are based o he followig key fiie-class lemma: Lemma 7. Le η be a B,B-valued ree of deph. For a fiie se W of A, A-valued rees of deph, i holds ha Emax ɛ η ɛw ɛ C w ɛ 2 αη ɛ 2 mi B 2 2C, A 2 2α log W 7 w W for ay C 0, α 0. I also holds ha Emax ɛ η ɛw ɛ w W B 2log W max w W,ɛ : w ɛ 2. 8 Remark 3. Le us compare he upper boud of Lemma 5 o he boud we may obai via a meric eropy approach, as i he work of Vovk 2. Assume ha F is a compac subse of C X equipped wih remum orm. The meric eropy, deoed by H ɛ,f, is he logarihm of he smalles ɛ-e wih respec o he orm o X. A aggregaig procedure over he elemes of he e gives a upper boud omiig cosas ad logarihmic facors ɛ + H ɛ,f 9 o regre. Here, ɛ is he amou we lose from resricig he aeio o he ɛ-e, ad he secod erm appears from aggregaio over a fiie se. While he balace 9 ca yield correc raes for small classes, i fails o capure he opimal behavior for large oparameric ses of fucios. Ideed, for a ɛ p behavior of meric eropy, Vovk cocludes he rae of O p p+. For p 2, his is slower ha he O p p+2 rae oe obais from Lemma 5 by rivially upper boudig he sequeial eropy by meric eropy. The gai is due o he chaiig echique, a pheomeo well-kow i saisical learig heory. Our coribuio is o iroduce he same coceps o he domai of olie learig. Le us also meio ha sequeial coverig umber of F is a empirical quaiy ad is fiie eve if we cao upper boud meric eropy. 4 Furher Examples For he sake of illusraio we show bouds o miimax raes for a couple of examples. Example Sparse liear predicors. Le G = g,..., g M be a se of M fucios such ha each g i : X,. Defie F o be he covex combiaio of a mos s ou of hese M fucios. Tha is s s F = α j g σj : σ :s M, j,α j 0, α j = For his example oe ha he sequeial coverig umber ca be easily upper bouded: we ca choose s ou of M fucios i M s ways ad furher he l meric eropy for covex combiaio of s bouded fucios a scale β is bouded as β s. We coclude ha em s N 2 β,f, β s s 7

8 From he mai heorem, he upper boud is s logm/s V 0 O Example 2 Besov Spaces. Le X be a compac subse of R d. Le F be a ball i Besov space Bp,q s X. Whe s > d/p, poiwise meric eropy bouds a scale β scale as Ωβ d/s 2, p. 20. O he oher had, whe s d/p,, oe ca show ha he space is a Baach space ha is p-uiformly covex. From 5, i ca be show ha sequeial Rademacher ca be upper bouded by O /p, yieldig a boud o sequeial eropy a scale β as Oβ p. These wo corols ogeher give he boud o he miimax rae. The geeric forecaser wih Rademacher complexiy as relaxaio see Secio 6, ejoys he bes of boh of hese raes. More specifically, we may ideify he followig regimes: If s d/2, he miimax rae is O 2s 2s+d. If s < d/2, he miimax rae depeds o he ieracio of p ad d, s: if p > + d 2s, he miimax rae is O 2s 2s+d, as above. oherwise, he miimax rae is O p 5 Lower Bouds The lower bouds will ivolve a oio of a dimesio of F called he sequeial fa-shaerig dimesio. Le us iroduce his oio. Defiiio 2. A X -valued ree of deph d is said o be β-shaered by F if here exiss a R-valued ree s of deph d such ha ɛ ± d, f ɛ F s.. ɛ f ɛ x ɛ s ɛ β/2 for all,...,d. The ree s is called a wiess. The larges d for which here exiss a β-shaered X -valued ree is called he sequeial fa-shaerig dimesio, deoed by fa β F. The sequeial fa-shaerig dimesio is relaed o sequeial coverig umbers as follows: Theorem 8 5. Le F be a class of fucios X,. For ay β > 0, Therefore, if logn 2 β,f, c/β p, he 2e N 2 β,f, N β,f, β fa β F c/β p /log2e/β. faβ F The lower bouds will ow be obaied assumig fa β F C /β p behavior of he fa-shaerig dimesio, ad he resulig saeme of Theorem 2 i erms of he sequeial eropy growh will ivolve exra logarihmic facors, hidde i he Ω oaio. Lemma 9. Cosider he problem of olie regressio wih resposes bouded by B = 4. For ay class F of fucios X, ad ay β > 0 ad = fa β F, V 0 β I paricular, if fa β F C /β p for p > 0, we have V 0 C /p.. 8

9 Lemma 0. For ay class F ad β > 0, here exiss a modified class F such ha fa β F 2fa β F + 4 ad for > fa β F, V 0 C 2 fa β F 2β β 2. I paricular, whe p 0,2 ad fa β F = C /β p, 6 Relaxaios ad Algorihms V 0 C To desig geeric forecasers for he problem of olie o-parameric regressio we follow he recipe provided i 3. I was show i ha paper ha if oe ca fid a relaxaio Rel a sequece of mappigs from observed daa o reals ha saisfies iiial ad admissibiliy codiios he oe ca build esimaors based o such relaxaios. Specifically, we look for relaxaios ha saisfy he followig iiial codiio Rel x:, y : if 2 p+2. f x y 2 ad he recursive admissibiliy codiio ha for ay ad ay x X if ŷ y 2 + Rel x:, y : Rel x:, y : ŷ B,B y B,B 20 If a relaxaio Rel saisfies hese wo codiios he oe ca defie a algorihm via ŷ = argmi ŷ y 2 + Rel x:, y : ŷ B,B y B,B ad for his forecas he associaed boud o regre is auomaically bouded as see 3 for deails : Reg Rel Now furher oe ha if ŷ y 2 + Rel x:,y :, y is a covex fucio of y he he predicio akes a very simple form, as he remum over y is aaied eiher a B or B. The predicio ca be wrie as ŷ = argmi max ŷ B 2 + Rel x:,y :,B,ŷ + B 2 + Rel x:,y :, B ŷ B,B Observe ha he firs erm decreases as ŷ icreases o B ad likewise he secod erm moooically decreases as ŷ decreases o B. Hece he soluio o he above is give whe boh erms are equal if his does happe wihi he rage B,B he we clip. I oher words, Rel x:,y :,B Rel x:,y :, B ŷ = Clip 4B Hece, for ay admissible relaxaio such ha ŷ y 2 +Rel x:,y :, y is a covex fucio of y, he above predicio based o he relaxaio ejoys he boud o regre Rel. We ow claim ha he followig codiioal versio of Equaio 3 gives a admissible relaxaio ad leads o a mehod ha ejoys he regre bouds show i he firs par of his paper. Lemma. The followig relaxaio is admissible : R x :, y : = E ɛ 4Bɛ j f x j ɛ µ j ɛ f x j ɛ µ j ɛ 2 f x j y j 2 j =+ 9

10 The forecas correspodig o his relaxaio is give by ŷ = R x :,y :,B R x :,y :, B 4B The above algorihm ejoys he regre boud of a offse Rademacher complexiy: Reg E ɛ 4Bɛ f x ɛ µ ɛ f x ɛ µ ɛ 2 Noice ha sice he regre boud for he above predicio based o he sequeial Rademacher relaxaio is exacly he oe give i Equaio 3, he upper bouds provided for V 0 i Theorem also hold for he above algorihm. 6. Recipe for desigig olie regressio algorihms We ow provide a schema for derivig forecasers for geeral olie o-parameric regressio problems:. Fid relaxaio Rel such ha R x:, y : Rel x:, y : ad s.. ŷ y 2 +R x:,y :, y is a covex fucio of y 2. Check he codiio x X,p B,B 3. Give x o roud, he predicio ŷ is give by E y p E y p y y 2 + E y p Rel x:, y : Rel x:, y : Rel x:,y :,B Rel x:,y :, B ŷ = Clip 4B Proposiio 2. Ay algorihm derived from he above schema usig relaxaio Rel ejoys a boud o regre. Reg Rel Example : Fiie class of expers As a example of esimaor derived from he schema we firs cosider he simple case F <. Corollary 3. The followig is a admissible relaxaio : I leads o he followig algorihm Rel x:, y : = B 2 log ŷ = Clip B 4 log exp exp ad ejoys a regre boud Reg B 2 log F. exp B 2 f x j y j 2 B 2 f x j y j 2 B 2 f x B 2 B 2 f x j y j 2 B 2 f x + B 2 0

11 Example : Liear regressio Nex, cosider he problem of olie liear regressio i R d. Here F is he class of liear fucios. For his problem we cosider a slighly modified oio of regre : ŷ y 2 if f x y 2 + λ f λ This regre ca be see aleraively as regre if we assume ha o rouds d + o 0 Naure plays λe,0,..., λe d,0 ad ha o hese rouds he learer kowig his predics 0, hus icurrig zero loss over hese iiial rouds. Hece we ca readily apply he schema for desigig a algorihm for his problem. Corollary 4. The followig is a admissible relaxaio 2 Rel x:, y : = y j z j + 4B 2 log z j z j +λi I leads o he Vovk-Azoury-Warmuh forecaser 9, 3: ŷ = Clip ad ejoys he followig upper boud o regre: Refereces y ŷ 2 x d d z j z j + λi x j x j + λi y j x j f x y 2 + λ 2 f dB2 log λd y 2 j. J.Y. Audiber. Fas learig raes i saisical iferece hrough aggregaio. The Aals of Saisics, 374:59 646, P. Auer, N. Cesa-Biachi, ad C. Geile. Adapive ad self-cofide o-lie learig algorihms. Joural of Compuer ad Sysem Scieces, 64:48 75, K. S. Azoury ad M. K. Warmuh. Relaive loss bouds for o-lie desiy esimaio wih he expoeial family of disribuios. Machie Learig, 433:2 246, Jue N. Cesa-Biachi. Aalysis of wo gradie-based algorihms for o-lie regressio. Joural of Compuer ad Sysem Scieces, 593:392 4, N. Cesa-Biachi, Y. Freud, D. Haussler, D. P. Helmbold, R. E. Schapire, ad M. K. Warmuh. How o use exper advice. Joural of he ACM, 443: , N. Cesa-Biachi ad G. Lugosi. Miimax regre uder log loss for geeral classes of expers. I Proceedigs of he Twelfh aual coferece o compuaioal learig heory, pages 2 8. ACM, N. Cesa-Biachi ad G. Lugosi. Predicio, Learig, ad Games. Cambridge Uiversiy Press, D. P. Foser. Predicio i he wors case. Aals of Saisics, 92: , S. Gerchioviz. Sparsiy regre bouds for idividual sequeces i olie liear regressio. Joural of Machie Learig Research, 4: , S. Gerchioviz ad J. Yu. Adapive ad opimal olie liear regressio o l -balls. Theoreical Compuer Sciece, 203.

12 J. Kivie ad M. K. Warmuh. Expoeiaed gradie versus gradie desce for liear predicors. If. Compu., 32: 63, S. Medelso. Learig wihou Coceraio. ArXiv e-pris, Jauary A. Rakhli, O. Shamir, ad K. Sridhara. Relax ad radomize: From value o algorihms. I Advaces i Neural Iformaio Processig Sysems 25, pages , A. Rakhli, K. Sridhara, ad A. Tewari. Olie learig: Radom averages, combiaorial parameers, ad learabiliy. Advaces i Neural Iformaio Processig Sysems 23, pages , A. Rakhli, K. Sridhara, ad A. Tewari. Sequeial complexiies ad uiform marigale laws of large umbers. Probabiliy Theory ad Relaed Fields, February A. Rakhli, K. Sridhara, ad A. Tsybakov. Eropy, miimax regre ad miimax risk. I submissio, S. Shalev-Shwarz. Olie Learig: Theory, Algorihms, ad Applicaios. PhD hesis, Hebrew Uiversiy, N. Srebro, K. Sridhara, ad A. Tewari. Smoohess, low oise ad fas raes. I Advaces i Neural Iformaio Processig Sysems, pages , V. Vovk. Compeiive o-lie liear regressio. I NIPS 97: Proceedigs of he 997 coferece o Advaces i eural iformaio processig sysems 0, pages , Cambridge, MA, USA, 998. MIT Press. 20 V. Vovk. Compeiive o-lie saisics. Ieraioal Saisical Review, 69:23 248, V. Vovk. Meric eropy i compeiive o-lie predicio. CoRR, abs/cs/ , V. Vovk. O-lie regressio compeiive wih reproducig kerel hilber spaces. I Theory ad Applicaios of Models of Compuaio, pages Spriger, V. Vovk. Compeig wih wild predicio rules. Machie Learig, 692:93 22,

13 A Proofs Proof of Lemma 4. Le us ow sudy he value 4. We will do so from iside ou by cosiderig he las sep =, he workig our way back o =. Give a value x, by he miimax heorem, if q p = p = E y p Eŷ q,y p αŷ y 2 + f x y 2 αife y ŷ y 2 + E y f x y 2 ŷ αµ y 2 + f x y 2 where µ = Ey uder he disribuio p wih por o B,B. Observe ha µ y 2 f x y 2 = 2y µ f x µ f x µ 2 ad hece he expressio i 2 ca be wrie as E y f x y 2 + 2y µ f x µ f x µ 2 αµ y 2 p Coiuig i his fashio back o =, he miimax value is equal o V α = x p E y 2y µ f x µ f x µ 2 αµ y The remum over p ca ow be upper bouded by he remum over he mea µ B,B ad a zero-mea disribuio p wih por o B,B. Deoig by η a radom variable wih his disribuio p, he variable µ + η is he i 2B,2B. We upper boud 23 by V α x p,µ E η η f x µ f x µ 2 αη Sice he αη 2 erm does o deped o f, we use lieariy of expecaio o wrie where V α = x p,µ E η 2η f x µ f x µ 2 Dp,..., p 25 Dp,..., p = αeη 2. We ow symmerize he liear erm. Le η be a sequece age o η ha is, η ad η are i.i.d. codiioally o η :. We wrie µ = Eη ad use covexiy of he remum o arrive a a upper boud V α E η 2η η f x µ f x µ 2 Dp x p,µ,..., p 26 = x E η,η E ɛ p,µ 2ɛ η η f x µ f x µ 2 Dp,..., p 27 3

14 where i he secod equaliy holds because η ad η are i.i.d. from p, codiioally o he pas observaios. We ow spli he above remum over f io wo pars, hus passig o he upper boud E η,η E ɛ x p,µ + E η,η E ɛ x p,µ = x = x E η p E ɛ p,µ E η p E ɛ p,µ E ɛ x µ,η = E ɛ,η 2ɛ η f x µ 2 f x µ 2 2 Dp,..., p 2ɛ η f x µ 2 f x µ 2 2 Dp,..., p 4ɛ η f x µ f x µ 2 Dp,..., p 4ɛ η f x µ f x µ 2 αη 2 4ɛ η f x µ f x µ 2 αη 2 4ɛ η ɛf x ɛ µ ɛ f x ɛ µ ɛ 2 αη ɛ 2 This proves he firs saeme. For he case α = 0, we have V 0 E ɛ 4ɛ η ɛf x ɛ µ ɛ f x ɛ µ ɛ 2,η E ɛ x,µ,η = 4ɛ η f x µ f x µ ɛ 2 Sice each η rage over B,B, we ca represe i as B imes he expecaio of a radom variable u,. Deoig his disribuio by q, by Jese s iequaliy V 0 E ɛ x,µ,q E u E ɛ x,µ,q = E ɛ x,µ,u = E ɛ x,µ 4ɛ Eu Bf x µ f x µ ɛ 2 4ɛ u Bf x µ f x µ ɛ 2 4ɛ u Bf x µ f x µ ɛ 2 4ɛ Bf x µ f x µ ɛ 2 which is he same as he desired upper boud i 3, i he ree oaio. As for he lower boud, Recall from Eq. 23 ha he value wih α = 0 is equal o V 0 = x p E y 2y µ f x µ f x µ For he purposes of a lower boud, le us pick paricular disribuios p as follows. Le ɛ,...,ɛ be idepede Rademacher radom variables. Fix a B/2,B/2-valued ree µ. Le y = µ ɛ : +B/2ɛ. Hece, y B,B as 4

15 required. We ca he lower boud he above expressio as V 0 µ x = E E ɛ 2ɛ f x µ ɛ f x µ ɛ 2 Bɛ f x ɛ µ ɛ f x ɛ µ ɛ 2 Proof of Lemma 7. For ay λ > 0, Emax ɛ η ɛw ɛ C w ɛ 2 αη ɛ 2 w W λ loge exp λɛ η ɛw ɛ λc w ɛ 2 λαη ɛ 2 w W Codiioig o ɛ :, we aalyze E exp w W = w W w W exp exp λɛ η ɛw ɛ λc w ɛ 2 λαη ɛ 2 ɛ : λɛ η ɛw ɛ λc w ɛ 2 λαη ɛ 2 λɛ η w ɛ λc w ɛ 2 E exp λɛ η ɛw ɛ ɛ: λαη ɛ 2 exp λ 2 η ɛ 2 w ɛ 2 /2 λc w ɛ 2 λαη ɛ 2 29 The choice λ = 2C /B 2 esures λ 2 η ɛ 2 w ɛ 2 /2 λc w ɛ 2 0 Aleraively, he choice λ = 2α/A 2 esures λ 2 η ɛ 2 w ɛ 2 /2 λαη ɛ 2 0 I boh cases, he expoeial facor peeled off i 29 is o greaer ha. We proceed all he way o = o arrive a a upper boud of λ log exp0 = mi B 2 2C, A 2 2α log W. w W The secod saeme which already appears i 4 is proved similarly, excep he uig value λ is chose a he ed, ad we eed o accou for he wors-case l 2 orm alog ay pahs. For ay ree w W, E exp λɛ η ɛw ɛ ɛ : exp λɛ η ɛw ɛ exp B 2 λ 2 w ɛ 2 /2 exp λɛ η ɛw ɛ maxexp B 2 λ 2 w ɛ 2 /2 ɛ Coiuig i his fashio backwards o =, for ay w W E exp λɛ η ɛw ɛ max exp B 2 λ 2 /2 ɛ,...,ɛ w ɛ 2 ad hus E exp w W λɛ η ɛw ɛ W max max exp ɛ,...,ɛ w W B 2 λ 2 /2 w ɛ 2. 5

16 Choosig λ = 2log W B 2 max ɛ:,w W w ɛ 2 we obai Emax ɛ η ɛw ɛ w W λ loge exp λɛ η ɛw ɛ B 2log W w W max w W,ɛ : w ɛ 2 Proof of Lemma 5. Le V be a sequeial γ-cover of G o z i he l 2 sese, i.e. ɛ, g G, v V s.. g z ɛ v ɛ 2 γ 2 Le us augme V o iclude he all-zero ree, ad deoe he resulig se by V = V 0. Deoe by vɛ, g a γ-close ree promised above, bu we leave he choice for laer. The for ay c 0, E 4ɛ η ɛg z ɛ g z ɛ 2 g G = E g G 4ɛ η ɛ E 4ɛ η ɛ g G + Emax v V g z ɛ vɛ, g ɛ 30 g z ɛ 2 c 2 vɛ, g ɛ ɛ η ɛvɛ, g ɛ c 2 vɛ, g ɛ 2 32 g z ɛ vɛ, g ɛ 4ɛ η ɛv ɛ c 2 v ɛ 2 We ow claim ha for ay ɛ, g here exiss a eleme vɛ, g V such ha g z ɛ 2 c 2 g z ɛ 2 c 2 vɛ, g ɛ vɛ, g ɛ 2 35 ad so we ca drop he correspodig egaive erm i he remum over G. To prove his claim, firs cosider he easy case g z ɛ 2 C 2 γ 2, where C = c c. The we may choose 0 V as a ree ha provides a sequeial Cγ-cover i he l 2 sese. Clearly, 35 is he saisfied wih his choice of vɛ, g = 0. Now, assume g z ɛ 2 > C 2 γ 2. Fix ay ree vɛ, g V ha is γ-close i he l 2 sese o g o he pah ɛ. Deoe u = vɛ, g ɛ,...,vɛ, g ɛ ad h = g z ɛ,..., g z ɛ. Thus, we have ha u h γ ad h Cγ for he orm h 2 = h 2. The u u h + h γ + h C + h ad hus h c u as desired. By choosig c = /2, we have C = ad hus he zero ree also provides a γ-cover. We coclude ha E 4ɛ η ɛg z ɛ g z ɛ 2 4E ɛ η ɛ g z ɛ vɛ, g ɛ 36 g G g G + Emax v V 4ɛ η ɛv ɛ /4v ɛ 2 where vɛ, g is defied o be he all-zero ree if g z ɛ 2 γ 2 ad oherwise as a eleme of he cover V ha is γ-close o g o he pah ɛ. 37 6

17 By Lemma 7, he erm 37 is upper bouded as E ɛ max 4ɛ η ɛv ɛ /4v ɛ 2 32B 2 logn 2 γ,g, z v V We ow ur o he aalysis of he firs erm o he righ-had side of 36. Le vɛ, g be deoed by vɛ, g 0 ad V be deoed by V 0. Le V j deoe a sequeial 2 j γ-cover of G o he ree z, for j =,..., N, N o be specified laer. We ca ow wrie g z ɛ vɛ, g 0 ɛ E g G = E g G E g G ɛ η ɛ ɛ η ɛ N g z ɛ vɛ, g N ɛ + N ɛ η ɛ g z ɛ vɛ, g N ɛ + ɛ η ɛ vɛ, g j E g G ɛ η ɛ ɛ vɛ, g j ɛ vɛ, g j ɛ vɛ, g j ɛ From here, he aalysis is very similar o he oe i 5, excep for he addiioal radom variables η ɛ muliplyig he differeces, ad also for he mior fac ha vɛ, g 0 is defied as 0 for some g,ɛ pairs. This laer fac, however, does o affec he proof sice 0 does provide a valid γ-cover whe i is used. Firs, by Cauchy-Schwarz iequaliy, E ɛ η ɛ g z ɛ vɛ, g N ɛ ɛ η E ɛ g z ɛ vɛ, g N ɛ g G g G /2 E η ɛ 2 β N Bβ N where β j = 2 j γ. For he secod erm, fix a paricular j ad cosider all pairs v s, v r wih v s V j ad v r V j. For each such pair, defie a ree w s,r by w s,r ɛ = v s ɛ v r ɛ if here exiss g G s.. vs = vg,ɛ j, v r = vg,ɛ j 0 oherwise. for all ad ɛ ±. Oe ca check ha he ree is well-defied, ad we se The for ay j N ad ɛ, W j = w s,r : s V j, r V j. ɛ η ɛ vɛ, g j ɛ vɛ, g j ɛ max ɛ η ɛw ɛ g G w W By he argume oulied i 5, for ay w W j ad ay pah ɛ, Puig everyhig ogeher, ad usig Lemma 7, w ɛ 2 3 β j. E ɛ ɛ η ɛ g z ɛ vɛ, g 0 ɛ Bβ N + B g G N 3β j 2log V j V j 7

18 ad he las erm is upper bouded by 6B N β j log V j 2B N β j β j + log V j 2B Give ay ρ 0,γ, we le N = maxj : β j > 2ρ. The β N < 4ρ ad β N+ > ρ, ad hus This cocludes he proof. E ɛ η ɛ g z ɛ vɛ, g 0 ɛ B g G if ρ 0,γ 4ρ + 2 γ ρ β0 β N+ logn 2 δg, zdδ logn 2 δ,g, zdδ Proof of Lemma 6. The proof closely follows ha of Lemma 5, excep for he way we use Lemma 7 o ake advaage of he subraced quadraic erm. We also employ a l oio of sequeial cover, raher ha l 2. To his ed, le V be a sequeial γ-cover of G o z i he l sese, i.e. ɛ, g G, v V s.. max g z ɛ v ɛ γ As before, le V = V 0 ad deoe by vɛ, g a γ-close ree promised by he defiiio. As i 30, for ay c 0,, E g G E g G 4ɛ η ɛg z ɛ g z ɛ 2 αη ɛ 2 + Emax v V 4ɛ η ɛ g z ɛ vɛ, g ɛ α 2 η ɛ2 4ɛ η ɛv ɛ c 2 v ɛ 2 α 2 η ɛ2 g z ɛ 2 c 2 vɛ, g ɛ 2 Followig he proof of Lemma 5, we claim ha for ay ɛ, g here exiss a eleme vɛ, g V such ha for ay, g z ɛ c vɛ, g ɛ 38 Firs cosider he easy case g z ɛ Cγ, where C = c c. The 0 V provides a sequeial Cγ-cover i he l sese. If, o he oher had, g z ɛ > Cγ, we fix ay ree vɛ, g V ha is γ-close i he l sese o g o he pah ɛ. Wih he same argume as i he proof of Lemma 5, we coclude 38. Hece, E 4ɛ η ɛg z ɛ g z ɛ 2 αη ɛ 2 g G 4E g G + Emax v V ɛ η ɛ By Lemma 7, he erm 40 is upper bouded as g z ɛ vɛ, g ɛ α 2 η ɛ2 4ɛ η ɛv ɛ /4v ɛ 2 α 2 η ɛ2 E ɛ max 4ɛ η ɛv ɛ /4v ɛ 2 α v V 2 η ɛ2 α 6A 2 logn γ,g, z

19 As for he erm i 39, we wrie E ɛ η ɛ g z ɛ vɛ, g 0 ɛ α g G 2 η ɛ2 g z ɛ vɛ, g N ɛ = E g G ɛ η ɛ + N E ɛ η ɛ g G N + ɛ η ɛ vɛ, g j α 4 η ɛ2 g z ɛ vɛ, g N ɛ α 4 η ɛ2 E ɛ η ɛ vɛ, g j g G Usig Cauchy-Schwarz iequaliy alog wih ab /2a 2 + b 2, ɛ η ɛ g z ɛ vɛ, g N ɛ ɛ vɛ, g j ɛ α 4N η ɛ2 ɛ vɛ, g j ɛ α 4N η ɛ2 αɛ η ɛ 2 g z ɛ vɛ, g N 2 α ɛ αη 4 ɛ 2 + g z ɛ vɛ, g N 2 α ɛ ad hus E ɛ η ɛ g z ɛ vɛ, g N ɛ α g G 4 η ɛ2 α β N where β j = 2 j γ. For he j -h lik i he chai, recall ha we ca defie w s,r ɛ = v s ɛ v r ɛ if here exiss g G s.. vs = vg,ɛ j, v r = vg,ɛ j 0 oherwise. for all ad ɛ ±. The for ay j N ad ɛ, ɛ η ɛ vɛ, g j ɛ vɛ, g j ɛ α g G 4N η ɛ2 max ɛ η ɛw ɛ α w W 4N η ɛ2 ad i mus hold by he defiiio of he cover ha w ɛ 2β j for ay w W j ad ay pah ɛ ad ay. Puig everyhig ogeher, ad usig Lemma 7, E ɛ ɛ η ɛ g z ɛ vɛ, g 0 ɛ α g G 2 η ɛ2 β N N 8Nβ 2 α + j α log V j V j Simplifyig ad usig β j = β j β j, we obai a upper boud of β N α + 6N α N β 2 j log V j = β N α β N α + 6N α + 6N α N β j β j β j log V j β0 β N+ δlogn δ,g, zdδ 9

20 Give ay ρ 0,γ, we le N = maxj : β j > 2ρ. The β N < 4ρ ad β N+ > ρ. Furher, N logγ/ρ. Thus E ɛ η ɛ g z ɛ vɛ, g 0 ɛ α g G 2 η ɛ2 α if ρ 0,γ Togeher wih 4 his cocludes he proof. 4ρ + 6logγ/ρ γ ρ δlogn δ,g, zdδ Proof of Lemma 9. Fix a β > 0, ad se = fa β F. Suppose x is a X -valued ree of deph ha is β-shaered by F : ɛ, f ɛ F s.. ɛ f ɛ x ɛ µ ɛ β/2 where µ is he wiess o shaerig. Sice fucios i F ake values i,, i is also he case ha µ is,- valued, ad hus f x ɛ µ ɛ 2 for all f F. The from 4 wih he paricular choices of x ad µ described above, V 0 E 4ɛ f x ɛ µ ɛ f x ɛ µ ɛ 2 E 4ɛ f x ɛ µ ɛ 2 f x ɛ µ ɛ E 4ɛ f ɛ x ɛ µ ɛ 2 f ɛ x ɛ µ ɛ Usig he defiiio of shaerig, we ca furher lower boud he above quaiy by E 4 f ɛ x ɛ µ ɛ 2 f ɛ x ɛ µ ɛ E β = β Now, pose fa β F = C /β p, p > 0. The = fa β F implies β = C /p. The resul follows. Proof of Lemma 0. Assume d = fa β F. Le z be a X -valued ree of deph d ha is β-shaered by F wih a wiess ree s. Observe ha he fucios f ɛ ha guaraee , ɛ f ɛ z ɛ s ɛ β/2 45 do o ecessarily ake o values close o he s ɛ ± β/2 ierval. We augme F wih 2 d fucios g ɛ ha ake o he same values as f ɛ, excep 45 is saisfied wih equaliy: ɛ g ɛ z ɛ s ɛ = β/2. Le F be he resulig class of fucios, ad G = F \ F. We ow argue ha fa β F cao be more ha 2d + 4, as we have oly added a mos 2 d fucios o F. Suppose for he sake of coradicio ha here exiss a ree z of deph a leas 2d + 5 shaered by F. There mus exis 2 2d+5 fucios ha shaer z ad oly a mos 2 d of hem ca be from G. Le us label he leaves of z wih he fucios ha shaer he correspodig pah from he roo; hese fucios are clearly disic. Order he leaves of he ree i ay way, ad observe ha here mus exis a pair of fucios from G wih idices differig by a leas 2 d+4. I is easy o see ha such wo leaves ca oly have a commo pare a d + 3 levels from he leaves, ad his yields a complee biary subree of size d + ha is shaered by fucios i F, a coradicio. We will ow use he fucio class F o prove a lower boud. Recall ha z is a X -valued ree of deph fa β ha is β-shaered by G F. Le s be he wiess ree for he shaerig. We will ow show a cosrucio of paricular rees of deph = fa β 46 fa β 20

21 usig he pair z, s. Defie k = fa β = fa β ad cosider he X -valued ree x ad he R-valued ree µ of deph cosruced as follows. For ay pah ɛ ± ad ay, se x ɛ = z k ɛ, µ ɛ = s k ɛ where ɛ ± fa β is he sequece of sigs specified as ɛ = sig k ɛ j,sig 2k j =k+ k fa β ɛ j,...,sig j =k ɛ j. fa β We ow lower boud 4 by choosig he paricular defied above: V 0 E = E 2ɛ f x ɛ µ ɛ f x ɛ µ ɛ 2 2ɛ f z k ɛ s k ɛ f z k ɛ s k ɛ2 Spliig he sum over io fa β blocks, he above expressio is equal o faβ E = E = E i k i= j =i k+ faβ 2ɛ j f z i ɛ s i ɛ f z i ɛ s i ɛ 2 2f z i ɛ s i ɛ i= faβ 2 ɛ i f z i ɛ s i ɛ i= i k j =i k+ i k j =i k+ ɛ j kf z i ɛ s i ɛ 2 ɛ j kf z i ɛ s i ɛ 2 where he las sep follows by he defiiio of ɛ. Recall ha z is shaered by he subse G ad ha he fucios i G say close o he wiess ree s. We obai a lower boud E g G 2 ɛ i f z i ɛ s i ɛ faβ i= i k j =i k+ fa β ɛ j kf z i ɛ s i ɛ 2 i k E β i= k fa β F β where we used Khichie s iequaliy i he las sep. By he defiiio of k, k fa β F β 2 = fa βf β 2fa β F = β fa β F 2. j =i k+ 2 kβ2 4 ɛ j kβ2 4 ad We coclude ha V 0 4 fa β F kβ2 4 = 4 β 2 2 2β fa β F β

22 Now pose fa β F = c/β p for some c > 0. Firs, we eed o esure ha fa β F = c/β p, as required by our cosrucio. This meas ha β c /p. Pluggig i he rae of fa β F io 47, 2 2β fa β F β 2 = 2 2c /2 β p/2 β 2 Observe ha he seig of β = 32c /2+p /p+2 yields a lower boud of c p p p+2 where c p deoes a cosa ha may deped o p, ad whose value may chage from lie o lie. Examiig 28, we see ha V 0 is odecreasig wih. To illusrae his, le >. For +,...,, we may choose p i 28 as a dela disribuio o f x, for ay sequece of x, where f is a opimal fucio over seps,...,. Clearly, V 0 V 0. I view of 46 ad he above discussio, V 0 V2 0, ad hus V 0 2 V 0 2 V 0 c p p p+2. Proof of Lemma. Firs oe ha whe = he iiial codiio is rivially saisfied as R x :, y : = f x j y j 2 = if f x j y j 2. Le us deoe ad A + f = j =+ L f = f x j y j 2 Bɛ j f x j ɛ µ j ɛ f x j ɛ µ j ɛ 2 To check admissibiliy oe ha we eed o check he iequaliy i Equaio 48. To do so oe ha for ay x X, p B,B, E y p E y p y y 2 + E y p R x:, y : = Ey p E y p y y 2 + Expadig he square i he firs erm ad he he loss of f a ime, we obai 2 E y p E y p y 2y E y p y + y 2 + E ɛ A + f L f = E y p E y p y 2 2y E y p y + Rearragig, he above is equal o A + f L f + 2 E y p = E y p which is E ɛ E ɛ E y p E y p E ɛ E ɛ A+ f L f A + f f 2 x + 2f x y L f 2 E y p y f 2 x E 2 y p y + 2f x E y p y y A + f L f + 2 E 2 y p y f 2 x E y p y 2f x E y p y + 2 f x E y p y y E ɛ A + f L f 2 E 2 y p y f 2 x E y p y 2 f x E y p y E y p y + 2 f x E y p y y E ɛ A + f L f f 2 x E y p y + 2 f x E y p y y E y p y 22

23 By Jese s iequaliy, he above ca be upper bouded by E y,y p E ɛ A + f L f = E y,y p,ɛ E ɛ f 2 x E y p y + 2 f x E y p y y y A + f L f f 2 x E y p y + 2ɛ f x E y p y y y Sice he iequaliies above hold for ay x X, p B,B, we have 2 E y p E y p y y + E y p R x:, y : x X,p B,B x X,p B,B x X y,y,µ B,B x X y,µ B,B E y,y p,ɛ E ɛ E ɛ E ɛ E ɛ A + f L f f 2 x E y p y + 2ɛ f x E y p y y y A + f L f 2 f x µ + 2ɛ f x µ y y E ɛ A + f L f 2 f x µ + 4ɛ f x µ y Sice he above is covex i y, we ca replace he remum over B,B o remum over B,B E ɛ E ɛ A + f L f 2 f x µ + 4ɛ f x µ y x X,µ B,B y B,B = x X µ B,B E ɛ = E ɛ j = E ɛ A + f L f 2 f x µ + 4Bɛ f x µ Bɛ j f x j ɛ µ j ɛ f x j ɛ µ j ɛ 2 L f = R x:, y : Thus we have show ha R is a admissible relaxaio. Furher, ŷ y 2 + R x:,y :, y is a covex fucio of y ad so for he esimaor oe ca use ŷ = R x :,y :,B R x :,y :, B 4B o clippig is eeded above as ŷ is always bewee B ad B. For he above esimaor oe ejoys he regre boud Reg R Noe ha his is exacly he boud i Eq. 3. Proof of Proposiio 2. Noice ha he above recipe closely follows he oio of relaxaio provided i 3. All we eed o do is check ha he relaxaio derived saisfies admissibiliy ad iiial codiios. By Sep of he recipe, sice he offse Rademacher relaxaio is admissible o sar wih, he derived relaxaio also saisfies iiial codiio. To show admissibiliy codiio oice ha he se B,B is compac ad covex ad ŷ y 2 + Rel x:, y : is a covex fucio of ŷ. Hece applyig miimax heorem, we see ha, if ŷ y 2 + Rel x:, y : ŷ B,B y B,B = p B,B ŷ = p B,B = p B,B if E y p ŷ y 2 + Rel x:, y : if E y p ŷ y 2 + E y p Rel x:, y : ŷ 2 E y p E y p y y + E y p Rel x:, y : 23

24 Hece he admissibiliy codiio ca be rewrie as : x X, 2 E y p E y p y y + E y p Rel x:, y : Rel x:, y : p B,B 48 Proof of Corollary 3. As doe i 3 for he case of fiie class of expers, i he Rademacher relaxaio oe ca replace he max wih a limi of sof-max as follows: R x :, y : = E ɛ max 4Bɛ j f x j ɛ µ j ɛ f x j ɛ µ j ɛ 2 f x j y j 2 j =+ = E ɛ if λ log exp λ 4Bɛ j f x j ɛ µ j ɛ f x j ɛ µ j ɛ 2 λ f x j y j 2 λ>0 j =+ if λ log exp λ f x j y j 2 λ>0 + λ log E ɛ exp λ 4Bɛ j f x j ɛ µ j ɛ f x j ɛ µ j ɛ 2 j =+ No oice ha if we se λ = B 2, he proof of Lemma 7 exacly shows ha λ log E ɛ exp Hece we arrive a our relaxaio Now o show admissibiliy, oe ha λ j =+ Rel x:, y : = B 2 log E y p x,p y E y 2 + Rel x:, y : = x,p E y p = x,p E y p = x,p E y p = x,p E y p = x,p E y p 4Bɛ j f x j ɛ µ j ɛ f x j ɛ µ j ɛ 2 B 2 log F exp B 2 f x j y j 2 y 2 E y 2 + B 2 log exp B 2 f x j y j 2 B 2 log exp B 2 y 2 B 2 E y 2 + B 2 log exp B 2 f x j y j 2 B 2 log exp B 2 y 2 B 2 E y 2 B 2 f x j y j 2 B 2 log exp B 2 E y 2 + 2B 2 f x y B 2 f 2 2 x B f x j y j 2 B 2 log exp B 2 E y f x 2 + 2B 2 f x y E y B 2 f x j y j 2 24

25 Now by covexiy see we ca ake he expecaio w.r.. y iside ad hece we ge, E y p y E y 2 + Rel x:, y : x,p B 2 log B 2 E y f x 2 B x,p B 2 log exp exp B = Rel x:, y : f x j y j 2 2 f x j y j 2 Agai as we used above see we have ha he relaxaio is such ha ŷ y 2 +Rel x:,y :, y is a covex fucio of y ad so he esimaor is give by 2 Rel x:,y :,B Rel x:,y :, B ŷ = Clip 4B = Clip B 4 log exp B 2 f x j y j 2 B 2 f x B 2 exp B 2 f x j y j 2 B 2 f x + B 2 Now he fial regre boud we obai is give by Reg Rel ad so we coclude ha Reg B 2 log F Proof of Corollary 4. For simpliciy, each ipu isace x R d we defie vecor i R d+ as z = 0, x, he vecor obaied by cocaeaig 0 before x. Furher give rees x ad µ, we wrie he z as he B,B X valued ree correspodig o x ad µ obaied by cocaeaig µ s before x s o every ode. Also for every liear predicor f F defie correspodig w =, f. The uormalized regre over he rouds d o ca be wrie as Hece, we have, ŷ y 2 if w, z y 2 + λ w 2 2 w R x :, y : = E ɛ 4Bɛ j w, z j ɛ w, z j ɛ 2 w, z j y j 2 λ w 2 2 z j =+ = 2E ɛ w, 2Bɛ j z j ɛ + y j z j z w j =+ 2 w z j ɛz j ɛ + z j z j + λi w y 2 j j =+ Le us deoe A +: z = j =+ z j ɛz j ɛ ad B = z j z. Usig Fechel-Youg iequaliy for j ad is cojugae we ge, R x :, y : z 2 w A +: z + B + λi w 2 E ɛ 2Bɛ j z j ɛ + y j z j y 2 j j =+ A +: z+b +λi 25

26 The idea ow is o obai a furher upper boud by removig he depedece o he ree z. Opeig he square wih oly he -h erm, he above expressio is equal o z 2 2Bɛ j z j ɛ + y j z j j =+ A +: z+b +λi E ɛ y 2 j + 4B 2 z ɛ A +: z + B + λi z ɛ By he sadard argume, z ɛ A +: z + B + λi z ɛ A +: z + B + λi A +: z + B + λi Usig he iequaliy x logx for x > 0, we obai a upper boud E 2 ɛ 2Bɛ z j z j ɛ + y j z j y 2 j + 4B 2 A+: z + B + λi E ɛ log j =+ A +: z+b +λi A +: z + B + λi Proceedig i similar fashio by peelig off erms from he orm, we arrive a, 2 R x :, y : y j z j y 2 j + 4B 2 A+: z + B + λi E ɛ log B +λi z B + λi 2 y j z j + 4B 2 /d d log y 2 B B +λi + λi j ad we ake his las expressio as our relaxaio Rel x:, y :. Now oice ha sice z s are 0 o he firs coordiae he relaxaio ca be rewrie as 2 Rel x:, y : = y j z j + 4B 2 /d d log y 2 j B B +λi + λi 2 = y j x j y 2 j + 4B 2 /d d log B +λi B + λi = = if 2 y j f, x j f B + λi f f x j y j 2 + λ f 2 2 Usig he above we prove admissibiliy of relaxaio as follows : E y p p y E y 2 + Rel x:, y : = E y p p y E y 2 if = E y p p 2y E y + E y 2 if + 4B 2 /d d log B + λi y 2 j + 4B 2 log + 4B 2 log /d d B + λi 2 f, x j y j + λ f 2 + λ + 4B 2 log /d d B + λi + λ /d d B + λi 2 f, x j y j + f, x 2 2y f, x + λ f 2 + λ 26

27 The above expressio is equal o E y p p 2E y f, x j y j E y 2 f, x 2 + 2y f, x E y + λ f 2 + λ + 4B 2 /d d log B + λi 2 = E y p f, x j y j E y 2 f, x 2 + 2f x E y + 2y E y f, x E y p + λ f 2 + λ + 4B 2 /d d log B + λi which is equal o E y p p µ E ɛ 2 f, x j y j f, x E y 2 + 2y E y f, x E y + λ f 2 + λ + 4B 2 log f, x j y j 2 f, x µ 2 + 4Bɛ f, x µ + λ f 2 + λ + 4B 2 log /d d B + λi /d d B + λi As earlier, rewriig he above usig Fechel-Youg cojugacy ad followig similar lie of proof coverig o he z oaio we arrive a a upper boud E 2 ɛ y z j z j + 2Bɛ z + 4B 2 /d d log y 2 B +z z +λi B + λi j 2 = y z j z j + 4B 2 z B +z z +λi B + z z + λi z + 4B 2 /d d log y 2 B + λi j which is furher upper bouded by 2 y z j z j + 4B 2 B + λi log z + 4B 2 /d d log y 2 B B +λi + λi B + λi j 2 = y j z j + 4B 2 /d d log y 2 B B +λi + λi j = Rel x:, y : Thus we have show admissibiliy ad furher his relaxaio is such ha ŷ y 2 + Rel x:,y :, y is a covex fucio of y ad so he esimaor associaed wih his relaxaio is simply y 2 j x j + B x B ŷ = Clip +λi y 2 j x j B x B +λi 4B Expadig ou he wo orm square erms we coclude ha ŷ = Clip x B + λi y j x j Noice ha his is exacly he clipped versio of he Vovk-Azoury-Warmuh forecaser. The fial regre boud we obai is give by Reg Rel ad so we coclude ha for ay f F, regre agais his liear predicor is bouded as : y ŷ 2 f x y 2 + λ 2 f dB2 log λd 27