A Local Regret in Nonconvex Online Learning

Transcription

1 Sergul Aydore Lee Dicker Dean Foser Absrac We consider an online learning process o forecas a sequence of oucomes for nonconvex models. A ypical measure o evaluae online learning policies is regre bu such sandard definiion of regre is inracable for nonconvex models even in offline seings. Gradien based definiion of regres are common for boh offline and online nonconvex problems. A noion of local gradien based regre was recenly inroduced. Inspired by he concep of calibraion, we provide anoher local gradien definiion and we discuss why our definiion is more inerpreable for forecasing problems.. Inroducion In ypical forecasing problems, we make probabilisic esimaes of fuure oucomes based on he previous observaions. Recenly, i has been shown ha forecasing models can be complex nonconvex models (Flunker e al., 07; Wen e al., 07). Frequen updae of hese models is desired as he relaionship beween he arges and oupus migh change over ime. However, re-raining hese models can be ime consuming. Online learning is a mehod of updaing he model on each paern as i is observed as opposed o bach learning where he raining is performed over groups of paern. I is a common echnique o dynamically adap o new paerns in he daa or when raining over he enire daa se is infeasible. The lieraure in online learning is rich wih ineresing heoreical and pracical applicaions bu i is usually limied o he convex problems where global opimizaion is compuaionally racable (Zinkevich, 003). On he oher hand, i is NP-hard o compue he global minimum of nonconvex funcions over a convex domain (Hazan e al., 07). Due o he inracabiliy of he nonconvex problems, various assumpions on he inpu have been used o design polynomial-ime algorihms (Arora e al., 04; Hsu e al., 0). However, hese were oo specific o he models and Amazon, NY, USA. Correspondence o: Sergul Aydore <sergulaydore@gmail.com>. Proceedings of he 35 h Inernaional Conference on Machine Learning, Sockholm, Sweden, PMLR 80, 08. Copyrigh 08 by he auhor(s). more generic approach was needed. One way o achieve his is by replacing he global opimaliy requiremen wih a more modes requiremen of saionariy (Allen-Zhu & Hazan, 06). The idea of online learning was borrowed from game heory where an online player answers a sequence of quesions. The rue answers o he quesions are unknown o he player a he ime of each decision and he player suffers a loss afer commiing o a decision. These losses are unknown o he player and he performance of he sequence of decisions will be evaluaed by he difference beween his accumulaed loss and he bes fixed decision in hindsigh. Mos recenly, Hazan e al. (07) proposed a noion of gradien based local regre for nonconvex games. Inspired by Hazan s approach and incorporaing he noion of calibraion, we inroduce a novel gradien based local regre for forecasing problems. Calibraion is a well-sudied concep in forecasing (Foser & Vohra, 998). From game heoreic poin of view, we call a forecasing procedure calibraed if he forecass are consisen in hindsigh. To he bes of our knowledge, such definiion of regre is new. We show ha he proposed regre has logarihmic bound. Furhermore, we provide insighs o he proposed regre for inerior poins of he feasible se.. Seing In online forecasing, our goal is o updae x a each in order o incorporae he mos recenly available informaion. Assume ha T {,, T } represens a collecion of T consecuive poins where T is an ineger and represens an iniial forecas poin. The follow-heleader algorihm for online opimizaion provides soluion as: x arg min x s f s (x; x, x,, x ) where f,, f T : K R are nonconvex funcions on some convex subse K R d... Regre Analysis The performance of online learning algorihms is commonly evaluaed by he regre, which is defined as he difference beween he real cumulaive loss and he minimum cumulaive loss across T : R f (x ) min x K f (x) ()

2 If he regre grows linearly wih T, i can be concluded ha he player is no learning. If, on he oher hand, he regre grows sub-linearly, he player is learning and is accuracy is improving. While such a definiion of regre makes sense for convex opimizaion problems, i is no appropriae for nonconvex problems, due o NP-hardness of nonconvex global opimizaion even in offline seings. Indeed, mos research on nonconvex problems focuses on finding local opima. In lieraure on nonconvex opimizaion algorihms, i is common o use he magniude of he gradien o analyze convergence. Hazan e al. (07) inroduced a local regre measure - a new noion of regre ha quanifies he objecive of predicing poins wih small gradiens on average. A each round of he game, he gradiens of he loss funcions from w where w T mos recen rounds of play are evaluaed a he curren forecas, and hese gradiens are hen averaged. Hazan e al. (07) s local regre is defined o be he sum of he squared magniude of he gradiens averages. Definiion..(Hazan s Local Regre) The w-local regre of an online algorihm is defined as: HR F,w (x ) () when K R d and F,w (x ) w w i0 f i(x ). Hazan e al. (07) proposed various gradien descen algorihms where he regre HR is sublinear... Proposed Local Regre In order o inroduce he concep of calibraion (Foser & Vohra, 998), le s consider he firs order Taylor series expansion of he cumulaive loss: f (proj K (x + u)) f (x ) + f (x + D u (x )) D u (x ), f (x ) (3) where D u (x ) proj K (x +u)x for any u R d. If he forecass {x,, x T } are well-calibraed, hen perurbing x by any u canno subsanially reduce he cumulaive loss. Hence, we can say ha he sequence {x,, x T } is asympoically calibraed wih respec o {f,, f T }, if: lim sup T sup u R d T D u (x ), f (x ) 0 (4) Definiion..(Proposed Regre) We propose a w-local regre as: PR D u (x s ), (5) w sw+ where f (x ) 0 for 0. To moivae equaion 5, we use he following equaliy: lim δ 0 δ sup u δ w sw+ D u (x s ), w sw+ which holds for he inerior poins. Using our definiion of regre, we effecively evaluae an online learning algorihm by compuing he average of losses a he corresponding forecas values over a sliding window. Hazan e al. (07) s definiion of regre, on he oher hand, compues average of previous losses compued on he mos recen forecas. We believe ha our definiion of regre is more applicable o forecasing problems as evaluaing oday s forecas on previous loss funcions migh be misleading..3. Toy example We illusrae he behavior of our and Hazan e al. (07) s definiion of regre via a oy example. We simulae wo nonconvex loss funcions: f (x) and f (x) and compare pairs of policies. We use he original definiion of regre given in equaion as ground ruh. Figure shows hese wo funcions ogeher wih heir gradiens. We inroduce following hree policies: - Good Policy: play 5, 0, 5, 0, - Bad Policy: play 0, 5, 0, 5, - Fixed Policy: play.5,.5,.5,.5, Among hese policies; good policy should give he lowes regre since i is he bes policy we can play. Bad policy should give he larges regre. The value we play in fixed policy corresponds o he dip of he dashed black line (i.e. he sum of gradiens of he wo funcions os zero) in Figure and he loss of his policy should be beween good and bad f (x) f (x) f (x) f (x) f (x) + f (x) x Figure. Simulaion of a game wih wo alernaing nonconvex loss funcions. (6)

3 Regre Regre(Good Policy) Regre(Fixed Policy) Regre(Bad Policy) Regre(Fixed Policy) Original Regre Hazan s regre Proposed regre Table. Comparison of policies using differen regre definiions. policies. We repor he resuls in Table ; our regre has he same sign as he original regre whereas Hazan s regre has opposie sign which indicaes ha Hazan s regre concludes ha he bad policy is beer. 3. Bound Analysis In his secion we assume ha all he poins considered are inerior poins of he feasible se. We consider hree scenarios: (i), w is fixed and K R d (ii) / and w (iii) / and w is fixed. 3.., w is fixed and K R d Since K R d, he updae rule becomes x + x f (x ); in oher words no projecion operaor is necessary. Hence, we can wrie: sw+ u, u, f (x ) sw+ u, sw+ f (x ) (x x + ) u, (x w+ x ) x w+ x M (7) Taking u sw+ f (x )/ sw+ f (x ) ; we can wrie sw+ f s(x s ) M /. Hence; he proposed regre bound can be wrien as: PR w w sw+ M T w (8) which can be made sublinear in T if w is seleced large enough. 3.. /, w Assuming x s + u is in inerior of he feasible se for all u and s and seing w, we can wrie he resul in heorem 5. as: u, u, s (9) s s f s(x s ) s f (0) s(x s ) () s + G () s fs(xs) where u is se o s fs(xs). Hence, we ge: f s (x s ) + G. (3) s Summing his over yields: f s (x s ) + G s + G log(t ). (4) 3.3. / and w is fixed The firs erm on he RHS of he resul in lemma 5. is non-negaive; using similar logic o 3. we can wrie: w + G w. s Summing his resul across yields: + G w s 3 + G T (T + ) (5) w which is quadraic in T bu w can be seleced accordingly o make he upper bound sub-linear. 4. Conclusion We inroduced a new definiion of a local regre o sudy nonconvex problems in forecasing. We used he concep

4 of calibraion and showed ha our regre can be wrien as a local regre for he inerior poins in he feasible se. Our regre differs from Hazan s regre in he sense ha i emphasizes oday s reward as opposed o pas reward. We also showed ha our definiion of regre has a logarihmic bound. As a fuure direcion, we plan o sudy he insighs of our regre for he boundary poins in he feasible se. 5. Appendix We provide a bound for he proposed regre in equaion 5 for he inerior poins in he feasible wih he following assumpions: sup x,y K x y M; sup x K, T f (x) G; he parameer updae a is: x + proj K (x f (x )) where / is he learning rae for some small > 0. Lemma ( 5.. D u (x ), f (x ) u u +, u + u+ x + u x ) G Proof. Le y + x f (x ) and u proj K (x + u). Noe ha x + proj K (y + ). Then we have: D u (x ), f (x ) u x, x y + u x, x x + + u x, x + y + u x, x x + + u x +, x + y + + x + x, x + y + u x, x x + + x x +, y + x + u x, x x + + x x +, x f (x )x + u x, x x + + x x + f (x ), x x + u x, x x + + x x + G ( u x + u x + x + x ) G u u +, u + x + + ( u+ x + u x ) + ( u+ u + x + x ) G u u +, u + u u +, u + (x + + u) + ( u+ x + u x ) + ( u+ u + x + x ) G u u +, u + ( u+ x + u x ) G We can use his resul in he following heorem: Theorem 5.. sw+ D u(x s ), G ( w + ) ( ) + G 3M Proof. sw+ D u (x s ), sw+ s u s u s+, u sw+ s G ( + us+ x s+ u s x s ) sw+ s w + u w+ x, u u + x, u + ( ) s s u x, u sw+ w + + u + x + u w+ x w+ ( s s ) u s x s G w+ sw+ u + x, u w + u w+ x, u + ( ) s s u x, u sw+ s ( ) M + G + G ( w + ) (6) Now, le s explore he bound for u x, u. By definiion of u, we can wrie: x + u x x + u u Hence, x + u x + x u + x + u x, x u x + u x + x u + x x, x u + u, x u x + u x M + u, x u. (7) u, u x M (8) Taking x u + and combining 6 wih 8, we ge: w + D u (x s ), M sw+ ( ) ( ) w + M M + G G ( w + ) + G (9)

5 References Allen-Zhu, Zeyuan and Hazan, Elad. Variance reducion for faser non-convex opimizaion. In Inernaional Conference on Machine Learning, pp , 06. Arora, Sanjeev, Ge, Rong, and Moira, Ankur. New algorihms for learning incoheren and overcomplee dicionaries. In Conference on Learning Theory, pp , 04. Flunker, Valenin, Salinas, David, and Gashaus, Jan. Deepar: Probabilisic forecasing wih auoregressive recurren neworks. arxiv preprin arxiv: , 07. Foser, Dean P and Vohra, Rakesh V. Asympoic calibraion. Biomerika, 85(): , 998. Hazan, Elad, Singh, Karan, and Zhang, Cyril. Efficien regre minimizaion in non-convex games. arxiv preprin arxiv: , 07. Hsu, Daniel, Kakade, Sham M, and Zhang, Tong. A specral algorihm for learning hidden markov models. Journal of Compuer and Sysem Sciences, 78(5): , 0. Wen, Ruofeng, Torkkola, Kari, and Narayanaswamy, Balakrishnan. A muli-horizon quanile recurren forecaser. arxiv preprin arxiv:7.053, 07. Zinkevich, Marin. Online convex programming and generalized infiniesimal gradien ascen. In Proceedings of he 0h Inernaional Conference on Machine Learning (ICML-03), pp , 003.