arxiv: v2 [cs.lg] 28 Dec 2018

Transcription

1 Dynamic Online Gradien Descen wih Improved Query Complexiy: A Theoreical Revisi arxiv:8.086v [cs.lg] 8 Dec 08 Yawei Zhao, En Zhu, Xinwang Liu, and Jianping Yin Naional Universiy of Defense Technology, Changsha, 40073, China. Dongguan Universiy of Technology, Dongguan, Guangdong, 53808, China. {zhaoyawei, enzhu, xinwangliu}@nud.edu.cn; jpyin@dgu.edu.cn. Absrac We provide a new heoreical analysis framework o invesigae online gradien descen in he dynamic environmen. Comparing wih he previous work, he new framework recovers he sae-of-he-ar dynamic regre, bu does no require exra gradien queries for every ieraion. Specifically, when funcions are srongly convex and β smooh, o achieve he sae-of-he-ar dynamic regre, he previous work requires Oκ wih κ = β queries of gradiens a every ieraion. Bu, our framework shows ha he query complexiy can be improved o be O, which does no depend on κ. The improvemen is significan for ill-condiioned problems because ha heir objecive funcion usually has a large κ. Inroducion Online Gradien DescenOGD has drawn much aenion in he communiy of machine learning Zhu and Xu [05], Hazan and Seshadhri[007], Hall and Wille [05], Shalev-Shwarz[0], Garber[08], Bedi e al. [08]. I is widely used in various applicaions such as online recommendaion Song e al. [008], search ranking Moon e al. [00]. Generally, OGD is formulaed as a game beween a learner and an adversary. A he -h round of he game, he learner submis x from he feasible se X, and he adversary selecs a funcion f : X R. Then, he funcion f is reurned o he learner, and incurs he loss f x. Recenly, here has been a surge of ineres in analyzing OGD by using he dynamic regre Zinkevich [003], Mokhari e al. [06], Yang e al. [06], Lei e al. [07]. The dynamic regre is usually defined as T RT = f x f x, where x argmin x X f x. Unforunaely, i is well-known ha a sublinear dynamic regre bound canno be achieved in he wors case Zinkevich [003]. The reason is ha he funcions f,...,f T may be changed arbirarily in he dynamic environmen. Bu, i is possible o upper bound he dynamic regre in erms of cerain regulariy of he comparaor sequence. Those regulariies are usually defined as he pah lengh Mokhari e al. [06], Yang e al. [06]: T PT := Px,...,x T = x x, or squared pah lengh Zhang e al. [07]: ST := Sx,...,x T = x x. = =

2 Table : Our mehod OGD recovers he sae-of-he-ar regre wih improved query complexiy. Algo. Obj. ype Dynamic regre Avg. queries Mokhari e al. [06] srongly convex OP O Zhang e al. [07] srongly convex Omin{ST,P } Oκ Ours srongly convex Omin{ST,P } O They capure he cumulaive Euclidean norm or he square of Euclidean norm of he difference beween successive comparaors. When all he funcions f,...,f T are -srongly convex and β-smooh, he dynamic regre is bounded by OPT Mokhari e al. [06]. When he local variaions are small, S T is much smaller han PT. Thus, he sae-of-he-ar dynamic regre of OGD is improved o be Omin{P T,S T } Zhang e al. [07]. Bu,oachievehesae-of-he-ardynamicregre,i.e.,Omin{PT,S T },hevarianofogdinzhang e al. [07] has o query Oκ gradiens for every ieraion. Here, κ := β represens he condiion number for he β smooh and srongly convex objecive funcion f. For a large κ, he exremely large query complexiy makes i no pracical in he online seing. In he paper, we invesigae he basic online gradien descen, and provide a new heoreical analysis framework. Using he new analysis framework, we show ha he dynamic regre Omin{PT,S T } can be achieved wih O, insead of Oκ queries of gradiens in Zhang e al. [07]. Main heoreical resuls are oulined in Table briefly. The improvemen of he query complexiy is vially imporan for ill-condiioned problems Taranola [004] whose objecive funcion usually has a large condiion number, i.e., κ. Le us ake he image deblurring problem as an example. Suppose we have a blurred image y, which is modeled by using an unknown real image x and a blurring marix A. Tha is, y = Ax. Here, A is usually a non-singular marix wih a large condiion number, e.g., κ = 0 6. We wan o recover he real image x from he blurred image y, ha is, x = A y. Comparing wih he mehod in Zhang e al. [07], our new analysis frameworkshows ha OGD is good enough, and he required queries of gradiens can be reduced by muliple orders. The paper is organized as follows. Secion reviews he relaed work. Secion 3 presens he preliminaries. Secion 4 presens our heoreical analysis framework. Secion 5 presens he improved bounds of regre and query complexiy for he srongly convex case. Secion 6 concludes he paper. Relaed work. Regres of OGD in he saic environmen. Online gradien descen in he saic environmen has been exensively invesigaed over he las en years. The sublinear saic regres for smooh or srongly convex funcions have been obained in many lieraures Shalev-Shwarz [0], Hazan [06], Duchi e al. [0], Zinkevich [003]. Specifically, when f is srongly convex, he regre of online gradien descen is OlogT Hazan [06]. When f is convex bu no srongly convex, he regre of online gradien descen is O T Hazan [06].. Regres of OGD in he dynamic environmen. When all he funcions f,...,f T are srongly-convex and β smooh, he dynamic regre of OGD is OPT Mokhari e al. [06], Yang e al. [06]. If OGD queries Oκ gradiens a every ieraion, he dynamic regre of OGD can be improved o be Omin{PT,S T } Zhang e al. [07]. Bu, our analysis framework shows ha he O gradien queries for every ieraion is enough o obain Omin{PT,S T } dynamic regre. Addiionally, here are some oher regulariies including he funcional variaion Zhu and Xu [05], Besbes, Omar e al. [05] and he gradien variaion Chiang e al. [0]. Those regulariies measure differen aspecs of he variaion in he dynamic environmen. Since hey are no comparable direcly, some ill-condiioned may be noaed by ill-posed or badly posed in some lieraures.

3 researchers consider o bound he dynamic regre by using he mixed regulariy Jadbabaie e al. [05]. Exending our heoreical framework o differen regulariies is an ineresing avenue for fuure work. Besides, he new proposed heoreical analysis framework is inspired by Joulani e al.[07]. Joulani e al. [07] provides a heoreical analysis framework in he saic environmen, bu our heoreical analysis framework works in he dynamic environmen. 3 Preliminaries 3. Noaions and assumpions We use he following noaion. The bold lower-case leers, e.g., x represen vecors. The normal leers, e.g., β represen a scalar number. η represens he learning rae of Algorihm a he -h ieraion, and η min := min{η,...,η T }. The condiion number κ is defined by κ := β for any β smooh and srongly convex funcion f. represens he l norm of a vecor. Π X represens he projecion o a se X. X := argmin x X f x represens he minimizer se a he -h ieraion. Bregman divergence B f x,y is defined by B f x,y := fx fy fy,x y for any funcion f. In he paper, funcions {f } T are assumed o be convex and β smooh defined as follows. Definiion β smoohness. A funcion f : X R is β smooh, if, for any x X and y X, we have fy fx+ fx,y x + β y x. If he funcion f is β smooh, according o he definiion of he Bregmandivergence, we have B f x,y β x y holds for any x X and y X. The oher assumpions used in he paper are presened as follows. Assumpion srong convexiy. For any, he funcion f : X R is srongly convex. Tha is, for any x X and y X, f y f x+ f x,y x + y x. Assumpion Boundedness of gradiens. We assume f x G for any. Assumpion 3 Boundedness of he domain of x. We assume x x R for any. The above assumpions, i.e., Assumpions -3, are he basic assumpions, which are used widely in previous researches Shalev-Shwarz [0], Hazan [06], Duchi e al. [0], Zinkevich [003]. Addiionally, we make he following assumpion, which is used o model he dynamic environmen. Assumpion 4 Boundedness of variaions in he dynamic environmen.. For any i [T] and j [T], here exiss V such ha x i+ x i V x j+ x j. Assumpion 4 models he dynamic environmen by using V. A small V means he environmen changes mildly. The environmen changes significanly wih he increase of V. 3

4 Algorihm OGD: Online Gradien Descen. Require: The learning rae η wih T. : for =,,...,T do : Submi x X and receive he funcion f wih f : X R. 3: Query he gradien f x of f. 4: x + = Π X x η f x. reurn x T+ Algorihm OMGD: Online Muliple Gradien Descen Zhang e al. [07]. Require: The learning rae η wih T. : for =,,...,T do : Submi x X and receive he funcion f wih f : X R. 3: z = x, and K = κ+. 4: for j =,,...,K do 5: Query he gradien f z j of f. 6: z j+ = Π X z j η f z j. 7: x + = z K+. reurn x T+ 3. Algorihm Recall he algorihm of he OGD. A he -h ieraion, i submis x, and receives he loss funcion f x. Querying he gradien of f x, i updaes x by using he projeced gradien descen mehod. The deails are presened in Algorihm. Comparing wih he sae-of-he-ar mehod, i.e., Algorihm, OGD only requires one query of gradien for everyieraion, while Algorihm requires κ+ queries of gradien. When κ is large, he query complexiy of Algorihm is much higher han OGD. Comparing wih OMGD, i.e., Algorihm, our new heoreical analysis framework shows ha OGD is good enough o recover he sae-of-he-ar dyanmic regre yielded by OMGD, bu i only leads o O query of gradien, insead of Oκ queries of gradien required by OMGD. 4 A new heoreical analysis framework In he secion, we firs provide a modular analysis framework, which does no depend on he assumpion on he funcions. Then, equipped wih he srongly convex assumpion, i yields specific resuls. 4. High-level hough Our original goal is equivalen o invesigae wheher he basic OGD, i.e., Algorihm can obain he saeof-he-ar dynamic regre, i.e., min{pt,s T }. Using he divide-and-conrol sraegy, we divide he dynamic regre of OGD ino wo pars.. The firs par, denoed by RT o, is caused by he online seing in he dynamic environmen. I does no depend on he srongly convex assumpion on he funcion f.. The second par, denoed by RT m, is due o he projeced gradien descen sep in Algorihm. I depends on he assumpion on he funcion f such as convexiy or srong convexiy. 4

5 In he paper, our firs conribuion is o provide an upper bound of RT o wihou he srongly convex assumpion of f. Then, benefiing from he rich heoreical ools in he saic opimizaion, we successfully bound RT m by using he srongly convex assumpion of f. 4. Mea framework Generally, he dynamic regre of OGD is bounded as follows. Theorem. For any η > 0 in Algorihm, he dynamic regre of OGD defined in is bounded by R T R o T +R m T where and R m T := T R o T := x x + + x x η η B ηf x,x +η f x f x + + η βη x + x. In Theorem, RT o represens he regre due o he online seing, and Rm T he projeced gradien descen updaing sep in Algorihm. represens he regre due o Remark. Noe ha he upper bound of R m T depends on he srongly convex assumpion of he funcion f. Theorem. Use Assumpion 4, and se η > 0 in Algorihm. Denoe x 0 = x and η min = min{η,...,η T }. For any 0 < ρ, he regre due o he online seing, i.e., RT o is bounded by RT o ρ+ρv η min ρ S T + x η x + T x + η + η x +. Remark. Noe ha his upper bound of RT o does no depend on he srongly convex assumpion of he funcion f. I sill holds for he convex funcion f. LemmaAppearedinProposiioninMokhari e al.[06]. Use Assumpion. Le v + = Π X v η f v and X := argmin v X f v. Denoe κ = β. If η β and ρ = κ κ, we have v + x ρ v x. According o Lemma, when f s are srongly convex, 0 < ρ < See Lemma. When f s are jus convex, ρ = ha is, 0. Recall ha RT m depends on he srongly convex assumpion of f s. Equipped by Lemma, we find ha as long as RT m is furher bounded, we are able o provide an upper bound for he dynamic regre. 5 Improved query complexiy for srongly convex f When all f s are smooh and srongly convex, he dynamic regre of our mehod OGD is upper bounded by he following heorem. Theorem 3. Use Assumpions,, 3 and 4. Seing η = η = β+β / in Algorihm, and ρ = κ κ <, we bound he dynamic regre of OGD as R T min{j,j }, 5

6 where and ρ+ρv J = S T + β + β ρ f x, ST + β + β x x + J = G x x PT ρ + G ρ P T. β + β f x Corollary. Suppose T bounded by f x = OST. According o Theorem 3, he dynamic regre of OGD is R T min{j,j } min{p T,S T}. Proof. RecallAssumpion3, andwehave x x R. When T f x = OST, wehavej ST. Similarly, we have J G R ρ P T + G ρ P T. Thus, we finally obain I complees he proof. R T min{j,j } min{p T,S T }. Recall he previous mehod, i.e., Algorihm. Is dynamic regre has been proved, and we presen i as follows. Lemma Appeared in Theorem 3 and Corollary 4 in Zhang e al. [07].. Use Assumpions,, and 3, and choose η β in Algorihm. Denoe he dynamic regre of Algorihm by R T. Then, for any consan σ > 0, R T where is bounded by R T min{j 3,J 4 }, J 3 =GPT +G x x P T, J 4 = f x σ +β +σ S T + x x S T + f x. Furhermore, suppose T f x = OST, and we hus have R T min{p T,S T }. Comparing wih Lemma, our new resul achieves he same bound of he regre. Bu, OGD, i.e., Algorihm, only requires one query of gradien for every ieraion, which does no depend on κ, and hus ouperforms Algorihm by reduing he query complexiy significanly. The following remarks highligh he advanages of our analysis framework. 6

7 Remark 3. Our analysis framework achieves he sae-of-he-ar dynamic regre presened in Zhang e al. [07] wih a consan facor, and ouperforms he dynamic regre OPT presened in Mokhari e al. [06]. Remark 4. Our analysis framework shows ha O queries of gradiens for every ieraion is enough o achieve he sae-of-he-ar dynamic regre, bu Zhang e al. [07] requires Oκ queries of gradiens for every ieraion. 6 Conclusion We provide a new heoreical analysis framework o analyze he regre and query complexiy of OGD in he dynamic environmen. Comparing wih he previous work, our framework achieves he sae-of-he-ar dynamic regre, and improve he required queries of gradien o be O. Proof of heorems. Proof of Theorem : Proof. T RT = η f x η f x η = η f x,x + x η B ηf }{{} x,x + η f x,x x +. η }{{} I I Now, we begin o bound I. According o Lemma 4, we obain I Afer ha, we begin o bound I. x x + + x x x + x. 3 I = η f x,x x + = η f x η f x + +η B f x +,x η f x f x + + βη x + x. 4 The las inequaliy holds because ha all f s are β smooh. Subsiuing 3 and 4 ino, we finally complee he proof. Proof of Theorem : Proof. According o he cosine heorem, we have x x + + x + x + x + x x+ x + x + x. 5 According o Lemma, if f is convex and smooh, x + x ρ x x holds for 0 < ρ. Specifically, 0 < ρ < holds when f is srongly convex, and ρ = holds when f is jus convex. We hus have x + x x + x + x + x ρ x x + x + x +. 7

8 Le A + = x+ x +, M+ = x + x, and we hus have ha is, A + M + ρ A. Thus, we have Summing up, we obain A + M + M + A + ρ A, A + M + ρa ρa ρm ρ A ρ A ρ M ρ A. A + ρ A + M + +ρm +...+ρ M + =ρ x x + ρ + i x i x i = holds due o leing x 0 = x. Subsiuing 6 ino 5, we obain, + i= i= ρ + i x i x i = x + x + ρ i x + i x i. 6 i= x x + + x + x + x + x A+ x + x x + x + x + x ρ i x + i x i i= x + x +V x + x ρ i i= Thus, we obain x + x + ρv x ρ + x = ρ+ρv x + ρ x. 7 R o T = T = x x + + x x η x x + + x η + x + + T η + η x + x + + x η x x T η x T+ T T x η x + + x + x + + T x η x + x η + η 8 + x +

9 T η ρ+ρv x ρ + x + T x x + η η + η x + η + η x + + x η x. ρ+ρv η min ρ S T + T x + x + Here, η min = min{η,η,...,η T }. holds due o 7. Dividing on boh sides, we complee he proof. Proof of Theorem 3: Proof. When he funcion f is srongly convex, we have B f x,x x x. 8 Subsiuing 8 ino Theorem, we obain R T η + + x x + + η x x + η η f x f x + x η x + + x x η β + η + β x + x + η x x + + x x + η ρ+ρv β + β ρ η η f x η f x ST + β + β x x + βη x + x β + β f x. holds due o 3 in Lemma 5 by seing θ = and θ = η. holds because of η = T. The las inequaliy holds due o Theorem. Combining Lemma 6, we finally complee he proof. β+ β for Proof of lemmas. Lemma 3. Denoe hx = η f x,x + x x. If x + = Π X x η f x, we have x + Argminhx. x X Proof. Consider he following convex opimizaion problem minhx = min η f x,x + x X x X x x 9 9

10 Denoe he opimum se is X, ha is, for any x X, hx = min x X hx holds. According o he firs-order opimaliy condiion Boyd and Vandenberghe [004], we have, for any z X and x X, 0 hx,z x Recall ha x + = Π X x η f x. Thus, we have = η f x +x x,z x. 0 η f x +x + x,z x + = Π X x η f x x η f x,z x + 0. Tha is, x + saisfies he firs-order opimaliy condiion of 9. I complees he proof. Lemma 4. Use Assumpion 3. For any minimizer x X and X := argmin x X f x, we have η f x,x + x x x + x + x + x x. Proof. Firs, we consruc an auxiliary funcion h = η f x, + x. According o Lemma 3, we have x + Argminhx. Le x = x + +τx x + wih τ 0,]. x X 0 h x hx + Dividing τ on boh sides, we obain = η f x,τx x + + x x + +τ x,x + x. 0 η f x,x x + + x x + + x,x + x τ η f x,x x + + lim τ 0 + τ x x + + x,x + x τ = η f x,x x + + lim τ 0 + x x + + x +,x x + + x,x + x = η f x,x x + + x +,x x + + x,x + x = η f x,x x + x x + x + x + x x. holds because ha holds for any τ 0,]. Re-arranging he iems, we prove he conclusion. Lemma 5. Suppose ha all f s are β smooh. For any θ > 0, θ > 0 and any minimizer x argmin x X f x, we have Proof. For any θ > 0 and θ > 0, we have f x f x + 3 θ x x β + + x + x + θ θ θ f x. f x f x + θ f x x x + θ f x,x x + f x,x x + 0

11 θ β f x f x + β θ x + x θ x x + β θ x + x. The las inequaliy holds because ha all f s are β smooh. Re-arranging he iems, we hus complee he proof. Lemma 6 Appeared in Theorem in Mokhari e al. [06]. Suppose ha Assumpions -3 hold, and all κ f are β smooh. Thus, ρ = κ < wih κ := β. Se η L in OGD, i.e., Algorihm. The dynamic regre of OGD is bounded as Acknowledgmen RT G x x PT + G ρ ρ. This work was suppored by he Naional Key R & D Program of China 08YFB00303and he Naional Naural Science Foundaion of China Gran No , , and References A. S. Bedi, P. Sarma, and K. Rajawa. Tracking moving agens via inexac online gradien descen algorihm. IEEE Journal of Seleced Topics in Signal Processing, :0 7, Feb 08. Besbes, Omar, Gur, Yonaan, and Zeevi, Assaf J. Non-Saionary Sochasic Opimizaion. Operaions Research, 635:7 44, 05. S. Boyd and L. Vandenberghe. Convex Opimizaion. Cambridge Universiy Press, New York, NY, USA, 004. ISBN C. K. Chiang, T. Yang, C. J. Lee, M. Mahdavi, C. J. Lu, R. Jin, and S. Zhu. Online opimizaion wih gradual variaions. Journal of Machine Learning Research, 3, 0. J. Duchi, E. Hazan, and Y. Singer. Adapive subgradien mehods for online learning and sochasic opimizaion. Journal of Machine Learning Research JMLR, : 59, 0. D. Garber. Fas Raes for Online Gradien Descen Wihou Srong Convexiy via Hoffman s Bound. arxiv.org, 08. E. C. Hall and R. M. Wille. Online Convex Opimizaion in Dynamic Environmens. IEEE Journal of Seleced Topics in Signal Processing, 94:647 66, 05. E. Hazan. Inroducion o online convex opimizaion. Foundaions and Trends in Opimizaion, 3-4: 57 35, 06. E. Hazan and C. Seshadhri. Adapive algorihms for online decision problems. Elecronic Colloquium on Compuaional Complexiy, 4, 007. A. Jadbabaie, A. Rakhlin, S. Shahrampour, and K. Sridharan. Online Opimizaion : Compeing wih Dynamic Comparaors. In Proceedings of Inernaional Conference on Arificial Inelligence and Saisics AISTATS, pages , 05.

12 P. Joulani, A. György, and C. Szepesvári. A Modular Analysis of Adapive Non-Convex Opimizaion - Opimism, Composie Objecives, and Variaional Bounds. In Proceedings of Inernaional Conference on Algorihmic Learning Theory ALT, 07. Y. Lei, L. Shi, and Z.-C. Guo. Convergence of unregularized online learning algorihms. Journal of Machine Learning Research, 8: , Jan. 07. A. Mokhari, S. Shahrampour, A. Jadbabaie, and A. Ribeiro. Online opimizaion in dynamic environmens: Improved regre raes for srongly convex problems. In Proceedings of IEEE Conference on Decision and Conrol CDC, pages IEEE, 06. T. Moon, L. Li, W. Chu, C. Liao, Z. Zheng, and Y. Chang. Online learning for recency search ranking using real-ime user feedback. In Proceedings of ACM Inernaional Conference on Informaion and Knowledge Managemen CIKM, CIKM 0, pages , New York, NY, USA, 00. ACM. S. Shalev-Shwarz. Online Learning and Online Convex Opimizaion. Foundaions and Trends R in Machine Learning, 4:07 94, 0. Y. Song, Z. Zhuang, H. Li, Q. Zhao, J. Li, W.-C. Lee, and C. L. Giles. Real-ime auomaic ag recommendaion. In Proceedings of ACM SIGIR Conference on Research and Developmen in Informaion Rerieval, SIGIR 08, pages 55 5, New York, NY, USA, 008. A. Taranola. Inverse Problem Theory and Mehods for Model Parameer Esimaion. Sociey for Indusrial and Applied Mahemaics, Philadelphia, PA, USA, 004. ISBN T. Yang, L. Zhang, R. Jin, and J. Yi. Tracking Slowly Moving Clairvoyan - Opimal Dynamic Regre of Online Learning wih True and Noisy Gradien. In Proceedings of he 34h Inernaional Conference on Machine Learning ICML, 06. L. Zhang, T. Yang, J. Yi, R. Jin, and Z.-H. Zhou. Improved Dynamic Regre for Non-degenerae Funcions. In Proceedings of Neural Informaion Processing Sysems NIPS, 07. C. Zhu and H. Xu. Online Gradien Descen in Funcion Space. arxiv.org, 05. M. Zinkevich. Online convex programming and generalized infiniesimal gradien ascen. In Proceedings of Inernaional Conference on Machine Learning ICML, pages , 003.