Convergence rates of proximal gradient methods via the convex conjugate

Transcription

1 Convergence rates of proxmal gradent methods va the convex conjugate Davd H Gutman Javer F Peña January 8, 018 Abstract We gve a novel proof of the O(1/ and O(1/ convergence rates of the proxmal gradent and accelerated proxmal gradent methods for composte convex mnmzaton The crux of the new proof s an upper bound constructed va the convex conjugate of the objectve functon 1 Introducton The development of accelerated versons of frst-order methods has had a profound nfluence n convex optmzaton In hs semnal paper [9] Nesterov devsed a frst-order algorthm wth optmal O(1/ rate of convergence for unconstraned convex optmzaton va a modfcaton of the standard gradent descent algorthm that ncludes momentum steps A later breathrough was the acceleraton of the proxmal gradent method ndependently developed by Bec and Teboulle [] and by Nesterov [11] The proxmal gradent method, also nown as the forward-bacward method [8], s an extenson of the gradent descent method to solve the composte mnmzaton problem mn ϕ(x + ψ(x (1 x Rn where ϕ : R n R s dfferentable and ψ : R n R { } s a closed convex functon such that for t > 0 the proxmal map Prox t (x := arg mn {ψ(y + 1t } x y ( y R n s computable The sgnfcance of Nesterov s and Bec and Teboulle s breathroughs has prompted nterest n new approaches to explan how acceleraton s acheved n frst-order methods [1,3 5,7,1,13] Some of these approaches are based on geometrc [3,4], control [7], and dfferental Department of Mathematcal Scences, Carnege Mellon Unversty, USA, dgutman@andrewcmuedu Tepper School of Busness, Carnege Mellon Unversty, USA, jfp@andrewcmuedu 1

2 equatons [13] technques The recent artcle [1] reles on the convex conjugate to gve a unfed and succnct dervaton of the O(1/, O(1/, and O(1/ convergence rates of the subgradent, gradent, and accelerated gradent methods for unconstraned smooth convex mnmzaton The crux of the approach n [1] s a generc upper bound on the terates generated by the subgradent, gradent, and accelerated gradent algorthms constructed va the convex conjugate of the objectve functon We extend the man constructon n [1] to gve a unfed dervaton of the convergence rates of the proxmal gradent and accelerated proxmal gradent algorthms for the composte convex mnmzaton problem (1 As n [1], the central result of ths paper (Theorem 1 s an upper bound on the terates generated by both the non-accelerated and the accelerated proxmal gradent methods Ths bound s constructed va the convex conjugate of the objectve functon Theorem 1 readly yelds the wdely nown O(1/ and O(1/ convergence rates of the proxmal gradent and accelerated proxmal gradent algorthms for (1 when the smooth component ϕ has Lpschtz gradent and the step szes are chosen judcously Theorem 1 hghlghts some ey smlartes and dfferences between the non-accelerated and the accelerated algorthms It s noteworthy that Theorem 1 and ts varant, Theorem, hold under certan condtons on the step szes and momentum used n the algorthm but do not requre any Lpschtz assumpton The convex conjugate approach underlyng Theorem 1 also extends to a proxmal subgradent algorthm when the component ϕ s merely convex but not necessarly smooth (See Algorthm and Proposton 1 Ths extenson automatcally yelds a novel dervaton of both classcal [10, Theorem 3] as well as modern convergence rates [6, Theorem 5] for the projected subgradent algorthm The latter dervatons are smlar to the dervaton of the convergence rates for the proxmal gradent and accelerated proxmal gradent algorthms Throughout the paper we assume that R n s endowed wth an nner product, and that denotes the correspondng Eucldean norm Proxmal gradent and accelerated proxmal gradent methods Let ϕ : R n R be a dfferentable convex functon and ψ : R n R { } be a closed convex functon such that the proxmal map ( s computable Let f := ϕ + ψ and consder the problem (1 that can be rewrtten as mn f(x (3 x Rn Algorthm 1 descrbes a template of a proxmal gradent algorthm for (3 Step 7 of Algorthm 1 ncorporates a momentum step The (non-accelerated proxmal gradent method s obtaned by choosng θ +1 = 1 n Step 6 In ths case Step 7 smply sets y +1 = x +1 and does not ncorporate any momentum Other choces of θ +1 (0, 1] yeld accelerated versons of the proxmal gradent method In partcular, the FISTA algorthm n [] s obtaned by choosng θ +1 (0, 1] va the rule θ +1 = θ (1 θ +1 In ths case θ (0, 1 for 1 and there s a non-trval momentum term n Step 7

3 Algorthm 1 Template for proxmal gradent method 1: nput: x 0 R n : y 0 := x 0 ; θ 0 := 1 3: for = 0, 1,, do 4: pc t > 0 5: x +1 := Prox t (y t ϕ(y 6: pc θ +1 (0, 1] 7: y +1 := x +1 + θ +1(1 θ θ (x +1 x 8: end for The man result n ths paper s Theorem 1 below whch subsumes the wdely nown convergence rates O(1/ and O(1/ of the proxmal gradent and accelerated proxmal gradent algorthms under sutable choces of t, θ, = 0, 1, Theorem 1 reles on a sutable constructed sequence z R n, = 1,, The constructon of z R n, = 1,, n turn s motvated by the dentty (5 below Consder Step 5 n Algorthm 1, namely The optmalty condtons for (4 mply that x +1 = Prox t (y t ϕ(y (4 x +1 = y t g where g := g ϕ + gψ for gϕ := ϕ(y and for some g ψ ψ(x +1 Step 5 and Step 7 of Algorthm 1 mply that for = 0, 1, y +1 (1 θ +1 x +1 θ +1 = x +1 (1 θ x θ Snce θ 0 = 1 and y 0 = x 0, t follows that for = 1,, y (1 θ x θ = x 0 f(x +1 mn x R n 1 =0 t θ g (1 θ (y x = θ = y (1 θ x θ ( x 0 y t θ g 1 =0 t θ g (5 As t s customary, we wll assume that the step szes t chosen at Step 4 n Algorthm 1 satsfy the followng decrease condton { ϕ(y + ϕ(y, x y + 1 = ϕ(y + ψ(x +1 + g ψ, y x +1 } x y + ψ(x t t g (6 The condton (6 holds n partcular when ϕ s Lpschtz and t, = 0, 1, are chosen va a standard bactracng procedure Observe that (6 mples f(x +1 f(y Theorem 1 also reles on the convex conjugate functon Recall that f h : R n R { } s a convex functon then ts convex conjugate h : R n R { } s defned as h (z = sup x R n { z, x h(x} 3

4 Theorem 1 Suppose θ (0, 1], = 0, 1,, and the step szes t > 0, = 0, 1,, are such that (6 holds Let x R n, = 1,, be the terates generated by Algorthm 1 Let z R n, = 1, be as follows Then z := 1 t θ g =0 1 t θ =0 (7 1 =0 LHS f (z + z, x 0 z, (8 where LHS s as follows dependng on the choce of θ (0, 1] and t > 0 (a When θ = 1, = 0, 1, let t θ LHS := =0 t f(x +1 =0 t (b When t > 0 and θ (0, 1], = 0, 1, are such that 1 =0 LHS = f(x t θ = (1 θ t =0 θ Theorem 1 readly mples that n both case (a and case (b { } LHS mn {f(u z 1, u} + mn z, u + u R n u R n 1 u x t 0 =0 θ { } 1 mn f(u + u R n 1 u x t 0 =0 θ 1 f(x + 1 x x t 0 =0 θ for all x R n Let f and X respectvely denote the optmal value and set of optmal solutons to (3 If f s fnte and X s nonempty then n both case (a and case (b of Theorem 1 we get f(x f dst(x 0, X 1 (9 t =0 θ Suppose t 1, = 0, 1,, for some constant L > 0 Ths holds n partcular f ϕ s L Lpschtz and t s chosen va a standard bactracng procedure Then nequalty (9 yelds the followng nown convergence bound for the proxmal gradent method f(x f L dst(x 0, X 4 let

5 On the other hand, suppose t = 1, = 0, 1,, for some constant L > 0 and θ L, = 0, 1,, are chosen va θ 0 = 1 and θ+1 = θ (1 θ +1 Then a straghtforward nducton shows that 1 =0 t θ = (1 θ =0 t = 1 θ Lθ 1 ( + 1 4L Thus case (b n Theorem 1 apples and nequalty (9 yelds the followng nown convergence bound for the accelerated proxmal gradent method f(x f L dst(x 0, X ( + 1 Although Theorem 1 yelds the conc O(1/ convergence rate of the accelerated proxmal gradent algorthm, t apples under the somewhat restrctve condtons stated n case (b above In partcular, case (b does not cover the more general case when t, = 0, 1, are chosen va bactracng as n the FISTA wth bactracng algorthm n [] The convergence rate n ths case, namely [, Theorem 44] s a consequence of Theorem below Theorem s a varant of Theorem 1(b that apples to more flexble choces of t, θ, = 0, 1, In partcular, Theorem apples to the popular choce θ =, = 0, 1, + Theorem Suppose f = mn f(x s fnte, θ x R n (0, 1], = 0, 1,, satsfy θ 0 = 1 and θ+1 θ (1 θ +1, and the step szes t > 0, = 0, 1,, are non-ncreasng and such that (6 holds Let x R n, = 1,, be the terates generated by Algorthm 1 Let z R n, = 1,, be as follows Then for = 1,, where R 1 = 1 and R +1 = t 1 z = θ 1 1 t g t 1 θ =0 f(x f (R (f f (z + z, x 0 t 1 z θ 1, (10 t R n : f(x = f} s nonempty then f(x f mn u R n θ θ 1 (1 θ R 1, = 1,, In partcular, f X = {x { R (f(u f + θ 1 t 1 u x 0 } = θ 1 dst(x 0, X t 1 Suppose the step szes t, = 0, 1,, are non-ncreasng, satsfy (6, and t 1, = L 0, 1,, for some constant L > 0 Ths holds n partcular when ϕ s Lpschtz and t s chosen va a sutable bactracng procedure as the one n [] If θ 0 = 1 and θ+1 θ (1 θ +1, = 0, 1, then Theorem mples that f(x f Lθ 1 dst(x 0, X Hence f θ+1 = θ (1 θ +1 or θ = for = 0, 1, then + f(x f L dst(x 0, X ( + 1 5

6 3 Proof of Theorem 1 and Theorem We wll use the followng propertes of the convex conjugate Suppose h : R n R { } s a convex functon Then h (z + h(x z, x (11 for all z, x R n, and equalty holds f z h(x Suppose f, ϕ, ψ : R n R { } are convex functons and f = ϕ + ψ Then f (z ϕ + z ψ ϕ (z ϕ + ψ (z ψ for all z ϕ, z ψ R n (1 Suppose f : R n R + { } s a convex functon and R 1 Then (R f (Rz = R (f (z, (13 and (R f (z f (z (14 31 Proof of Theorem 1 1 We prove (8 by nducton To ease notaton, let µ := 1 t throughout ths proof For =0 θ = 1 we have LHS 1 = f(x 1 ϕ(x 0 + ψ(x 1 + g ψ0, x 0 x 1 t 0 g 0 = ϕ(x 0 g ϕ 0, x 0 + ψ(x 1 g ψ0, x 1 = ϕ (g ϕ 0 ψ (g ψ 0 + g 0, x 0 t 0 g 0 f (z 1 + z 1, x 0 z 1 µ 1 + g 0, x 0 t 0 g 0 The frst step follows from (6 The thrd step follows from (11 and g ϕ 0 = ϕ(x 0, g ψ 0 ψ(x 1 The last step follows from (1 and the choce of z 1 = g 0 = g ϕ 0 + g ψ 0 and µ 1 = 1 t 0 Suppose (8 holds for and let γ = t /θ =0 t /θ The constructon (7 mples that Therefore, z +1, x 0 z +1 µ +1 = (1 γ z +1 = (1 γ z + γ g µ +1 = (1 γ µ ( z, x 0 z µ ( +γ g, x 0 z γ g µ (1 γ µ (15 6

7 In addton, the convexty of f, propertes (11, (1, and g ϕ = ϕ(y, g ψ ψ(x +1, g = g ϕ + gψ mply f (z +1 (1 γ f (z γ f (g (1 γ f (z γ (ϕ (g ϕ + ψ (g ψ (16 = (1 γ f (z γ ( g ϕ, y ϕ(y + g ψ, x +1 ψ(x +1 Let RHS denote the rght-hand sde n (8 From (15 and (16 t follows that RHS +1 (1 γ RHS ( γ g, x 0 y z + ϕ(y + ψ(x +1 + g ψ µ, y x +1 Hence to complete the proof of (8 by nducton t suffces to show that LHS +1 (1 γ LHS ( γ g, x 0 y z + ϕ(y + ψ(x +1 + g ψ µ, y x +1 To that end, we consder case (a and case (b separately Case (a In ths case γ = x 0 y z µ = 0 Therefore t =0 t 1 and y = x Thus µ = 1, =0 t γ (1 γ µ g γ (1 γ µ g γ (1 γ µ (17 (18 = t, and LHS +1 (1 γ LHS = γ f(x +1 γ (ϕ(y + ψ(x +1 + g ψ, y x +1 t g = γ (ϕ(y + ψ(x +1 + g ψ, y γ x +1 g (1 γ µ ( = γ g, x 0 y z + ϕ(y + ψ(x +1 + g ψ µ, y γ x +1 g (1 γ µ The second step follows from (6 The thrd and fourth steps follow from x 0 y z µ = 0 respectvely Thus (18 holds n case (a Case (b In ths case γ = θ and γ (1 γ µ = t Therefore γ (1 γ µ LHS +1 (1 γ LHS = f(x +1 (1 γ (ϕ(x + ψ(x ϕ(y + ψ(x +1 + g ψ, y x +1 t g ( (1 γ ϕ(y + g ϕ, x y + ψ(x +1 + g ψ, x x +1 = γ (ϕ(y + ψ(x +1 + g ψ, y x +1 + (1 γ g, y x t g ( = γ g, x 0 y z + ϕ(y + ψ(x +1 + g ψ µ, y x +1 7 = t and γ (1 γ µ g

8 The second step follows from (6 and the convexty of ϕ and ψ The last step follows from θ = γ, equaton (5, and = t Thus (18 holds n case (b as well 3 Proof of Theorem γ (1 γ µ The proof of Theorem s a modfcaton of the proof of Theorem 1 Wthout loss of generalty assume f = 0 as otherwse we can wor wth f f n place of f Agan we prove (10 by nducton To ease notaton, let µ := θ 1 t 1 throughout ths proof For = 1 nequalty (10 s dentcal to (8 snce R 1 = 1 and θ 0 = 1 Hence ths case follows from the proof of Theorem 1 for = 1 Suppose (10 holds for Observe that for ρ := R +1 R Frst, = t 1 t θ θ 1 (1 θ = µ +1 µ (1 θ z +1 = ρ (1 θ z + θ g µ +1 = ρ (1 θ µ 1 Next, proceed as n the proof of Theorem 1 z +1, x 0 z +1 = ρ (1 θ ( z, x 0 z + θ g, x 0 z µ +1 µ µ = ρ (1 θ ( z, x 0 z + θ g, x 0 z µ µ Second, the convexty of f and the fact that f f = 0 mply (R +1 f (z +1 (1 θ (R +1 f (ρ z θ (R +1 f (g θ µ +1 g t g (19 (1 θ (ρ R f (ρ z θ f (g (0 ρ (1 θ (R f (z θ (ϕ (g ϕ + ψ (g ψ = ρ (1 θ (R f (z θ ( g ϕ, y ϕ(y + g ψ, x +1 ψ(x +1 The frst step follows from the convexty of f The second step follows from (14 The thrd step follows from (1 and (13 The last step follows from (11 and g ϕ = ϕ(y, g ψ ψ(x +1 Let RHS denote the rght-hand sde n (10 The nducton hypothess mples that RHS f(x 0 Thus from (19, (0, and ρ 1 t follows that RHS +1 (1 θ RHS RHS +1 ρ (1 θ RHS (1 ( θ g, x 0 y z + ϕ(y + ψ(x +1 + g ψ µ, y x +1 t g 8

9 Fnally, proceedng exactly as n case (b n the proof of Theorem 1 we get f(x +1 (1 θ f(x θ (ϕ(y + ψ(x +1 + g ψ, y x +1 + (1 θ g, y x t g ( = θ g, x 0 y z + ϕ(y + ψ(x +1 + g ψ µ, y x +1 t g RHS +1 (1 θ RHS The second step follows from (5 The thrd step follows from (1 Ths completes the proof by nducton 4 Proxmal subgradent method Algorthm descrbes a varant of Algorthm 1 for the case when ϕ : R n R s merely convex Algorthm Proxmal subgradent method 1: nput: x 0 R n : for = 0, 1,, do 3: pc g ϕ ϕ(x and t > 0 4: x +1 := Prox t (x t g ϕ 5: end for When ψ s the ndcator functon I C of a closed convex set C, Step 4 n Algorthm can be rewrtten as x +1 = arg mn x t g ϕ x = Π C(x t g ϕ Hence when ψ = I C x C Algorthm becomes the projected subgradent method for mn ϕ(x ( x C The classcal convergence rate for the projected gradent s an mmedate consequence of Proposton 1 as we detal below Proposton 1 n turn s obtaned va a mnor twea on the constructon and proof of Theorem 1 Observe that where g = g ϕ + gψ x +1 = Prox t (x t g ϕ x +1 = x t g for some gψ ψ(x +1 Next, let z R n, = 0, 1, be as follows z = =0 t g =0 t (3 Proposton 1 Let x R n, = 0, 1,, be the sequence of terates generated by Algorthm and let z R n, = 0, 1, be defned by (3 Then for = 0, 1,, =0 t (ϕ(x + ψ(x +1 1 =0 t g ϕ =0 t f =0 (z + z, x 0 t z (4 { } 1 mn f(u + u R n =0 t u x 0 9

10 In partcular, for all x R n =0 t (ϕ(x + ψ(x +1 1 =0 t g ϕ =0 t f(x + x 0 x =0 t Proof Let LHS and RHS denote respectvely the left-hand and rght-hand sdes n (4 We proceed by nducton For = 0 we have LHS 0 = ϕ(x 0 + ψ(x 1 t 0 g ϕ 0 = ϕ (g ϕ 0 + g ϕ 0, x 0 ψ (g ψ 0 + g ψ0, x 1 t 0 g ϕ 0 f (g 0 + g 0, x 0 t 0 g 0 = RHS 0 The second step follows from (11 and g ϕ 0 ϕ(x 0, g ψ 0 ψ(x 1 The thrd step follows from (1 and g 0 = g ϕ 0 + g ψ 0, x 1 = x 0 t 0 g 0 Next we show the man nductve step to +1 Observe that z +1 = (1 γ z +γ g +1 for = 0, 1, where γ = t (0, 1 Proceedng exactly as n the proof of Theorem 1 =0 t we get RHS +1 (1 γ RHS γ (ϕ(x +1 + ψ(x + + g ψ+1, x +1 x + t +1 g +1 = γ ( ϕ(x +1 + ψ(x + + t +1 g ψ +1 t +1 g ϕ +1 The second step follows because g +1 = g ϕ +1 + gψ +1 and x + = x +1 t +1 g +1 The proof s thus completed by observng that ( LHS +1 (1 γ LHS = γ ϕ(x +1 + ψ(x + t +1 g ϕ +1 ( γ ϕ(x +1 + ψ(x + + t +1 g ψ +1 t +1 g ϕ +1 Let C R n be a nonempty closed convex set and ψ = I C As noted above, n ths case Algorthm becomes the projected subgradent algorthm for problem ( We next show that n ths case Proposton 1 yelds the classcal convergence rates (6 and (7, as well and the modern and more general one (8 recently establshed by Grmmer [6, Theorem 5] Suppose ϕ = mn ϕ(x s fnte and X := {x C : ϕ(x = ϕ} s nonempty From x C Proposton 1 t follows that =0 t (ϕ(x ϕ t g ϕ + dst(x 0, X (5 =0 10

11 In partcular, f g L for all x C and g ϕ(x then (5 mples mn (ϕ(x =0 ϕ t L + dst(x 0, X =0,, =0 t (6 Let α := t g ϕ, = 0, 1, Then Step 4 n Algorthm can be rewrtten as x +1 = g ϕ Π C (x α provded g ϕ g ϕ > 0, whch occurs as long as x s not an optmal soluton to ( If g ϕ > 0 for = 0, 1,, then (5 mples mn (ϕ(x =0 ϕ L α + dst(x 0, X =0,, =0 α (7 Let L : R + R + Followng Grmmer [6], the subgradent oracle for ϕ s L-steep on C f for all x C and g ϕ(x g L(ϕ(x ϕ As dscussed by Grmmer [6], L-steepness s a more general and weaer condton than the tradtonal bound g L for all x C and g ϕ(x Indeed, the latter bound s precsely L-steepness for the constant functon L(t = L and holds when ϕ s L-Lpschtz on C Suppose the subgradent oracle for ϕ s L-steep for some L : R + R + If α := t g ϕ > 0 for = 0, 1,, then (5 mples and thus ϕ(x ϕ α L(ϕ(x ϕ =0 mn (ϕ(x ϕ sup =0,, { t : =0 α + dst(x 0, X, t L(t =0 α + dst(x 0, X } =0 α (8 Acnowledgements Ths research has been funded by NSF grant CMMI References [1] Z Allen-Zhu and L Oreccha Lnear couplng: An ultmate unfcaton of gradent and mrror descent arxv preprnt arxv: , 014 [] A Bec and M Teboulle A fast teratve shrnage-thresholdng algorthm for lnear nverse problems SIAM Journal on Imagng Scences, (1:183 0, 009 [3] S Bubec, Y Lee, and M Sngh A geometrc alternatve to Nesterov s accelerated gradent descent arxv preprnt arxv: , 015 [4] D Drusvyatsy, M Fazel, and S Roy An optmal frst order method based on optmal quadratc averagng arxv preprnt arxv: ,

12 [5] N Flammaron and F Bach From averagng to acceleraton, there s only a step-sze In COLT, pages , 015 [6] B Grmmer Convergence rates for determnstc and stochastc subgradent methods wthout Lpschtz contnuty arxv preprnt arxv: , 017 [7] L Lessard, B Recht, and A Pacard Analyss and desgn of optmzaton algorthms va ntegral quadratc constrants SIAM Journal on Optmzaton, 6(1:57 95, 016 [8] P Lons and B Mercer Splttng algorthms for the sum of two nonlnear operators SIAM Journal on Numercal Analyss, 16(6: , 1979 [9] Y Nesterov A method for unconstraned convex mnmzaton problem wth rate of convergence O(1/ Dolady AN SSSR (n Russan (Englsh translaton Sovet Math Dol, 69: , 1983 [10] Y Nesterov Introductory Lectures on Convex Optmzaton: A Basc Course Appled Optmzaton Kluwer Academc Publshers, 004 [11] Y Nesterov Gradent methods for mnmzng composte functons Mathematcal Programmng, 140(1:15 161, 013 [1] J Peña Convergence of frst-order methods va the convex conjugate Operatons Research Letters, 45: , 017 [13] W Su, S Boyd, and E Candès A dfferental equaton for modelng Nesterov s accelerated gradent method: Theory and nsghts In Advances n Neural Informaton Processng Systems, pages , 014 1