FAST CONVERGENCE OF INERTIAL DYNAMICS AND ALGORITHMS WITH ASYMPTOTIC VANISHING DAMPING

Transcription

1 FAST CONVERGENCE OF INERTIAL DYNAMICS AND ALGORITHMS WITH ASYMPTOTIC VANISHING DAMPING HEDY ATTOUCH, ZAKI CHBANI, JUAN PEYPOUQUET, AND PATRICK REDONT Absrac. In a Hilber space seing H, we sudy he fas convergence properies as + of he rajecories of he second-order differenial equaion ẍ + α ẋ + Φx = g, where Φ is he gradien of a convex coninuously differeniable funcion Φ : H R, α is a posiive parameer, and g : [, + [ H is a small perurbaion erm. In his inerial sysem, he viscous damping coefficien α vanishes asympoically, bu no oo rapidly. For α 3, and + g d < +, jus assuming ha argmin Φ, we show 0 ha any rajecory of he above sysem saisfies he fas convergence propery Φx min Φ C H. Moreover, for α > 3, we show ha any rajecory converges wealy o a minimizer of Φ. The srong convergence is esablished in various pracical siuaions. These resuls complemen he O rae of convergence for he values obained by Su, Boyd and Candès in he unperurbed case g = 0. Time discreizaion of his sysem, and some of is varians, provides new fas converging algorihms, expanding he field of rapid mehods for srucured convex minimizaion inroduced by Neserov, and furher developed by Bec and Teboulle wih FISTA. This sudy also complemens recen advances due o Chambolle and Dossal.. Inroducion Le H be a real Hilber space, which is endowed wih he scalar produc, and norm, and le Φ : H R be a convex differeniable funcion. In his paper, we sudy he soluion rajecories of he second-order differenial equaion ẍ + α ẋ + Φx = 0, wih α > 0, in erms of heir asympoic behavior as +. This will serve us as a guideline for he sudy of he corresponding algorihms. We ae for graned he exisence and uniqueness of a global soluion o he Cauchy problem associaed wih. Alhough his is no our main concern, we poin ou ha, given > 0, for any x 0 H, v 0 H, he exisence of a unique global soluion on [, + [ for he Cauchy problem wih iniial condiion x = x 0 and ẋ = v 0 can be guaraneed, for insance, if Φ is Lipschiz-coninuous on bounded ses, and Φ is minorized. Throughou he paper, unless oherwise indicaed, we simply assume ha Φ : H R is a coninuously differeniable convex funcion. As we shall see, mos of he convergence properies of he rajecories are valid under his general assumpion. This approach paves he way for he exension of our resuls o non-smooh convex poenial funcions replacing he gradien by he subdifferenial. In preparaion for a sabiliy sudy of his sysem and he associaed algorihms, we will also consider he following perurbed version of : ẍ + α ẋ + Φx = g, where he second-member g : [, + [ H is an inegrable source erm, such ha g is small for large. The imporance of he evoluion sysem is hreefold:. Mechanical inerpreaion: I describes he moion of a paricle wih uni mass, subjec o a poenial energy funcion Φ, and an isoropic linear damping wih a viscosiy parameer ha vanishes asympoically. This provides a simple model for a progressive reducion of he fricion, possibly due o maerial faigue. Key words and phrases. Convex opimizaion, fas convergen mehods, dynamical sysems, gradien flows, inerial dynamics, vanishing viscosiy, Neserov mehod. Effor sponsored by he Air Force Office of Scienific Research, Air Force Maerial Command, USAF, under gran number FA Also suppored by Fondecy Gran 4089, Conicy Anillo ACT-06, ECOS-Conicy Projec C3E03, Millenium Nucleus ICM/FIC RC30003, Conicy Projec MATHAMSUD 5MATH-0, Conicy Redes 4083, and Basal Projec CMM Universidad de Chile.

2 HEDY ATTOUCH, ZAKI CHBANI, JUAN PEYPOUQUET, AND PATRICK REDONT. Fas minimizaion of funcion Φ: Equaion is a paricular case of he inerial gradien-lie sysem 3 ẍ + aẋ + Φx = 0, wih asympoic vanishing damping, sudied by Cabo, Engler and Gada in [5, 6]. As shown in [5, Corollary 3.] under some addiional condiions on Φ, every soluion x of 3 saisfies lim + Φx = min Φ, provided ad = +. The specific case was sudied by Su, Boyd and Candès in [44] in erms of he rae of convergence 0 for he values. More precisely, [44, Theorem 4.] esablishes ha Φx min Φ = O, whenever α 3. Unforunaely, heir analysis does no enail he convergence of he rajecory iself. 3. Relaionship wih fas numerical opimizaion mehods: As poined ou in [44, Secion ], for α = 3, can be seen as a coninuous version of he fas convergen mehod of Neserov see [3, 33, 34, 35], and is widely used successors, such as he Fas Ieraive Shrinage-Thresholding Algorihm FISTA, sudied in [0]. These mehods have a convergence rae of Φx min Φ = O, where is he number of ieraions. As for he coninuous-ime sysem, convergence of he sequences generaed by FISTA and relaed mehods has no been esablished so far. This is a cenral and long-sanding quesion in he sudy of numerical opimizaion mehods. The purpose of his research is o esablish he convergence of he rajecories saisfying, as well as he sequences generaed by he corresponding numerical mehods wih Neserov-ype acceleraion. We also complee he sudy wih several sabiliy properies concerning boh he coninuous-ime sysem and he algorihms. The main conribuions of his wor are he following: In Secion, we firs esablish he minimizing propery in he general case where α > 0, and inf Φ is no necessarily aained. As a consequence, every wea limi poin of he rajecory mus be a minimizer of Φ, and so, he exisence of a bounded rajecory characerizes he exisence of minimizers. Nex, assuming argmin Φ and α 3, we recover he O convergence raes, and give several examples and counerexamples concerning he opimaliy of hese resuls. Nex, we show ha every soluion of converges wealy o a minimizer of Φ provided α > 3 and argmin Φ. We rely on a Lyapunov analysis, which was firs used by Alvarez [3] in he conex of he heavy ball wih fricion. For he limiing case α = 3, which corresponds exacly o Neserov s mehod, he convergence of he rajecories is sill a puzzling open quesion. We finish his secion b! y providing an ergodic convergence resul for he acceleraion of he sysem in case Φ is Lipschiz-coninuous on sublevel ses of Φ. In Secion 3, srong convergence is esablished in various pracical siuaions enjoying furher geomeric feaures, such as srong convexiy, symmery, or nonempyness of he inerior of he soluion se. In he srongly convex case, we obained a surprising resul: convergence of he values occurs a a rae of O α 3. In Secion 4, we analyze he asympoic behavior, as +, of he soluions of he perurbed differenial sysem, and obain similar convergence resuls under inegrabiliy assumpions on he perurbaion erm g. Secion 5 conains he analogous resuls for he associaed Neserov-ype algorihms also for α > 3. To avoid repeaing similar argumens, we sae he resuls and develop he proofs direcly for he perurbed version. As a guideline, we follow he proof of he convergence of he coninuous dynamic. We provide discree versions of he differenial inequaliies ha we used in he Lyapunov convergence analysis. The convergence resuls are parallel o hose for he coninuous case, under summabiliy condiions on he errors. As we were preparing he final version of his manuscrip, we discovered he preprin [7] by Chambolle and Dossal, where he wea convergence resul of he algorihm in he unperurbed case is obained by a similar, bu differen argumen see [7, Theorem 3]. This approach has been furher developed by Aujol-Dossal [6] in he perurbed case.. Minimizing propery, convergence raes and wea convergence of he rajecories We begin his secion by providing some preliminary esimaions concerning he global energy of he sysem and he disance o he minimizers of Φ. These allow us o show he minimizing propery of he rajecories under minimal assumpions. Nex, we recover he convergence raes for he values originally given in [44], and obain furher decay esimaes ha ulimaely imply he convergence of he soluions of. We finish he sudy by proving an ergodic convergence resul for he acceleraion. Several examples and counerexamples are given hroughou he secion... Preliminary remars and esimaions. The exisence of global soluions o has been examined, for insance, in [5, Proposiion.] in he case of a general asympoic vanishing damping coefficien. In our seing, for any > 0, α > 0, and x 0, v 0 H H, here exiss a unique global soluion x : [, + [ H of, saisfying he iniial condiion x = x 0, ẋ = v 0, under he sole assumpion ha Φ is Lipschiz-coninuous on bounded ses, and inf Φ >. Taing > 0 comes from he singulariy of he damping coefficien a = α a zero. Indeed, since we are only concerned abou he asympoic behavior of he rajecories, we do no really care abou he origin of ime. If one insiss in saring from = 0, hen all he resuls remain valid wih a = α +. A differen poins, we shall use he global energy of he sysem, given by W : [, + [ R 4 W = ẋ + Φx.

3 FAST CONVERGENCE OF INERTIAL DYNAMICS AND ALGORITHMS WITH ASYMPTOTIC VANISHING DAMPING 3 Using, we immediaely obain Lemma.. Le W be defined by 4. For each >, we have Ẇ = α ẋ. Hence, W is nonincreasing, and W = lim + W exiss in R { }. If Φ is bounded from below, W is finie. Now, given z H, we define h z : [, + [ R by 5 h z = x z. By he Chain Rule, we have Using, we obain ḣ z = x z, ẋ and ḧ z = x z, ẍ + ẋ. 6 ḧ z + α ḣz = ẋ + x z, ẍ + α ẋ = ẋ + x z, Φx. The convexiy of Φ implies and we deduce ha x z, Φx Φx Φz, 7 ḧ z + α ḣz + Φx Φz ẋ. We have he following relaionship beween h z and W : Lemma.. Tae z H, and le W and h z be defined by 4 and 5, respecively. There is a consan C such ha s W s Φz ds C ḣz 3 α W. Proof. Divide 7 by, and use he definiion of W given in 4, o obain ḧz + α ḣz + W Φz 3 ẋ. Inegrae his expression from o > use inegraion by pars for he firs erm, o obain 8 s W s Φz ds ḣ z ḣz α + On he one hand, Lemma. gives s ḣzsds + 3 s ẋs ds = 3 α W W. On he oher hand, anoher inegraion by pars yields s ḣzsds = h z h z + 0 s 3 h zsds h z. 0 Combining hese inequaliies wih 8, we ge 3 s ẋs ds. s W s Φz ds ḣ z ḣz + α + h z α W W = C ḣz 3 α W, where C collecs he consan erms... Minimizing propery. I urns ou ha he rajecories of minimize Φ in he compleely general seing, where α > 0, argmin Φ is possibly empy, and Φ is no necessarily bounded from below recall ha we assume he exisence and uniqueness of a global soluion o he Cauchy problem associaed wih, which is no guaraneed by hese general assumpions. This propery was obained by Alvarez in [3, Theorem.] for he heavy ball wih fricion where he damping is consan. Similar resuls can be found in [5]. We have he following: Theorem.3. Le α > 0, and suppose x : [, + [ H is a soluion of. Then i W = lim + W = lim + Φx = inf Φ R { }. ii As +, every wea limi poin of x lies in argmin Φ. iii If argmin Φ =, hen lim + x = +. iv If x is bounded, hen argmin Φ. v If Φ is bounded from below, hen lim + ẋ = 0. In fac, W decreases sricly, as long as he rajecory is no saionary.

4 4 HEDY ATTOUCH, ZAKI CHBANI, JUAN PEYPOUQUET, AND PATRICK REDONT vi If Φ is bounded from below and x is bounded, hen lim + ḣ z = 0 for each z H. Moreover, Φx min Φd < +. Proof. To prove i, firs se z H and τ >. By Lemma., W in nonincreasing. Hence, Lemma. gives which we rewrie as and hen Inegrae from = o = τ o obain Bu W τ Φz Hence, W τ Φz ds s + 3 α W τ C ḣz, ds W τ Φz s + 3 C 3 α α Φz ḣz, W τ Φz ln + 3 τ lnτ ln + τ + τ α ln C 3 α Φz ḣz. 3α ln τ C 3 τ α Φz τ ḣzd. ḣ z d = h zτ h z τ h z + τ d h z. W τ Φzτ lnτ + Aτ + B Cτ + D, for suiable consans A, B, C and D. This immediaely yields W Φz, and hence W inf Φ. I suffices o observe ha inf Φ lim inf Φx lim sup Φx lim W = W o obain i. Nex, ii follows from i by he lower-semiconinuiy of Φ for he wea opology. Clearly, iii and iv are immediae consequences of ii. We obain v by using i and he definiion of W given in 4. For vi, since ḣz = x z, ẋ and x is bounded, v implies z argmin Φ, we ge which complees he proof. lim + ḣ z = 0. Finally, using he definiion of W ogeher wih Lemma. wih 3 Φx min Φd C min Φ < +, α Remar.4. We shall see in Theorem.4 ha, for α 3, he exisence of minimizers implies ha every soluion of is bounded. This gives a converse o par iv of Theorem.3. If Φ is no bounded from below, i may be he case ha ẋ does no end o zero, as shown in he following example: Example.5. Le H = R and α > 0. The funcion x = saisfies wih Φx = α + x. Then lim + Φx = = inf Φ, and lim + ẋ = Two anchored energy funcions. We begin by inroducing wo imporan auxiliary funcions, and showing heir basic properies. From now on, we assume argmin Φ. Fix λ 0, ξ 0, p 0 and x argmin Φ. Le x : [, + [ H be a soluion of. For define E λ,ξ = Φx min Φ + λx x + ẋ + ξ x x, E p λ = p E λ,0 = p Φx min Φ + λx x + ẋ, and noice ha E λ,ξ and E p λ are sums of nonnegaive erms. These generalize he energy funcions E and Ẽ inroduced in [44]. More precisely, E = E α,0 and Ẽ = E α 3/3. We need some preparaory calculaions prior o differeniaing E λ,ξ and E p λ. For simpliciy of noaion, we do no mae he dependence of x or ẋ on explici. Noice ha we use in he second line o dispose of ẍ. d d Φx min Φ = Φx min ϕ + ẋ, Φx d d λx x + ẋ = λ x x, Φx λα λ x x, ẋ α λ ẋ ẋ, Φx d d x x = x x, ẋ.

5 FAST CONVERGENCE OF INERTIAL DYNAMICS AND ALGORITHMS WITH ASYMPTOTIC VANISHING DAMPING 5 Whence, we deduce 9 0 d d E λ,ξ = Φx min Φ λ x x, Φx + ξ λα λ x x, ẋ α λ ẋ d d E p λ = p + p+ Φx min Φ λ p+ x x, Φx λα λ p p x x, ẋ + λ p p x x α λ p p+ ẋ. Remar.6. If y H and x argmin Φ, he convexiy of Φ gives min Φ = Φx Φy + Φy, x y. Using his in 9 wih y = x, we obain d d E λ,ξ λ Φx min Φ + ξ λα λ x x, ẋ α λ ẋ. If one chooses ξ = λα λ, hen d d E λ,ξ λ Φx min Φ α λ ẋ. Therefore, if α 3 and λ α, hen E λ,ξ is nonincreasing. The exreme cases λ = and λ = α are of special imporance, as we shall see shorly..4. Rae of convergence for he values. We now recover convergence rae resuls for he value of Φ along a rajecory, already esablished in [44, Theorem 4.]: Theorem.7. Le x : [, + [ H be a soluion of, and assume argmin Φ. If α 3, hen If α > 3, hen + Φx min Φ E α,0. Φx min Φ d E α,0 α 3 < +. Proof. Suppose α 3. Choose λ = α and ξ = 0, so ha ξ λα λ = α λ = 0 and λ = α 3. Remar.6 gives d d E α,0 α 3 Φx min Φ, and E α,0 is nonincreasing. Since Φx min Φ E α,0, we obain Φx min Φ E α,0. If α > 3, inegraing from o we obain sφxs min Φds α 3 E α,0 E α,0 α 3 E α,0, which allows us o conclude. Remar.8. I would be ineresing o now wheher α = 3 is criical for he convergence rae given above. Remar.9. For he firs-order seepes descen dynamical sysem, he ypical rae of convergence is O/ see, for insance, [40, Secion 3.]. For he second-order sysem, we have obained a rae of O/. I would be ineresing o now wheher higher-order sysems give he corresponding raes of convergence. Anoher challenging quesion is he convergence rae of he rajecories defined by differenial equaions involving fracional ime derivaives, as well as inegro-differenial equaions. Remar.0. The consan in he order of convergence given by Theorem.7 is Kx 0, v 0 = E α,0 = 0Φx 0 min Φ + α x 0 x + v 0, where x 0 = x and v 0 = ẋ. This quaniy is minimized when x 0 argmin Φ and v0 = α x x 0, wih min K = 0. If x 0 x, he rajecory will no be saionary, bu he value Φx will be consanly equal o min Φ. Of course, selecing x 0 argmin Φ is no realisic, and he poin x is unnown. Keeping ˆx 0 fixed, he funcion v 0 Kˆx 0, v 0 is minimized a ˆv 0 = α x ˆx 0. This suggess aing he iniial velociy as a muliple of an approximaion of x ˆx 0, such as he gradien direcion ˆv 0 = Φˆx 0, Newon or Levenberg-Marquard direcion ˆv 0 = [εi + Φˆx 0 ] Φˆx 0 ε 0, or he proximal poin direcion ˆv 0 = [ ] I + γ Φ ˆx 0 ˆx 0 γ >> 0.

6 6 HEDY ATTOUCH, ZAKI CHBANI, JUAN PEYPOUQUET, AND PATRICK REDONT.5. Some examples and counerexamples. A convergence rae of O/ may be aained, even if argmin Φ = and α < 3. This is illusraed in he following example: Example.. Le H = R and ae Φx = α e x wih α. Le us verify ha x = ln is a soluion of. On he one hand, ẍ + α ẋ = α. On he oher hand, Φx = α e x which gives Φx = α e ln = α. Thus, x = ln is a soluion of. Le us examine he minimizing propery. We have inf Φ = 0, and Φx = α e ln = α. Therefore, one may wonder wheher he rapid convergence of he values is rue in general. The following example shows ha his is no he case: Example.. Le H = R and ae Φx = c, wih θ > 0, α x θ x = +θ is a soluion of. On he one hand, On he oher hand, Φx = cθx θ which gives θ +θ ẍ + α ẋ = +θ α + θα +θ + θ. Φx = cθ +θ +θ +θ = α + θα + θ α+θα and c = θ+θ. Le us verify ha Thus, x = +θ is soluion of. Le us examine he minimizing propery. We have inf Φ = 0, and Φx = c θ +θ, wih θ + θ <. We conclude ha he order of convergence may be sricly slower han O/ when argmin Φ =. In he Example., his occurs no maer how large α is. The speed of convergence of Φx o inf Φ depends on he behavior of Φx as x +. The above examples sugges ha, when Φx decreases rapidly and aains is infimal value as x, we can expec fas convergence of Φx. Even when argmin Φ, O/ is he wors possible case for he rae of convergence, aained as a limi in he following example: Example.3. Tae H = R and Φx = c x γ, where c and γ are posiive parameers. Le us loo for nonnegaive soluions of of he form x =, wih θ > 0. This means ha he rajecory is no oscillaing, i is a compleely θ damped rajecory. We begin by deermining he values of c, γ and θ ha provide such soluions. On he one hand, On he oher hand, Φx = cγ x γ x, which gives ẍ + α ẋ = θθ + α θ+. Φx = cγ. θγ Thus, x = is soluion of if, and only if, θ i θ + = θγ, which is equivalen o γ > and θ = γ ; and ii cγ = θα θ, which is equivalen o α > γ γ and c = γγ α γ γ. We have min Φ = 0 and Φx = γγ α γ γ. γ γ The speed of convergence of Φx o 0 depends on he parameer γ. As γ ends o infiniy, he exponen ends o. This limiing siuaion is obained by aing a funcion Φ ha becomes very fla around he se of is minimizers. Therefore, wihou oher geomeric assumpions on Φ, we canno expec a convergence rae beer han O/. By conras, in Secion 3, we will show beer raes of convergence under some geomerical assumpions, lie srong convexiy of Φ. +θ. γ γ

7 FAST CONVERGENCE OF INERTIAL DYNAMICS AND ALGORITHMS WITH ASYMPTOTIC VANISHING DAMPING 7.6. Wea convergence of he rajecories. In his subsecion, we show he convergence of he soluions of, provided α > 3. We begin by esablishing some preliminary esimaions ha canno be derived from he analysis carried ou in [44]. The firs saemen improves par v of Theorem.3, while he second one is he ey o proving he convergence of he rajecories of : Theorem.4. Le x : [, + [ H be a soluion of wih argmin Φ. i If α 3 and x is bounded, hen ẋ = O/. More precisely, ẋ E α,0 + α sup x x. 3 ii If α > 3, hen x is bounded and + ẋ d E,α 3 α 3 < +. Proof. To prove i, assume α 3 and x is bounded. From he definiion of E λ,ξ, we have λx x + ẋ E λ,ξ, and so ẋ E λ,ξ + λ x x. By Remar.6, E α,0 is nonincreasing, and we immediaely obain. In order o show ii, suppose now ha α > 3. Choose λ = and ξ = α 3. By Remar.6, we have 4 d d E λ,ξ α 3 ẋ, and E λ,ξ is nonincreasing. From he definiion of E λ,ξ, we deduce ha x x ξ E λ,ξ, which gives 5 x x E,α 3 α 3 E,α 3, α 3 and esablishes de boundedness of x. Inegraing 4 from o, and recalling ha E λ,ξ is nonnegaive, we obain as required. s ẋs ds E,α 3 α 3 Remar.5. In view of and 5, when α > 3, we obain he following explici bound for ẋ, namely ẋ E,α 3 E α,0 + α. α 3 Since lim + ẋ = 0 by Theorem.3, we also have lim + ẋ = 0. We are now in a posiion o prove he wea convergence of he rajecories of, which is he main resul of his secion: Theorem.6. Le Φ : H R be a coninuously differeniable convex funcion such ha argmin Φ, and le x : [, + [ H be a soluion of wih α > 3. Then x converges wealy, as +, o a poin in argmin Φ. Proof. We shall use Opial s Lemma A.. To his end, le x argmin Φ and recall from 7 ha where h z is given by 5. This yields ḧ x + α ḣx + Φx min Φ ẋ, ḧx + αḣx ẋ. In view of Theorem.4, par ii, he righ-hand side is inegrable on [, + [. Lemma A.4 hen implies lim + h x exiss. This gives he firs hypohesis in Opial s Lemma. The second one was esablished in par ii of Theorem.3. Remar.7. A puzzling quesion concerns he convergence of he rajecories for α = 3, a quesion which is direcly relaed o he convergence of he sequences generaed by Neserov s mehod..7. Furher sabilizaion resuls. Le us complemen he sudy of equaion by examining he asympoic behavior of he acceleraion ẍ. To his end, we shall use an addiional regulariy assumpion on he gradien of Φ. Proposiion.8. Le α > 3 and le x : [, + [ H be a soluion of wih argmin Φ. Assume Φ Lipschiz-coninuous on bounded ses. Then ẍ is bounded, globally Lipschiz coninuous on [, + [, and saisfies lim + α s α ẍs ds = 0.,

8 8 HEDY ATTOUCH, ZAKI CHBANI, JUAN PEYPOUQUET, AND PATRICK REDONT Proof. Firs recall ha x and ẋ are bounded, by virue of Theorems.4 and.3, respecively. By, we have 6 ẍ = α ẋ Φx. Since Φ is Lipschiz-coninuous on bounded ses, i follows from 6, and he boundedness of x and ẋ, ha ẍ is bounded on [, + [. As a consequence, ẋ is Lipschiz-coninuous on [, + [. Reurning o 6, we deduce ha ẍ is Lipschiz-coninuous on [, + [. Pic x argmin Φ, se h = h x o simplify he noaion and use 6 o obain 7 ḧ + α ḣ + x x, Φx = ẋ. Le L be a Lipschiz consan for Φ on some ball conaining he minimizer x and he rajecory x. By virue of he Baillon-Haddad Theorem see, for insance, [8], [39, Theorem 3.3] or [33, Theorem..5], Φ is L -cocoercive on ha ball, which means ha x x, Φx Φx L Φx Φx. Subsiuing his inequaliy in 7, and using he fac ha Φx = 0, we obain In view of 6, his gives ḧ + α ḣ + L Φx ẋ. ḧ + α ḣ + L ẍ + α ẋ ẋ. Developing he square on he lef-hand side, and neglecing he nonnegaive erm α ẋ / /L, we obain ḧ + α ḣ + L ẍ + α d L d ẋ ẋ. We muliply his inequaliy by α o obain d α ḣ + d L α ẍ + α d L α d ẋ α ẋ. Inegraion from o yields α ḣ α 0 ḣ+ s α ẍs ds+ α α ẋ α 0 ẋ α ẋs s α ds s α ẋs ds. L L Neglecing he nonnegaive erm α α ẋ /L, we obain 8 α ḣ + L where C = α 0 ḣ + α α 0 ẋ /L. If <, we have α s α ẍs ds C + α ẋs s α ds + s α ẋs ds, s α ẍs ds = α s α ẍs ds + α s α ẍs ds for all. Since he firs erm on he righ-hand side ends o 0 as +, we may assume, wihou loss of generaliy, ha. Observe now ha s α s α, whenever s. Whence, inequaliy 8 simplifies o α ḣ + L s α ẍs ds C + α s α ẋs ds. Dividing by α and inegraing again, we obain h h + τ τ α s α ẍs ds dτ C τ L α α+ 0 α+ + α τ α s α ẋs ds dτ. Seing C = h +C α+ 0 /α, and neglecing he nonnegaive erm h of he lef-hand side and he nonposiive erm C α+ /α of he righ-hand side, we ge L τ α τ s α ẍs ds dτ C + α τ α τ s α ẋs ds dτ. τ Se gτ = τ α s α ẍs ds and use Fubini s Theorem on he second inegral o ge L gτdτ C + α s α ẋs s α+ α+ ds C + α s ẋs ds. α α

9 FAST CONVERGENCE OF INERTIAL DYNAMICS AND ALGORITHMS WITH ASYMPTOTIC VANISHING DAMPING 9 By par ii of Theorem.4, he inegral on he righ-hand side is finie. We have 9 The derivaive of g is + gτdτ < +. τ ġτ = ατ α s α ẍs ds + ẍτ. Le C be an upper bound for ẍ. We have τ 0 ġτ C + ατ α s α ds = C + α α + τ α τ α+ α+ C + α α + From 9 and 0 we deduce ha lim τ + gτ = 0 by virue of Lemma A.. Remar.9. Since s α ds = α+ α+ α+ 0, Proposiion.8 expresses a fas ergodic convergence of ẍs o 0 wih respec o he weigh s α as +, namely s α ẍs ds = o. s α ds 3. Srong convergence resuls A counerexample due o Baillon [7] shows ha he rajecories of he seepes descen dynamical sysem may converge wealy bu no srongly. Neverheless, under some addiional geomerical or opological assumpions on Φ, he seepes descen rajecories do converge srongly. This has been proved in he case where he funcion Φ is eiher even or srongly convex see [3], or when inargmin Φ see [, heorem 3.3]. Some of hese resuls have been exended o inerial dynamics, see [3] for he heavy ball wih fricion, and [6] for an inerial version of Newon s mehod. This suggess ha convexiy alone may no be sufficien for he rajecories of o converge srongly, bu one can reasonably expec i o be he case under some addiional condiions. The purpose of his secion is o esablish his fac. The differen ypes of hypoheses will be sudied in independen subsecions s! ince differen echniques are required. 3.. Se of minimizers wih nonempy inerior. Le us begin by sudying he case where inargmin Φ. Theorem 3.. Le Φ : H R be a coninuously differeniable convex funcion. Le inargmin Φ, and le x : [, + [ H be a soluion of wih α > 3. Then x converges srongly, as +, o a poin in argmin Φ. Moreover, Φx d < +. Proof. Since inargmin Φ, here exis x argmin Φ and some ρ > 0 such ha Φz = 0 for all z H such ha z x < ρ. By he monooniciy of Φ, for all y H, we have Φy, y z 0. Hence, Φy, y x Φy, z x. Taing he supremum wih respec o z H such ha z x < ρ, we infer ha Φy, y x ρ Φy for all y H. In paricular, Φx, x x ρ Φx. By using his inequaliy in 9 wih λ = α and ξ = 0, we obain d d E α,0 + α ρ Φx Φx min Φ, whence we derive, by inegraing from o E α,0 E α,0 + α ρ s Φxs ds Since E α,0 is nonnegaive, par ii of Theorem.7 gives Φx d < +. s Φxs min Φ ds. Finally, rewrie as ẍ + αẋ = Φx. Since he righ-hand side is inegrable, we conclude by applying Lemma A.7 and Theorem.6..

10 0 HEDY ATTOUCH, ZAKI CHBANI, JUAN PEYPOUQUET, AND PATRICK REDONT 3.. Even funcions. Le us recall ha Φ : H R is even if Φ x = Φx for every x H. In his case he se argmin Φ is nonempy, and conains he origin. Theorem 3.. Le Φ : H R be a coninuously differeniable convex even funcion, and le x : [, + [ H be a soluion of wih α > 3. Then x converges srongly, as +, o a poin in argmin Φ. Proof. For τ s, se qτ = xτ xs xτ xs. We have qτ = ẋτ, xτ + xs and qτ = ẋτ + ẍτ, xτ + xs. Combining hese wo equaliies and using, we obain qτ + α τ qτ = ẋτ + ẍτ + α τ ẋτ, xτ + xs = ẋτ Φxτ, xτ + xs. Recall ha he energy W τ = ẋτ + Φxτ is nonincreasing. Therefore, ẋτ + Φxτ ẋs + Φxs = ẋs + Φ xs ẋs + Φxτ Φxτ, xτ + xs, by convexiy. Afer simplificaion, we obain 3 ẋτ Φxτ, xτ + xs. Combining and 3, we obain As in he proof of Lemma A.4, we have τ qτ + α qτ 3 τ ẋτ. qτ τ := C τ α + 3 τ α τ u α ẋu du, where C = ẋ x. The funcion does no depend on s. Moreover, using Fubini s Theorem, we deduce ha + + C τ dτ α 0 α + 3 u ẋu du < +, α by par ii of Theorem.4. Inegraing qτ τ from o s, we obain x xs x xs + s τdτ. Since Φ is even, we have 0 argmin Φ. Hence lim + x exiss see he proof of Theorem.6. As a consequence, x has he Cauchy propery as +, and hence converges Uniformly convex funcions. Following [9], a funcion Φ : H R is uniformly convex on bounded ses if, for each r > 0, here is an increasing funcion ω r : [0, + [ [0, + [ vanishing only a, and such ha 4 Φy Φx + Φx, y x + ω r x y for all x, y H such ha x r and y r. Uniformly convex funcions are sricly convex and coercive. Theorem 3.3. Le Φ be uniformly convex on bounded ses, and le x : [, + [ H be a soluion of wih α > 3. Then x converges srongly, as +, o he unique x argmin Φ. Proof. Recall ha he rajecory x is bounded, by par ii in Theorem.4. Le r > 0 be such ha x is conained in he ball of radius r cenered a he origin. This ball also conains x, which is he wea limi of he rajecory in view of he wea lower-semiconinuiy of he norm and Theorem.6. Wriing y = x and x = x in 4, we obain ω r x x Φx min Φ. The righ-hand side ends o 0 as + by virue of Theorem.3. I follows ha x converges srongly o x as +. Le us recall ha a funcion Φ : H R is srongly convex if here exiss a posiive consan µ such ha Φy Φx + Φx, y x + µ x y for all x, y H. Clearly, srongly convex funcions are uniformly convex on bounded ses. However, a sriing fac is ha convergence raes increase indefiniely wih larger values of α for hese funcions.

11 FAST CONVERGENCE OF INERTIAL DYNAMICS AND ALGORITHMS WITH ASYMPTOTIC VANISHING DAMPING Theorem 3.4. Le Φ : H R be srongly convex, and le x : [, + [ H be a soluion of wih α > 3. Then x converges srongly, as +, o he unique elemen x argmin Φ. Moreover 5 Φx min Φ = O α 3, x x = O α 3, and ẋ = O α 3. Proof. Srong convergence follows from Theorem 3.3 because srongly convex funcions are uniformly convex on bounded ses. From 0 and he srong convexiy of Φ, we deduce ha d d E p λ p + λp+ Φx min Φ λα λ p p x x, ẋ λ µ pλ p x x α λ p p+ ẋ for any λ 0 and any p 0. Now fix p = 3 α 3 and λ = 3α, so ha p + λ = α λ p/ = 0 and α λ p = p/. The above inequaliy becomes { Define = max, } pλ µ, so ha d d E p λp λ p x x, ẋ λ µ pλ p x x. d d E p λp λ p x x, ẋ for all. Inegrae his inequaliy from o use inegraion by pars on he righ-hand side o ge E p λ E p λ + λp p x x p 4 x x p s p xs x ds. Hence, 6 E p λ E p λ + λp 4 p x x E p λ + λp µ p Φx min Φ, in view of he srong convexiy of Φ. By he definiion of E p λ, we have p+ Φx min Φ E p λ E p λ + λp µ p Φx min Φ. Dividing by p+ and using he definiion of, along wih he fac ha, we obain Φx min Φ E p λ p + λp µ Φx min Φ E p λ p + λp µ Φx min Φ E p λ p + Φx min Φ. Recalling ha p = 3 α 3 and λ = 3α, we deduce ha [ 7 Φx min Φ E p λ p = E 3 α 3 3 α ] 3 α. The srong convexiy of Φ hen gives 8 x x [ ] [ ] 4 µ Φx min Φ µ E p 4 λ p = µ E 3 α 3 3 α 3 α. Inequaliies 7 and 8 sele he firs wo poins in 5. Now, using 6 and 7, we derive E p λ E p λ + λp µ p Φx min Φ E p λ + λp µ E p λ E p λ + λp µ E p λ E p λ. The definiion of E p λ hen gives Hence and Bu using 8, we deduce ha p λx x + ẋ E p λ E p λ τ. λx x + ẋ 4 p E p λ, ẋ p/ E p λ + λ x x. λ x x λ µ p/ E p λ.

12 HEDY ATTOUCH, ZAKI CHBANI, JUAN PEYPOUQUET, AND PATRICK REDONT The las wo inequaliies ogeher give ẋ E p/ p λ + λ E p/ p µ λ λ +. p Taing squares, and rearranging he erms, we obain [ ẋ 4 + α α 3 which shows he las poin in 5 and complees he proof. ] E 3 α 3 3 α 3 α, The preceding heorem exends [44, Theorem 4.], which saes ha if α > 9/, hen Φx min Φ = O/ Asympoic behavior of he rajecory under perurbaions In his secion, we analyze he asympoic behavior, as +, of he soluions of he differenial equaion 9 ẍ + α ẋ + Φx = g. From he Cauchy-Lipschiz-Picard Theorem, for any iniial condiion x = x 0 H, ẋ = v 0 H, we deduce he exisence and uniqueness of a local soluion o 9, if Φ is Lipschiz-coninuous on bounded ses, Φ is minorized, and g is locally inegrable. The global exisence follows from he energy esimae proved in Lemma 4., in he nex subsecion. This being said, our main concern here is o obain sufficien condiions on he perurbaions in order o guaranee ha he convergence properies esablished in he preceding secions are preserved. The analysis follows very closely he argumens given in Secions and 3. I is developed in he same general seing, jus assuming ha Φ : H R is a coninuously differeniable convex funcion. Therefore, we shall sae he main resuls and sech he proofs, underlining he pars where addiional echniques are required. 4.. Energy esimaes and minimizaion propery. The following resul is obained by considering he global energy of he sysem, and showing ha i is a sric Lyapunov funcion: Lemma 4.. Le Φ be bounded from below, and le x : [, + [ H be a soluion of 9 wih α > 0 and g d < +. Then, sup ẋ < + and > τ ẋτ dτ < +. Moreover, lim Φx = inf Φ. + H Proof. Se T >. For T, define he energy funcion 30 W T := ẋ + Φx inf Φ + H T ẋτ, gτ dτ. Since ẋ is coninuous and g is inegrable, he funcion W T is well defined. Derivaing W T wih respec o ime, and using 9, we obain ẋ + W T = ẋ, ẍ + Φx g = ẋ, α ẋ = α ẋ 0. Hence W T is a decreasing funcion. In paricular, W T W T, which is T Φx inf Φ + ẋτ, gτ dτ H ẋ + As a consequence, ẋ ẋ + Applying Lemma A.8, we obain I follows ha ẋ 3 sup > ẋ Φx 0 inf Φ + H Φx 0 inf Φ + ẋτ gτ dτ. H ẋ + Φx 0 inf Φ + gτ dτ. H ẋ + Φx 0 inf Φ + H T gτ dτ < +. ẋτ, gτ dτ. As a consequence, we may define a funcion W : ], + [ R by W := [ ] ẋ + Φx inf Φ + ẋτ, gτ dτ sup ẋ gτ dτ, H > by 3. From he definiion of W T and W, we have 3 Ẇ = ẆT = α ẋ.

13 FAST CONVERGENCE OF INERTIAL DYNAMICS AND ALGORITHMS WITH ASYMPTOTIC VANISHING DAMPING 3 Inegraing from o, and using 3, we obain α τ ẋτ dτ = W W ẋ + [ ] Φx inf Φ + sup ẋ gτ dτ < +, H > which gives a bound for he second improper inegral. For he minimizaion propery, consider he funcion h : ], + [ R, defined by h = x z, where z is an arbirary elemen of H. We can easily verify ha By convexiy of Φ, we obain ḧ + α ḣ = ẋ Φx, x z + g, x z. 33 ḧ + α ḣ + Φx Φz ẋ + g, x z. Recall ha he funcion W defined above is nonincreasing and bounded from below. Hence, W converges, as +, o some W R. Moreover, using he definiion of W in 33, we deduce ha 34 ḧ + α ḣ + W + inf Φ Φz 3 ẋ + g, x z + Seing B = W + inf Φ Φz, we may wrie B 3 ẋ + g x z + θ ẋs, gs ds. sup ẋ gs ds d α d α ḣ. Muliplying his las equaion by, and inegraing beween and θ >, we ge B ln θ 3 θ θ g x z θ ẋ d + d + sup ẋ d α+ d α ḣd. Le us esimae he inegrals in he second member of he las inequaliy: The firs erm is finie, in view of Lemma 4.. gs ds d The second erm is also finie, since he relaion x z x z + ẋs ds implies θ g x z x0 z + d + sup ẋ g d < +. 3 For he hird erm, inegraion by pars gives θ gs ds d = ln θ gs ds ln gs ds + θ 4 For he fourh erm, se I = θ d α+ d [ ḣ ] θ θ I = + α + ḣd = C 0 + θ for some consan C 0, because h 0. Finally, noice ha ḣθ = ẋθ, xθ z sup ẋ Collecing he above resuls, we deduce ha B ln θ C + ln θ θ g ln d. αḣd, and inegrae by pars wice o obain θ ḣθ + + α θ hθ + + α θ x0 z + θ sup ẋ 3 hd C 0 + θ ḣθ,. θ gs ds + sup ẋ g ln d, for some oher consan C. Dividing by ln θ, and leing θ +, we conclude ha B 0, by using Lemma A.6 wih ψ = ln. This implies ha W Φz inf Φ for every z H, which leads o W 0. On he oher hand, i is easy o see ha + W Φx inf Φ sup ẋ gsds. Passing o he limi, as +, we deduce ha 0 W lim sup Φx inf Φ. Since we always have inf Φ lim inf Φx, we conclude ha lim + Φx = inf Φ.

14 4 HEDY ATTOUCH, ZAKI CHBANI, JUAN PEYPOUQUET, AND PATRICK REDONT 4.. Fas convergence of he values. We are now in posiion o prove he following: Theorem 4.. Le argmin Φ, and le x : [, + [ H be a soluion of 9 wih α 3 and Then Φx min H Φ = O. g d < +. Proof. The proof follows he argumens used for Theorem.7. Tae x S = argmin Φ. For T, define he energy funcion E α,g,t := α Φx inf Φ + α x H x + T + τ xτ x + τ gτ dτ. α ẋ α ẋτ, Le us show ha Derivaion of E α,g,t gives E α,g,t := = = 4 Φx inf α E α,g,t + α 3 α +α x x + 4 Φx inf α +α x x + 4 Φx inf α x x + Φx min Φ 0. H Φ + H α Φx, ẋ x x + ẋ + α ẋ, α ẋ + α ẍ Φ + H α Φx, ẋ α ẋ, α Φ + H α Φx, ẋ Φx α ẋ, α ẋ + ẍ g g α ẋ, 4 = Φx inf α Φ x H x, Φx. Using he subdifferenial inequaliy for Φ, and rearranging he erms, we obain 35 E α,g,t + α 3 Φx inf Φ 0. α H As a consequence, for α 3, he funcion E α,g,t is nonincreasing. In paricular, E α,g,t E α,g,t, which gives 36 wih α Φx inf Φ + α x H x + C + α ẋ From 36, we infer ha τ xτ x + C = α Φx 0 inf H Φ + α x 0 x + α ẋ. x x + α ẋ Applying Lemma A.8, we obain 37 x x + α ẋ and so 38 C α + xτ x + α C + τ gτ dτ, α α sup x x + < +. α ẋ Using 37 in 36, we conclude ha C α Φx inf Φ C + + H α α and he resul follows. Remar 4.3. As a consequence, he energy funcion E α,g := α Φx inf Φ + α x H x + + α ẋ τ α ẋτ τgτ dτ. τ gτ dτ τ gτ dτ, + is well defined on [, + [, and is a Lyapunov funcion for he dynamical sysem 9. τ xτ x + τ gτ dτ, α ẋτ, τ gτ dτ α ẋτ,

15 FAST CONVERGENCE OF INERTIAL DYNAMICS AND ALGORITHMS WITH ASYMPTOTIC VANISHING DAMPING Convergence of he rajecories. In he case α > 3, provided ha he second member g is sufficienly small for large, we are going o show he convergence of he rajecories of 9, as i occurs for he unperurbed sysem sudied in he previous secions Theorem 4.4. Le argmin Φ, and le x : [, + [ H be a soluion of 9 wih α > 3 and Then, x converges wealy, as +, o a poin in argmin Φ. g d < +. Proof. Sep. Recall, from he proof of Theorem 4., ha he energy funcion E α,g defined in Remar 4.3 saisfies Inegraing his inequaliy, we obain E α,g + α 3 Φx inf Φ 0. α H E α,g + α 3 α τφxτ inf H Φdτ E α,g. By he definiion of E α,g, and neglecing is nonnegaive erms, we infer ha and so + τ xτ x + τ 3 gτ dτ + α α ẋτ, α τφxτ inf H Φdτ E α,g, α 3 + τφxτ inf α Φdτ E α,g + xτ x + τ H 0 α ẋτ τgτ dτ by 38. Since α > 3, we deduce ha 39 Sep. Le us show ha + E α,g + sup x x + α ẋ τφxτ inf Φdτ < +. H By aing he scalar produc of 9 by ẋ, we ge ẋ d < +. ẍ, ẋ + α ẋ + Φx, ẋ = g, ẋ. Using he Chain Rule and he Cauchy-Schwarz inequaliy, we obain Inegraion by pars yields ẋ As a consequence, 40 ẋ d d ẋ + α ẋ + d Φx g ẋ. d s ẋs ds + α + Φx inf H Φ Φx inf H Φ ẋ + α s ẋs ds C 0 + s ẋs ds sφxs inf H Φds sφxs inf H Φds + + τgτ dτ. sgs sẋs ds. sgs sẋs ds for some consan C 0 depending only on he Cauchy daa. Since sφxs inf H Φds < + by 39, and α >, we deduce ha 4 ẋ C + sgs sẋs ds for some oher consan C, which we may assume o be nonnegaive. Applying Lemma A.8, we obain and so ẋ C + 4 sup ẋ < +. sgs ds,

16 6 HEDY ATTOUCH, ZAKI CHBANI, JUAN PEYPOUQUET, AND PATRICK REDONT Reurning o 40, we deduce ha 43 α which gives s ẋs ds C + Moreover, combining 38 and 4, we deduce ha sφxs inf Φds + sup ẋ H ẋ d < sup x < +, and he rajecory x is bounded. sgs ds, Sep 3. As before, we prove he wea convergence by means of Opial s Lemma A.. Tae x argmin Φ, and define h : [0, + [ R + by By he Chain Rule, h = x x. ḣ = x x, ẋ, Combining hese wo equaions, and using 9, we obain ḧ = x x, ẍ + ẋ. ḧ + α ḣ = ẋ + x x, ẍ + α ẋ, = ẋ + x x, Φx + g. By he monooniciy of Φ and he fac ha Φx = 0, we have 45 x x, Φx 0, and we infer ha Equivalenly ḧ + α ḣ ẋ + x x g. 46 ḧ + α ḣ, wih := ẋ + x x g ẋ + C g because he rajecory x is bounded, by 44. Recall ha + g d < + by assumpion, and ẋ d < + by 3. Hence, he funcion belongs o L, +. Applying Lemma A.4, wih w = ḣ, we deduce ha ḣ+ L, +, which implies ha he limi of h exiss, as +. This proves iem i of Opial s Lemma A.. For iem ii, observe ha every wea limi poin of x as + mus minimize Φ, since lim + Φx = inf Φ. Remar 4.5. Throughou he proof of Theorem 4.4, we proved ha Φx min Φ d < +, and H We also proved ha sup ẋ < +, and hence lim ẋ = Srong convergence resuls. For srong convergence, we have he following: ẋ d < +. Theorem 4.6. Le Φ : H R be a coninuously differeniable convex funcion, and le x : [, + [ H be a soluion of 9 wih α > 3 and cases: i The se argmin Φ has nonempy inerior; ii The funcion Φ is even; or iii The funcion Φ is uniformly convex. g d < +. Then x converges srongly, as +, in any of he following In order o prove his resul, i suffices o adap he argumens given in Secion 3 for he unperurbed case. Since i is relaively sraighforward, we leave i as an exercise o he reader.

17 FAST CONVERGENCE OF INERTIAL DYNAMICS AND ALGORITHMS WITH ASYMPTOTIC VANISHING DAMPING 7 5. Convergence of he associaed algorihms In his secion, we analyze he fas convergence properies of he associaed Neserov-ype algorihms. To avoid repeaing similar argumens, we sae he resuls and develop he proofs direcly for he perurbed version. 5.. A dynamical inroducion of he algorihm. Time discreizaion of dissipaive gradien-based dynamical sysems leads naurally o algorihms, which, under appropriae assumpions, have similar convergence properies. This approach has been followed successfully in a variey of siuaions. For a general absrac discussion see [7], and [8]. For dynamics wih inerial feaures, see [3], [4], [6], [4]. To cover pracical siuaions involving consrains or nonsmooh daa, we need o broaden our scope. This leads us o consider he non-smooh srucured convex minimizaion problem 47 min {Φx + Ψx : x H} where Φ : H R {+ } is a proper lower semiconinuous convex funcion; and Ψ : H R is a coninuously differeniable convex funcion, whose gradien is Lipschiz coninuous. The opimal soluions of 47 saisfy Φx + Ψx 0, where Φ is he subdifferenial of Φ, in he sense of convex analysis. In order o adap our dynamic o his non-smooh siuaion, we will consider he corresponding differenial inclusion 48 ẍ + α ẋ + Φx + Ψx g. This dynamical sysem is wihin he following framewor 49 ẍ + aẋ + Θx g, where Θ : H R {+ } is a proper lower semiconinuous convex funcion, and a is a posiive damping parameer. I is ineresing o esablish he asympoic properies, as +, of he soluions of he differenial inclusion 48. Beyond global exisence issues, one mus chec ha he Lyapunov analysis is sill valid. In view of he validiy of he subdifferenial inequaliy for convex funcions, he generalized chain rule for derivaives over curves see [], mos resuls presened in he previous secions can be ransposed o his more general conex, excep for he sabilizaion of he acceleraion, which relies on he Lipschiz characer of he gradien. However, a deailed sudy of his differenial inclusion goes far beyond he scope of he presen aricle. See [0] for some resuls in he case of a fixed posiive damping parameer, i.e., a = γ > 0 fixed, and g = 0. Thus, seing Θx = Φx + Ψx, we can reasonably assume ha, for α > 3, and + g d < +, for each rajecor! y of 48, here is rapid convergence of he values Θx min Θ C, and wea convergence of he rajecory o an opimal soluion. We shall use hese ideas as a guideline, in order o inroduce corresponding fas converging algorihms, maing he lin wih Neserov [3]-[35] and Bec-Teboulle [0]; and so, exending he recen wors of Chambolle-Dossal [7] and Su-Boyd-Candès [44] o he perurbed case. In order o preserve he fas convergence properies of he dynamical sysem 48, we are going o discreize i impliciely wih respec o he nonsmooh funcion Φ, and expliciely wih respec o he smooh funcion Ψ. Taing a fixed ime sep size h > 0, and seing = h, x = x he implici/explici finie difference scheme for 48 gives 50 h x + x + x + α h x x + Φx + + Ψy g, where y is a linear combinaion of x and x, ha will be made precise laer on. Afer developing 50, we obain 5 x + + h Φx + x + α x x h Ψy + h g. A naural choice for y leading o a simple formulaion of he algorihm oher choices are possible, offering new direcions of research for he fuure is 5 y = x + α x x. Using he classical proximal operaor equivalenly, he resolven of he maximal monoone operaor Φ { 53 prox γφ x = argmin ξ H Φξ + } ξ x = I + γ Φ x, γ and seing s = h, he algorihm can be wrien as y = x + α x x 54 x + = prox sφ y s Ψy g.

18 8 HEDY ATTOUCH, ZAKI CHBANI, JUAN PEYPOUQUET, AND PATRICK REDONT For pracical purposes, and in order o fi wih he exising lieraure on he subjec, i is convenien o wor wih he following equivalen formulaion y = x + +α x x 55 x + = prox sφ y s Ψy g. Indeed, we have +α = α +α. When α is an ineger, we obain he same sequences x and y, up o he reindexaion + α. For general α > 0, we can easily verify ha he algorihm 55 is sill associaed wih he dynamical sysem 48. This algorihm is wihin he scope of he proximal-based inerial algorihms [4], [3], [4], and forward-bacward mehods. In he unperurbed case, g = 0, i has been recenly considered by Chambolle-Dossal [7] and Su-Boyd- Candès [44]. I enjoys fas convergence properies which are very similar o ha of he coninuous dynamic. For α = 3 and g = 0, we recover he classical algorihm based on Neserov and Güler ideas, and developed by Bec-Teboulle FISTA 56 y = x + + x x x + = prox sφ y s Ψy. An imporan quesion regarding he FISTA mehod, as described in 56, is he convergence of sequences x and y, which is sill an open quesion. A major ineres o consider he broader conex of algorihms 55 is ha, for α > 3, hese sequences converge, even when inexacly compued, provided he errors or perurbaions are sufficienly small. 5.. Fas convergence of he values. We will see ha he fas convergence properies of algorihm 54 can be obained in a parallel way wih he convergence analysis in he coninuous case in Theorem 4.. Theorem 5.. Le Φ : H R {+ } be proper, lower-semiconinuous and convex, and le Ψ : H R be convex and coninuously differeniable wih L-Lipschiz coninuous gradien. Suppose ha S = argminφ + Ψ, and le x be a sequence generaed by algorihm 55 wih α 3, 0 < s < L, and N g < +. Then, 57 Φ + Ψx minφ + Ψ j=0 Cα s + α, where C = s α α Θx 0 Θ +α y 0 x +s j + α g j E0 α + s α j + α g j Proof. To simplify noaions, we se Θ = Φ + Ψ, and ae x argmin Θ. As in he coninuous case, we shall prove ha he energy sequence E given by 58 E := s α + α Θx Θx + α z x + wih 59 z := + α y α α x, j=0 s j + α g j, z j+ x, is non-increasing we shall jusify furher ha i is well defined. Noe ha E equals he Lyapunov funcion considered by Su-Boyd-Candès in [44, Theorem 4.3], plus a perurbaion erm. For each y H, we se Ψ y := Ψy g, y, and Θ y = Φy + Ψ y. Since Ψ y = Ψy g, we deduce ha Ψ is sill L-Lipschiz coninuous. By inroducing he operaor G s, : H H, defined by j= G s, y = s y prox sφ y s Ψ y for each y H, we can wrie and rewrie algorihm 55 as 60 prox sφ y s Ψ y = y sg s, y, y = x + +α x x ; x + = y sg s, y.

19 FAST CONVERGENCE OF INERTIAL DYNAMICS AND ALGORITHMS WITH ASYMPTOTIC VANISHING DAMPING 9 The variable z, defined in 59, will play an imporan role. I comes naurally ino play as a discree version of he erm α ẋ + x x which eners E α,g. Simple algebraic manipulaions give 6 z + = + α x + α α x = + α x + x + x, α and also 6 z + = + α α The operaor G s, saisfies y sg s, y α x = z s α + α G s,y. 63 Θ y sg s, y Θ x + G s, y, y x s G s,y. for all x, y H see [0], [7], [38], [44], since s L, and Ψ is L-lipschiz coninuous. Le us wrie successively his formula a y = y and x = x, hen a y = y and x = x. We obain Θ y sg s, y Θ x + G s, y, y x s G s,y, and Θ y sg s, y Θ x + G s, y, y x s G s,y, +α respecively. Muliplying he firs inequaliy by, and he second one by α inequaliies, and using he fac ha x + = y sg s, y, we obain Θ x + + α Θ x + α + α Θ x s G s,y + G s, y, + α y x + α + α y x. We rewrie he scalar produc above as G s, y, + α y x + α + α y x We obain 64 Θ x + = α + α = α + α G s, y, +α, hen adding he wo resuling α y x + y x G s, y, + α y α = α + α G s,y, z x. α x x + α Θ x + α + α Θ x + α + α G s,y, z x s G s,y. We shall obain a recursion from 64. To his end, observe ha 6 gives Afer developing z + x = z x s α + α G s,y. z + x = z x s α + α z x s, G s, y + α + α G s, y, and muliplying he above expression by α s+α, we obain α s + α z x z + x = α + α G s,y, z x s G s,y. Replacing his expression in 64, we obain Equivalenly, Θ x + Θ x + Θ x + α Θ x + α + α Θ x α + s + α z x z + x. + α Θ x Θ x α + s + α z x z + x.

20 0 HEDY ATTOUCH, ZAKI CHBANI, JUAN PEYPOUQUET, AND PATRICK REDONT Recalling ha Θy = Θ y + g, y, we obain Θx + Θx + α Θx Θx α + s + α z x z + x + g, x + x + α g, x x = + α Θx Θx α + s + α z x z + x + g, x + x + α + α x x. Muliplying by s α + α, we obain s α + α Θx + Θx s α + α Θx Θx + α z x z + x + s + α g, x + x + α α + α x x, which implies 65 in view of s α + α Θx + Θx + s α 3 + α z x z + x + s + α α α Θx Θx s α + α Θx Θx g, x + x + α + α x x, + α = + α α 3 α + α α 3. Seing G = s α + α Θx Θ + α z x, we can reformulae 65 as G + + s α 3 α Θx Θx G + s + α g, x + x + α α + α x x. Equivalenly, G + + s α 3 α Θx Θx G + s + α g, + α x + x + x x. α Using 6, we deduce ha 66 G + + s α 3 α Θx Θx G + s + α g, z + x. Fix an ineger K, and se K E K = G + s j + α g j, z j+ x, so ha 66 is equivalen o j= E K + + s α 3 α Θx Θx E K, and we deduce ha he sequence E K is nonincreasing. In paricular, E K E K 0, which gives As a consequence, K K G + s j + α g j, z j+ x G0 + s j + α g j, z j+ x. j= 67 G G0 + s j + α g j, z j+ x. j=0 By he definiion of G, neglecing some posiive erms, and using he Cauchy-Schwarz inequaliy, we infer ha z x α G0 + s α j=0 j + α g j z j x. j=

21 FAST CONVERGENCE OF INERTIAL DYNAMICS AND ALGORITHMS WITH ASYMPTOTIC VANISHING DAMPING Applying Lemma A.9 wih a = z x, we deduce ha G0 68 z x M := α + s α j + α g j. Noe ha M is finie, because N g < +. Reurning o 67 we obain G C := G0 + s j + α g j G0 α + s j + α g j. α j=0 j=0 j=0 By he definiion of G, we finally obain which gives 57 and complees he proof. s α + α Θx Θ C. Remar 5.. In [9], Kim-Fessler inroduce an exra inerial erm in he FISTA mehod ha allows hem o reduce he consan by a facor of in he complexiy esimaion. I would be ineresing o now wheher his varian can be obained by anoher discreizaion in ime of our inerial dynamic, or a differen one Convergence of he sequence x. Le us now sudy he convergence of he sequence x. Theorem 5.3. Le Φ : H R {+ } be proper, lower-semiconinuous and convex, and le Ψ : H R be convex and coninuously differeniable wih L-Lipschiz coninuous gradien. Suppose ha S = argminφ + Ψ, and le x be a sequence generaed by algorihm 55 wih α > 3, 0 < s < L, and N g < +. Then, i Φ + Ψx infφ + Ψ < + ; ii x + x < + ; and iii x converges wealy, as +, o some x argmin Φ. Proof. We follow he same seps as hose of Theorem 4.4: Sep. Recall from 66 ha G + + s α 3 α Θx Θx G + s + α g, z + x. By 68, he sequence z is bounded. Summing he above inequaliies, and using α > 3, we obain iem i. Sep. Rewrie inequaliy 63 as Θ y sg s, y + s y sg s,y x Θ x + s x y. Tae y = y, and x = x. Since x + = y sg s, y, and y x = By he definiion of Θ, his is +α x x, we obain Θ x + + s x + x Θ x + s + α x x. 69 Θx + + s x + x Θx + s + α x x + g, x + x. Se θ = Θx Θx, d = x x, a = α. By he Cauchy-Schwarz inequaliy, 69 gives d + s + a d θ θ + + g x + x. Muliply by + a o ge + a d + d + a θ θ a g x + x. s Summing for =,..., K, we obain K + a d + K K d s + a θ θ + + s + a g x + x. = = =

22 HEDY ATTOUCH, ZAKI CHBANI, JUAN PEYPOUQUET, AND PATRICK REDONT Performing a similar compuaion as in Chambolle-Dossal [7, Corollary ], we can wrie K K + a d K+ + a + a d 70 = K K s a + θ K + a θ K+ + + a θ + + a g x + x. By iem i, we have + a θ < +. Hence here exiss some consan C such ha = 7 K + a x K+ x K C + s = K + a g x + x for all K N. We now proceed as in he proof of Theorem 4.4. To his end, wrie 7 as a C + s j + a g j a j, j= where a j := j + a x j+ x j. Recalling ha g < +, apply Lemma A.9 wih β j = j + a g j o deduce ha 7 sup x + x < +. Injecing his informaion in 70, we obain a + a d C + = + a θ + sup + a x + x Since a = α, iem i and he definiion of d, ogeher give x+ x < +, + a g. which is ii. Sep 3. We finish by applying Opial s Lemma A.3 wih S = argminφ+ψ. By Theorem 5., we have Φ+Ψx minφ + Ψ. The wea lower-semiconinuiy of Φ + Ψ gives iem ii of Opial s Lemma. Thus, he only poin o verify is ha lim x x exiss for each x argminφ + Ψ. Tae any such x. We shall show ha lim h exiss, where h := x x. The beginning of he proof is similar o [4] or [7], and consiss in esablishing a discree version of he second-order differenial inequaliy 46. We use he ideniy a b + a c = b c + a b, a c, which holds for any a, b, c H. Taing b = x, a = x +, c = x, we obain x + x + x + x = x x + x + x, x + x, which is equivalen o 73 h h + = x + x + x + x, x x +. By he definiion of y, we have Therefore, x x + = y x + + α x x. 74 h h + = x + x + x + x, y x + + α x + x, x x. We now use he monooniciy of Φ. Since s Ψx s Φx, and y x + s Ψy + sg s Φx +, we have y x + s Ψy + sg + s Ψx, x + x 0. Equivalenly, Replacing in 74, we obain y x +, x + x + s Ψx Ψy + g, x + x h + h + x + x + s Ψy Ψx g, x + x + α x + x, x x 0.