A forward-backward penalty scheme with inertial effects for montone inclusions. Applications to convex bilevel programming

Transcription

1 A forward-backward enalty scheme with inertial effects for montone inclusions. Alications to convex bilevel rogramming Radu Ioan Boţ Dang-Khoa Nguyen January 13, 2018 Abstract. We investigate forward-backward slitting algorithm of enalty tye with inertial effects for finding the zeros of the sum of a maximally monotone oerator and a cocoercive one and the convex normal cone to the set of zeroes of an another cocoercive oerator. Weak ergodic convergence is obtained for the iterates, rovided that a condition exress via the Fitzatrick function of the oerator describing the underlying set of the normal cone is verified. Under strong monotonicity assumtions, strong convergence for the sequence of generated iterates can be roved. As a articular instance we consider a convex bilevel minimization roblems including the sum of a nonsmooth and a smooth function in the uer level and another smooth function in the lower level. We show that in this context weak nonergodic and strong convergence can be also achieved under inf-comactness assumtions for the involved functions. Keywords. maximally monotone oerator, Fitzatrick function, forward-backward slitting algorithm, convex bilevel otimization AMS subject classification. 47H05, 65K05, 90C25 1 Introduction and reliminaries 1.1 Motivation and roblems formulation During the last coule years one can observe in the otimization community an increasing interest in numerical schemes for solving variational inequalities exressed as monotone inclusion roblems of the form 0 P Ax ` N M xq, (1.1) where H is a real Hilbert sace, A: H Ñ H is a maximally monotone oerator, M : arg min h is the set of global minima of the roer, convex and lower semicontinuous function h: R Ñ sr : R Y t 8u and N M : H Ñ H is the normal cone of the set M. The article [7] was starting oint for a series of aers [6, 9, 10, 12, 18, 19, 24, 25, 33, 37, 38] addressing this toic or related ones. All these aers share the common feature that the roosed iterative schemes use enalization strategies, namely, by evaluating the enalized h by its gradient, in case the function is smooth (see, for instance, [9]), and by its roximal oerator, in case it is nonsmooth (see, for instance,[10]). Weak ergodic convergence has been obtained in [9, 10] under the hyothesis: For all P RanN M, ÿ j λ n β n h σ M ă `8, (1.2) β n β n Faculty of Mathematics, University of Vienna, Oskar-Morgenstern-Platz 1, 1090 Vienna, Austria, radu.bot@univie.ac.at. Research artially suorted by FWF (Austrian Science Fund), roject I 2419-N32. Faculty of Mathematics, University of Vienna, Oskar-Morgenstern-Platz 1, 1090 Vienna, Austria, dang-khoa.nguyen@univie.ac.at. The author gratefully acknowledges the financial suort of the Doctoral Programme Vienna Graduate School on Comutational Otimization (VGSCO) which is funded by Austrian Science Fund (FWF, roject W1260-N35). 1

2 with λ n q, the sequence of ste sizes, β n q, the sequence of enalty arameters, h : H Ñ sr, the Fenchel conjugate function of h, and RanN M the range of the normal cone oerator N M : H Ñ H. Let us mention that (1.2) is the discretized counterart of a condition introduced in [7] for continuous-time nonautonomous differential inclusions. One motivation for studying numerical algorithms for monotone inclusions of tye (1.1) comes from the fact that, when A Bf is the convex subdifferential of a roer, convex and lower semicontinuous function f : H Ñ R, s they furnish iterative methods for solving bilevel otimization roblems of the form min tf xq : x P arg min hu. (1.3) xph Among the alications where bilevel rogramming roblems lay an imortant role we mention ithe modelling of Stackelberg games, the determination of Wardro equilibria for network flows, convex feasibility roblems [5], domain decomosition methods for PDEs [4], image rocessing roblems [18], and otimal control roblems [10]. Later on, in [19], the following monotone inclusion roblem, which turned out to be more suitable for alications, has been addressed in the same sirit of enalty algorithms 0 P Ax ` Dx ` N M xq, (1.4) where A: H Ñ H is a maximally monotone oerator, D : H Ñ H is cocoercive oerator and the constraint set M is the set of zeros of another cocoercive oerator B : H Ñ H. The rovided algorithm of forward-backward tye evaluates the oerator A by a backward ste and the two single-valued oerators by forward stes. For the convergence analysis, (1.2) has been relaced by a condition formulated in terms of the Fitzatrick function associated with the oerator B, which we will also use in this aer. In [12], several articular situations for which this new condition is fulfilled have been rovided. The aim of this work is to endow the forward-backward enalty scheme for solving (1.4) from [19] with inertial effects, which means that the new iterate is defined in terms of the revious two iterates. Inertial algorithms have their roots in the time discretization of second order differential systems [3]. They can accelerate the convergence of iterates when minimizing a differentiable function [39] and the convergence of the objective function values when minimizing the sum of a convex nonsmooth and a convex smooth function [15, 28]. Moreover, as emhasized in [16], see also [23], algorithms with inertial effects may detect otimal solutions of minimization roblems which cannot be found by their noninertial variants. In the last years, a huge interest in inertial algorithms can be notices (see, for instance, [1, 2, 3, 8, 11, 15, 20, 21, 22, 23, 24, 25, 29, 30, 34, 35, 36]). We rove weak ergodic convergence of the sequence generated by the inertial forwardbackward enalty algorithm to a solution of the monotone inclusion roblem (1.4), under reasonable assumtions for the sequences of ste sizes, enalty and inertial arameters. When the oerator A is assumed to be strongly monotone, we also rove strong convergence of the generated iterates to the unique solution of (1.4). In Section 3, we address the minimization of the sum of a convex nonsmooth and a convex smooth function with resect to the set of minimizes of another convex and smooth function. Besides the convergence results obtained from the general case, we achieve weak nonergodic and strong convergence statements under inf-comactness assumtions for the involved functions. The weak nonergodic theorem is an useful alternative to the one in [25], where a similar statement has been obtained for the inertial forward-bacward enalty algorithm with constant inertial arameter under assumtions which are quite comlicated and hard to verify (see also [37, 38]). 2

3 1.2 Notations and reliminaries In this subsection we introduce some notions and basic results which we will use throughout this aer (see [13, 17, 40]). Let H be a real Hilbert sace with inner roduct x, y and associated norm a x, y. For a function Ψ: H Ñ R s : R Y t 8u, we denote DomΨ tx P H: Ψ xq ă `8u its effective domain and say that Ψ is roer, if DomΨ H and Ψ xq ą 8 for all x P H. The conjugate function of Ψ is Ψ : H Ñ R, s Ψ uq su xph txx, uy Ψ xqu. The convex subdifferential of Ψ at the oint x P H is the set BΨ xq t P H: xy x, y ď Ψ yq Ψ P Hu, whenever Ψ xq P R. We take by convention BΨ xq H, if Ψ xq P t 8u. Let M be a nonemty subset of H. The indicator function of M, which is denoted by δ M : H Ñ R, s takes the value 0 on M and `8 otherwise. The convex subdifferential of the indicator function is the normal cone of M, that is N M xq t P H: xy x, y ď P Hu, if x P M, and N M xq H otherwise. Notice that for x P M we have P N M xq if and only if σ M xq xx, y, where σ M δ M is the suort function of M. For an arbitrary set-value oerator A: H Ñ H we denote by GrA tx, vq P H ˆ H: v P Axu its grah, by DomA tx P H: Ax Hu its domain, by RanA tv P H: Dx P H with v P Axu its range and by A 1 : H Ñ H its inverse oerator, defined by v, xq P GrA 1 if and only if x, vq P GrA. We use also the notation ZerA tx P H: 0 P Axu for the set of zeros of the oerator A. We say that A is monotone, if xx y, v wy ě 0 for all x, vq, y, wq P GrA. A monotone oerator A is said to be maximally monotone, if there exists no roer monotone extension of the grah of A on H ˆ H. Let us mention that if A is maximally monotone, then ZerA is a convex and closed set, [13, Proosition 23.39]. We refer to [13, Section 23.4] for conditions ensuring that ZerA is nonemty. If A is maximally monotone, then one has the following characterization for the set of its zeros z P ZerA if and only if xu z, yy ě 0 for all u, yq P GrA. (1.5) The oerator A is said to be γ strongly monotone with γ ą 0, if xx y, v wy ě x y 2 for all x, vq, y, wq P GrA. If A is maximally monotone and strongly monotone, then ZerA is a singleton, thus nonemty, [13, Corollary 23.27]. The resolvent of A, J A : H Ñ H, is defined by J A : Id ` Aq 1, where Id: H Ñ H denotes the identity oerator on H. If A is maximally monotone, then J A : H Ñ H is single-value and maximally monotone, [13, Proosition 23.7, Corollary 23.10]. For an arbitrary γ ą 0, we have the following identity ([13, Proosition 23.18]) J γa ` γj γ 1 A 1 γ 1 Id Id. We denote Γ Hq the family of roer, convex and lower semicontinuous extended real-valued functions defined on H. When Ψ P Γ Hq and γ ą 0, we denote by rox γψ xq the roximal oint with arameter γ of function Ψ at oint x P H, which is the unique otimal solution of the otimization roblem " inf Ψ yq ` 1 * y x 2. yph 2γ Notice that J γbψ Id ` γbψq 1 rox γψ, thus rox γψ : H Ñ H is a single-valued oerator fulfilling the so-called Moreau s decomosition formula: rox γψ ` γrox γ 1 Ψ γ 1 Id Id. The function Ψ: H Ñ s R is said to be γ strongly convex with γ ą 0, if Ψ γ 2 2 is a convex function. This roerty imlies that BΨ is γ strongly monotone. 3

4 The Fitzatrick function ([32]) associated to a monotone oerator A is defined as ϕ A : H ˆ H Ñ R, s ϕ A x, uq : su txx, vy ` xy, uy xy, vyu y,vqpgra and it is a convex and lower semicontinuous function. For insights in the outstanding role layed by the Fitzatrick function in relatin the convex analysis with the theory of monotone oerators we refer to [13, 14, 17, 26, 27] and the references therein. If A is maximally monotone, then ϕ A is roer and it fulfills ϕ A x, uq ě xx, x, uq P H ˆ H, with equality if and only if x, uq P GrA. Notice that if Ψ P Γ Hq, then BΨ is a maximally monotone oerator and it holds BΨq 1 BΨ. Furthermore, the following inequality is true (see [14]): ϕ BΨ x, uq ď Ψ xq ` Ψ x, vq P H ˆ H. (1.6) We resent as follows some statements that will be essential when carrying out the convergence analysis. Let x n q ně0 be a sequence in H and λ n q be a sequenceof ositive real numbers. The sequence of weighted averages z n q is defined for every n ě 1 as z n : 1 τ n nÿ λ k x k, where τ n : k 1 nÿ λ k. (1.7) Lemma 1.1 (Oial-Passty). Let Z be a nonemty subset of H and assume that the limit lim x n u exists for every element u P Z. If every sequential weak cluster oint of x n q ně0, resectively z n q, lies in Z, then the sequence x n q ně0, resectively z n q, converges weakly to an element in Z as n Ñ `8. Two following result can be found in [12, 19]. Lemma 1.2. Let θ n q ně0, ξ n q and δ n q be sequences in R` with δ n q P l 1. If there exists n 0 ě 1 such that and α such that then the following statements are true: k 1 θ n`1 θ n ď α n θ n θ n 1 q ξ n ` δ ě n 0 0 ď α n ď α ă ě 1, iq ÿ rθ n θ n 1 s` ă `8, where rss` : max ts, 0u; iiq the limit lim nñ8 θ n exists. iiiq the sequence ξ n q belongs to l 1. The following result follows from Lemma 1.2, alied in case α n : 0 and θ n : ρ n ρ for all n ě 1, where ρ is a lower bound for ρ n q. Lemma 1.3. Let ρ n q be a sequence in R, which is bounded from below, and ξ n q, δ n q be sequences in R` with δ n q P l 1. If there exists n 0 ě 1 such that then the following statements are true: iq the sequence ρ n q is convergent. ρ n`1 ď ρ n ξ n ` δ ě n 0, 4

5 iiq the sequence ξ n q belongs to l 1. The following result, which will be useful in this work, shows that statement (ii) in Lemma 1.3 can be obtained also when ρ n q is not bounded by below, but it has a articular form. Lemma 1.4. Let ρ n q be a sequence in R and ξ n q, δ n q be sequences in R` with δ n q P l 1 and ρ n : θ n α n θ n 1 ` χ ě 1, where θ n q ně0, χ n q are sequences in R` and there exists α such that If there exists n 0 ě 1 such that then the sequence ξ n q belongs to l 1. 0 ď α n ď α ă ě 1. ρ n`1 ρ n ď ξ n ` δ ě n 0, (1.8) Proof. We fix an integer s N ě n 0, sum u the inequalities in (1.8) for n n 0, n 0 ` 1,, s N and obtain snÿ snÿ ÿ ρ N`1 s ρ n0 ď ξ n ` δ n ď δ n ă `8. (1.9) n n 0 n n 0 Hence the sequence tρ n u is bounded from above. sequence. For all n ě 1 it holds Let sρ ą 0 be an uer bound of this from which we deduce that By induction we obtain for all n ě n 0 ` 1 θ n αθ n 1 ď θ n α n θ n 1 ` χ n ρ n ď sρ, ρ n ď θ n ` αθ n 1 ď αθ n 1. (1.10) n n ÿ 0 θ n ď αθ n 1 ` sρ ď ď α n n 0 θ n0 ` sρ k 1 α k 1 ď α n n 0 θ n0 ` Then inequality (1.9) combined with (1.10) and (1.11) leads to sρ 1 α. (1.11) snÿ ÿ ξ n ď ρ n0 ρ N`1 s ` δ n ď ρ n0 ` αθ N s ` n n 0 n n 0 snÿ ď ρ n0 ` α s N n 0`1 θ n0 ` αsρ 1 α ` ÿ δ n δ n ă `8 (1.12) We let s N converge to `8 and obtain that ÿ ξ n ă `8. 2 The general monotone inclusion roblem In this section we address the following monotone inclusion roblem. Problem 2.1. Let H be a real Hilbert sace, A: H Ñ H a maximally monotone oerator, D : H Ñ H an η cocoercive with η ą 0, B : H Ñ H a µ cocoercive with µ ą 0 and assume that M : Zer B H. The monotone inclusion roblem to solve reads 0 P Ax ` Dx ` N M xq. 5

6 The following forward-backward enalty algorithm with inertial effects for solving Problem 2.1 will be in the focus of our investigations in this aer. Algorithm 2.2. Let α n q, λ n q and β n q be sequences of ositive real numbers such that C 1 q tλ n u P l 2 z l 1 ; C 2 q tα n u is nondecreasing; C 3 q 0 ď α n ď α ă `8 for all n ě 1. H fitz Let x 0, x 1 P H. For all n ě 1 we set x n`1 : J λna x n λ n Dx n λ n β n Bx n ` α n x n x n 1 qq. When D 0 and B h, where h : H Ñ R is a convex and differentiable function with µ 1 Lischitz continuous gradient with µ ą 0 fulfilling min h 0, then Problem 2.1 recovers the monotone inclusion roblem addressed in [9, Section 3] and Algorithm 2.2 can be seen as an inertial version of the iterative scheme considered in this aer. When B 0, we have that N M t0u and Algorithm 2.2 is nothing else than the inertial version of the classical forward-backward algorithm (see for instance [13, 31]). Hyotheses 2.3. The convergence analysis will be carry out in the following hyotheses (see also [19]): 1 q A ` N M is maximally monotone and Zer A ` D ` N M q H; H fitz 2 q for every P RanN M, ÿ j λ n β n su ϕ B ˆu, σ M ă `8. upm β n β n Since A and N M are maximally monotone oerators, the sum A ` N M is maximally monotone, rovided some secific regularity conditions are fulfilled (see [13, 17, 26, 40]). Furthermore, since D is also maximally monotone [13, Examle 20.28] and DomD H, if A ` N M is maximally monotone, then A ` D ` N M is also maximally monotone. Let us also notice that for P RanN M there exists u P M such that P N M uq, hence, for every β ą 0 it holds ˆ su ϕ B u, upm β B σ M ě u, F σ M 0. β β β For situations where H fitz 2 q is satisfied we refer the reader [12, 24, 25, 37]. Before formulating the main theorem of this section we will rove some useful technical results. Lemma 2.4. Let x n q ně0 be the sequence generated by Algorithm 2.2 and u, yq be an element in Gr A ` D ` N M q such that y v ` Du ` with v P Au and P N M uq. Further, let ε 1, ε 2, ε 3 ą 0 be such that 1 ε 3 ą 0. Then the following inequality holds for all n ě 1 x n`1 u 2 x n u 2 ď α n x n u 2 α n x n 1 u 2 1 ε 2 q x n`1 x n 2 ` α n ` α2 n x n x n ` λ 2 ε nβn 2 2µ 1 ε 3 q λ n β n Bx n ` λ 2 n 2ηλ n Dx n Du 2 ` 4 λ 2 n Du ` v 2 ε 2 ε 2 j ` 2ε 3 λ n β n su ϕ B ˆu, σ M ` 2λ n xu x n, yy. (2.1) upm ε 3 β n ε 3 β n 6

7 Proof. Let n ě 1 be fixed. According to definition of the resolvent of the oerator A we have x n x n`1 λ n Dx n ` β n Bx n q ` α n x n x n 1 q P λ n Ax n`1 (2.2) and, since λ n v P λ n Au, the monotonicity of A guarantees or, equivalently, and xx n`1 u, x n x n`1 λ n Dx n ` β n Bx n ` vq ` α n x n x n 1 qy ě 0 (2.3) 2 xu x n`1, x n x n`1 y ď 2λ n xu x n`1, β n Bx n ` Dx n ` vy 2α n xu x n`1, x n x n 1 y. (2.4) For the term in the left-hand side of (2.4) we have Since 2 xu x n`1, x n x n`1 y x n`1 u 2 ` x n`1 x n 2 x n u 2. (2.5) 2α n xu x n, x n x n 1 y α n u x n 1 2 ` α n u x n 2 ` α n x n x n xx n`1 x n, α n x n x n 1 qy ď x n`1 x n 2 ` α2 n x n x n 1 2, by adding the two inequalities, we obtain the following estimation for the second term in the right-hand side of (2.4) 2α n xu x n`1, x n x n 1 y ď α n x n u 2 α n x n 1 u 2 ` x n`1 x n 2 ` α n ` α2 n x n x n 1 2. (2.6) We turn now our attention to the first term in the right-hand side of (2.4), which can be written as 2λ n xu x n`1, β n Bx n ` Dx n ` vy 2λ n xu x n, β n Bx n ` Dx n ` vy ` 2λ n β n xx n x n`1, Bx n y ` 2λ n xx n x n`1, Dx n ` vy. (2.7) We have and 2λ n β n xx n x n`1, Bx n y ď ε 2 2 x n`1 x n 2 ` 2 ε 2 λ 2 nβ 2 n Bx n 2 (2.8) 2λ n xx n x n`1, Dx n ` vy ď ε 2 2 x n`1 x n 2 ` 2 ε 2 λ 2 n Dx n ` v 2 On the other hand, we have 2λ n xu x n, β n Bx n ` Dx n ` vy ď ε 2 2 x n`1 x n 2 ` 4 ε 2 λ 2 n Dx n Du 2 ` 4 ε 2 λ 2 n Du ` v 2. (2.9) 2λ n β n xu x n, Bx n y ` 2λ n xu x n, Dx n Duy ` 2λ n xu x n, Du ` vy. (2.10) Since 0 ă ε 3 ă 1 and Bu 0, the cocoercivity of B gives us 2λ n β n xu x n, Bx n y ď 2µ 1 ε 3 q λ n β n Bx n 2 ` 2ε 3 λ n β n xu x n, Bx n y. (2.11) 7

8 Similarly, the cocoercivity of D gives us 2λ n xu x n, Dx n Duy ď 2ηλ n Dx n Du 2. (2.12) Combining (2.11) ˆ - (2.12) Bwith (2.10) F and by using the definition Fitzatrick function and the fact that σ M u,, we obtain ε 3 β n ε 3 β n 2λ n xu x n, β n Bx n ` Dx n ` vy ď 2µ 1 ε 3 q λ n β n Bx n 2 ` 2ε 3 λ n β n xu x n, Bx n y 2ηλ n Dx n Du 2 ` 2λ n xu x n, Du ` vy 2µ 1 ε 3 q λ n β n Bx n 2 ` 2ε 3 λ n β n xu x n, Bx n y 2ηλ n Dx n Du 2 ` 2λ n xu x n, y y 2µ 1 ε 3 q λ n β n Bx n 2 2ηλ n Dx n Du 2 ` 2λ n xu x n, yy B F B F ` 2ε 3 λ n β n ˆxu, Bx n y ` x n, xx n, Bx n y u, ε 3 β n ε 3 β n ď 2µ 1 ε 3 q λ n β n Bx n 2 2ηλ n Dx n Du 2 ` 2λ n xu x n, yy j ` 2ε 3 λ n β n su ϕ B ˆu, σ M. (2.13) upm ε 3 β n ε 3 β n The inequalities (2.8), (2.9) and (2.13) lead to 2λ n xu x n`1, β n Bx n ` Dx n ` vy 2 ď λ 2 ε nβn 2 2µ 1 ε 3 q λ n β n Bx n 2 4 ` λ 2 n 2ηλ n Dx n Du 2 ` ε 2 x n`1 x n 2 2 ε 2 ` 4 j λ 2 n Du ` v 2 ` 2ε 3 λ n β n su ϕ B ˆu, σ M ` 2λ n xu x n, yy. (2.14) ε 2 upm ε 3 β n ε 3 β n Finally, by combining (2.5), (2.6) and (2.14), we obtain (2.1). From now on we will assume that for 0 ă α ă 1 3 the constants ε 1, ε 2, ε 3 ą 0 and the sequences λ n q and β n q are chosen such that C 4 q 1 ε 3 ą 0, ε 2 ă 1 α α2 and su λ n β n ă µε 2 1 ε 3 q. As a consequence, there exists 0 ă s ď 1 1 3ε 1 ε 2 n ě 1 it holds α n`1 ` α2 n`1 ε 1 ˆ 1 ` α 2, which means that for all 2ε 1 1 ε 3 q ď α ` α2 1 ε 3 q ă s, (2.15) On the other hand, there exists 0 ă t ď µ 1 ε 2 q 1 su λ n β n, which means that for all n ě 1 ε 3 ně0 it holds 1 λ n β n µ 1 ε 2 q ď t. (2.16) ε 3 Remark 2.5. iq Since 0 ă α ă 1 3, one can always find ε 1, ε 2 ą 0 such that ε 2 ă 1 α α 2. One ossible choice is ε 1 α 4 and 0 ă ε 2 ă 1 3α. From the second inequality in C 4 q it follows that 1 3ε 1 ε 2 ą ε 1 ` α ` α2 ą 0. 8

9 iiq As ε ε 1 ε 2 ˆ 1 ` α 2 2ε 1 it is always ossible to choose s such that 0 ă s ď 1 this case s ă 1 ε 2 α α2, one has (2.15). 1 ˆ1 ε 2 α α2 ą 0, 1 3ε 1 ε 2 The following roosition brings us closer to the convergence result. ˆ ε 1 1 ` α 2. Since in 1 3ε 1 ε 2ε 1 Proosition 2.6. Let 0 ă α ă 1 3, ε 1, ε 2, ε 3 ą 0 and the sequences λ n q and β n q satisfy condition C 4 q. Let x n q ně0 be the sequence generated by Algorithm 2.2 and assume that the Hyotheses 2.3 are verified. Then the following statements are true: iq the sequence x n`1 x n q ně0 belongs to l 2 and the sequence λ 2 n β n Bx n belongs to l 1 ; iiq if, moreover, lim inf λ nβ n ą 0, then sequence x n q ně0 lies in M. iiiq for every u P Zer A ` D ` N M q, the limit lim Bx n 0 and thus every cluster oint of the lim x n u exists. Proof. Since lim λ n 0, there exists a integer n 1 ě 1 such that λ n ď 2 η for all n ě n 0. ε 2 According to Lemma 2.4, for every u, yq P Gr A ` D ` N M q such that y v ` Du `, with v P Au and P N M uq, and all n ě n 0 the following inequality holds x n`1 u 2 x n u 2 ď α n x n u 2 α n x n 1 u 2 1 ε 2 q x n`1 x n 2 ` α n ` α2 n x n x n ` λ n β n 2µ 1 ε 3 q λ n β n Bx n 2 ε 2 ` 4 j λ 2 n Du ` v 2 ` 2ε 3 λ n β n su ϕ B ˆu, σ M ` 2λ n xu x n, yy. (2.17) ε 2 upm ε 3 β n ε 3 β n We consider u P Zer A ` D ` N M q, which means that we can take y 0 in (2.17). For all n ě 1 we denote θ n : x n u 2, ρ n : θ n α n θ n 1 ` α n ` α2 n x n x n 1 2 (2.18) and δ n : 4 λ 2 n Du ` v 2 ` 2ε 3 λ n β n su ϕ B ˆu, ε 2 upm Using that α n q is nondecreasing, for all n ě n 0 it yields ˆ ρ n`1 ρ n ď α n`1 ` α2 n`1 ε 3 β n 1 ε 2 q ˆ 2 ` λ n β n 2µ 1 ε 2 q ε 3 j σ M. (2.19) ε 3 β n x n`1 x n 2 λ n β n Bx n 2 ` δ n ď s x n`1 x n 2 2tλ n β n Bx n 2 ` δ n, (2.20) where s, t ą 0 are chosen according to (2.15) and (2.16), resectively. 9

10 Thanks to H fitz 2 q and C 1q it holds ÿ δ n 4 ε 2 Du ` v 2 ÿ λ 2 n ` 2 ÿ Hence, according to Lemma 1.4, we obtain ÿ ε 3 λ n β n su ϕ B ˆu, upm ně0 x n`1 x n 2 ă `8 and ÿ ε 3 β n j σ M ă `8. ε 3 β n (2.21) λ n β n Bx n 2 ă `8, (2.22) which roves (i). If, in addtion lim inf λ nβ n ą 0, then lim Bx n 0, which means every nñ8 cluster oint of the sequence x n q ně0 lies in Zer B M. In order to rove (iii), we consider again the inequality (2.17) for an arbitrary element u P Zer A ` D ` N M q and y 0. With the notations in (2.18) and (2.19), we get for all n ě n 0 θ n`1 θ n ď α n θ n θ n 1 q ` α n ` α2 n x n x n 1 2 ` δ n. (2.23) According to (2.21) and (2.22) we have ÿ α n ` α2 n x n x n 1 2 ` ÿ ÿ δ n ď ˆα ` α2 x n x n 1 2 ` ÿ δ n ă `8, (2.24) 4ε 1 4ε 1 therefore, by Lemma 1.2, the limit limit lim x n u exists, too. lim θ n lim x n u 2 exists, which means that the Remark 2.7. The condition C 3 q that we imosed in combination with 0 ă α ă 1 3 on the sequence of inertial arameters α n q is the one roosed in [3, Proosition 2.1] when addressing the convergence of the inertial roximal oint algorithm. However, the statements in roosition above and in the following convergence theorem remain valid if one alternatively assumes that there exists α 1 such that 0 ď α n ď α 1 ă 1 for all n ě 1 and ÿ α n ` α2 n x n x n 1 2 ă `8. This can be realized if one chooses for a fixed ą 1 ˆ α n ď min "α 1, 2ε 1 1 ` b1 ` n x n x n 1 2 ě 1. Indeed, in this situation we have that α2 n `α n ÿ 1 n 2 ď 0 for all n ě 1, which gives x n x n 1 α n ` α2 n x n x n 1 2 ď ÿ 1 n ă `8. Now we are ready to rove the main theorem of this section, which addresses the convergence of the sequence generated by Algorithm 2.2. Theorem 2.8. Let 0 ă α ă 1 3, ε 1, ε 2, ε 3 ą 0 and the sequences λ n q and β n q satisfy condition C 4 q. Let x n q ně0 be the sequence generated by Algorithm 2.2, z n q be the sequence defined in (1.7) and assume that the Hyotheses 2.3 are verified. Then the following statements are true: 10

11 iq the sequence z n q converges weakly to an element in Zer A ` D ` N M q as n Ñ `8. iiq if A is γ strongly monotone with γ ą 0, then x n q ně0 converges strongly to the unique element in Zer A ` D ` N M q as n Ñ `8. Proof. iq According to Proosition 2.6 (iii), the limit lim }x n u} exisits for every u P Zer A ` D ` N M q. Let z be a sequential weak cluster oint of z n q. We will show that z P Zer A ` D ` N M q, by using the characterization (1.5) of the maximal monotonicity, and the conclusion will follow by Lemma 1.1. To this end we consider an arbitrary u, yq P Gr A ` D ` N M q such that y v ` Du `, where v P Au and P N M uq. From (2.17), with the notations (2.18) and (2.19), we have for all n ě n 0 ρ n`1 ρ n ď s x n`1 x n 2 2tλ n β n Bx n 2 ` δ n ` 2λ n xu x n, yy ď δ n ` 2λ n xu x n, yy. (2.25) Recall that from (2.21) that ÿ δ n ă `8. Since x n q ně0 is bounded, the sequence ρ n q is also bounded. We fix an arbitrary integer s N ě n 0 and sum u the inequalities in (2.25) for n n 0 `1, n 0 ` 2,, s N. This yields ρ s N`1 ρ n0`1 ď ÿ δ n ` 2 C ÿn 0 n 1 λ n u ` After dividing this last inequality by 2τ s N 2 where T : ÿ δ n ` 2 ÿn 0 n 1 snÿ n 1 λ n x n, y G ` 2 λ n, we obtain C τ s N u snÿ n 1 λ n x n, y 1 1 `ρ 2τ sn`1 ρ n0`1 ď T ` 2 xu z N s 2τ N s, yy, (2.26) N s C G ÿn 0 n 1 and by using that lim NÑ8 τ s N lim λ n u ` snñ8 n 1 ÿn 0 n 1 snÿ lim inf snñ8 λ n x n, y λ n `8, we get xu z s N, yy ě 0. G P R. By assing in (2.26) to the limit As z is a sequential weak cluster oint of z n q, the above inequality gives us xu z, yy ě 0, which finally means that z P Zer A ` D ` N M q. iiq Let u P H be the unique element in Zer A ` D ` N M q. Since A is γ strongly monotone with γ ą 0, the formula in (2.3) reads for all n ě 1 xx n`1 u, x n x n`1 λ n Dx n ` β n Bx n ` vq ` α n x n x n 1 qy ě γλ n x n`1 u 2 or, equivalently, 2γλ n x n`1 u 2 ` 2 xu x n`1, x n x n`1 y ď 2λ n xu x n`1, β n Bx n ` Dx n ` vy 2α n xu x n`1, x n x n 1 y.. 11

12 By using again (2.5), (2.6) and (2.14) we obtain for all n ě 1 2γλ n x n`1 u 2 ` x n`1 u 2 x n u 2 ď α n x n u 2 α n x n 1 u 2 1 ε 2 q x n`1 x n 2 ` α n ` α2 n x n x n ` λ 2 ε nβn 2 2µ 1 ε 3 q λ n β n Bx n ` λ 2 n 2ηλ n Dx n Du 2 ` 4 λ 2 n Du ` v 2 ε 2 ε 2 j ` 2ε 3 λ n β n su ϕ B ˆu, σ M ` 2λ n xu x n, yy. upm ε 3 β n ε 3 β n By using the notations in (2.18) and (2.19), this yields for all n ě 1 2γλ n x n`1 u 2 ` θ n`1 θ n ď α n θ n θ n 1 q ` α n ` α2 n x n x n 1 2 ` δ n By taking into account (2.24), from Lemma 1.2 we get 2γ ÿ λ n x n u 2 ă `8. According to C 1 q we have ÿ λ n `8, which imlies that the limit lim nñ8 x n u must be equal to zero. This rovides the desired conclusion. 3 Alications to convex bilevel rogramming We will emloy the results obtained in the revious section, in the context of monotone inclusions, to the solving of convex bilevel rogramming roblems. Problem 3.1. Let H be a real Hilbert sace, f : H Ñ s R a roer, convex and lower semicontinuous function and g, h: H Ñ R differentiable functions with L g Lischitz continuous and, resectively, L h Lischitz continuous gradients. Suose that arg min h H and min h 0. The bilevel rogramming roblem to solve reads min f xq ` g xq. xparg min h The assumtion min h 0 is not a restricttive as, otherwise, one can relace h with h min h. Hyotheses 3.2. The convergence analysis will be carry out in the following hyotheses: H rog 1 q Bf ` N arg min h is maximally monotone and S : arg min tf xq ` g xqu H; xparg min h H rog 2 q for every P RanN arg min h, ÿ j λ n β n h σ arg min h ă `8. β n β n In the above hyotheses, we have that Bf ` g ` N arg min h B f ` g ` δ arg min h q and hence S Zer Bf ` g ` N arg min h q H. Since according to the Theorem of Baillon-Haddad (see, for examle, [13, Corollary 18.16]), g and h are L 1 g -cocoercive and, resectively, L 1 h - cocoercive, and arg min h Zer h solving the bilevel rogramming roblem in Problem 3.1 reduces to solving the monotone inclusion 0 P Bfxq ` gxq ` N arg min h xq. By using to this end Algorithm 2.2, we recieve the following iterative scheme. 12

13 Algorithm 3.3. Let α n q, λ n q and β n q be sequences of ositive real numbers such that C 1 q tλ n u P l 2 z l 1 ; C 2 q tα n u is nondecreasing; C 3 q there exists α with 0 ď α n ď α ă 1{3 for all n ě 1. Let x 0, x 1 P H. For all n ě 1 we set x n`1 : rox λnf x n λ n g x n q λ n β n h x n q ` α n x n x n 1 qq. By using the inequality (1.6), one can easily notice, that H rog 2 q imlies H fitz 2 q, which means that the convergence statements for Algorithm 3.3 can be derived as articular instances of the ones derived in the revious section. Alternatively, one can use to this end the following lemma and emloy the same ideas and techniques as in Section 2. Lemma 3.4 is similar to Lemma 2.4, however, it will allow us to rovide convergence statements also for the sequence of function values hx n qq ně0. Lemma 3.4. Let x n q ně0 be the sequence generated by Algorithm 3.3 and u, yq be an element in Gr Bf ` g ` N arg min h q such that y v ` guq ` with v P Bfuq and P N arg minh uq. Further, let ε 1, ε 2, ε 3 ą 0 be such that 1 ε 3 ą 0. Then the following inequality holds for all n ě 1 x n`1 u 2 x n u 2 ď α n x n u 2 α n x n 1 u 2 1 ε 2 q x n`1 x n 2 ` α n ` α2 n x n x n λ 2 ε nβn 2 2µ 1 ε 3 q λ n β n h x n q 2 4 ` λ 2 n 2ηλ n g x n q g uq 2 2 ε 2 ` λ n β n rh uq h x n qs ` 4 λ 2 n v ` g uq 2 ε 2 j 2 2 ` ε 3 λ n β n h σ arg min h ` 2λ n xu x n, yy. ε 3 β n ε 3 β n Proof. Let be n ě 1 fixed. The roof follows by combining the estimates used in the roof of Lemma 2.4 with some inequalities which better exloits the convexity of h. From (2.11) we have 2λ n β n xu x n, h x n qy ď 2µ 1 ε 3 q λ n β n h x n q 2 ` 2ε 3 λ n β n xu x n, h x n qy. Since h is convex, the following relation also hold 2λ n β n xu x n, h x n qy ď 2λ n β n rh uq h x n qs. Summing u the two inequalities above give us 2λ n β n xu x n, h x n qy ď µ 1 ε 3 q λ n β n h x n q 2 ` ε 3 λ n β n xu x n, h x n qy ` λ n β n rh uq h x n qs. Using the same techniques as in the derivation of (2.13), we get 2λ n xu x n, v ` g x n q ` β n h x n qy ď µ 1 ε 3 q λ n β n h x n q 2 2ηλ n g x n q g uq 2 ` λ n β n rh uq h x n qs j 2 2 ` 2λ n xu x n, yy ` ε 3 λ n β n h u, σ arg min h. ε 3 β n ε 3 β n With this imroved estimates, the conclusion follows as in the roof of Lemma

14 By using now Lemma 3.4, one obains, after slightly adating the roof of Proosition 2.6, the following result. Proosition 3.5. Let 0 ă α ă 1 3, ε 1, ε 2, ε 3 ą 0 and the sequences λ n q and β n q satisfy condition C 4 q. Let x n q ně0 be the sequence generated by Algorithm 3.3 and assume that the Hyotheses 3.2 are verified. Then the following statements are true: iq the sequence x n`1 x n q ně0 belongs to l 2 and the sequences λ 2 n β n hx n q λ n β n hx n qq belong to l 1 ; iiq if, moreover, lim inf λ nβ n ą 0, then lim hx nq cluster oint of the sequence x n q ně0 lies in arg min h. iiiq for every u P S, the limit lim x n u exists. Finally, the above roosition leads to the following convergence result. and lim h x nq 0 and thus every Theorem 3.6. Let 0 ă α ă 1 3, ε 1, ε 2, ε 3 ą 0 and the sequences λ n q and β n q satisfy condition C 4 q. Let x n q ně0 be the sequence generated by Algorithm 3.3, z n q be the sequence defined in (1.7) and assume that the Hyotheses 3.2 are verified. Then the following statements are true: iq the sequence z n q converges weakly to an element in S as n Ñ `8. iiq if f is γ strongly convex with γ ą 0, then x n q ně0 converges strongly to the unique element in S as n Ñ `8. As follows we will show that under inf-comactness assumtions one can achieve weak nonergodic convergence for the sequence x n q ně0. Weak nonergodic convergence has been obtained for Algorithm 3.3 in [25] when α n α for all n ě 1 and for restrictive choices for both the sequence of ste sizes and enalty arameters. We denote by f ` gq min xparg min h fxq ` gxqq. For every element x in H, we denote by dist x, Sq inf x u the distance from x to S. In articular, dist x, Sq x Pr Sx, ups where Pr S x denotes the rojection of x onto S. The rojection oerator Pr S is firmly nonexansive ([13, Proosition 4.8]), this means Pr S xq Pr S yq 2 ` rid Pr S s xq rid Pr S s yq 2 ď x y y P H. (3.1) Denoting d xq 1 2 dist x, Sq2 1 2 x Pr Sx 2 for all x P H, one has that x ÞÑ dxq is differentiable and it holds d xq x Pr S x for all x P H. Lemma 3.7. Let x n q ně0 be the sequence generated by Algorithm 3.3 and assume that the Hyotheses 3.2 are verified. Then the following inequality holds for all n ě 1 d x n`1 q d x n q α n d x n q d x n 1 qq ` λ n rf ` gq x n`1 q f ` gq s ˆLg ď 2 λ n ` Lh 4 λ nβ n ` αn x n`1 x n 2 ` α n x n x n 1 2. (3.2) 2 Proof. Let n ě 1 be fixed. Since d is convex, we have d x n`1 q d x n q ď xx n`1 Pr S x n`1 q, x n`1 x n y. (3.3) Then there exists v n`1 P Bfx n`1 q such that (see (2.2)) x n x n`1 λ n gx n q ` β n hx n qq ` α n x n x n 1 q λ n v n`1 14

15 and, so, xx n`1 Pr S x n`1 q, x n`1 x n y xx n`1 Pr S x n`1 q, λ n v n`1 λ n g x n q λ n β n h x n q ` α n x n x n 1 qy λ n β n xx n`1 Pr S x n`1 q, h x n qy ` α n xx n`1 Pr S x n`1 q, x n x n 1 y. (3.4) Since v n`1 P Bf x n`1 q, we get λ n xx n`1 Pr S x n`1 q, v n`1 y ď λ n rf Pr S x n`1 qq f x n`1 qs. (3.5) Using the convexity of g it follows g x n q g Pr S x n`1 qq ď x g x n q, x n Pr S x n`1 qy. (3.6) On the other hand, the Descent Lemma gives g x n`1 q ď g x n q ` x g x n q, x n`1 x n y ` Lg 2 x n`1 x n 2. (3.7) By adding (3.6) and (3.7), it yields λ n xx n`1 Pr S x n`1 q, g x n qy ď λ n rg Pr S x n`1 qq g x n`1 qs ` Lgλ n 2 Using the x n`1 x n 2. (3.8) 1 L h cocoercivity of h combined with the fact that h Pr S x n`1 qq 0 (as Pr S x n`1 q belongs to S), it yields xx n Pr S x n`1 q, h x n qy ď 1 L h h x n q 2. Therefore ˆ λ n β n xx n`1 Pr S x n`1 q, h x n qy ď λ n β n xx n x n`1, h x n qy 1 h x n q 2 L h Further, we have ď λ n β n L h 4 x n`1 x n 2. (3.9) and α n xx n`1 Pr S x n`1 q x n Pr S x n qq, x n x n 1 y ď α n 2 rid Pr Ss x n`1 q rid Pr S s x n q 2 ` αn 2 x n x n 1 2 ď α n 2 x n`1 x n 2 ` αn 2 x n x n 1 2, α n xx n Pr S x n q, x n x n 1 y α n d x n q ` αn 2 x n x n 1 2 αn 2 x n 1 Pr S x n q 2 ď α n d x n q ` αn 2 x n x n 1 2 α n d x n 1 q. By adding two relations above, we obtain α n xx n`1 Pr S x n`1 q, x n x n 1 y α n xx n`1 Pr S x n`1 q x n Pr S x n qq, x n x n 1 y ` α n xx n Pr S x n q, x n x n 1 y ď α n 2 x n`1 x n 2 ` α n x n x n 1 2 ` α n d x n q d x n 1 qq. (3.10) By combining (3.5), (3.8), (3.9) and (3.10) with (3.4) we obtain the desired conclusion. 15

16 Definition 3.8. A function Ψ: H Ñ s R is sad to be inf-comact if for every r ą 0 and κ P R the set Lev r κ Ψq : tx P H: x ď r, Ψ xq ď κu is relatively comact in H. An useful roerty of inf-comact functions follow. Lemma 3.9. Let Ψ: H Ñ s R be inf-comact and x n q ně0 be a bounded sequence in H such that Ψ x n qq ně0 is bounded as well. If the sequence x n q ně0 converges weakly to an element in x as n Ñ `8, then it converges strongly to this element. Proof. Let be sr ą 0 and sκ P R such that for all n ě 1 x n ď sr and Ψ x n q ď sκ. Hence, x n q ně0 belongs to the set Lev sr sκ Ψq, which is relatively comact. Then x n q ně0 has at least one strongly convergente subsequence. Since every strongly convergent subsequence x nl q lě0 of x n q ně0 has as limit x, the desired conclusion follows. We can formulate now the weak nonergodic convergence result. Theorem Let 0 ă α ă 1 3, ε 1, ε 2, ε 3 ą 0, the sequences λ n q and β n q satisfy the condition 0 ă lim inf λ nβ n ď su λ n β n ď µ, x n q ně0 be the sequence generated by Algorithm nñ8 ně0 3.3, and assume that the Hyotheses 3.2 are verified and that either f ` g or h is inf-comact. Then the following statements are true: iq lim d x nq 0; iiq the sequence x n q ně0 converges weakly to an element in S as n Ñ `8; iiiq if h is inf-comact, then the sequence x n q ně0 converges strongly to an element in S as n Ñ `8. Proof. iq Thanks to Lemma 3.7, for all n ě 1 we have d x n`1 q d x n q ` λ n rf ` gq x n`1 q f ` gq s ď α n d x n q d x n 1 qq ` ζ n, (3.11) where ζ n : ˆLg 2 λ n ` Lh 4 λ nβ n ` αn x n`1 x n 2 ` α n x n x n From Proosition 3.5 iq, combined with the fact that both sequences λ n q and β n q are bounded, it follows that ÿ ζ n ă `8. In general, since x n q ně0 is not necessarily included in arg min h, we have to treat two different cases. Case 1: There exists an integer n 1 ě 1 such that f ` gq x n q ě f ` gq for all n ě n 1. In this case, we obtain from Lemma 1.2 that: the limit lim d x nq exists. ÿ něn 2 λ n rf ` gq x n`1 q f ` gq s ă `8. Moreover, since λ n q R l 1, we must have lim inf f ` gq x nq ď f ` gq. (3.12) 16

17 Consider a subsequence x nk q kě0 of x n q ně0 such that lim f ` gq x n k q lim inf f ` gq x nq kñ`8 and note that, thanks to (3.12), the sequence f ` gqx nk qq kě0 is bounded. From Proosition 3.5 (ii)-(iii) we get that also x nk q kě0 and hx nk qq kě0 are bounded. Thus, since either f ` g or h is inf-comact, there exists a subsequence x nl q lě0 of x nk q kě0, which converges strongly to an element x as l Ñ `8. According to Proosition 3.5 (ii)-(iii), x belongs to arg min h. On the other hand, lim f ` gq x n l q lim inf f ` gq x nq ě f ` gq xq ě f ` gq. (3.13) lñ`8 We deduce from (3.12) - (3.13) that f ` gq xq f ` gq, or in other words, that x P S. In conclusion, thanks to the continuity of d, lim d x nq lim d x nl q d xq 0. lñ8 Case 2 : For all n ě 1 there exists some n 1 ą n such that f ` gq x n 1q ă f ` gq. We define the set V n 1 ě 1: f ` gq x n 1q ă f ` gq (. There exist an integer n 2 ě 2 such that for all n ě n 2 the set tk ď n: k P V u is nonemty. Hence, for all n ě n 2 the number t n : max tk ď n: k P V u is well-defined. By definition t n ď n for all n ě n 3 and moreover the sequence tt n u něn2 is nondecreasing and lim t n 8. Indeed, if lim t n t P R, then for all n 1 ą t it holds nñ8 f ` gq x n 1q ě f ` gq, contradiction. Choose an integer N ě n 2. If t N ă N, then, for all n t N,, N 1, since f ` gq x n q ě f ` gq, the inequality (3.11) gives d x n`1 q d x n q ď d x n`1 q d x n q ` λ n rf x n`1 q F s ď α n d x n q d x n 1 qq ` ζ n. (3.14) Summing (3.14) for n t N,, N 1 and using tht tα n u is nondecreasing, it yields N 1 ÿ d x N q d x tn q ď α n d x n q α n 1 d x n 1 qq ` n t N ď αd x N 1 q ` ÿ If t N N, then d x N q d x tn q and we have d x N q αd x N 1 q ď d x tn q ` ÿ N 1 ÿ n t N ζ n nět N ζ n. (3.15) For all n ě 1 we define a n : d x n q αd x n 1 q. In both cases it yields nět N ζ n. (3.16) Nÿ ÿ a N ď d x tn q ` ζ n ď d x tn q ` ζ n. (3.17) n t N nět N 17

18 Passing in (3.17) to limit as N Ñ `8 we obtain that lim su Let be u P S. For all n ě 1 we have which shows that d x n qq ně0 is bounded, as lim su nñ8 a n lim su nñ8 a n ď lim su d x tn q. (3.18) d x n q 1 2 dist x n, Sq 2 ď 1 2 x n u 2, lim x n u exists. We obtain rd x n q αd x n 1 qs ě 1 αq lim su d x n q ě 0. (3.19) nñ8 Further, for all n ě 1 we have f ` gq x tn q ă f ` gq, which gives lim suf ` gq x tn q ď f ` gq. (3.20) This means that the sequence f ` gq x tn qq ně0 is bounded from above. Consider a subsequence x tk q kě0 of x tn q ně0 such that lim d x t k q lim su d x tn q. kñ`8 From Proosition 3.5 (ii)-(iii) we get that also x tk q kě0 and hx tk qq kě0 are bounded. Thus, since either f ` g or h is inf-comact, there exists a subsequence x tl q lě0 of x tk q kě0, which converges strongly to an element x as l Ñ `8. According to Proosition 3.5 (ii)-(iii), x belongs to arg min h. Furthermore, it holds We deduce from (3.20) and (3.21) that lim inf lñ`8 f ` gq x t l q ě f ` gq xq ě f ` gq. (3.21) f ` gq ď f ` gq xq ď lim suf ` gq x tl q ď lim suf ` gq x tn q ď f ` gq, which gives x P S. Thanks to the continuity of d we get By combining (3.18), (3.19) and (3.22), it yields 0 ď 1 αq lim su which imlies lim su d x n q 0 and thus lim su d x tn q lim d x t l q d xq 0. (3.22) lñ`8 d x n q ď lim su a n ď lim su d x tn q 0, lim d x nq lim inf d x nq lim su d x n q 0. iiq According to iq we have lim nñ8 d x n q 0, thus every weak cluster oint of the sequence x n q ně0 belongs to S. From Lemma 1.1 it follows that x n q ně0 converges weakly to a oint in S as n Ñ `8. iiiq Since lim inf nñ8 λ nβ n ą 0, from Proosition 3.5(ii) we have that lim h x nq lim h x nq 0. Since x n q ně0 is bounded, there exist sr ą 0 and sκ P R such that for all n ě 1 x n ď sr and h x n q ď sκ. Thanks to iiq the sequence x n q ně0 converges weakly to an element in S. Therefore, according to Lemma 3.9, it converges strongly to this element in S. 18

19 References [1] F. Alvarez. On the minimizing roerty of a second order dissiative system in Hilbert saces. SIAM Journal on Control and Otimization, Vol. 38(4): (2000) [2] F. Alvarez. Weak convergence of a relaxed and inertial hybrid rojection-roximal oint algorithm for maximal monotone oerators in Hilbert sace. SIAM Journal on Otimization, Vol. 14(3): (2004) [3] F. Alvarez and H. Attouch. An inertial roximal method for maximal monotone oerators via discretization of a nonlinear oscillator with daming. Set-Valued Analysis, Vol. 9(1): 3 11 (2001) [4] H. Attouch, J. Bolte, P. Redont and A. Soubeyran. Alternating roximal algorithms for weakly couled convex minimization roblems, alications to dynamical games and PDE s. Journal of Convex Analysis, Vol. 15(3): (2008) [5] H. Attouch, L. M. Briceno-Arias and P. L. Combettes. A arallel slitting method for couled monotone inclusions. SIAM Journal on Control and Otimization, Vol. 48(5): (2010) [6] H. Attouch, A. Cabot and M.-O. Czarnecki Asymtotic behavior of nonautonomous monotone and subgradient evolution equations. Transactions of the American Mathematical Society, Vol. 370(2): (2018) [7] H. Attouch and M.-O. Czarnecki. Asymtotic behavior of couled dynamical systems with multiscale asects. Journal of Differential Equations, Vol. 248(6): (2010) [8] H. Attouch and M.-O. Czarnecki. Asymtotic behavior of gradient-like dynamical systems involving inertia and multiscale asects. Journal of Differential Equations, Vol. 262(3): (2017) [9] H. Attouch, M.-O. Czarnecki and J. Peyouquet. Couling forward-backward with enalty schemes and arallel slitting for constrained variational inequalities. SIAM Journal on Otimization, Vol. 21(4): (2011) [10] H. Attouch, M.-O. Czarnecki and J. Peyouquet. Prox-enalization and slitting methods for constrained variational roblems. SIAM Journal on Otimization, Vol. 21(1): (2011) [11] H. Attouch, J. Peyouquet and P. Redont. A dynamical aroach to an inertial forward-backward algorithm for convex minimization. SIAM Journal on Otimization, Vol. 24(1): (2014) [12] S. Banert and R. I. Boţ. Backward enalty schemes for monotone inclusion roblems. Journal of Otimization Theory and Alications, Vol. 166(3): (2015) [13] H. H. Bauschke and P. L. Combettes. Convex Analysis Monotone Oerator Theory in Hilbert Saces. CMS Books in Mathematics, Sringer, New York (NY) (2011) [14] H. H. Bauschke, D.A. McLaren and H.S. Sendov. Fitzatrick functions: inequalities, examles and remarks on a roblem by S. Fitzatrick. Journal of Convex Analysis, Vol. 13(3): (2006) [15] A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse roblems. SIAM Journal on Imaging Sciences, Vol. 2(1): (2009) [16] D. P. Bertsekas. Nonlinear rogramming. Athena Scientific, Cambridge (MA) (1999) [17] R. I. Boţ. Conjugate Duality in Convex Otimization. Lecture Notes in Economics and Mathematical Systems, Vol. 637, Sringer, Berlin Heidelberg (2010) [18] R. I. Boţ and E. R. Csetnek. A Tseng s tye enalty scheme for solving inclusion roblems involving linearly comosed and arallel-sum tye monotone oerators. Vietnam Journal of Mathematics, Vol. 42(4): (2014) [19] R. I. Boţ and E. R. Csetnek. Forward-backward and Tseng s tye enalty schemes for monotone inclusion roblems. Set-Valued Variational Analysis, Vol. 22(2): (2014) [20] R. I. Boţ and E. R. Csetnek. An inertial forward-backward-forward rimal-dual slitting algorithm for solving monotone inclusion roblems. Numerical Algorithms, Vol. 71(3): (2016) [21] R. I. Boţ and E. R. Csetnek. Penalty schemes with inertial effects for monotone inclusion roblems. Otimization, Vol. 66(6): (2017) [22] R. I. Boţ, E. R. Csetnek and C. Hendrich. Inertial Douglas-Rachford slitting for monotone inclusion roblems. Alied Mathematics and Comutation, Vol. 256(1): (2015) [23] R. I. Boţ, E. R. Csetnek and S. C. Laszlo. An inertial forward-backward algorithm for the minimization of the sum of two nonconvex functions. EURO Journal on Comutational Otimization, Vol.4(1): 3 25 (2016). [24] R. I. Boţ, E. R. Csetnek and N. Nimana. Gradient-tye enalty method with inertial effects for solving constrained convex otimization roblems with smooth data. Otimization Letters, Vol.12(1): (2018). [25] R. I. Boţ, E. R. Csetnek and N. Nimana. An inertial roximal-gradient enalization scheme for constrained convex otimization roblems. Vietnam Journal of Mathematics, Vol.46(1): (2018). 19

20 [26] J. M. Borwein. Maximal monotonicity via convex analysis. Journal of Convex Analysis, Vol. 13(3): (2006) [27] R. S. Burachik and B. F. Svaiter. Maximal monotone oerators, convex functions and a secial family of enlargements. Set-Valued Analysis, Vol. 10(4): (2002) [28] A. Chambolle and Ch. Dossal. On the convergence of the iterates of the Fast Iterative Shrinkage/Thresholding Algorithm. Journal of Otimization Theory and Alications, Vol. 166(3): (2015) [29] C. Chen, R. H. Chan, S. Ma and J. Yang. Inertial roximal ADMM for linearly constrained searable convex otimization. SIAM Journal on Imaging Science, Vol. 8(4): (2015) [30] C. Chen, S. Ma and J. Yang. A general inertial roximal oint algorithm for mixed variational inequality roblem. SIAM Journal on Otimization, Vol. 25(4): (2015) [31] P. L. Combettes. Solving monotone inclusions via comositions of nonexansive averaged oerators. Otimization, Vol. 53(5-6): (2004) [32] S. Fitzatrick. Reresenting monotone oerators by convex functions. In Proceedings of the Centre for Mathematical Analysis. Worksho/Miniconference on Functional Analysis and Otimization, Vol.20, Australian National University, Canberra (1988) [33] P. Frankel and J. Peyouquet. Lagrangian-Penalization Algorithm for Constrained Otimization and Variational Inequalities. Set-Valued and Variational Analysis, Vol. 20(2): (2012) [34] P.-E. Maingé. Convergence theorems for inertial Krasnosel skiĭ-mann tye algorithms. Journal of Comutational and Alied Mathematics, Vol. 219(1): (2008) [35] P.-E. Maingé and A. Moudafi. Convergence of new inertial roximal methods for dc rogramming. SIAM Journal on Otimization, Vol. 19(1): (2008) [36] A. Moudafi and M. Oliny. Convergence of a slitting inertial roximal method for monotone oerators. Journal of Comutational and Alied Mathematics, Vol. 155(2): (2003) [37] N. Noun and J. Peyouquet. Forward-backward enalty scheme for constrained convex minimization without inf-comactness. Journal of Otimization Theory and Alications, Vol. 158(3): (2013) [38] J. Peyouquet. Couling the Gradient Method with a General Exterior Penalization Scheme for Convex Minimization. Journal of Otimization Theory and Alications, Vol. 153(1): (2012) [39] B. T. Polyak. Introduction to Otimization. Translations Series in Mathematics and Engineering. Publications Division, Otimization Software Inc., New York (1987) [40] C. Zălinescu. Convex Analysis in General Vector Saces. World Scientific, Singaore (2002) 20