Levenberg-Marquardt dynamics associated to variational inequalities

Transcription

1 Levenberg-Marquardt dynamics associated to variational inequalities Radu Ioan Boţ Ernö Robert Csetnek Aril 1, 217 Abstract. In connection with the otimization roblem inf {Φx + Θx}, x argmin Ψ where Φ is a roer, convex and lower semicontinuous function and Θ and Ψ are convex and smooth functions defined on a real Hilbert sace, we investigate the asymtotic behavior of the trajectories of the nonautonomous Levenberg-Marquardt dynamical system { vt Φxt λtẋt + vt + vt + Θxt + βt Ψxt =, where λ and β are functions of time controlling the velocity and the enalty term, resectively. We show weak convergence of the generated trajectory to an otimal solution as well as convergence of the objective function values along the trajectories, rovided λ is monotonically decreasing, β satisfies a growth condition and a relation exressed via the Fenchel conjugate of Ψ is fulfilled. When the objective function is assumed to be strongly convex, we can even show strong convergence of the trajectories. Key Words. nonautonomous systems, Levenberg-Marquardt dynamics, regularized Newtonlike dynamics, Lyaunov analysis, convex otimization, variational inequalities, enalization techniques AMS subject classification. 34G25, 47J25, 47H5, 9C25 1 Introduction Throughout this manuscrit H is assumed to be a real Hilbert sace endowed with inner roduct, and associated norm =,. When T : H H is a C 1 oerator with derivative T, the solving of the equation find x H such that T x = can be aroached by the classical Newton method, which generates an aroximating sequence x n n of a solution of the oerator equation through T x n + T x n x n+1 x n = n. University of Vienna, Faculty of Mathematics, Oskar-Morgenstern-Platz 1, A-19 Vienna, Austria, radu.bot@univie.ac.at. University of Vienna, Faculty of Mathematics, Oskar-Morgenstern-Platz 1, A-19 Vienna, Austria, ernoe.robert.csetnek@univie.ac.at. Research suorted by FWF Austrian Science Fund, Lise Meitner Programme, roject M 1682-N25. 1

2 In order to overcome the fact that the classical Newton method assumes the solving of an equation which is in general not well-osed, one can use instead the Levenberg-Marquardt method T x n + λ n Id +T x n+1 x n x n = n, t n where Id : H H denotes the identity oerator on H, λ n a regularizing arameter and t n > the ste size. When T : H H is a set-valued maximally monotone oerator, Attouch and Svaiter showed in [13] that the above Levenberg-Marquardt algorithm can be seen as a time discretization of the dynamical system { vt T xt 1 λtẋt + vt + vt = for aroaching the inclusion roblem find x H such that T x. 2 This includes as a secial instance the roblem of minimizing a roer, convex and lower semicontinuous function, when T is taken as its convex subdifferential. Later on, this investigation has been continued in [2] in the context of minimizing the sum of a roer, convex and lower semicontinuous function with a convex and smooth one. In the sirit of [13], we aroach in this aer the otimization roblem inf {Φx + Θx}, 3 x argmin Ψ where Φ : H R {+ } is a roer, convex and lower semicontinuous function and Ψ, Θ : H R are convex and smooth functions, via the Levenberg-Marquardt dynamical system { vt Φxt 4 λtẋt + vt + vt + Θxt + βt Ψxt =, where λ and β are functions of time controlling the velocity and the enalty term, resectively. If Φ + N argmin Ψ is maximally monotone, then determining an otimal solution x H of 3 means nothing else than solving the subdifferential inclusion roblem find x H such that Φx + Θx + N argmin Ψ x 5 or, equivalently, solving the variational inequality find x argmin Ψ and v Φx such that v + Θx, y x y argmin Ψ. 6 We show weak convergence of the trajectory x generated by 4 to an otimal solution of 3 as well as convergence of the objective function values along the trajectory to the otimal objective value, rovided the assumtion + ran N argmin Ψ βt [ Ψ βt ] σ argmin Ψ dt < + 7 βt is fulfilled and the functions λ, β satisfy some mild conditions. If the objective function of 3 is strongly convex, the trajectory x converges even strongly to the unique otimal solution of 3. 2

3 of The condition 7 has its origins in the aer of Attouch and Czarnecki [7], where the solving inf Φx, 8 x argmin Ψ for Φ, Ψ : H R {+ } roer, convex and lower semicontinuous functions, is aroached through the nonautonomous first order dynamical system ẋt + Φxt + βt Ψxt, 9 by assuming that the enalizing function β : [, +, + tend to + as t +. Several ergodic and nonergodic convergence results have been reorted in [7] under the key assumtion 7. The aer of Attouch and Czarnecki [7] was the starting oint of a remarkable number of research articles devoted to enalization techniques for solving otimization roblems of tye 3, but also generalizations of the latter in form of variational inequalities exressed with maximal monotone oerators see [5,7,9,1,12,15,17,19 21,26,27]. In the literature enumerated above, the monotone inclusions roblems have been aroached either through continuous dynamical systems or through their discrete counterarts formulated as slitting algorithms. We seak in both cases about methods of enalty tye, which means in this context that the oerator describing the underlying set of the variational inequality under investigation is evaluated as a enalty functional. In the above-listed references one can find more general formulations of the key assumtion 7, but also further examles for which these conditions are satisfied. In Remark 5 and Remark 6 we rovide more insights into the relations of the dynamical system 14 to other continuous systems and their discrete counterarts from the literature. The results we obtain in this aer are in the sirit of Attouch-Czarnecki [7]. However, since the dynamical system we focus on is a combination of two different tyes of dynamical systems, the the asymtotic analysis is more involved, in the sense that one has to take into consideration the articularities of both continuous systems. 2 Preliminaries In this section we resent some reliminary definitions, results and tools that will be useful throughout the aer. We consider the following definition of an absolutely continuous function. Definition 1 see, for instance, [2, 13] A function x : [, b] H where b > is said to be absolutely continuous if one of the following equivalent roerties holds: i there exists an integrable function y : [, b] H such that xt = x + t ysds t [, b]; ii x is continuous and its distributional derivative is Lebesgue integrable on [, b]; iii for every ε >, there exists η > such that for any finite family of intervals I k = a k, b k [, b] we have the imlication I k I j = and k b k a k < η = k xb k xa k < ε. A function x : [, + H where b > is said to be locally absolutely continuous if it is absolutely continuous on each interval [, b] for < b < +. 3

4 Remark 1 a It follows from the definition that an absolutely continuous function is differentiable almost everywhere, its derivative coincides with its distributional derivative almost everywhere and one can recover the function from its derivative ẋ = y by the integration formula i. b If x : [, b] H, where b >, is absolutely continuous and B : H H is L-Lischitz continuous for L, then the function z = B x is absolutely continuous, too. This can be easily seen by using the characterization of absolute continuity in Definition 1iii. Moreover, z is differentiable almost everywhere on [, b] and the inequality żt L ẋt holds for almost every t [, b]. The following results, which can be interreted as continuous counterarts of the quasi- Fejér monotonicity for sequences, will lay an imortant role in the asymtotic analysis of the trajectories of the dynamical system investigated in this aer. For the roof of Lemma 2 we refer the reader to [2, Lemma 5.1]. Lemma 3 follows by using similar arguments as used in [2, Lemma 5.2]. Lemma 2 Suose that F : [, + R is locally absolutely continuous and bounded from below and that there exists G L 1 [, + such that for almost every t [, + Then there exists lim F t R. d F t Gt. dt Lemma 3 If 1 <, 1 r, F : [, + [, + is locally absolutely continuous, F L [, + L [, +, G 1, G 2 : [, + R, G 1 L 1 [, +, G 2 L r [, + and for almost every t [, + then lim F t =. d dt F t G 1t + G 2 t, 1 Proof. In case r = 1 this follows from Lemma 2 and the fact that F L [, +. Assume now that r > 1 and define q := r > 1, which fulfills the relation q r Further, from 1 we derive for almost every t [, + = d dt F tq qf t q 1 G 1 t + qf t q 1 G 2 t. 12 Since F L [, + and G 1 L 1 [, +, the function t F t q 1 G 1 t is L 1 -integrable on [, +. Moreover, due to F q 1 L q 1 [, +, G 2 L r [, + and 11, the function t F t q 1 G 2 t is also L 1 -integrable on [, +. We conclude that the function on the right-hand side of inequality 12 belongs to L 1 [, +. Alying now Lemma 2 we obtain that there exists lim F t R, which combined again with F L [, + delivers the conclusion. The next result which we recall here is the continuous version of the Oial Lemma. Lemma 4 Let S H be a nonemty set and x : [, + H a given function. Assume that i for every x S, lim xt x exists; ii every weak sequential cluster oint of the ma x belongs to S. Then there exists x S such that xt converges weakly to x as t +. 4

5 3 A Levenberg-Marquardt dynamical system: existence and uniqueness of the trajectories Consider the otimization roblem inf {Φx + Θx}, 13 x argmin Ψ where H is a real Hilbert sace and the following conditions hold: H Ψ Ψ : H [, + is convex, Fréchet differentiable with Lischitz continuous gradient and argmin Ψ = Ψ 1 ; H Θ Θ : H R is convex, Fréchet differentiable with Lischitz continuous gradient; H Φ Φ : H R {+ } is convex, lower semicontinuous and fulfills the relation S := {z argmin Ψ dom Φ : Φz + Θz Φx + Θx x argmin Ψ}. Here, dom Φ = {x H : Φx < + } denotes the effective domain of the function Φ. In connection with 13, we investigate the nonautonomous dynamical system vt Φxt λtẋt + vt + vt + Θxt + βt Ψxt = x = x, v = v Φx, 14 where x, v H and Φ : H H, Φx := { H : Φy Φx +, y x y H}, for Φx R and Φx := for Φx R, denotes the convex subdifferential of Φ. We denote by dom Φ = {x H : Φx } the domain of the oerator Φ. Furthermore, we make the following assumtions regarding the functions of time controlling the velocity and the enalty: Hλ 1 λ : [, +, + is locally absolutely continuous; Hβ 1 β : [, + [, + is locally integrable. Let us mention that due to H 1 λ, λt exists for almost every t. Remark 5 a In case Φx = for all x H, the dynamical system 14 becomes { λtẋt + Θxt + βt Ψxt = x = x, 15 The asymtotic convergence of the trajectories generated by 15 has been investigated in [7] under the assumtion λt = 1 for all t, for Θ and Ψ nonsmooth functions, by relacing their gradients with convex subdifferentials and, consequently, by treating the differential equation as a monotone inclusion see 9. b In case Ψx = for all x H, the dynamical system vt Φxt λtẋt + vt + vt + Θxt = x = x, v = v Φx, has been investigated in [2] see, also, [13], for the situation when Θx = for all x H. 16 5

6 c In case Θx = and Ψx = 1 2 x 2 for all x H and λt = λ R for every t [, +, the Levenberg-Marquardt dynamical system 14 becomes vt Φxt λẋt + vt + vt + βtxt = x = x, v = v Φx. The dynamical system 17 has been considered in [1] in connection with the roblem of finding the minimal norm elements among the minima of Φ, namely, see also [6] and [12, Section 3] 17 inf x argmin Φ x In contrast to 14, where the function describing the constrained set of 13 is enalized, in 17 the objective function of 18 is enalized via a vanishing enalization function see [1]. In the following we secify what we understand under a solution of the dynamical system 14. Definition 2 We say that the air x, v is a strong global solution of 14, if the following roerties are satisfied: i x, v : [, + H are locally absolutely continuous functions; ii vt Φxt for every t [, + ; iii λtẋt + vt + vt + Θxt + βt Ψxt = for almost every t [, + ; iv x = x, v = v. Similarly to the techniques used in [13], we will show the existence and uniqueness of the trajectories generated by 14 by converting it to an equivalent first order differential equation with resect to z, defined by zt = xt + µtvt, 19 where µt = 1 λt t. To this end we will make use of the resolvent and Yosida aroximation of the convex subdifferential of Ψ. For γ >, we denote by J γ Φ = Id +γ Φ 1 the resolvent of γ Φ. Due to the maximal monotonicity of Φ, the resolvent J γ Φ : H H is a single-valued oerator with full-domain, which is, furthermore, nonexansive, that is 1-Lischitz continuous. Let us notice that the resolvent of the convex subdifferential is nothing else than the the roximal oint oerator and for all x H we have J γ Φ x = rox γφ x = argmin u H The Yosida regularization of Φ is defined by Φ γ = 1 γ Id J γ Φ { Φu + 1 u x 2 2γ }. 6

7 and it is γ 1 -Lischitz continuous. For more roerties of these oerators we refer the reader to [14]. Assume now that x, v is a strong global solution of 14. From 19 we have for every t [, + vt Φxt zt xt + µt Φxt = Id +µt Φxt, thus, from the definition of the resolvent we derive that relation ii in Definition 2 is equivalent to xt = J µt Φ zt. 2 From 19, 2 and the definition of the Yosida regularization we obtain vt = Φ µt zt. 21 Further, by differentiating 19 and taking into account iii in Definition 2, we get for almost every t [, + żt = ẋt + µtvt + µt vt = µtvt µtvt µt Θxt βtµt Ψxt. 22 Taking into account 2, 21 and 22 we conclude that z defined in 19 is a strong global solution of the dynamical system { żt + µt µt Φ µt zt + µt Θ J µt Φ zt + βtµt Ψ J µt Φ zt = z = x + µv. 23 Vice versa, if z is a strong global solution of 23, then one obtains via 2 and 21 a strong global solution of 14. Remark 6 By considering the time discretization żt z n+1 z n h n of the above dynamical system and by taking µ constant, from 2 and 23 we obtain the iterative scheme { xn = rox n µφ z n z n+1 = 1 h n z n + h n xn µ Θx n µβ n Ψx n 24, which for h n = 1 yields the following algorithm n x n+1 = rox µφ xn µ Θx n µβ n Ψx n. 25 The convergence of the above algorithm has been investigated in [2] in the more general framework of monotone inclusion roblems, under the use of variable ste sizes µ n n and by assuming that [ ] ran N argmin Ψ µ n β n Ψ σ argmin Ψ < +, β n β n n N which is a condition that can be seen as a discretized version of the one stated in 7. The case Θx = for all x H has been treated in [1] see also the references therein. Next we show that, given x, v H and by assuming Hλ 1 and H1 β, there exists a unique strong global solution of the dynamical system 23. This will be done in the framework of the Cauchy-Lischitz Theorem for absolutely continuous trajectories see for examle [25, Proosition 6.2.1], [28, Theorem 54]. To this end we will make use of the following Lischitz roerty of the resolvent oerator as a function of the ste size, which actually is a consequence of the classical results [24, Proosition 2.6] and [14, Proosition 23.28] see also [13, Proosition 2.3] and [2, Proosition 3.1]. 7

8 Proosition 7 Assume that H Φ holds, x H and < δ < +. Then the maing τ J τ Φ x is Lischitz continuous on [δ, +. More recisely, for any λ 1, λ 2 [δ, + the following inequality holds: J λ1 Φx J λ2 Φx λ 1 λ 2 Φ δ x. 26 Furthermore, the function λ Φ λ x is nonincreasing. Notice that the dynamical system 23 can be written as { żt = ft, zt z = z, 27 where z = x + µv and f : [, + H H is defined by ft, w = µt µt Φ µt w µt Θ J µt Φ w βtµt Ψ J µt Φ w. 28 In the following we denote by L Φ and L Ψ the Lischitz constants of Φ and Ψ, resectively. a Notice that for every t and every w 1, w 2 H we have ft, w 1 ft, w λt λt + L Θ λt + L βt Ψ w 1 w λt Indeed, this follows 28, the Lischitz roerties of the oerators involved and the definition of µt. Further, notice that due to H 1 λ and H1 β, L f : [, + R, L f t = 1 + λt λt + L Θ λt + L βt Ψ λt, which is for every t equal to the Lischitz-constant of ft,, satisfies hence b We show now that L f L 1 [, b] for any < b < +. w H, b >, f, w L 1 [, b], H. 3 We fix w H and b >. Due to H 1 λ, there exist λ min, λ max > such that < λ min λt λ max t [, b], < 1 λ max µt 1 λ min t [, b]. Relying on Proosition 7 we obtain for all t [, b] the following chain of inequalities: ft, w µt µt Φ 1 λmax w + µt ΘJ µt Φ w ΘJ 1 Φw + µt ΘJ 1 λmax λmax + βtµt ΨJ µt Φ w ΨJ 1 Φw + βtµt ΨJ 1 λmax λmax µt µt Φ 1 λmax w + L Θ µt + L Ψ βtµt µt 1 λ max µt 1 λ max ΘJ 1 λmax Φw + µt ΘJ 1 λmax Φw ΨJ 1 Φw + βtµt ΨJ 1 Φw. λmax λmax 8

9 Now 3 follows from the roerties of the functions µ and β, and the fact that µt µt λt for almost every t. λ min λ min In the light of the statements roven in a and b, the existence and uniqueness of a strong global solution of the dynamical system 23 follow from [25, Proosition 6.2.1] see also [28, Theorem 54]. Finally, similarly to the roof of [13, Theorem 2.4ii], one can guarantee the existence and uniqueness of the trajectories generated by 14 by relying on the roerties of the dynamical system 23 and on 2 and 21. The details are left to the reader. 4 Convergence of the trajectories and of the objective function values In this section we rove weak convergence for the trajectory generated by the dynamical system 14 to an otimal solution of 13 as well as convergence for the objective function values of the latter along the trajectory. Some techniques from [7] and [13] will be useful in this context. To this end we will make the following sulementary assumtions: H 2 λ λ : [, +, + is locally absolutely continuous and λt for almost every t [, + ; Hβ 2 β : [, +, + is measurable and bounded from above on each interval [, b], where < b < + ; H ran N argmin Ψ + H Φ + Θ + δ argmin Ψ = Φ + Θ + N argmin Ψ, [ ] βt Ψ σ argmin Ψ dt < + ; βt βt N argmin Ψ is the normal cone to the set argmin Ψ: N argmin Ψ x = { H :, y x y argmin Ψ} for x argmin Ψ and N argmin Ψ x = for x argmin Ψ; ran N argmin Ψ is the range of the normal cone N argmin Ψ : ran N argmin ψ if and only if there exists x argmin Ψ such that N argmin Ψ x; Ψ : H R {+ } is the Fenchel conjugate of Ψ: Ψ = su x H {, x Ψx} H; σ argmin Ψ : H R {+ } is the suort function of the set argmin Ψ: σ argmin Ψ = su x argmin Ψ, x for all H; δ argmin Ψ : H R {+ } is the indicator function of argmin Ψ: it takes the value on the set argmin Ψ and +, otherwise. We have N argmin Ψ = δ argmin Ψ. Moreover, N argmin Ψ x if and only if x argmin Ψ and σ argmin Ψ =, x. Remark 8 a The condition λt for almost every t [, + has been used in [13] in the study of the asymtotic convergence of the dynamical system 1, when aroaching the monotone inclusion roblem 2. b Under H Ψ, due to Ψ δ argmin Ψ, we have Ψ δ argmin Ψ = σ argmin Ψ. 9

10 c When Ψ = see Remark 5b, it holds N argmin Ψ x = {} for every x argmin Ψ = H, Ψ = σ argmin Ψ = δ {}, which shows that in this case H trivially holds. d A nontrivial situation in which condition H is fulfilled is when ψx = 1 2 inf y C x y 2, for a nonemty, convex and closed set C H see [7]. Then 7 holds if and only if + 1 dt < +, βt which is trivially satisfied for βt = 1 + t α with α > 1. e Due to the continuity of Θ, the condition H is equivalent to Φ + δ argmin Ψ = Φ + N argmin Ψ, which holds when sqridom Φ argmin Ψ, a condition that is fulfilled, if Φ is continuous at a oint in dom Φ argmin Ψ or intargmin Ψ dom Φ we invite the reader to consult also [14], [16] and [29] for other sufficient conditions for the above subdifferential sum formula. Here, for M H a convex set, sqri M := {x M : λ> λm x is a closed linear subsace of H} denotes its strong quasi-relative interior. We always have int M sqri M in general this inclusion may be strict. If H is finite-dimensional, then sqri M coincides with ri M, the relative interior of M, which is the interior of M with resect to its affine hull. The following differentiability result of the comosition of convex functions with absolutely continuous trajectories that is due to Brézis see [24, Lemme 4,. 73] and also [7, Lemma 3.2] will lay an imortant role in our analysis. Lemma 9 Let f : H R {+ } be a roer, convex and lower semicontinuous function. Let x L 2 [, T ], H be absolutely continuous such that ẋ L 2 [, T ], H and xt dom f for almost every t [, T ]. Assume that there exists ξ L 2 [, T ], H such that ξt fxt for almost every t [, T ]. Then the function t fxt is absolutely continuous and for every t such that xt dom f we have d fxt = ẋt, h h fxt. dt We start our convergence analysis with the following technical result. Lemma 1 Assume that H Ψ, H Θ, H Φ, Hλ 1 and H2 β hold and let x, v : [, + H H be a strong stable solution of the dynamical system 14. Then the following statements are true: i ẋt, vt for almost every t [, + ; ii d dtφxt = ẋt, vt for almost every t [, +. Proof. i See [13, Proosition 3.1]. The roof relies on the first relation in 14 and the monotonicity of the convex subdifferential. ii The roof makes use of Lemma 9. Let T > be fixed. Due to the continuity of x and v we obviously have x, v L 2 [, T ], H. 1

11 The only condition which has to be checked is ẋ L 2 [, T ], H. By considering the second relation in 14 and by inner multilying it with ẋt, we derive for almost every t [, T ] λt ẋt 2 + ẋt, vt + ẋt, vt + ẋt, Θxt + βt ẋt, Ψxt =. Using i we obtain for almost every t [, T ] λt ẋt 2 + ẋt, vt + ẋt, Θxt + βt ẋt, Ψxt. 31 Since x, v are continuous on [, T ], they are bounded on [, T ], a roerty which is shared also by t βt Ψxt, due to H 2 β and H Ψ, and by t Θxt, due to H Θ. Since λ is bounded from below by a ositive constant on [, T ], from 31 one easily obtains that ẋ L 2 [, T ], H and the conclusion follows by alying Lemma 9. Lemma 11 Assume that H Ψ, H Θ, H Φ, H 2 λ, H2 β, H and H hold and let x, v : [, + H H be a strong stable solution of the dynamical system 14. Choose arbitrary z S and N argmin Ψ z such that Θz Φz. Define g z, h z : [, + [, + as g z t = Φz Φxt + vt, xt z and The following statements are true: i lim λt 2 xt z 2 + g z t ii + βtψxtdt < + ; iii lim t iv lim t h z t = Θz Θxt + Θxt, xt z. [, + ;, xs z ds R; Φ + Θxs Φ + Θz + βsψxs ds R; t v lim vs, xs z + Θxs, xs z + βsψxs ds R; vi lim t Φ + Θxs Φ + Θz ds R; t vii lim vs, xs z + Θxs, xs z ds R; viii g z L 1 [, + L [, + and h z L 1 [, +. Proof. For the beginning, we notice that from the definition of S and H we have Φ + Θ + δ argmin Ψ z = Φz + Θz + N argmin Ψ z, hence there exists such that N argmin Ψ z 32 Θz Φz

12 For almost every t it holds according to 14 d λt xt z 2 = λt dt 2 2 xt z 2 + λt ẋt, xt z = λt 2 xt z 2 vt, xt z vt, xt z Θxt, xt z βt Ψxt, xt z. 34 From 14 and the convexity of Φ, Θ and Ψ we have for every t [, + and Φz Φxt + vt, z xt 35 Θz Θxt + Θxt, z xt 36 = Ψz Ψxt + Ψxt, z xt. 37 From 33 and the convexity Φ and Θ we obtain for every t [, + and Φxt Φz + Θz, xt z 38 Θxt Θz + Θz, xt z. 39 Further, due to Lemma 1ii it holds for almost every t [, + d dt g zt = ẋt, vt + vt, xt z + vt, ẋt = vt, xt z. 4 On the other hand, using 32 and the Young-Fenchel inequality we obtain for every t [, + βtψxt +, xt z = βt Ψxt + βt, xt z = βt Ψxt βt, xt + σ argmin Ψ βt βt Ψ + σ argmin Ψ. 41 βt βt Finally, we obtain for almost every t [, + d λt dt 2 xt z 2 + g z t + βt Ψ + σ argmin Ψ βt βt d λt dt 2 xt z 2 + g z t + βtψxt +, xt z d λt dt 2 xt z 2 + g z t + Φ + Θxt Φ + Θz + βtψxt d λt dt 2 xt z 2 + g z t + vt, xt z + Θxt, xt z + βtψxt, where the first inequality follows from 41, the second one from 38 and 39, the next one from 35 and 36, and the last one from Hλ 2, 34, 4 and

13 i Since for almost every t [, + we have see 42 d λt dt 2 xt z 2 + g z t βt Ψ βt σ argmin Ψ βt, the conclusion follows from Lemma 2, H and the fact that g z t for every t. ii Let F : [, + R be defined by F t = t From 42 we have for almost every s [, + βsψxs +, xs z ds t [, +. βsψxs +, xs z d ds By integration we obtain for every t [, + λs 2 xs z 2 + g z s. F t λt 2 xt z 2 + g z t λ 2 x z 2 g z λ 2 x z 2 g z, hence F is bounded from below. Furthermore, from 41 we derive for every t [, + d dt F t = βtψxt +, xt z βt Ψ βt σ argmin Ψ βt. From H and Lemma 2 it follows that lim F t exists and it is a real number. Hence lim t Further, since ψ, we obtain for every t [, + βsψs +, xs z ds R. 43 βtψxt +, xt z βt Ψxt +, xt z. 2 Similarly to 41one can show that for every t [, + βt βt Ψxt +, xt z 2 2 Ψ 2 βt 2 + σ argmin Ψ, βt while from 42 we obtain that for almost every t [, + it holds d λt dt 2 xt z 2 + g z t + βt 2 2 Ψ + σ argmin Ψ 2 βt βt d λt dt 2 xt z 2 + g z t + βt Ψxt +, xt z 2 d λt dt 2 xt z 2 + g z t + βtψxt +, xt z 13

14 By using the same arguments as used in the roof of 43 it yields that lim t βs 2 Ψs +, xs z ds R. 44 Finally, from 43 and 44 we obtain ii. iii Follows from 43 and ii. iv-v These statements follow from 42 and 41, by using similar arguments as used for roving 43. vi-vii These statements are direct consequences of iv, v and ii. viii Combining vi and vii with g z, h z, we easily derive that Since g z + h z L 1 [, +. g z g z + h z, we deduce that g z L 1 [, + and h z L 1 [, +. Finally, notice that due to i there exists T > such that g z is bounded on [T, +. The boundedness of g z on [, T ] follows from 38 and the continuity of x and v. Thus, g z L [, +. In order to roceed with the asymtotic analysis of the dynamical system 14, we make the following more involved assumtions on the functions λ and β, resectively: H 3 λ λ : [, +, + is locally absolutely continuous, λt for almost every t [, + and lim λt > ; Hβ 3 β : [, +, + is locally absolutely continuous, it satisfies for some k the growth condition βt kβt for almost every t [, + and lim βt = +. Lemma 12 Assume that H Ψ, H Θ, H Φ, H 3 λ, H3 β, H and H hold and let x, v : [, + H H be a strong stable solution of the dynamical system 14. The following statements are true: i x is bounded; ii lim Ψxt =. Proof. Take an arbitrary z S and according to H N argmin Ψ z such that Θz Φz and consider the functions g z, h z defined in Lemma 11. i According to Lemma 11i, since g z, we have that t λt xt z 2 is bounded, which combined with lim λt > imlies that x is bounded. ii Consider the function E 1 : [, + R defined for every t [, + by E 1 t = Φ + Θxt βt + Ψxt. 14

15 Using Lemma 1 and 14 we obtain for almost every t [, + Ė 1 t = 1 βt vt, ẋt + Θxt, ẋt βt β 2 Φ + Θxt + Ψxt, ẋt t = 1 vt + Θxt + βt Ψxt, ẋt βt βt β 2 Φ + Θxt t = 1 λtẋt vt, ẋt βt βt β 2 Φ + Θxt t = λt βt ẋt 2 1 βt βt β 2 t inf t vt, ẋt βt β 2 Φ + Θxt t Φ + Θxt, 45 where we used that, according to 38, 39 and i, Φ + Θxt is bounded from below. From 45 and Lemma 2 it follows that there exists lim E 1 t R. Using now Lemma 11iv we get lim inf Φ + Θxt Φ + Θz + βtψxt 46 and, since Φ + Θxt is bounded from below, this limes inferior is a real number. Let t n n N be a sequence with lim n + t n = + such that lim Φ + Θxt n Φ + Θz + βt n Ψxt n = n + lim inf Φ + Θxt Φ + Θz + βtψxt R. Since E 1 t n = 1 Φ + Θz Φ + Θxt n Φ + Θz + βt n Ψxt n + βt n βt n and lim n + βt n = +, it yields that lim n + E 1 t n =. exists, lim E 1t =. The statement follows by taking into consideration that for every t [, + n N Thus, since lim E 1 t Ψxt Ψxt+ 1 Φ+Θxt inf βt Φ+Θxs = E 1 t 1 s βt inf Φ+Θxs s in combination with lim βt = +. Lemma 13 Assume that H Ψ, H Θ, H Φ, H 3 λ, H3 β, H and H hold and let x, v : [, + H H be a strong stable solution of the dynamical system 14. Then lim infφ + Θxt Φ + Θz z S. Proof. Take an arbitrary z S. From H there exists N argmin Ψ z such that Θz Φz. From Lemma 11iii we get lim inf, xt z

16 We claim that lim inf, xt z =. 48 Since according to the revious lemma x is bounded, this limit inferior is a real number. Let t n n N be a sequence with lim n + t n = + such that lim, xt n z =lim inf, xt z R. 49 n + Using again that x is bounded, there exists x H and a subsequence xt nk such that xt nk k converges weakly to x as k +. From 49 we derive lim inf, xt z =, x z. 5 Since Ψ is weak lower semicontinuous, from Lemma 12ii we get Ψx lim inf k + Ψxt n k =, hence x argmin Ψ. Combining this with N argmin Ψ z we derive, x z. From 5 and 47 we conclude that 48 is true. Moreover, due to Θz Φz, 38 and 39 we obtain Φ + Θxt Φ + Θz +, xt z and the conclusion follows from 48. Remark 14 One can notice that the condition β kβ has not been used in the roofs of Lemma 12 and Lemma 13. We come now to the main results of the aer. Theorem 15 Assume that H Ψ, H Θ, H Φ, H 3 λ, H3 β, H and H hold and let x, v : [, + H H be a strong stable solution of the dynamical system 14. The following statements are true: i + βtψxtdt < + ; ii ẋ L 2 [, + ; H; iii ẋ, v L 1 [, + ; iv Φ + Θxt converges to the otimal objective value of 13 as t + ; v lim Ψxt = lim βtψxt = ; vi xt converges weakly to an otimal solution of 13 as t +. Proof. Take an arbitrary z S. From H there exists N argmin Ψ z such that Θz Φz. Consider again the functions g z, h z defined in Lemma 11. Notice that statement i has been already roved in Lemma 11. Further, consider the function E 2 : [, + R defined for every t [, + as E 2 t = Φ + Θxt + βtψxt. 16

17 By using Lemma 1, relation 14 and Hβ 3 we derive for almost every t [, + Ė 2 t = vt, ẋt + Θxt, ẋt + βt Ψxt, ẋt + βtψxt = vt + Θxt + βt Ψxt, ẋt + βtψxt = λtẋt vt, ẋt + βtψxt λt ẋt 2 ẋt, vt + kβtψxt. 51 Since E 2 is bounded from below, a simle integration rocedure in 51 combined with i, Lemma 1i and Lemma 2 yields lim E 2t R, 52 + λt ẋt 2 dt < + and + ẋt, vt dt < +, which is statement iii. Statement ii follows by taking into account that lim inf λt >. Further, since βtψxt, from 46 and Lemma 13 we get that lim inf Φ + Θxt Φ + Θz + βtψxt =. 53 Taking into account the definition of E 2 and the fact that lim E 2 t R, we conclude that Further, we have lim su Φ + Θxt lim su which combined with Lemma 13 yields lim E 2t = Φ + Θz. 54 Φ + Θxt + βtψxt = lim E 2t = Φ + Θz, lim Φ + Θxt = Φ + Θz, 55 hence iv holds. The statement v is a consequence of Lemma 12ii, 54, 55 and the definition of E 2. In order to rove statement vi, we will make use of the Oial Lemma 4. From 42 we have for almost every t [, + λt 2 xt z 2 + λt ẋt, xt z + d dt g zt βt Ψ σ argmin Ψ, βt βt hence d dt g zt βt Ψ σ argmin Ψ λt βt βt 2 xt z 2 + λt ẋt xt z = G 1 t + G 2 t, 56 where G 1 t = βt Ψ σ argmin Ψ βt βt λt xt z

18 and G 2 t = λt ẋt xt z. Now using that x is bounded, from ii and H we derive that and G 1 L 1 [, + G 2 L 2 [, +. From 56, a direct alication of Lemma 3 and Lemma 11viii yields lim g zt =. By combining this with Lemma 11i and the fact that lim λt >, we conclude that there exists lim xt z R. Since z S has been chose arbitrary, the first condition of the Oial Lemma is fulfilled. Let t n n N be a sequence of ositive numbers such that lim n + t n = + and xt n converges weakly to x as n +. By using the weak lower semicontinuity of Ψ and Lemma 12ii we obtain Ψx lim inf n + Ψxt n =, hence x argmin Ψ. Moreover, the weak lower semicontinuity of Φ + Θ and 55 yield Φ + Θx lim inf n + Φ + Θxt n = Φ + Θz, thus x S. Remark 16 An anonymous reviewer raised the question whether the trajectories generated by 14 diverge in norm in case the otimization roblem 13 has no solution. In the following we give a ositive answer to this question. We assume that dom Φ argmin Ψ otherwise the otimization roblem 13 is degenerate. We suose that all the hyotheses of Theorem 15, exceting the assumtion that the set of otimal solutions S is nonemty, hold. Let x, v : [. + H H be a strong stable solution of 14 and assume that x is bounded. In other words, there exists M > such that Take z dom Φ argmin Ψ and r > such that xt M t. 57 r > max{ z, M}. 58 In the following we denote by B, r the closed ball centered at origin with radius r. We will use at follows several times the fact that the normal cone to a set at an element belonging to the interior of this set reduces to {}. Due to 58, we consequently have Φ + δ B,r xt = Φxt + N B,r xt = Φxt t. This means that x, v is a strong global solution of the system { vt Φ + δ B,r xt λtẋt + vt + vt + Θxt + βt Ψxt = 59 18

19 too. This continuous system can be associated to the otimization roblem inf {Φx + δ x argmin Ψ B,r x + Θx}. 6 Notice that the set of otimal solutions to 6 is nonemty, since its objective function is coercive. Further, by combining H with 58, we obtain: Φ + δ B,r + Θ + δ argmin Ψ = Φ + δ B,r + Θ + N argmin Ψ. 61 By alying Theorem 15 to the otimization roblem 6 and the continuous system 59, it follows that there exists an otimal slution u to 6 such that xt converges weakly to u as t +. Due to the weak-lower semicontinuity of the norm function, from 57 and 58 it follows that u belongs to the interior of the closed ball B, r. Thus, from 61, 58 and H we derive Φ + δ B,r + Θ + δ argmin Ψ u = Φ + Θ + δ argmin Ψ u. The later imlies that the set of otimal solutions to 13 is nonemty, which contradicts the assumtion we made. In this way, our claim is roved. In the last result we show that if the objective function of 13 is strongly convex, then the trajectory x generated by 14 converges strongly to the unique otimal solution of 13. Theorem 17 Assume that H Ψ, H Θ, H Φ, H 3 λ, H3 β, H and H hold and let x, v : [, + H H be a strong stable solution of the dynamical system 14. If Φ + Θ is strongly convex, then xt converges strongly to the unique otimal solution of 13 as t +. Proof. Let γ > be such that Φ + Θ is γ-strongly convex. It is a well-known fact that in case the otimization roblem 13 has a unique otimal solution, which we denote by z. From H there exists N argmin Ψ z such that Θz Φz. Consider again the functions g z, h z defined in Lemma 11. By combining 41 with the stronger inequality Φ + Θxt Φ + Θz, xt z + γ 2 xt z 2 t [, +, 62 we obtain this time see the roof of Lemma 11 for almost every t [, + d λt dt 2 xt z 2 + g z t + γ 2 xt z 2 + βt Ψ + σ argmin Ψ βt βt d λt dt 2 xt z 2 + g z t + γ 2 xt z 2 + βtψxt +, xt z d λt dt 2 xt z 2 + g z t + Φ + Θxt Φ + Θz + βtψxt. Taking into account H, by integration of the above inequality we obtain + xt z 2 dt < +. Since according to the roof of Theorem 15, lim xt z exists, we conclude that xt z converges to as t + and the roof is comlete

20 Remark 18 The results resented in this aer remain true even if the assumed growth condition is satisfied starting with a t, that is, if there exists t such that βt kβt for almost every t [t, +. Acknowledgements. The authors are thankful to two anonymous reviewers for comments and remarks which imroved the resentation of the aer. References [1] B. Abbas, An asymtotic viscosity selection result for the regularized Newton dynamic, arxiv: v1, 215 [2] B. Abbas, H. Attouch, B.F. Svaiter, Newton-like dynamics and forward-backward methods for structured monotone inclusions in Hilbert saces, Journal of Otimization Theory and its Alications 1612, , 214 [3] F. Alvarez, H. Attouch, J. Bolte, P. Redont, A second-order gradient-like dissiative dynamical system with Hessian-driven daming. Alication to otimization and mechanics, Journal de Mathématiques Pures et Aliquées 818, , 22 [4] H. Attouch, F. Alvarez, The heavy ball with friction dynamical system for convex constrained minimization roblems, in: Otimization Namur, 1998, 25 35, in: Lecture Notes in Economics and Mathematical Systems 481, Sringer, Berlin, 2 [5] H. Attouch, A. Cabot, M.-O. Czarnecki, Asymtotic behavior of nonautonomous monotone and subgradient evolution equations, to aear in Transactions of the American Mathematical Society, arxiv: , 216 [6] H. Attouch, M.-O. Czarnecki, Asymtotic control and stabilization of nonlinear oscillators with non-isolated equilibria, Journal of Differential Equations 1791, , 22 [7] H. Attouch, M.-O. Czarnecki, Asymtotic behavior of couled dynamical systems with multiscale asects, Journal of Differential Equations 2486, , 21 [8] H. Attouch, M.-O. Czarnecki, Asymtotic behavior of gradient-like dynamical systems involving inertia and multiscale asects, Journal of Differential Equations 2623, , 217 [9] H. Attouch, M.-O. Czarnecki, J. Peyouquet, Prox-enalization and slitting methods for constrained variational roblems, SIAM Journal on Otimization 211, , 211 [1] H. Attouch, M.-O. Czarnecki, J. Peyouquet, Couling forward-backward with enalty schemes and arallel slitting for constrained variational inequalities, SIAM Journal on Otimization 214, , 211 [11] H. Attouch, X. Goudou, P. Redont, The heavy ball with friction method. I. The continuous dynamical system: global exloration of the local minima of a real-valued function by asymtotic analysis of a dissiative dynamical system, Communications in Contemorary Mathematics 21, 1 34, 2 [12] H. Attouch, P.-E. Maingé, Asymtotic behavior of second-order dissiative evolution equations combining otential with non-otential effects, ESAIM. Control, Otimisation and Calculus of Variations 173, , 211 2

21 [13] H. Attouch, B.F. Svaiter, A continuous dynamical Newton-like aroach to solving monotone inclusions, SIAM Journal on Control and Otimization 492, , 211 [14] H.H. Bauschke, P.L. Combettes, Convex Analysis and Monotone Oerator Theory in Hilbert Saces, CMS Books in Mathematics, Sringer, New York, 211 [15] S. Banert, R.I. Boţ, Backward enalty schemes for monotone inclusion roblems, Journal of Otimization Theory and Alications 1663, , 215 [16] R.I. Boţ, Conjugate Duality in Convex Otimization, Lecture Notes in Economics and Mathematical Systems, Vol. 637, Sringer, Berlin Heidelberg, 21 [17] R.I. Boţ, E.R. Csetnek, Second order dynamical systems associated to variational inequalities, Alicable Analysis 965, , 217 [18] R.I. Boţ, E.R. Csetnek, A dynamical system associated with the fixed oints set of a nonexansive oerator, Journal of Dynamics and Differential Equations 291, , 217 [19] R.I. Boţ, E.R. Csetnek, Aroaching the solving of constrained variational inequalities via enalty term-based dynamical systems, Journal of Mathematical Analysis and Alications 4352, , 216 [2] R.I. Boţ, E.R. Csetnek, Forward-backward and Tseng s tye enalty schemes for monotone inclusion roblems, Set-Valued and Variational Analysis 22, , 214 [21] R.I. Boţ, E.R. Csetnek, A Tseng s tye enalty scheme for solving inclusion roblems involving linearly comosed and arallel-sum tye monotone oerators, Vietnam Journal of Mathematics 424, , 214 [22] R.I. Boţ, E.R. Csetnek, Penalty schemes with inertial effects for monotone inclusion roblems, Otimization, DOI: 1.18/ , 216 [23] R.I. Boţ, E.R. Csetnek, Second order forward-backward dynamical systems for monotone inclusion roblems, SIAM Journal on Control and Otimization 543, , 216 [24] H. Brézis, Oérateurs maximaux monotones et semi-groues de contractions dans les esaces de Hilbert, North-Holland Mathematics Studies No. 5, Notas de Matemática 5, North- Holland/Elsevier, New York, 1973 [25] A. Haraux, Systèmes Dynamiques Dissiatifs et Alications, Recherches en Mathé- matiques Aliquées 17, Masson, Paris, 1991 [26] N. Noun, J. Peyouquet, Forward-backward enalty scheme for constrained convex minimization without inf-comactness, Journal of Otimization Theory and Alications, 1583, , 213 [27] J. Peyouquet, Couling the gradient method with a general exterior enalization scheme for convex minimization, Journal of Otimizaton Theory and Alications 1531, , 212 [28] E.D. Sontag, Mathematical control theory. Deterministic finite-dimensional systems, Second edition, Texts in Alied Mathematics 6, Sringer-Verlag, New York, 1998 [29] C. Zălinescu, Convex Analysis in General Vector Saces, World Scientific, Singaore, 22 21