arxiv: v1 [math.oc] 12 Mar 2013

Transcription

1 On the convergence rate improvement of a primal-dual splitting algorithm for solving monotone inclusion problems arxiv: v [math.oc] Mar 03 Radu Ioan Boţ Ernö Robert Csetnek André Heinrich February 7, 08 Abstract. We present two modified versions of the primal-dual splitting algorithm relying on forward-backward splitting proposed in [] for solving monotone inclusion problems. Under strong monotonicity assumptions for some of the operators involved we obtain for the sequences of iterates that approach the solution orders of convergence of O n and Oω n, for ω 0,, respectively. The investigated primal-dual algorithms are fully decomposable, in the sense that the operators are processed individually at each iteration. We also discuss the modified algorithms in the context of convex optimization problems and present numerical experiments in image processing and support vector machines classification. Key Words. maximally monotone operator, strongly monotone operator, resolvent, operator splitting, subdifferential, strongly convex function, convex optimization algorithm, duality AMS subject classification. 47H05, 65K05, 90C5 Introduction and preliminaries The problem of finding the zeros of the sum of two or more maximally monotone operators in Hilbert spaces continues to be a very active research field, with applications in convex optimization, partial differential equations, signal and image processing, etc. see [,5 7,9,,3,]. To the most prominent methods in this area belong the proximal point algorithm for finding the zeros of a maximally monotone operator see [7] and the Douglas-Rachford splitting algorithm for finding the zeros of the sum of two maximally monotone operators see [4]. However, also motivated by different applications, the research community was interested in considering more general problems, in which the sum of finitely many operators appear, some of them being composed with linear continuous Department of Mathematics, Chemnitz University of Technology, D-0907 Chemnitz, Germany, e- mail: radu.bot@mathematik.tu-chemnitz.de. Research partially supported by DFG German Research Foundation, project BO 56/4-. Department of Mathematics, Chemnitz University of Technology, D-0907 Chemnitz, Germany, robert.csetnek@mathematik.tu-chemnitz.de. Research supported by DFG German Research Foundation, project BO 56/4-. Department of Mathematics, Chemnitz University of Technology, D-0907 Chemnitz, Germany, andre.heinrich@mathematik.tu-chemnitz.de. Research supported by the European Union, the European Social Fund ESF and prudsys AG in Chemnitz.

2 operators [, 9, ]. In the last years, even more complex structures were considered, in which also parallel sums are involved, see [6, 7, 3, ]. The algorithms introduced in the literature for these issues have the remarkable property that the operators involved are evaluated separately in each iteration, either by forward steps in the case of the single-valued ones including here the linear continuous operators and their adjoints or by backward steps for the set-valued ones, by using the corresponding resolvents. More than that they share the common feature to be of primal-dual type, meaning that they solve not only the primal inclusion problem, but also its Attouch- Théra-type dual. In this context we mention the primal-dual algorithms relying on Tseng s forward-backward-forward splitting method see [9, 3], on the forward-backward splitting method see [] and on the Douglas-Rachford splitting method see [7]. A relevant task is to adapt these iterative methods in order be able to investigate their convergence, namely, to eventually determine convergence rates for the sequences generated by the schemes in discussion. This could be important when one is interested in obtaining an optimal solution more rapidly than in their initial formulation, which furnish only the convergence statement. Accelerated versions of the primal-dual algorithm from [3] were already provided in [6], whereby the reported numerical experiments emphasize the advantages of the first over the original iterative scheme. The aim of this paper is to provide modified versions of the algorithm proposed by Vũ in [] for which an evaluation of their convergence behaviour is possible. By assuming that some of the operators involved are strongly monotone, we are able to obtain for the sequences of iterates orders of convergence of O n and Oωn, for ω 0,, respectively. For the readers convenience we present first some notations which are used throughout the paper see [ 3, 5, 9, ]. Let H be a real Hilbert space with inner product, and associated norm =,. The symbols and denote weak and strong convergence, respectively. When G is another Hilbert space and K : H G a linear continuous operator, then the norm of K is defined as K = sup{ Kx : x H, x }, while K : G H, defined by K y, x = y, Kx for all x, y H G, denotes the adjoint operator of K. For an arbitrary set-valued operator A : H H we denote by Gr A = {x, u H H : u Ax} its graph, by dom A = {x H : Ax } its domain and by A : H H its inverse operator, defined by u, x Gr A if and only if x, u Gr A. We say that A is monotone if x y, u v 0 for all x, u, y, v Gr A. A monotone operator A is said to be maximally monotone, if there exists no proper monotone extension of the graph of A on H H. The resolvent of A, J A : H H, is defined by J A = Id H +A, where Id H : H H, Id H x = x for all x H, is the identity operator on H. Moreover, if A is maximally monotone, then J A : H H is single-valued and maximally monotone cf. [, Proposition 3.7 and Corollary 3.0]. For an arbitrary γ > 0 we have see [, Proposition 3.] and see [, Proposition 3.8] p J γa x if and only if p, γ x p Gr A J γa + γj γ A γ Id H = Id H. Let γ > 0 be arbitrary. We say that A is γ-strongly monotone if x y, u v γ x y for all x, u, y, v Gr A. A single-valued operator A : H H is said to be γ-cocoercive

3 if x y, Ax Ay γ Ax Ay for all x, y H H. Moreover, A is γ-lipschitzian if Ax Ay γ x y for all x, y H H. A single-valued linear operator A : H H is said to be skew, if x, Ax = 0 for all x H. Finally, the parallel sum of two operators A, B : H H is defined by A B : H H, A B = A + B. The following problem represents the starting point of our investigations see []. Problem Let H be a real Hilbert space, z H, A : H H a maximally monotone operator and C : H H an η-cocoercive operator for η > 0. Let m be a strictly positive integer and for any i {,..., m} let G i be a real Hilbert space, r i G i, B i : G i G i a maximally monotone operator, D i : G i G i a maximally monotone and ν i -strongly monotone operator for ν i > 0 and L i : H G i a nonzero linear continuous operator. The problem is to solve the primal inclusion find x H such that z Ax + together with the dual inclusion L i Bi D i L i x r i + Cx, find v G,..., v m G m such that x H : { z m L i v i Ax + Cx v i B i D i L i x r i, i =,..., m. 3 if We say that x, v,..., v m H G... G m is a primal-dual solution to Problem, z L i v i Ax + Cx and v i B i D i L i x r i, i =,..., m. 4 If x H is a solution to, then there exists v,..., v m G... G m such that x, v,..., v m is a primal-dual solution to Problem and, if v,..., v m G... G m is a solution to 3, than there exists x H such that x, v,..., v m is a primal-dual solution to Problem. Moreover, if x, v,..., v m H G... G m is a primal-dual solution to Problem, then x is a solution to and v,..., v m G... G m is a solution to 3. By employing the classical forward-backward algorithm see [, 0] in a renormed product space, Vũ proposed in [] an iterative scheme for solving a slightly modified version of Problem formulated in the presence of some given weights w i 0, ], i =,..., m, with m w i = for the terms occurring in the second summand of the primal inclusion problem. The following result is an adaption of [, Theorem 3.] to Problem in the error-free case and when n = for any n 0. Theorem In Problem suppose that z ran A + L i Bi D i L i r i + C. Let τ and σ i, i =,..., m, be strictly positive numbers such that min{τ, σ,..., σ m } min{η, ν,..., ν m } m τ σ i L i >. 3

4 Let x 0, v,0,..., v m,0 H G... G m and for all n 0 set: [ x n+ = J τa xn τ m L i v i,n + Cx n z ] y n = x n+ x n v i,n+ = J σi B [v i,n + σ i L i y n Di v i,n r i ], i =,..., m. i Then there exists a primal-dual solution x, v,..., v m to Problem such that x n x and v,n,..., v m,n v,..., v m as n +. The structure of the paper is as follows. In the next section we propose under appropriate strong monotonicity assumptions two modified versions of the above algorithm which ensure for the sequences of iterates orders of convergence of O n and Oωn, for ω 0,, respectively. In Section 3 we show how to particularize the general results in the context of nondifferentiable convex optimization problems, where some of the functions occurring in the objective are strongly convex. In the last section we present some numerical experiments in image denoising and support vector machines classification and emphasize also the practical advantages of the modified iterative schemes over the initial one provided in Theorem. Two modified primal-dual algorithms In this section we propose in two different settings modified versions of the algorithm in Theorem and discuss the orders of convergence of the sequences of iterates generated by the new schemes.. The case A + C is strongly monotone For the beginning, we show that in case A + C is strongly monotone one can guarantee an order of convergence of O n for the sequence x n n 0. To this end, we update in each iteration the parameters τ and σ i, i =,..., m, and use a modified formula for the sequence y n n 0. Due to technical reasons, we apply this method in case Di is equal to zero for i =,..., m, that is D i 0 = G i and D i x = for x 0. Let us notice that, by using the approach proposed in [6, Remark 3.], one can extend the statement of Theorem 7 below, which is the main result of this subsection, to the primal-dual pair of monotone inclusions as stated in Problem. More precisely, the problem we consider throughout this subsection is as follows. Problem 3 Let H be a real Hilbert space, z H, A : H H a maximally monotone operator and C : H H a monotone and η-lipschitzian operator for η > 0. Let m be a strictly positive integer and for any i {,..., m} let G i be a real Hilbert space, r i G i, B i : G i G i a maximally monotone operator and L i : H G i a nonzero linear continuous operator. The problem is to solve the primal inclusion find x H such that z Ax + L i B i L i x r i + Cx, 5 together with the dual inclusion find v G,..., v m G m such that x H : { z m L i v i Ax + Cx v i B i L i x r i, i =,..., m. 6 4

5 As for Problem, we say that x, v,..., v m H G... G m is a primal-dual solution to Problem 3, if z L i v i Ax + Cx and v i B i L i x r i, i =,..., m. 7 Remark 4 One can notice that, in comparison to Problem, we relax in Problem 3 the assumptions made on the operator C. It is obvious that, if C is a η-cocoercive operator for η > 0, then C is monotone and /η-lipschitzian. Although in case C is the gradient of a convex and differentiable function, due to the celebrated Baillon-Haddad Theorem see, for instance, [, Corollary 8.6], the two classes of operators coincide, in general the second one is larger. Indeed, nonzero linear, skew and Lipschitzian operators are not cocoercive. For example, when H and G are real Hilbert spaces and L : H G is nonzero linear continuous, then x, v L v, Lx is an operator having all these properties. This operator appears in a natural way when considering primal-dual monotone inclusion problems as done in [9]. Under the assumption that A + C is γ-strongly monotone for γ > 0 we propose the following modification of the iterative scheme in Theorem. Algorithm 5 Initialization: For n 0 set: Choose τ 0 > 0, σ i,0 > 0, i =,.., m, such that τ 0 < γ/η, η +, τ m 0 σ i,0 L i + τ 0 γ ητ 0 / and x 0, v,0,..., [ v m,0 H G... G m. x n+ = J τn/a xn τ n / m L i v i,n + Cx n z ] θ n = / + τ n γ ητ n / y n = x n+ + θ n x n+ x n v i,n+ = J σi,n B [v i,n + σ i,n L i y n r i ], i =,..., m i τ n+ = θ n τ n, σ i,n+ = σ i,n /θ n+, i =,..., m. Remark 6 Notice that the assumption τ m 0 σ i,0 L i + τ 0 γ ητ 0 / in Algorithm 5 is equivalent to τ m σ i,0 L i, being fulfilled if τ 0 > 0 is chosen such that τ 0 γ/ + γ / + m σ i,0 L i + η/ m σ i,0 L i. + η/ Theorem 7 Suppose that A+C is γ-strongly monotone for γ > 0 and let x, v,..., v m be a primal-dual solution to Problem 3. Then the sequences generated by Algorithm 5 fulfill for any n 0 x n+ x m m + τ σ i,0 L i v i,n v i τ σ i,0 Moreover, τ n+ x x + τ v i,0 v i τ σ i,0 + x x 0 τ 0 + m L i x x 0, v i,0 v i. τ 0 lim nτ n = n + γ, hence one obtains for x n n 0 an order of convergence of O n. 5

6 Proof. The idea of the proof relies on showing that the following Fejér-type inequality is true for any n 0 τ n+ τ n+ τ n+ τ n x n+ x v i,n+ v i + τ σ i,0 + x n+ x n+ τn+ L i x n+ x n+, v i,n+ + v i 8 x n+ x + L i x n+ x n, v i,n + v i. v i,n v i + x n+ x n τ σ i,0 τn To this end we use first that in the light of the definition of the resolvents it holds for any n 0 m x n+ x n+ L i v i,n+ + Cx n+ z + Cx n+ A + Cx n+. 9 τ n+ Since A + C is γ-strongly monotone, 7 and 9 yield for any n 0 γ x n+ x x n+ x, x n+ x n+ + τ n+ m x n+ x, L i v i,n+ + Cx n+ z + Cx n+ z L i v i = x n+ x, x n+ x n+ + x n+ x, Cx n+ Cx n+ + τ n+ L i x n+ x, v i v i,n+. Further, we have x n+ x, x n+ x n+ = x n+ x and, since C is η-lipschitzian, x n+ x x n+ x n+ x n+ x, Cx n+ Cx n+ ητ n+ x n+ x + η x n+ x n+, τ n+ 0 hence for any n 0 it yields + γ ητ n+ x n+ x τ n+ τ n+ x n+ x η τ n+ x n+ x n+ + L i x n+ x, v i v i,n+. 6

7 Taking into account that η + we obtain for any n 0 that + γ ητ n+ x n+ x τ n+ x n+ x x n+ x n+ + τ n+ τ n+ On the other hand, for every i =,..., m and any n 0, from L i x n+ x, v i v i,n+. σ i,n v i,n v i,n+ + L i y n r i B i v i,n+, the monotonicity of Bi and 7 we obtain 0 v i,n v i,n+ + L i y n r i L i x r i, v i,n+ v i σ i,n = v i,n v i,n+, v i,n+ v i + L i y n x, v i,n+ v i σ i,n = v i,n v i v i,n v i,n+ v i,n+ v i σ i,n σ i,n σ i,n + L i y n x, v i,n+ v i, hence v i,n+ v i σ i,n v i,n v i σ i,n v i,n v i,n+ σ i,n + L i y n x, v i,n+ v i. 3 Summing up the inequalities in and 3 we obtain for any n 0 + γ ητ n+ x n+ x + τ n+ τ n+ x n+ x + + L i x n+ y n, v i,n+ + v i. v i,n+ v i σ i,n v i,n v i x n+ x n+ σ i,n τ n+ v i,n v i,n+ σ i,n 4 Further, since y n = x n+ + θ n x n+ x n, for every i =,..., m and any n 0 it holds L i x n+ y n, v i,n+ + v i = L i xn+ x n+ θ n x n+ x n, v i,n+ + v i = L i x n+ x n+, v i,n+ + v i θ n L i x n+ x n, v i,n + v i + θ n L i x n+ x n, v i,n + v i,n+ L i x n+ x n+, v i,n+ + v i θ n L i x n+ x n, v i,n + v i + θn L i σ i,n x n+ x n + v i,n v i,n+. σ i,n 7

8 By combining the last inequality with 4 we obtain for any n 0 + γ ητ n+ x n+ x + τ n+ v i,n+ v i + x n+ x n+ σ i,n τ n+ L i x n+ x n+, v i,n+ + v i 5 τ n+ x n+ x + θ n L i x n+ x n, v i,n + v i. v i,n v i m + L i σ i,n θ σ n x n+ x n i,n After dividing 5 by τ n+ and noticing that for any n 0 τ n+ + γ τ n+ η = τ n+, and τ n+ σ i,n = τ n σ i,n =... = τ σ i,0 m L i σ i,n θ n = τ m n+ L i σ i,n τ n+ τn = τ m L i σ i,0 τn τn, it follows that the Fejér-type inequality 8 is true. Let N N, N. Summing up the inequality in 8 from n = 0 to N, it yields τ N+ x N+ x + x x + τ v i,n v i τ σ i,0 + x N+ x N τ N v i,0 v i + x x 0 τ σ i,0 τ0 + 6 τ N L i x N+ x N, v i,n + v i τ 0 L i x x 0, v i,0 + v i Further, for every i =,..., m we use the inequality and obtain finally L i x N+ x N, v i,n + v i τ N σ i,0 L i m τn m σ i,0 L i x N+ x N + σ i,0 L i v i,n v i σ i,0. x N+ x τ N+ + v i,n v i x x τ σ i,0 τ + v i,0 v i τ σ i,0 + x x 0 τ 0 + m m j= L i x x 0, v i,0 v i + σ j,0 L j v i,n v i, τ 0 σ i,0 8

9 which rapidly yields the inequality in the statement of the theorem. We close the proof by showing that lim nτ n = /γ. Notice that for any n 0, n + τ n+ = τ n + τ n γ ητ n. 7 Since 0 < τ 0 < γ/η, it follows by induction that 0 < τ n+ < τ n < τ 0 < γ/η for any n, hence the sequence τ n n 0 converges. In the light of 7 one easily obtains that lim τ τ n = 0 and, further, that lim n n + n + τ n+ =. As τ n n 0 is a strictly increasing and unbounded sequence, by applying the Stolz-Cesàro Theorem it yields lim nτ n = n + lim n + = lim n + = lim n + n n + n τ n+ = lim n + τ n τ n τ n+ τ n + τ n+ τn τn+ τ n + τ n+ τ n+ γ η τ n = n + τ n = lim = lim n + lim n + τ n τ n+ τ n τ n+ τ n τ n+ τ n + τ n+ τ n+ τn γ ητ n τ n τ n+ + γ η τ n = γ. Remark 8 Let us mention that, if A + C is γ-strongly monotone with γ > 0, then the operator A + m L i B il i r i + C is strongly monotone, as well, thus the monotone inclusion problem 5 has at most one solution. Hence, if x, v,..., v m is a primal-dual solution to Problem 3, then x is the unique solution to 5. Notice that the problem 6 may not have an unique solution.. The case A + C and B i + D i, i =,..., m, are strongly monotone In this subsection we propose a modified version of the algorithm in Theorem which guarantees when A + C and Bi + Di, i =,..., m, are strongly monotone orders of convergence of Oω n, for ω 0,, for the sequences x n n 0 and v i,n n 0, i =,..., m. The algorithm aims to solve the primal-dual pair of monotone inclusions stated in Problem under relaxed assumptions for the operators C and Di, i =,..., m. Problem 9 Let H be a real Hilbert space, z H, A : H H a maximally monotone operator and C : H H a monotone and η-lipschitzian operator for η > 0. Let m be a strictly positive integer and for any i {,..., m} let G i be a real Hilbert space, r i G i, B i : G i G i a maximally monotone operator, D i : G i G i a monotone operators such that Di is ν i -Lipschitzian for ν i > 0 and L i : H G i a nonzero linear continuous operator. The problem is to solve the primal inclusion find x H such that z Ax + together with the dual inclusion L i Bi D i L i x r i + Cx, 8 find v G,..., v m G m such that x H : { z m L i v i Ax + Cx v i B i D i L i x r i, i =,..., m. 9 9

10 Under the assumption that A + C is γ-strongly monotone for γ > 0 and Bi + Di is δ i -strongly monotone with δ i > 0, i =,...m, we propose the following modification of the iterative scheme in Theorem. Algorithm 0 Initialization: For n 0 set: Choose µ { > 0 such that µ min γ /η, δ /ν,..., δ m/νm, γ/ } m L i /δ i, τ = µ/γ, σ i = µ/δ i, i =,.., m, θ [/ + [ µ, ] and x 0, v,0,..., v m,0 H G... G m. x n+ = J τa xn τ m L i v i,n + Cx n z ] y n = x n+ + θx n+ x n v i,n+ = J σi B [v i,n + σ i L i y n Di v i,n r i ], i =,..., m. i Theorem Suppose that A + C is γ-strongly monotone for γ > 0, Bi + Di is δ i - strongly monotone for δ i > 0, i =,..., m, and let x, v,..., v m be a primal-dual solution to Problem 9. Then the sequences generated by Algorithm 0 fulfill for any n 0 γ x n+ x + ω ω n γ x x + where 0 < ω = +θ 4+µ <. δ i v i,n v i δ i v i,0 v i + γ ω x x 0 + µω L i x x 0, v i,0 v i, Proof. For any n 0 we have m τ x n+ x n+ L i v i,n+ + Cx n+ z + Cx n+ A + Cx n+, 0 thus, since A + C is γ-strongly monotone, 9 yields γ x n+ x x n+ x, τ x n+ x n+ + m x n+ x, L i v i,n+ + Cx n+ z + Cx n+ z τ x n+ x, x n+ x n+ + x n+ x, Cx n+ Cx n+ + L i x n+ x, v i v i,n+. Further, by using 0 and L i v i = x n+ x, Cx n+ Cx n+ γ x n+ x + η γ x n+ x n+, 0

11 we get for any n 0 τ + γ x n+ x τ x n+ x τ η x n+ x n+ + γ L i x n+ x, v i v i,n+. After multiplying this inequality with µ and taking into account that µ τ = γ, µ τ + γ = γ + µ and µ τ η = γ η γ γ µ γ, we obtain for any n 0 γ + µ x n+ x γ x n+ x γ x n+ x n+ + µ L i x n+ x, v i v i,n+. On the other hand, for every i =,..., m and any n 0, from v i,n v i,n+ + L i y n Di v i,n r i + Di v i,n+ Bi + Di v i,n+, σ i the δ i -strong monotonicity of Bi + Di and 9 we obtain δ i v i,n+ v i v i,n v i,n+, v i,n+ v i σ i + L i y n r i Di v i,n + Di v i,n+ L i x r i, v i,n+ v i. Further, for every i =,..., m and any n 0 we have v i,n v i,n+, v i,n+ v i = v i,n v i v i,n v i,n+ v i,n+ v i σ i σ i σ i σ i and, since D i is a ν i -Lipschitzian operator, D i v i,n+ D i v i,n, v i,n+ v i δ i v i,n+ v i + ν i δ i v i,n+ v i,n. Consequently, for every i =,..., m and any n 0 it holds + δ i v i,n+ v i σ i σ i v i,n v i ν i v i,n+ v i,n + L i x y n, v i v i,n+, σ i δ i which, after multiplying it by µ here is the initial choice of µ determinant, yields δ i + µ v i,n+ v i δ i v i,n v i δ i v i,n+ v i,n +µ L i x y n, v i v i,n+. 3

12 We denote a n := γ x n+ x + δ i v i,n v i n 0. Summing up the inequalities in and 3 we obtain for any n 0 + µ a n+ a n 4 γ x n+ x n+ δ i v i,n v i,n+ + µ L i x n+ y n, v i v i,n+. Further, since y n = x n+ + θx n+ x n and ω θ, for every i =,..., m and any n 0 it holds L i x n+ y n, v i v i,n+ = L i x n+ x n+ θx n+ x n, v i v i,n+ = L i x n+ x n+, v i v i,n+ ω L i x n+ x n, v i v i,n + ω L i x n+ x n, v i,n+ v i,n + θ ω L i x n+ x n, v i,n+ v i L i x n+ x n+, v i v i,n+ ω L i x n+ x n, v i v i,n + ω L i µω L i x n+ x n v i,n+ v i,n + δ i + δ i µω L i θ ω L i µω L i x n+ x n v i,n+ v i + δ i = δ i µω L i L i x n+ x n+, v i v i,n+ ω L i x n+ x n, v i v i,n + θµω L i x n+ x n v i,n+ v i,n + δ i δ i µ Taking into consideration that µ θω L i from 4 we obtain for any n 0 δ i v i,n+ v i + θ ωδ i. µω γθ ω γ ω and + µ = ω + θ ω ω, ω a n+ + γ x n+ x n+ a n + γ ω x n+ x n θ ω δ i a n+ ω v i,n+ v i + µ Li x n+ x n+, v i v i,n+ ω L i x n+ x n, v i v i,n. As ω θ and a n+ m δ i v i,n+ v i 0, we further get after multiplying the last inequality with ω n the following Fejér-type inequality that holds for any n 0 ω n+ a n+ + γ ω n x n+ x n+ + µω n ω n a n + γ ω n x n+ x n + µω n m m L i x n+ x n+, v i,n+ v i L i x n+ x n, v i,n v i. 5

13 Let N N, N. Summing up the inequality in 5 from n = 0 to N, it yields Using that ω N a N + γ ω N+ x N x N+ + µω N+ a 0 + γ ω x x 0 + µω m L i x x 0, v i,0 v i. L i x N+ x N, v i,n v i µ L i 4δ i L i x N+ x N, v i,n v i x N+ x N δ i µ v i,n v i, i =,..., m, this further yields ω N a N + ω N+ γ µ L i m x N x N+ ω N+ δ i v i,n v i 4 δ i a 0 + γ ω x x 0 + µω L i x x 0, v i,0 v i. 6 Taking into account the way µ has been chosen, we have γ µ 4 L i δ i γ γ 4 > 0, hence, after multiplying 6 with ω N it yields a N ω δ i v i,n v i ω a N 0 + γ ω x x 0 + µω L i x x 0, v i,0 v i. The conclusion follows by taking into account the definition of the sequence a n n 0. Remark If A + C is γ-strongly monotone for γ > 0 and Bi + Di is δ i -strongly monotone for δ i > 0, i =,..., m, then there exists at most one primal-dual solution to Problem 9. Hence, if x, v,..., v m is a primal-dual solution to Problem 9, then x is the unique solution to the primal inclusion 8 and v,..., v m is the unique solution to the dual inclusion 9. 3 Convex optimization problems The aim of this section is to show that the two algorithms proposed in this paper and investigated from the point of view of their convergence properties can be employed when solving a primal-dual pair of convex optimization problems. For a function f : H R, where R := R {± } is the extended real line, we denote by dom f = {x H : fx < + } its effective domain and say that f is proper if dom f and fx for all x H. We denote by ΓH the family of proper convex and lower semi-continuous extended real-valued functions defined on H. Let f : H R, f u = sup x H { u, x fx} for all u H, be the conjugate function of f. The subdifferential of 3

14 f at x H, with fx R, is the set fx := {v H : fy fx + v, y x y H}. We take by convention fx :=, if fx {± }. Notice that if f ΓH, then f is a maximally monotone operator cf. [6] and it holds f = f. For f, g : H R two proper functions, we consider their infimal convolution, which is the function f g : H R, defined by f gx = inf y H {fy + gx y}, for all x H. Let S H be a nonempty set. The indicator function of S, δ S : H R, is the function which takes the value 0 on S and + otherwise. The subdifferential of the indicator function is the normal cone of S, that is N S x = {u H : u, y x 0 y S}, if x S and N S x = for x / S. When f ΓH and γ > 0, for every x H we denote by prox γf x the proximal point of parameter γ of f at x, which is the unique optimal solution of the optimization problem { inf fy + } y x. 7 y H γ Notice that J γ f = Id H +γ f = prox γf, thus prox γf : H H is a single-valued operator fulfilling the extended Moreau s decomposition formula prox γf +γ prox /γf γ Id H = Id H. 8 Let us also recall that the function f : H R is said to be γ-strongly convex for γ > 0, if f γ is a convex function. Let us mention that this property implies γ-strong monotonicity of f see [, Example.3]. Finally, we notice that for f = δ S, where S H is a nonempty convex and closed set, it holds J γns = J NS = J δs = Id H +N S = prox δs = P S, 9 where P S : H C denotes the projection operator on S see [, Example 3.3 and Example 3.4]. In order to investigate the applicability of the algorithm introduced in Subsection. we consider the following primal-dual pair of convex optimization problems. Problem 3 Let H be a real Hilbert space, z H, f ΓH and h : H R a convex and differentiable function with a η-lipschitzian gradient for η > 0. Let m be a strictly positive integer and for any i {,..., m} let G i be a real Hilbert space, r i G i, g i ΓG i and L i : H G i a nonzero linear continuous operator. Consider the convex optimization problem { } inf x H fx + and its Fenchel-type dual problem { sup v i G i,,...,m g i L i x r i + hx x, z f h z L i v i Considering maximal monotone operators 30 g i v i + v i, r i }. 3 A = f, C = h and B i = g i, i =,..., m, 4

15 the monotone inclusion problem 5 reads find x H such that z fx + while the dual inclusion problem 6 reads find v G,..., v m G m such that x H : L i g i L i x r i + hx, 3 { z m L i v i fx + hx v i g i L i x r i, i =,..., m. If x, v,..., v m H G... G m is a primal-dual solution to 3-33, namely, z L i v i fx + hx and v i g i L i x r i, i =,..., m, 34 then x is an optimal solution of the problem 30, v,..., v m is an optimal solution of 3 and the optimal objective values of the two problems coincide. Notice that 34 is nothing else than the system of optimality conditions for the primal-dual pair of convex optimization problems In case a qualification condition is fulfilled, these optimality conditions are also necessary. For the readers convenience, let us present some qualification conditions which are suitable in this context. One of the weakest qualification conditions of interiority-type reads see, for instance, [3, Proposition 4.3, Remark 4.4] m r,..., r m sqri dom g i {L x,..., L m x : x dom f}. 35 Here, for H a real Hilbert space and S H a convex set, we denote by sqri S := {x S : >0 S x is a closed linear subspace of H} its strong quasi-relative interior. Notice that we always have int S sqri S in general this inclusion may be strict. If H is finite-dimensional, then sqri S coincides with ri S, the relative interior of S, which is the interior of S with respect to its affine hull. The condition 35 is fulfilled if i dom g i = G i, i =,..., m or ii H and G i are finitedimensional and there exists x ri dom f such that L i x r i ri dom g i, i =,..., m see [3, Proposition 4.3]. Another useful and easily verifiable qualification condition guaranteing the optimality conditions 34 has the following formulation: there exists x dom f m L i r i + dom g i such that g i is continuous at L i x r i, i =,..., m see [3, Remark.5] and [5]. The following two statements are particular instances of Algorithm 5 and Theorem 7, respectively. Algorithm 4 Initialization: For n 0 set: Choose τ 0 > 0, σ i,0 > 0, i =,.., m, such that τ 0 < γ/η, η +, τ m 0 σ i,0 L i + τ 0 γ ητ 0 / and x 0, v,0,..., v m,0 H G... G m. [ xn τ n / m L i v i,n + hx n z ] x n+ = prox τn/f θ n = / + τ n γ ητ n / y n = x n+ + θ n x n+ x n v i,n+ = prox σi,n gi [v i,n + σ i,n L i y n r i ], i =,..., m τ n+ = θ n τ n, σ i,n+ = σ i,n /θ n+, i =,..., m. 5 33

16 Theorem 5 Suppose that f + h is γ-strongly convex for γ > 0 and the qualification condition 35 holds. Then there exists a unique optimal solution x to 30, an optimal solution v,..., v m to 3 fulfilling the optimality conditions 34 and such that the optimal objective values of the problems 30 and 3 coincide. The sequences generated by Algorithm 4 fulfill for any n 0 x n+ x m m + τ σ i,0 L i v i,n v i τ σ i,0 Moreover, τ n+ x x + τ v i,0 v i τ σ i,0 + x x 0 τ 0 + m L i x x 0, v i,0 v i. τ 0 lim nτ n = n + γ, hence one obtains for x n n 0 an order of convergence of O n. Remark 6 The uniqueness of the solution of 30 in the above theorem follows from [, Corollary.6]. Remark 7 In case hx = 0 for all x H, one has to choose in Algorithm 4 as initial points τ 0 > 0, σ i,0 > 0, i =,.., m, with τ 0 m σ i,0 L i + τ 0 γ/ and and to update the sequence θ n n 0 via θ n = / + τ n γ/ for any n 0, in order to obtain a suitable iterative scheme for solving the pair of primal-dual optimization problems 30-3 with the same convergence behavior as of Algorithm 4. We turn now our attention to the algorithm introduced in Subsection. and consider to this end the following primal-dual pair of convex optimization problems. Problem 8 Let H be a real Hilbert space, z H, f ΓH and h : H R a convex and differentiable function with a η-lipschitzian gradient for η > 0. Let m be a strictly positive integer and for any i {,..., m} let G i be a real Hilbert space, r i G i, g i, l i ΓG i such that l i is νi -strongly convex for ν i > 0 and L i : H G i a nonzero linear continuous operator. Consider the convex optimization problem { } inf x H fx + and its Fenchel-type dual problem { sup v i G i,,...,m g i l i L i x r i + hx x, z f h z L i v i Considering the maximal monotone operators 36 g i v i + li v i + v i, r i }. 37 A = f, C = h, B i = g i and D i = l i, i =,..., m, according to [, Proposition 7.0, Theorem 8.5], D i = l i is a monotone and ν i - Lipschitzian operator for i =,..., m. The monotone inclusion problem 8 reads find x H such that z fx + L i g i l i L i x r i + hx, 38 6

17 while the dual inclusion problem 9 reads { z m find v G,..., v m G m such that x H : L i v i fx + hx v i g i l i L i x r i, i =,..., m. 39 If x, v,..., v m H G... G m is a primal-dual solution to 38-39, namely, z L i v i fx + hx and v i g i l i L i x r i, i =,..., m, 40 then x is an optimal solution of the problem 36, v,..., v m is an optimal solution of 37 and the optimal objective values of the two problems coincide. Notice that 40 is nothing else than the system of optimality condition for the primal-dual pair of convex optimization problems The assumptions made on l i guarantees that g i l i ΓG i see [, Corollary.6, Proposition.4] and, since domg i l i = dom g i + dom l i, i =,..., m, one can can consider the following qualification condition of interiority-type in order to guarantee 40 m r,..., r m sqri dom g i + dom l i {L x,..., L m x : x dom f}. 4 The following two statements are particular instances of Algorithm 0 and Theorem, respectively. Algorithm 9 Initialization: For n 0 set: Choose µ { > 0 such that µ min γ /η, δ /ν,..., δ m/νm, γ/ } m L i /δ i, τ = µ/γ, σ i = µ/δ i, i =,.., m, θ [/ + µ, [ ] and x 0, v,0,..., v m,0 H G... G m. x n+ = prox τf xn τ m L i v i,n + hx n z ] y n = x n+ + θx n+ x n v i,n+ = prox σi gi [v i,n + σ i L i y n li v i,n r i ], i =,..., m. Theorem 0 Suppose that f + h is γ-strongly convex for γ > 0, gi + l i is δ i -strongly convex for δ i > 0, i =,..., m, and the qualification condition 4 holds. Then there exists a unique optimal solution x to 36, a unique optimal solution v,..., v m to 37 fulfilling the optimality conditions 40 and such that the optimal objective values of the problems 36 and 37 coincide. The sequences generated by Algorithm 9 fulfill for any n 0 γ x n+ x + ω ω n γ x x + where 0 < ω = +θ 4+µ <. δ i v i,n v i δ i v i,0 v i + γ ω x x 0 + µω L i x x 0, v i,0 v i, 7

18 4 Numerical experiments In this section we illustrate the applicability of the theoretical results in the context of two numerical experiments in image processing and support vector machines classification. 4. Image processing In this subsection we compare the numerical performances of Algorithm 4 with the ones in the iterative scheme in Theorem for an image denoising problem. To this end we treat the nonsmooth regularized convex optimization problem inf x [0,] k { x b + T V x + W x }, 4 where T V : R k R denotes a discrete isotropic total variation, W : R k R k a the discrete Haar wavelet transform with four levels,, > 0 the regularization parameters and b R k the observed noisy image. Notice that we consider images of size k = M N as vectors x R k, where each pixel denoted by x i,j, i M, j N, ranges in the closed interval from 0 pure black to pure white. Two popular choices for the discrete total variation functional are the isotropic total variation T V iso : R k R, T V iso x = M N j= M + x i+,j x i,j + x i,j+ x i,j N x i+,n x i,n + and the anisotropic total variation T V aniso : R k R, T V aniso x = M N j= M + j= x M,j+ x M,j, x i+,j x i,j + x i,j+ x i,j N x i+,n x i,n + j= x M,j+ x M,j, where in both cases reflexive Neumann boundary conditions are assumed. Obviously, in both situations the qualification condition in Theorem 5 is fulfilled. Denote Y = R k R k and define the linear operator L : R k Y, x i,j L x i,j, L x i,j, where { { xi+,j x L x i,j = i,j, if i < M 0, if i = M and L xi,j+ x x i,j = i,j, if j < N 0, if j = N. The operator L represents a discretization of the gradient in horizontal and vertical direction. One can easily check that L 8 and that its adjoint L : Y R k is as easy to implement as the operator itself cf. [0]. Moreover, since W is an orthogonal wavelet, it holds W =. 8

19 a Noisy image, σ = 0.06 b Noisy image, σ = 0. c Denoised image, = 0.035, = 0.0, isotropic TV d Denoised image, = 0.07, = 0.0, isotropic TV Figure : The noisy images in a and b were obtained after adding white Gaussian noise with standard deviation σ = 0.06 and σ = 0., respectively, to the original lichtenstein test image. The outputs of Algorithm 4 after 00 iterations when solving 4 with isotropic total variation are shown in c and d, respectively. When considering the isotropic total variation, the problem 4 can be formulated as inf {fx + g Lx + g W x}, 43 x R k where f : R k R, fx = x b + δ [0,] kx is -strongly convex, g : Y R is defined as g u, v = u, v, where, : Y R, u, v = M N j= u i,j + v i,j, is a norm on the Hilbert space Y and g : R k R, g x = x. Take p, q Y and σ > 0. We have prox σf p = P [0,] k + σ p + σb. Moreover, g p, q = δ Sp, q and prox σg p, q = P S p, q, where cf. [6] S = p, q Y : max p i,j + q i,j i M j N 9

20 ε = 0 4 isotropic TV anisotropic TV σ = 0.06 σ = 0. σ = 0.06 σ = 0. Algorithm in [] Algorithm Table : Comparison of the algorithm in [] and its modification, Algorithm 4, when denoising the images in Figure a and b for isotropic TV and anisotropic TV. The entries represent the number of iterations needed for attaining a root mean squared error for the primal iterates below the tolerance level of ε = 0 4. RMSE anisotropic isotropic anisotropic modified isotropic modified Iterations Figure : RMSE curves for image denoising: the case of white Gaussian noise with standard deviation σ = 0.06 and the projection operator P S : Y S is defined via p i,j, q i,j p i,j, q i,j { }, i M, j N. max, p i,j + q i,j Finally, g p = δ [, ] kp, hence prox σg p = P [, ] kp p Rk. On the other hand, when considering the anisotropic total variation, the problem 4 can be formulated as inf {fx + g Lx + g W x}, 44 x R k where the functions f, g are taken as above and g : Y R is defined as g u, v = u, v. For every p, q Y we have g p = δ [, ] k [, ] kp, q and prox σ g p, q = P [, ] k [, ] kp, q. We experimented with the lichtenstein test image to which we added white Gaussian noise with standard deviation σ = 0.06 and σ = 0., respectively. We solved 4 by Algorithm 4 with the modifications mentioned in Remark 7 for both instances of the discrete total variation functional. For the first noisy image added noise with standard deviation σ = 0.06, we took as regularization parameters = and = 0.0 and for the second one added noise with standard deviation σ = 0., =

21 RMSE anisotropic isotropic anisotropic modified isotropic modified Iterations Figure 3: RMSE curves for image denoising: the case of white Gaussian noise with standard deviation σ = 0. and = 0.0. As initial choices in Algorithm 4 we opted for =, τ 0 = 50, σ,0 = 0.04 and σ,0 = The reconstructed images after 00 iterations for isotropic total variation are shown in Figure. We compared Algorithm 4 from the point of view of the number of iterations needed for a good recovery with the iterative method from Theorem see [] for both isotropic and anisotropic total variation. For the iterative scheme from [] we have chosen for both noisy images and both discrete total variation functionals the optimal initializations τ = 0.35, σ = 0. and σ = 0.0. The comparisons concerning the number of iterations needed for a good recovery were made via the root mean squared error RMSE. We refer the reader to Table for the achieved results, which show that Algorithm 4 outperforms the iterative scheme in []. Figures and 3 show the evolution of the RMSE curves when solving 43 and 44 with the algorithm from [] and with its modified version, Algorithm 4, respectively. 4. Support vector machines classifications The numerical experiments we present in this subsection refer to the class of kernel based learning methods and address the problem of classifying images via support vector machines. We make use of a data set of 800 training images and 983 test images of size 8 8 from the website roweis/data.html. The problem consists in determining a decision function based on a pool of handwritten digits showing either the number eight or the number nine, labeled by and +, respectively see Figure 4. We evaluate the quality of the decision function on a test data set by computing the percentage of misclassified images. Notice that we use only a half of the available images from the training data set, in order to reduce the computational effort. Figure 4: A sample of images belonging to the classes and +, respectively. The classifier functional f is assumed to be an element of the Reproducing Kernel

22 Hilbert Space RKHS H κ, which in our case is induced by the symmetric and finitely positive definite Gaussian kernel function κ : R d R d x y R, κx, y = exp σ. Let, κ be the inner product on H κ, κ the corresponding norm and K R n n the Gram matrix with respect to the training data set Z = {X, Y,..., X n, Y n } R d {+, }, namely the symmetric and positive definite matrix with entries K ij = κx i, X j for i, j =,..., n. We make use of the hinge loss function v : R R R, vx, y = max{ xy, 0}, which penalizes the deviation between the predicted value fx and the true value y {+, }. The smoothness of the decision function f H κ is employed by means of the smoothness functional Ω : H κ R, Ωf = f κ, taking high values for non-smooth functions and low values for smooth ones. The decision function is the optimal solution of the Tikhonov regularization problem inf f H κ { } n Ωf + C vfx i, Y i, 45 where C > 0 denotes the regularization parameter controlling the tradeoff between the loss function and the smoothness functional. By taking into account the representer theorem see [8], there exists a vector c = c,..., c n T R n such that the minimizer f of 45 can be expressed as a kernel expansion in terms of the training data, namely, it holds f = n c iκ, X i. In this case the smoothness functional becomes Ωf = f κ = f, f κ = n n j= c ic j κx i, X j = c T Kc and for i =,..., n it holds fx i = n j= c jκx i, X j = Kc i. This means that in order to find the decision function it is enough to solve the convex optimization problem { } inf c R n hc + n g i Kc, 46 where h : R n R, hc = ct Kc is convex, differentiable with K -Lipschitz gradient and g i : R n R, g i c = Cvc i, Y i, i =,..., n, are convex functions. The optimization problem 46 has the structure of 30, where f is taken to be identically 0. Notice that in this case the function f + h = h is γ-strongly convex with γ = min, where min is the smallest eigenvalue of the matrix K. Due to the continuity of the functions g i, i =,..., n, the qualification condition required in Theorem 5 is guaranteed. We solved 46 by Algorithm 4 and used for µ > 0 the following formula see [8] prox µg i c = 0,..., P Yi [ C,0]c i µy i,..., 0 T. With respect to the considered data set, we denote by D= {X i, Y i, i =,..., 5899} R 784 {+, } the set of available training data consisting of 974 images in the class and 95 images in the class +. A sample from each class of images is shown in Figure 4. The images have been vectorized and normalized by dividing each of them by the quantity X i. As initial choices in Algorithm 4 we took τ 0 = 0.99 γ K, = K + and σ i,0 = + τ0 γ K τ 0 //nτ 0 K, i =,..., n, and tested different combinations of the

23 σ training error test error Table : Misclassification rate in percentage for different choices of the kernel parameter σ and for both the training and the test data set. kernel parameter σ over a fixed number of 500 iterations. In Table we present the misclassification rate in percentage for the training and for the test data the error for the training data is less than the one for the test data. One can notice that for certain choices of σ the misclassification rate outperforms the one reported in the literature dealing with numerical methods for support vector classification. Let us mention that the numerical results are given for the case C =. We tested also other choices for C, however we did not observe great impact on the results. References [] H.H. Bauschke, P.L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces, CMS Books in Mathematics, Springer, New York, 0 [] J.M. Borwein and J.D. Vanderwerff, Convex Functions: Constructions, Characterizations and Counterexamples, Cambridge University Press, Cambridge, 00 [3] R.I. Boţ, Conjugate Duality in Convex Optimization, Lecture Notes in Economics and Mathematical Systems, Vol. 637, Springer, Berlin Heidelberg, 00 [4] R.I. Boţ, E.R. Csetnek, Regularity conditions via generalized interiority notions in convex optimization: new achievements and their relation to some classical statements, Optimization 6, 35 65, 0 [5] R.I. Boţ, E.R. Csetnek, A. Heinrich, A primal-dual splitting algorithm for fnding zeros of sums of maximally monotone operators, arxiv: , 0 [6] R.I. Boţ, C. Hendrich, Convergence analysis for a primal-dual monotone + skew splitting algorithm with applications to total variation minimization, arxiv:.706, 0 [7] R.I. Boţ, C. Hendrich, A Douglas-Rachford type primal-dual method for solving inclusions with mixtures of composite and parallel-sum type monotone operators, arxiv:.036, 0 [8] R.I. Boţ, C. Hendrich, A variable smoothing algorithm for solving convex optimization problems, arxiv:07.354, 0 [9] L.M. Briceño-Arias, P.L. Combettes, A monotone + skew splitting model for composite monotone inclusions in duality, SIAM Journal on Optimization 4, 30-50, 0 3

24 [0] A. Chambolle, An algorithm for total variation minimization and applications, Journal of Mathematical Imaging and Vision, 0, 89 97, 004 [] A. Chambolle, T. Pock, A first-order primal-dual algorithm for convex problems with applications to imaging, Journal of Mathematical Imaging and Vision 40, 0-45, 0 [] P.L. Combettes, Solving monotone inclusions via compositions of nonexpansive averaged operators, Optimization 535-6, , 004 [3] P.L. Combettes, J.-C. Pesquet, Primal-dual splitting algorithm for solving inclusions with mixtures of composite, Lipschitzian, and parallel-sum type monotone operators, Set-Valued and Variational Analysis 0, , 0 [4] J. Douglas, H.H. Rachford, On the numerical solution of the heat conduction problem in and 3 space variables, Transactions of the American Mathematical Society 8, 4 439, 956 [5] I. Ekeland, R. Temam, Convex Analysis and Variational Problems, North-Holland Publishing Company, Amsterdam, 976 [6] R.T. Rockafellar, On the maximal monotonicity of subdifferential mappings, Pacific Journal of Mathematics 33, 09 6, 970 [7] R.T. Rockafellar, Monotone operators and the proximal point algorithm, SIAM Journal on Control and Optimization 45, , 976 [8] B. Schölkopf, A.J. Smola, Learning with Kernels. Support Vector Machines, Regularization, Optimization, and Beyond, MIT Press, Cambridge, 00 [9] S. Simons, From Hahn-Banach to Monotonicity, Springer, Berlin, 008 [0] P. Tseng, A modified forward-backward splitting method for maximal monotone mappings, SIAM Journal on Control and Optimization 38, , 000 [] B.C. Vũ, A splitting algorithm for dual monotone inclusions involving cocoercive operators, Advances in Computational Mathematics, DOI 0.007/s [] C. Zălinescu, Convex Analysis in General Vector Spaces, World Scientific, Singapore, 00 4