Inexact Alternating-Direction-Based Contraction Methods for Separable Linearly Constrained Convex Optimization
|
|
- Catherine Reed
- 5 years ago
- Views:
Transcription
1 J Optim Theory Appl 04) 63:05 9 DOI 0007/s z Inexact Alternating-Direction-Based Contraction Methods for Separable Linearly Constrained Convex Optimization Guoyong Gu Bingsheng He Junfeng Yang Received: 9 February 03 / Accepted: 7 November 03 / Published online: 3 December 03 Springer Science+Business Media New Yor 03 Abstract Alternating direction method of multipliers has been well studied in the context of linearly constrained convex optimization In the last few years, we have witnessed a number of novel applications arising from image processing, compressive sensing and statistics, etc, where the approach is surprisingly efficient In the early applications, the objective function of the linearly constrained convex optimization problem is separable into two parts Recently, the alternating direction method of multipliers has been extended to the case where the number of the separable parts in the objective function is finite However, in each iteration, the subproblems are required to be solved exactly In this paper, by introducing some reasonable inexactness criteria, we propose two inexact alternating-direction-based contraction methods, which substantially broaden the applicable scope of the approach The convergence and complexity results for both methods are derived in the framewor of variational inequalities Keywords Alternating direction method of multipliers Separable linearly constrained convex optimization Contraction method Complexity Communicated by Panos M Pardalos G Gu B) B He J Yang Department of Mathematics, Nanjing University, Nanjing, 0093, China ggu@njueducn B He hebma@njueducn J Yang jfyang@njueducn
2 06 J Optim Theory Appl 04) 63:05 9 Introduction Because of its significant efficiency and easy implementation, the alternating direction method of multipliers ADMM) has attracted wide attention in various areas [, ] In particular, some novel and attractive applications of the ADMM have been discovered very recently; eg, total-variation regularization problems in image processing [3], l -norm minimization in compressive sensing [4], semidefinite optimization problems [5], the covariance selection problem and semidefinite least squares problem in statistics [6, 7], the sparse and low-ran recovery problem in engineering [8], etc The paper is organized as follows As a preparation, in the next section, we give a brief review of ADMM, which serves as a motivation of our inexact alternatingdirection-based contraction methods Some basic properties of the projection mappings and variational inequalities are recalled as well Then, in Sect 3 we present two contraction methods, which are based on two descent directions generated from an inexact alternating minimization of L A defined in 4) The rationale of the two descent search directions follows in Sect 4, and the convergence and complexity results are proved in Sects 5 and 6, respectively Finally, some conclusions are drawn in Sect 7 Preliminaries Before introducing the ADMM, we briefly review the augmented Lagrangian method ALM), for which we consider the following linearly constrained convex optimization problem: min { θx) Ax = b,x X } ) Here θx) : R n R is a convex function not necessarily smooth), A R m n, b R m, and X R n is a closed and convex set For solving problem ), the classical ALM generates a sequence of iterates via the following scheme: { x + = arg min{θx) λ,ax b + Ax b H x X }, λ + = λ HAx + b), where H R m m is a positive definite scaling matrix penalizing the violation of the linear constraints, and λ R m is the associated Lagrange multiplier; see, eg, [9, 0] for more details An important special scenario of ), which captures concrete applications in many fields [6, 3], is the following case, where the objective function is separable into two parts: min { θ x ) + θ x ) A x + A x = b,x i X i,i =, } ) Here n + n = n, and, for i =,, θ i : R n i R are convex functions not necessarily smooth), A i R m n i, and X i R n i are closed and convex sets For solving ), the ADMM, which dates bac to [4] and is closely related to the Douglas Rachford
3 J Optim Theory Appl 04) 63: operator splitting method [5], is perhaps one of the most popular methods Given x,λ ), the ADMM generates x +,x +,λ + ) via the following scheme: x + = arg min{θ x ) λ,a x + A x b + A x + A x b H x X }, x + = arg min{θ x ) λ,a x + + A x b + A x + + A x b H x X }, λ + = λ HA x + + A x + b) Therefore, ADMM can be viewed as a practical and structure-exploiting variant in a split or relaxed form) of the classical ALM for solving the separable problem ), with the adaptation of minimizing the involved separable variables separately in an alternating order In this paper, we consider a more general separable case of ) in the sense that the objective function is separable into finitely many parts: { N } N min θ i x i ) A i x i = b,x i X i,i =,,N, 3) i= i= where N i= n i = n, and, for i =,,N, θ i : R n i R are convex functions not necessarily smooth), A i R m n i, b R m, and X i R n i are closed and convex sets Without loss of generality, we assume that the solution set of 3) is nonempty Recently, the ADMM was extended to handle 3)byHeetal[6] For convenience, we denote the augmented Lagrangian function of 3) by N N L A x,,x N,λ):= θ i x i ) λ, A i x i b + N A i x i b, 4) i= i= where H R m m is a symmetric positive definite scaling matrix, and λ R m is the Lagrangian multiplier Given x,,x N,λ ), the algorithm in [6] generates the next iterate x +,,xn +,λ+ ) in two steps First, the algorithm produces a trial point x,, x N, λ ) by the following alternating minimization scheme: { x i = arg min{l A x,, x i,x i,xi+,,x N,λ ) x i X i }, i =,,N, i= H λ = λ H N i= A i x i b) 5) Then, the next iterate x +,,xn +,λ+ ) is obtained from x,,x N,λ ) and the trial point x,, x N, λ ) To the best of our nowledge, it is the first time that tractable algorithms for 3) based on the full utilization of their separable structure have been developed As demonstrated in [6], the resulting method falls into the framewors of both the descent-lie methods in the sense that the iterates generated by the extended ADMM scheme 5) can be used to construct descent directions and the contraction-type methods according to the definition in [7]) as the distance between the iterates and the solution set of 3) is monotonically decreasing Therefore, the method is called the alternating-direction-based contraction method
4 08 J Optim Theory Appl 04) 63:05 9 From a practical point of view, however, in many cases, solving each subproblem in 5) accurately is either expensive or even impossible On the other hand, there seems to be little justification of the effort required to calculate the accurate solutions at each iteration In this paper, we develop two inexact methods for solving problem 3), in which the subproblems in 5) are solved inexactly So, the methods presented in this paper are named inexact alternating-direction-based contraction methods In the sequel, we briefly review some basic properties and related definitions that will be used in the forthcoming analysis We consider the generally separable linearly constrained convex optimization problem 3) For i =,,N,welet f i x i ) θ i x i )), where θ i x i )) denotes the subdifferential of θ i at x i Moreover, we let W := X X N R m Then, it is evident that the first-order optimality condition of 3) is equivalent to the following variational inequality for finding x,,x N,λ ) W: { xi x i,f ix i ) AT i λ 0, i =,,N, λ λ, N i= A i x i b 0, x,,x N,λ) W 6) By defining x w := x N λ f x ) A T λ and Fw):= f N x N ) A T N λ Ni= A i x i b the problem 6) can be rewritten in a more compact form as w w,f w ) 0 w W, which we denote by VIW,F) Recall that Fw)is said to be monotone iff u v,fu) Fv) 0 u, v W One can easily verify that Fw) is monotone whenever all f i, i =,,N,are monotone Under the nonempty assumption of the solution set of 3), the solution set, W,ofVIW,F)is nonempty and convex Given a positive definite matrix G of size n + m) n + m), we define the G- norm of u R n+m as u G = u, Gu The projection onto W under the G-norm is defined as P W,G v) := arg min { v w G w W } From the above definition it follows that v PW,G v), G w P W,G v) ) 0 v R n+m, w W 7) Consequently, we have P W,G u) P W,G v) G u v G u, v R n+m
5 J Optim Theory Appl 04) 63: and PW,G v) w G v w G v PW,G v) G v Rn+m, w W 8) An important property of the projection is contained in the following lemma, for which the omitted proof can be found in [8, pp 67] Lemma Let W be a closed convex set in R n+m, and let G be any n + m) n + m) positive definite matrix Then w is a solution of VIW,F)if and only if w [ = P W,G w αg F w )] α>0 In other words, we have w [ = P W,G w αg F w )] w W, w W w w,f w ) 0 9) 3 Two Contraction Methods In this section, we present two algorithms, each of which consists of a search direction and a step length The search directions are based on the inexact alternating minimization of L A, while both algorithms use the same step length 3 Search Directions For given w = x,,x N,λ ), the alternating direction scheme of the th iteration generates a trial point w = x,, x N, λ ) via the following procedure: For i =,,N, x i is computed as { [ x i = P X i x i ) i f i x i A T i λ + A T i H A j x j + j= where ξi A i x i x i ) H and xi x i,ξ i N j=i+ ) ]} A j xj b + ξi, 0) A i x 4 i x i ) H ) Then, we set N λ = λ H A j x j ) b ) j= We claim that the above approximation scheme is appropriate for inexactly solving the minimization subproblems, ie, min { L A x,, x i,x i,x i+,,x N,λ) x i X i } 3)
6 0 J Optim Theory Appl 04) 63:05 9 Assume that x i is an optimal solution of 3) By the definition of L A in 4), the first-order optimality condition of 3) reduces to finding x i X i such that [ x i x i,f ) i x i A T i λ H i A j x j +A i x i + j= N j=i+ A j x j b )] 0 x i X i where f i x i ) is a subgradient of θ ix i ) at x i, ie, f i x i ) θ i x i )) According to Lemma, the above variational inequality is equivalent to { [ x i = P X i x i ) i f i x i A T i λ + A T i H A j x j + A i x i + j= N j=i+ A j x j b )]} If x i is an optimal solution of 3), then by setting ξi = 0in0) the approximation conditions in ) are satisfied In fact, if A i xi x i ) H = 0, then the approximation conditions in ) require that the corresponding subproblem 0) besolved exactly Thus, without loss of generality, we may assume that Ai x i x i ) H 0 Suppose that ˆx i is an approximate solution of 3), ie, { [ ˆx i P X i ˆx i ) i f i ˆx i A T i λ + A T i H A j x j + A i ˆx i + By setting j= { [ x i = P X i ˆx i ) i f i ˆx i A T i λ + A T i H A j x j + A i ˆx i + j= N j=i+ N j=i+ simple manipulation shows that 0) holds with ξi = x i ) ) )) ˆx i f x i f ˆx i A T i HA i x i ˆx i ) A j x j b )]} A j x j b )]} When ˆx i is sufficiently close to the exact solution x i, the relations between x i, ˆx i, and x i ensure that ˆx i x i and Ai x i x i ) H Ai x i x i ) H Therefore, for a suitable approximate solution ˆx i, we can define an x i such that the inexactness conditions 0) and ) are satisfied
7 J Optim Theory Appl 04) 63:05 9 Instead of accepting w as a new iterate, we use it to generate descent search directions In the sequel, we define the matrix M and the vector ξ as A T HA A T M := HA A T HA 0 0 A T N HA A T N HA A T N HA N H Then, our two search directions are, respectively, given by ξ ξ and ξ := ξn 0 4) d w, w,ξ ) = M w w ) ξ 5) and d w, w,ξ ) = F w ) + A T A T N 0 N H A j x j x j ) ) 6) j= Based on w, w, and ξ, we define ϕ w, w,ξ ) = w w,d w, w,ξ ) N + λ λ, A j x j x j ) 7) The function ϕw, w,ξ ) is a ey component in analyzing the proposed methods In the subsequent section, we prove in Theorem 4) that j= { ϕw, w,ξ ) 4 N j= A j x j x j ) H + λ λ H ), ϕw, w,ξ ) = 0 w W 8) Hence, ϕw, w,ξ ) can be viewed as an error measuring function, which measures how much w fails to be in W Furthermore, by utilizing ϕw, w,ξ ) we will prove in Theorems 4 and 43 that, for any given positive definite matrix G R n+m) n+m), both G d w, w,ξ ) and G d w, w,ξ ) are descent directions with respect to the unnown distance function w w G 3 The Step Length In this subsection, we provide a step length for the search directions d w, w,ξ ) and d w, w,ξ ) Later, we will justify the choice of the step length to be defined and show that the step length has a positive lower bound
8 J Optim Theory Appl 04) 63:05 9 Given a positive definite matrix G R n+m) n+m), the new iterate w + is generated as w + = w α G d w, w,ξ ) 9) or w + [ = P W,G w α G d w, w,ξ )], 0) where α = γα, α = ϕw, w,ξ ) G d w, w,ξ ), and γ 0, ) ) G The sequence {w } generated by 9) is not necessarily contained in W, while the sequence produced by 0) lies in W Note that the step length α in both 9) and 0) depends merely on ϕw, w,ξ ), d w, w,ξ ), and γ The proposed methods utilize different search directions but the same step length According to our numerical experiments in [9], the update form 0) usually outperforms 9), provided that the projection onto W can easily be carried out We mention that the proposed inexact ADMMs 9) and 0) are different from those in the literature [9, 0] In fact, the inexact methods proposed in [9, 0] belong to the proximal-type methods [ 4], while the ADMM subproblems in this paper do not include proximal terms of any form 4 Rationale of the Two Directions To derive the convergence of the proposed methods, we use similar arguments as those in the general framewor proposed in [5] By Lemma the equality in 0) is equivalent to [ x i x i,f ) i N i x i A T i λ H A j x j + A j xj )]+ξ b i 0 x i X i j= j=i+ By substituting λ given in ), the above inequality can be rewritten as x i x i,f ) N i x i A T i λ + A T i H A j x j x j ) ) + ξi 0 x i X i ) j=i+ According to the general framewor [5], for the pair w, w ), the following conditions are required to guarantee the convergence: w { = P W w [ d w, w,ξ ) d w, w,ξ )]}, 3) w w,d w, w,ξ ) ϕ w, w,ξ ) w w,d w, w,ξ ), 4) for which we give proofs in Lemmas 4 and 4, respectively
9 J Optim Theory Appl 04) 63: Lemma 4 Let w = x,, x N, λ ) be generated by the inexact alternating direction scheme 0) ) from the given vector w = x,,x N,λ ) Then, we have w W and w w,d w, w,ξ ) d w, w,ξ ) 0 w W, where d w, w,ξ ) and d w, w,ξ ) are defined in 5) and 6), respectively Proof Denote x := x,, x N ) and X := X X N It is clear from 0) that x X From), for all x X,wehave f x ) AT λ A T H N j= A j xj x j )) x x T x x f x ) AT λ + A T H N j=3 A j xj x j )) + x N x N f N x N ) AT λ N 0 ξ ξ ξ N 0 5) By adding x x T A T x x H j= A j xj x j )) A T H j= A j xj x j )) x N x N A T N H N j= A j xj x j )) to both sides of 5), we get x x T f x x x ) AT λ + A T H N j= A j xj x j )) ξ f x ) AT λ + A T H N j= A j x j x j )) ξ + x N x N f N x N ) AT λ N + A T N H N j= A j xj x j )) ξn x x T A T x x H j= A j xj x j )) A T H j= A j xj x j )) x N x N A T N H N j= A j xj x j )) x x T A T HA 0 0 x x = x A T HA A T HA x x x 0 6) x N x N A T N HA A T N HA A T N HA x N N x N
10 4 J Optim Theory Appl 04) 63:05 9 for all x X Since N i= A i x i b = H λ λ ), by embedding the equality N λ λ, A i x i b = λ λ,h λ λ ) i= into 6) we obtain w W and x x T f x x x ) AT λ + A T H N j= A j xj x j )) ξ f x ) AT λ + A T H N j= A j x j x j )) ξ x N + x N f N x λ λ N ) AT λ N + A T N H N j= A j xj x j )) ξn Ni= A i x i b 0 x x T A T HA x x x A T HA A T HA x x x x N 0 0 x N A T λ λ N HA A T N HA A T N HA N 0 xn x N H λ λ for all w W Recalling the definitions of d w, w,ξ ) and M see 6) and 4)), the last inequality can be rewritten in the following compact form: w W, w w,d w, w,ξ ) + ξ w w,m w w ) w W The assertion of this lemma follows directly from the above inequality and the definition of d w, w,ξ ) in 5) According to 9),theassertioninLemma4 is equivalent to 3) Lemma 4 Let w = x,, x N, λ ) be generated by the inexact alternating direction scheme 0) ) from the given vector w = x,,x N,λ ) Then, we have w w,d w, w,ξ ) ϕ w, w,ξ ) w w,d w, w,ξ ) w W 7) where d w, w,ξ ), d w, w,ξ ), and ϕw, w,ξ ) are defined in 5), 6), and 7) Proof According to 6), we have w w,d w, w,ξ ) = w w,f N w ) + A j x j xj ) N,H A j x j x j ) ) 8) j= j=
11 J Optim Theory Appl 04) 63: From the monotonicity of Fw)and the fact that w W it follows that w w,f w ) w w,f w ) 0 Substituting the above inequality into 8), we obtain w w N,d w, w,ξ ) A j x j xj ) N,H A j x j x j ) ) j= Since N j= A j xj = b and H N j= A j x j b) = λ λ, the above inequality becomes w w,d w, w,ξ ) N λ λ, A j x j x j ) w W Inequality 7) follows immediately by further considering the definition of ϕw, w,ξ ) Lemma 43 Let w = x,, x N, λ ) be generated by the inexact alternating direction scheme 0) ) from the given vector w = x,,x N,λ ) Then, we have w w,m w w ) N + λ λ, A j x j x j ) j= j= j= = N A j x j x j ) H + λ λ N ) H + A j x j b j= Proof From the definition of the matrix M in 4)wehave A T HA x M w w ) A T HA A T HA x x = x 0 0, A T N HA A T N HA A T N HA N 0 xn x N H λ λ from which we can easily verify the following equality: w w,m w w ) = N ) A j x j x j ) H + λ λ H j= j= H
12 6 J Optim Theory Appl 04) 63:05 9 A x x ) T H H H 0 A x + A x x ) H H H 0 x ) A x A N xn x N ) x ) H H H 0 A N x λ λ H N x N ) λ λ Simple calculations show that N λ λ, A j x j x j ) j= A x x ) A x = x ) A N xn x N ) λ λ T 9) I A x x ) I A x x ) 30) I A N xn x N ) I I I 0 λ λ The addition of 9) and 30) yields w w,m w w ) N + λ λ, A j x j x j ) j= A x x ) T H H H I A x x ) A x = x ) H H H I A x x ) A N xn x N ) H H H I A N xn x N ) λ λ I I I H λ λ + N A j x j x ) j H + ) λ λ H j= = N A j x j x j ) + H λ λ ) j= + N Aj x j x ) j H + ) λ λ H j= = N A j xj b + N Aj x j x j ) H + ) λ λ H, j= H j= H where the last equality follows from ) The lemma is proved
13 J Optim Theory Appl 04) 63: Now, we prove 8) and the descent properties of d w, w,ξ ) and d w, w,ξ ) Theorem 4 Let w = x,, x N, λ ) be generated by the inexact alternating direction scheme 0) ) from the given vector w = x,,x N,λ ) If the inexactness criteria ) are satisfied, then ϕ w, w,ξ ) 4 N j= Aj x j x j ) H + λ λ H In addition, if ϕw, w,ξ ) = 0, then w W is a solution of VIW,F) ) 3) Proof First, it follows from 5), 7), and Lemma 43 that ϕ w, w,ξ ) = N A j x j x ) j H + ) λ λ H j= + N A j x j b w w,ξ 3) j= From the inexactness criteria )wehave w w,ξ 4 H N Aj x j x j ) Substituting the above inequality into 3), the first part of the theorem follows immediately Consequently, if ϕw, w,ξ ) = 0, it follows that j= A j x j x j ) = 0, j =,,N, and λ = λ H Moreover, it follows from ) that ξj equations into ) we get = 0, j =,,N By substituting the above x i X i, xi x i,f ) i x i A T i λ 0 xi X i,i =,,N, 33) and N A j x j b = H λ λ ) = 0 34) j= Combining 33) and 34), we get w W, w w,f w ) 0 w W, and thus w is a solution of VIW,F)
14 8 J Optim Theory Appl 04) 63:05 9 Theorem 4 Let w = x,, x N, λ ) be generated by the inexact alternating direction scheme 0) ) from the given vector w = x,,x N,λ ) Then, we have w w,d w, w,ξ ) ϕ w, w,ξ ) w W, 35) where d w, w,ξ ) and ϕw, w,ξ ) are defined in 5) and 7), respectively Proof First, it follows from Lemma 4 that w w,d w, w,ξ ) w w,d w, w,ξ ) w W Combining with Lemma 4, wehave w w,d w, w,ξ ) ϕ w, w,ξ ) w w,d w, w,ξ ) w W which implies 35) Recall that the sequence {w } generated by 9) is not necessarily contained in W Therefore, the w in Theorem 4 must be allowed to be any point in R m+n In contrast, the sequence produced by 0) lies in W, and thus it is required in the next theorem that w belongs to W Theorem 43 Let w = x,, x N, λ ) be generated by the inexact alternating directions scheme 0) ) from the given vector w = x,,x N,λ ) If w W, then w w,d w, w,ξ ) ϕ w, w,ξ ) w W, 36) where d w, w,ξ ) and ϕw, w,ξ ) are defined in 6) and 7), respectively Proof Since w W, it follows from Lemma 4 that w w,d w, w,ξ ) w w,d w, w,ξ ) 37) Then, the addition of 7) to both sides of 37) yields 36) 5 Convergence Using the directions d w, w,ξ ) and d w, w,ξ ) offered by 5) and 6), the new iterate w + is determined by the positive definite matrix G and the step length α,see9) and 0) In order to explain how to determine the step length, we define the new step-length-dependent iterate by w + α ) = w α G d w, w,ξ ) 38) and w + α ) = P W,G [ w α G d w, w,ξ )] 39)
15 J Optim Theory Appl 04) 63: In this way, ϑ α ) = w w G w + α ) w 40) G and ϑ α ) = w w G w + α ) w 4) G measure the improvement in the th iteration by using updating forms 38) and 39), respectively Since the optimal solution w W is unnown, it is generally infeasible to maximize the improvement directly The following theorem introduces a tight lower bound on ϑ α ) and ϑ α ), which does not depend on the unnown vector w Theorem 5 For any w W and α 0, we have ϑ α ) qα ) and ϑ α ) qα ), where qα ) = α ϕ w, w,ξ ) α G d w, w,ξ ) G 4) Proof From 38) and 40) wehave ϑ α ) = w w G w w α G d w, w,ξ ) G = w w,α d w, w,ξ ) α G d w, w,ξ ) G α ϕ w, w,ξ ) α G d w, w,ξ ) G = qα ), where the last inequality follows from 35) Hence, the first assertion of this theorem is proved By setting v = w α G d w, w,ξ ) and u = w in 8) we get w + α ) w G w α G d w, w,ξ ) w G w α G d w, w,ξ ) w + α ) G Substituting the above inequality into 4), we obtain ϑ α ) w w G w w α G d w, w,ξ ) G + w w + α ) α G d w, w,ξ ) G = w w + α ) G + w + α ) w,α d w, w,ξ ) 43) Since w + α ) W, it follows from Lemma 4 that w + α ) w,d w, w,ξ ) w + α ) w,d w, w,ξ ) 44) The addition of 7) to both sides of 44) yields w + α ) w,d w, w,ξ ) ϕ w, w,ξ ) + w + α ) w,d w, w,ξ ) 45)
16 0 J Optim Theory Appl 04) 63:05 9 Substituting 45) into the right-hand side of 43), we obtain ϑ α ) w w + α ) G + α ϕ w, w,ξ ) + α w + α ) w ) T d w, w,ξ ) = w w + α ) α G d w, w,ξ ) G α G d w, w,ξ ) G + α ϕ w, w,ξ ) α ϕ w, w,ξ ) α G d w, w,ξ ) G = qα ) Hence, the proof is completed Denote ϑα ) := min { ϑ α ), ϑ α ) } qα ) 46) Note that qα ) is a quadratic function of α, and it reaches its maximum at α = ϕw, w,ξ ) G d w, w,ξ ), 47) G which is just the same as defined in ) Consequently, it follows from Theorem 5 that ϑ α ) q α ) = α ϕ w, w,ξ ) Since inequality 35) is used in the proof of Theorem 5, in practice, multiplication of the optimal step length α by a factor γ>may result in faster convergence By using 4) and 47)wehave q γα ) = γα ϕ w, w,ξ ) γα ) G d w, w,ξ ) G = γ γ)α ϕ w, w,ξ ) 48) In order to guarantee that the right-hand side of 48) is positive, we choose γ [, ) The following theorem shows that the sequence {w } generated by the proposed method is Fejèr monotone with respect to W Theorem 5 For any w W, the sequence {w } generated by each of the proposed methods with update form 9) or 0)) satisfies w + w G w w G γ γ)α ϕ w, w,ξ ) w W 49) Proof It follows from Theorem 5 that ϑγα ) qγα ), which is equivalent to w + w G w w G q γα ) Then, the result of this theorem directly follows from 48)
17 J Optim Theory Appl 04) 63:05 9 Theorem 5 indicates that the sequence {w } converges to the solution set monotonically in the Fejèr sense Thus, according to [6, 7], the proposed method belongs to the class of contraction methods In the following, we show that α > 0 has a positive lower bound Lemma 5 For any given but fixed) positive definite matrix G, there exists a constant c 0 > 0 such that α c 0 for all >0 Proof From the definition of M in 4) it is easy to show that A T H / M w w ) A T = H / A T H / 0 0 A T N H / A T N H / A T N H / H / H / A x x ) H / A x x ) H / A N xn x N ) H / λ λ ) Therefore, there exists a constant K>0such that M w w ) N K Aj x j x j ) H + ) λ λ H j= Moreover, according to the inexactness criteria ), we have ξ N A j x j x j ) j= Since d w, w,ξ ) = Mw w ) ξ,wehave d w, w,ξ ) M w w ) + ξ N K + ) A j x j x j ) H + ) λ λ H j= Using 3) and recalling that any two norms in a finite-dimensional space are equivalent, we have α = ϕw, w,ξ ) d w, w,ξ ) c ϕw, w,ξ ) d G w, w,ξ ) c c 0 := > 0, 50) 8K + ) H where c > 0 is a constant This completes the proof of the lemma
18 J Optim Theory Appl 04) 63:05 9 Theorem 53 Let {w } and { w } be the sequences generated by the proposed alternating-direction-based contraction methods 9) and 0) for problem 3) Then, we have i) The sequence {w } is bounded ii) lim { N j= A j xj x j ) + λ λ }=0 iii) Any accumulation point of { w } is a solution of 3) iv) When A i, i =,,N, are full column ran matrices, the sequence { w } converges to a unique w W Proof The first assertion follows from 49) Moreover, by combining the recursion of 49) and the fact that α c 0 > 0 it is easy to show that lim ϕ w, w,ξ ) = 0 Consequently, it follows from Theorem 4 that lim Ai x i x i ) = 0, i =,,N, and lim λ λ = 0, 5) and the second assertion is proved Using similar arguments as in the proof of Theorem 4, we obtain w W, lim w w,f w ) 0 w W, 5) and thus any accumulation point of { w } is a solution of VIW,F), ie, a solution of 3) If all A i are full column ran matrices, it follows from the first assertion and 5) that { w } is also bounded Let w be an accumulation point of { w } Then, there exists a subsequence { w j } that converges to w It follows from 5) that w j W, lim w w j,f w ) j 0 w W, and consequently, we have w W, w w,f w ) 0 w W, which implies that w W Since {w } is Fejèr monotone and lim w w =0, the sequence { w } cannot have any other accumulation point and thus must converge to w 6 Complexity The analysis in this section is inspired by [6] It is based on a ey inequality see Lemmas6 and 6)that is similarto that in [6] In the current framewor of variational inequalities, the analysis becomes much simpler and more elegant As a prepa-
19 J Optim Theory Appl 04) 63: ration for the proof of the complexity result, we first give an alternative characterization of the optimal solution set W, namely, W = { w W : w w,fw) 0 } w W For a proof of this characterization, we refer to Theorem 35 in the boo [7] According to the above alternative characterization, we have that ŵ W is an ɛ- optimal solution of VIW,F)if it satisfies ŵ W and {ŵ } sup w,fw) ɛ 53) w W In general, our complexity analysis follows the lines of [8], but instead of using w directly, we need to introduce an auxiliary vector, namely, x ŵ = x N, ˆλ where ˆλ = λ H N A j x j x j ) Since λ = λ H N j= A j x j b) see )), we have ˆλ = λ H N j= A j x j b) or, equivalently, H λ λ ) = j= N N A j x j b = A j x j x j ) + H λ ˆλ ) j= As a consequence, we may rewrite the two descent directions separately as d w, w,ξ ) = M w w ) ξ = ˆM w ŵ ) ξ =: ˆd w, ŵ,ξ ), 54) where and j= A T HA A T ˆM := HA A T HA 0 0 A T N HA A T N HA A T N HA N 0 A A A N H d w, w,ξ ) = F w ) + A T A T N 0 N H A j x j x j ) ) = F ŵ ) 55) j=
20 4 J Optim Theory Appl 04) 63:05 9 In fact, we constructed ŵ and ˆM such that ˆd w, ŵ,ξ ) is exactly the direction d w, w,ξ ) and Fŵ ) is exactly the direction d w, w,ξ ) Moreover, the assertioninlemma4 can be rewritten accordingly as w ŵ,f ŵ ) ˆd w, ŵ,ξ ) 0 w W 56) Now we are ready to prove the ey inequality for both algorithms, which is given in the following two lemmas Lemma 6 If the new iterate w + is updated by 9), then we have w ŵ,γα F ŵ ) + w w G w w + ) G 0 w W Proof Due to 56), we have w ŵ,γα F ŵ ) w ŵ,γα ˆd w, ŵ,ξ ) w W 57) In addition, we have, by using 54) and 9), γα ˆd w, ŵ,ξ ) = γα d w, w,ξ ) = G w w +) Substitution of the last equation into 57) yields w ŵ,γα F ŵ ) w ŵ,g w w +) Thus, it suffices to show that w ŵ,g w w +) + w w G w w + ) G 0 w W 58) Applying the equality a b,gc d) + a c G a d G) = c b G d b ) G, we derive that w ŵ,g w w +) + w w G w w+ G) = w ŵ G w + ŵ G) 59) In view of 9), we have w ŵ G w + ŵ G = w ŵ G w ŵ γα G d w, w,ξ ) G = w ŵ,γα d w, w,ξ ) γ α ) G d w, w,ξ ) G
21 J Optim Theory Appl 04) 63: Moreover, we have w ŵ,d w, w,ξ ) = w w,d w, w,ξ ) + w ŵ,d w, w,ξ ) = w w,d w, w,ξ ) N + H A j x j x j ),H λ λ ) j= = w w,d w, w,ξ ) N + λ λ, A j x j x j ) = ϕ w, w,ξ ), 60) where the second equality follows from the definitions of w, ŵ, and d w, w,ξ ) By combining the last two equations and using ) we obtain w ŵ G w + ŵ G j= = γα ϕ w, w,ξ ) γ α ) G d w, w,ξ ) G = γ γ)α ϕ w, w,ξ ) Substituting the last equation into 59) yields w ŵ,g w w +) + w w G w w+ G) = γ γ)α ϕ w, w,ξ ) 0, which is just 58) Thus, the proof is complete Lemma 6 If the new iterate w + is updated by 0), then we have w ŵ,γα F ŵ ) + w w G w w+ ) G 0 w W Proof To begin with, we separate the term w ŵ ) T γα Fŵ ) into two as w ŵ,γα F ŵ ) = w + ŵ,γα F ŵ ) + w w +,γα F ŵ ) 6) In the sequel, we will deal with the above two terms separately Since w + W, we have, by substituting w with w + in 56), w + ŵ,γα F ŵ ) w + ŵ,γα ˆd w, ŵ,ξ ) = w + ŵ,γα d w, w,ξ ) = w ŵ,γα d w, w,ξ ) w w +,γα d w, w,ξ ) 6)
22 6 J Optim Theory Appl 04) 63:05 9 Recall that the first term on the right-hand side of 6) was calculated in 60) As to the second term, we have w w +,γα d w, w,ξ ) Thus, we obtain w + ŵ,γα F ŵ ) = w w +,G γα G d w, w,ξ )) w w + G + γ α ) G d w, w,ξ ) G γα ϕ w, w,ξ ) γ α ) G d w, w,ξ ) G w w + G = γ γ)α ϕ w, w,ξ ) w w + G, 63) where the equality follows from ) Now, we turn to consider the second term w w +,γα Fŵ ) in 6) Since w + is updated by 0), w + is the projection of [w γα G Fŵ )] onto W under the G-norm It follows from 7) that [ w γα G F ŵ )] w +,G w w +) 0 w W As a consequence, we have w w +,γα F ŵ ) w w +,G w w +) By applying the formula a,gb = a G a b G + b G ) to the right-hand side of the last inequality we derive that w w +,γα F ŵ ) w w + G w w G) + w w + G 64) By incorporating inequalities 63) and 64) into Eq 6), the assertion of Lemma 6 follows Having the same ey inequality for both methods, the O/t) rate of convergence in an ergodic sense) can be obtained easily Theorem 6 For any integer t>0, we have a ŵ t W that satisfies ŵt w,fw) γυ t w w 0 G w W, where ŵ t = Υ t t t α ŵ and Υ t = α =0 =0
23 J Optim Theory Appl 04) 63: Proof In Lemmas 6 and 6, we have proved the same ey inequality for both methods, namely, w ŵ,γα F ŵ ) + w w G w w+ ) G 0 w W Since F is monotone, we have w ŵ,γα Fw) + w w G w w+ ) G 0 w W, or, equivalently, ŵ w,γ α Fw) + w w + G w w ) G 0 w W, When taing the sum of the above inequalities over = 0,,t, we obtain t t ) α ŵ α w,fw) + w t+ w 0 G γ w w0 ) G 0 =0 =0 By dropping the term w t+ w 0 G and incorporating Υ t and ŵ t into the above inequality, we have ŵt w,fw) w w0 G γυ t w W Hence, the proof is complete Since it follows from 50) that t Υ t = α t + )c 0, we have, by Theorem 6, =0 ŵt w,fw) γυ t w w 0 G w w0 G γc 0 t + ) w W According to 53), the above inequality implies the O/t)-rate of convergence immediately We emphasize that our convergence rate is in the ergodic sense From a theoretical point of view, this suggests to use a larger parameter γ 0, ) in implementations 7 Conclusions Attracted by the practical efficiency of the alternating direction method of multipliers, an alternating-direction-based contraction method was proposed in [6] The
24 8 J Optim Theory Appl 04) 63:05 9 new method deals with the general separable and linearly constrained convex optimization problem, where the objective function is separable into finitely many parts However, the new method requires the exact solution of ADMM subproblems, which limits its applicability To overcome this limitation, this paper presents two inexact alternating-direction-based contraction methods These methods are practically more viable since the subproblems are solved inexactly The convergence properties and complexity results O/t) rate of convergence) of the proposed methods were derived We emphasize that even for the simplest case, where the objective function is separable into two parts, our methods are different from the common inexact methods in the literature [9, 0], as our subproblems for computing search directions do not include proximal terms of any form In addition, the complexity results, which are proved in the framewor of variational inequalities, are new for this ind of inexact ADMs Acnowledgements Authors wish to than Prof Panos M Pardalos the associate editor) and Prof Cornelis Roos Delft University of Technology) for useful comments and suggestions G Gu was supported by the NSFC grant 004 B He was supported by the NSFC grant J Yang was supported by the NSFC grant 379 References Floudas, CA, Pardalos, PM eds): Encyclopedia of Optimization, nd edn Springer, Berlin 009) Pardalos, PM, Resende, MGC eds): Handboo of Applied Optimization Oxford University Press, Oxford 00) 3 Ng, MK, Weiss, P, Yuan, X: Solving constrained total-variation image restoration and reconstruction problems via alternating direction methods SIAM J Sci Comput 35), ) 4 Yang, J, Zhang, Y: Alternating direction algorithms for l -problems in compressive sensing SIAM J Sci Comput 33), ) 5 Wen, Z, Goldfarb, D, Yin, W: Alternating direction augmented Lagrangian methods for semidefinite programming Math Program Comput 3 4), ) 6 He, B, Xu, M, Yuan, X: Solving large-scale least squares semidefinite programming by alternating direction methods SIAM J Matrix Anal Appl 3), ) 7 Yuan, X: Alternating direction method for covariance selection models J Sci Comput 5), ) 8 Yuan, X, Yang, J: Sparse and low-ran matrix decomposition via alternating direction method Pac J Optim 9), ) 9 Bertseas, DP: Constrained Optimization and Lagrange Multiplier Methods Computer Science and Applied Mathematics Academic Press/Harcourt Brace Jovanovich Publishers, New Yor 98) 0 Nocedal, J, Wright, SJ: Numerical Optimization, nd edn Springer Series in Operations Research and Financial Engineering Springer, New Yor 006) Candès, EJ, Romberg, J, Tao, T: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information IEEE Trans Inf Theory 5), ) Chan, RH, Yang, J, Yuan, X: Alternating direction method for image inpainting in wavelet domains SIAM J Imaging Sci 43), ) 3 Goldstein, T, Osher, S: The split Bregman method for L-regularized problems SIAM J Imaging Sci ), ) 4 Gabay, D, Mercier, B: A dual algorithm for the solution of nonlinear variational problems via finite element approximation Comput Math Appl ), ) 5 Douglas, J Jr, Rachford, HH Jr: On the numerical solution of heat conduction problems in two and three space variables Trans Am Math Soc 8, ) 6 He, B, Tao, M, Xu, M, Yuan, X: An alternating direction-based contraction method for linearly constrained separable convex programming problems Optimization 64), )
25 J Optim Theory Appl 04) 63: Blum, E, Oettli, W: Grundlagen und Verfahren Springer, Berlin 975) Grundlagen und Verfahren, Mit einem Anhang Bibliographie zur Nichtlinearer Programmierung, Öonometrie und Unternehmensforschung, No XX 8 Bertseas, DP, Tsitsilis, JN: Parallel and Distributed Computation: Numerical Methods Prentice- Hall, Upper Saddle River 989) 9 He, B, Liao, L, Qian, M: Alternating projection based prediction-correction methods for structured variational inequalities J Comput Math 46), ) 0 He, B, Liao, L, Han, D, Yang, H: A new inexact alternating directions method for monotone variational inequalities Math Program, Ser A 9), ) Chen, G, Teboulle, M: A proximal-based decomposition method for convex minimization problems Math Program, Ser A 64), ) Rocafellar, RT: Monotone operators and the proximal point algorithm SIAM J Control Optim 45), ) 3 Teboulle, M: Convergence of proximal-lie algorithms SIAM J Optim 74), ) 4 Tseng, P: Alternating projection-proximal methods for convex programming and variational inequalities SIAM J Optim 74), ) 5 He, B, Xu, M: A general framewor of contraction methods for monotone variational inequalities Pac J Optim 4), ) 6 Nemirovsi, A: Prox-method with rate of convergence O/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems SIAM J Optim 5), ) Electronic) 7 Facchinei, F, Pang, JS: Finite-Dimensional Variational Inequalities and Complementarity Problems, vol I Springer Series in Operations Research Springer, New Yor 003) 8 Cai, X, Gu, G, He, B: Onthe O/t) convergence rate of the projection and contraction methods for variational inequalities with Lipschitz continuous monotone operators Comput Optim Appl 03) doi:0007/s
Contraction Methods for Convex Optimization and Monotone Variational Inequalities No.16
XVI - 1 Contraction Methods for Convex Optimization and Monotone Variational Inequalities No.16 A slightly changed ADMM for convex optimization with three separable operators Bingsheng He Department of
More informationContraction Methods for Convex Optimization and Monotone Variational Inequalities No.18
XVIII - 1 Contraction Methods for Convex Optimization and Monotone Variational Inequalities No18 Linearized alternating direction method with Gaussian back substitution for separable convex optimization
More informationOn the Iteration Complexity of Some Projection Methods for Monotone Linear Variational Inequalities
On the Iteration Complexity of Some Projection Methods for Monotone Linear Variational Inequalities Caihua Chen Xiaoling Fu Bingsheng He Xiaoming Yuan January 13, 2015 Abstract. Projection type methods
More informationContraction Methods for Convex Optimization and monotone variational inequalities No.12
XII - 1 Contraction Methods for Convex Optimization and monotone variational inequalities No.12 Linearized alternating direction methods of multipliers for separable convex programming Bingsheng He Department
More informationOn convergence rate of the Douglas-Rachford operator splitting method
On convergence rate of the Douglas-Rachford operator splitting method Bingsheng He and Xiaoming Yuan 2 Abstract. This note provides a simple proof on a O(/k) convergence rate for the Douglas- Rachford
More informationOn the acceleration of augmented Lagrangian method for linearly constrained optimization
On the acceleration of augmented Lagrangian method for linearly constrained optimization Bingsheng He and Xiaoming Yuan October, 2 Abstract. The classical augmented Lagrangian method (ALM plays a fundamental
More informationContraction Methods for Convex Optimization and Monotone Variational Inequalities No.11
XI - 1 Contraction Methods for Convex Optimization and Monotone Variational Inequalities No.11 Alternating direction methods of multipliers for separable convex programming Bingsheng He Department of Mathematics
More informationOn the O(1/t) convergence rate of the projection and contraction methods for variational inequalities with Lipschitz continuous monotone operators
On the O1/t) convergence rate of the projection and contraction methods for variational inequalities with Lipschitz continuous monotone operators Bingsheng He 1 Department of Mathematics, Nanjing University,
More informationThe Direct Extension of ADMM for Multi-block Convex Minimization Problems is Not Necessarily Convergent
The Direct Extension of ADMM for Multi-block Convex Minimization Problems is Not Necessarily Convergent Caihua Chen Bingsheng He Yinyu Ye Xiaoming Yuan 4 December, Abstract. The alternating direction method
More informationProximal-like contraction methods for monotone variational inequalities in a unified framework
Proximal-like contraction methods for monotone variational inequalities in a unified framework Bingsheng He 1 Li-Zhi Liao 2 Xiang Wang Department of Mathematics, Nanjing University, Nanjing, 210093, China
More informationOptimal Linearized Alternating Direction Method of Multipliers for Convex Programming 1
Optimal Linearized Alternating Direction Method of Multipliers for Convex Programming Bingsheng He 2 Feng Ma 3 Xiaoming Yuan 4 October 4, 207 Abstract. The alternating direction method of multipliers ADMM
More informationShiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 9. Alternating Direction Method of Multipliers
Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 9 Alternating Direction Method of Multipliers Shiqian Ma, MAT-258A: Numerical Optimization 2 Separable convex optimization a special case is min f(x)
More informationAn Algorithmic Framework of Generalized Primal-Dual Hybrid Gradient Methods for Saddle Point Problems
An Algorithmic Framework of Generalized Primal-Dual Hybrid Gradient Methods for Saddle Point Problems Bingsheng He Feng Ma 2 Xiaoming Yuan 3 January 30, 206 Abstract. The primal-dual hybrid gradient method
More informationLinearized Alternating Direction Method of Multipliers via Positive-Indefinite Proximal Regularization for Convex Programming.
Linearized Alternating Direction Method of Multipliers via Positive-Indefinite Proximal Regularization for Convex Programming Bingsheng He Feng Ma 2 Xiaoming Yuan 3 July 3, 206 Abstract. The alternating
More informationInexact Alternating Direction Method of Multipliers for Separable Convex Optimization
Inexact Alternating Direction Method of Multipliers for Separable Convex Optimization Hongchao Zhang hozhang@math.lsu.edu Department of Mathematics Center for Computation and Technology Louisiana State
More informationRecent Developments of Alternating Direction Method of Multipliers with Multi-Block Variables
Recent Developments of Alternating Direction Method of Multipliers with Multi-Block Variables Department of Systems Engineering and Engineering Management The Chinese University of Hong Kong 2014 Workshop
More informationSpectral gradient projection method for solving nonlinear monotone equations
Journal of Computational and Applied Mathematics 196 (2006) 478 484 www.elsevier.com/locate/cam Spectral gradient projection method for solving nonlinear monotone equations Li Zhang, Weijun Zhou Department
More informationOn the Convergence and O(1/N) Complexity of a Class of Nonlinear Proximal Point Algorithms for Monotonic Variational Inequalities
STATISTICS,OPTIMIZATION AND INFORMATION COMPUTING Stat., Optim. Inf. Comput., Vol. 2, June 204, pp 05 3. Published online in International Academic Press (www.iapress.org) On the Convergence and O(/N)
More informationA relaxed customized proximal point algorithm for separable convex programming
A relaxed customized proximal point algorithm for separable convex programming Xingju Cai Guoyong Gu Bingsheng He Xiaoming Yuan August 22, 2011 Abstract The alternating direction method (ADM) is classical
More informationThe Direct Extension of ADMM for Multi-block Convex Minimization Problems is Not Necessarily Convergent
The Direct Extension of ADMM for Multi-block Convex Minimization Problems is Not Necessarily Convergent Yinyu Ye K. T. Li Professor of Engineering Department of Management Science and Engineering Stanford
More informationCoordinate Update Algorithm Short Course Operator Splitting
Coordinate Update Algorithm Short Course Operator Splitting Instructor: Wotao Yin (UCLA Math) Summer 2016 1 / 25 Operator splitting pipeline 1. Formulate a problem as 0 A(x) + B(x) with monotone operators
More informationSplitting methods for decomposing separable convex programs
Splitting methods for decomposing separable convex programs Philippe Mahey LIMOS - ISIMA - Université Blaise Pascal PGMO, ENSTA 2013 October 4, 2013 1 / 30 Plan 1 Max Monotone Operators Proximal techniques
More informationSparse Optimization Lecture: Dual Methods, Part I
Sparse Optimization Lecture: Dual Methods, Part I Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know dual (sub)gradient iteration augmented l 1 iteration
More informationResearch Article Solving the Matrix Nearness Problem in the Maximum Norm by Applying a Projection and Contraction Method
Advances in Operations Research Volume 01, Article ID 357954, 15 pages doi:10.1155/01/357954 Research Article Solving the Matrix Nearness Problem in the Maximum Norm by Applying a Projection and Contraction
More informationA class of Smoothing Method for Linear Second-Order Cone Programming
Columbia International Publishing Journal of Advanced Computing (13) 1: 9-4 doi:1776/jac1313 Research Article A class of Smoothing Method for Linear Second-Order Cone Programming Zhuqing Gui *, Zhibin
More information496 B.S. HE, S.L. WANG AND H. YANG where w = x y 0 A ; Q(w) f(x) AT g(y) B T Ax + By b A ; W = X Y R r : (5) Problem (4)-(5) is denoted as MVI
Journal of Computational Mathematics, Vol., No.4, 003, 495504. A MODIFIED VARIABLE-PENALTY ALTERNATING DIRECTIONS METHOD FOR MONOTONE VARIATIONAL INEQUALITIES Λ) Bing-sheng He Sheng-li Wang (Department
More informationSome Properties of the Augmented Lagrangian in Cone Constrained Optimization
MATHEMATICS OF OPERATIONS RESEARCH Vol. 29, No. 3, August 2004, pp. 479 491 issn 0364-765X eissn 1526-5471 04 2903 0479 informs doi 10.1287/moor.1040.0103 2004 INFORMS Some Properties of the Augmented
More informationOn the Weak Convergence of the Extragradient Method for Solving Pseudo-Monotone Variational Inequalities
J Optim Theory Appl 208) 76:399 409 https://doi.org/0.007/s0957-07-24-0 On the Weak Convergence of the Extragradient Method for Solving Pseudo-Monotone Variational Inequalities Phan Tu Vuong Received:
More informationDual and primal-dual methods
ELE 538B: Large-Scale Optimization for Data Science Dual and primal-dual methods Yuxin Chen Princeton University, Spring 2018 Outline Dual proximal gradient method Primal-dual proximal gradient method
More informationPrediction-based adaptive control of a class of discrete-time nonlinear systems with nonlinear growth rate
www.scichina.com info.scichina.com www.springerlin.com Prediction-based adaptive control of a class of discrete-time nonlinear systems with nonlinear growth rate WEI Chen & CHEN ZongJi School of Automation
More informationAccelerated primal-dual methods for linearly constrained convex problems
Accelerated primal-dual methods for linearly constrained convex problems Yangyang Xu SIAM Conference on Optimization May 24, 2017 1 / 23 Accelerated proximal gradient For convex composite problem: minimize
More informationResearch Article Strong Convergence of a Projected Gradient Method
Applied Mathematics Volume 2012, Article ID 410137, 10 pages doi:10.1155/2012/410137 Research Article Strong Convergence of a Projected Gradient Method Shunhou Fan and Yonghong Yao Department of Mathematics,
More informationConvex Optimization Notes
Convex Optimization Notes Jonathan Siegel January 2017 1 Convex Analysis This section is devoted to the study of convex functions f : B R {+ } and convex sets U B, for B a Banach space. The case of B =
More informationA Unified Approach to Proximal Algorithms using Bregman Distance
A Unified Approach to Proximal Algorithms using Bregman Distance Yi Zhou a,, Yingbin Liang a, Lixin Shen b a Department of Electrical Engineering and Computer Science, Syracuse University b Department
More informationCoordinate Update Algorithm Short Course Proximal Operators and Algorithms
Coordinate Update Algorithm Short Course Proximal Operators and Algorithms Instructor: Wotao Yin (UCLA Math) Summer 2016 1 / 36 Why proximal? Newton s method: for C 2 -smooth, unconstrained problems allow
More informationSOR- and Jacobi-type Iterative Methods for Solving l 1 -l 2 Problems by Way of Fenchel Duality 1
SOR- and Jacobi-type Iterative Methods for Solving l 1 -l 2 Problems by Way of Fenchel Duality 1 Masao Fukushima 2 July 17 2010; revised February 4 2011 Abstract We present an SOR-type algorithm and a
More informationA Multilevel Proximal Algorithm for Large Scale Composite Convex Optimization
A Multilevel Proximal Algorithm for Large Scale Composite Convex Optimization Panos Parpas Department of Computing Imperial College London www.doc.ic.ac.uk/ pp500 p.parpas@imperial.ac.uk jointly with D.V.
More informationINERTIAL ACCELERATED ALGORITHMS FOR SOLVING SPLIT FEASIBILITY PROBLEMS. Yazheng Dang. Jie Sun. Honglei Xu
Manuscript submitted to AIMS Journals Volume X, Number 0X, XX 200X doi:10.3934/xx.xx.xx.xx pp. X XX INERTIAL ACCELERATED ALGORITHMS FOR SOLVING SPLIT FEASIBILITY PROBLEMS Yazheng Dang School of Management
More informationA primal-dual fixed point algorithm for multi-block convex minimization *
Journal of Computational Mathematics Vol.xx, No.x, 201x, 1 16. http://www.global-sci.org/jcm doi:?? A primal-dual fixed point algorithm for multi-block convex minimization * Peijun Chen School of Mathematical
More informationProximal ADMM with larger step size for two-block separable convex programming and its application to the correlation matrices calibrating problems
Available online at www.isr-publications.com/jnsa J. Nonlinear Sci. Appl., (7), 538 55 Research Article Journal Homepage: www.tjnsa.com - www.isr-publications.com/jnsa Proximal ADMM with larger step size
More informationSOME STABILITY RESULTS FOR THE SEMI-AFFINE VARIATIONAL INEQUALITY PROBLEM. 1. Introduction
ACTA MATHEMATICA VIETNAMICA 271 Volume 29, Number 3, 2004, pp. 271-280 SOME STABILITY RESULTS FOR THE SEMI-AFFINE VARIATIONAL INEQUALITY PROBLEM NGUYEN NANG TAM Abstract. This paper establishes two theorems
More informationOn the Local Quadratic Convergence of the Primal-Dual Augmented Lagrangian Method
Optimization Methods and Software Vol. 00, No. 00, Month 200x, 1 11 On the Local Quadratic Convergence of the Primal-Dual Augmented Lagrangian Method ROMAN A. POLYAK Department of SEOR and Mathematical
More informationDual Proximal Gradient Method
Dual Proximal Gradient Method http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes Outline 2/19 1 proximal gradient method
More informationA convergence result for an Outer Approximation Scheme
A convergence result for an Outer Approximation Scheme R. S. Burachik Engenharia de Sistemas e Computação, COPPE-UFRJ, CP 68511, Rio de Janeiro, RJ, CEP 21941-972, Brazil regi@cos.ufrj.br J. O. Lopes Departamento
More informationPacific Journal of Optimization (Vol. 2, No. 3, September 2006) ABSTRACT
Pacific Journal of Optimization Vol., No. 3, September 006) PRIMAL ERROR BOUNDS BASED ON THE AUGMENTED LAGRANGIAN AND LAGRANGIAN RELAXATION ALGORITHMS A. F. Izmailov and M. V. Solodov ABSTRACT For a given
More informationResearch Article Modified Halfspace-Relaxation Projection Methods for Solving the Split Feasibility Problem
Advances in Operations Research Volume 01, Article ID 483479, 17 pages doi:10.1155/01/483479 Research Article Modified Halfspace-Relaxation Projection Methods for Solving the Split Feasibility Problem
More informationApplication of the Strictly Contractive Peaceman-Rachford Splitting Method to Multi-block Separable Convex Programming
Application of the Strictly Contractive Peaceman-Rachford Splitting Method to Multi-block Separable Convex Programming Bingsheng He, Han Liu, Juwei Lu, and Xiaoming Yuan Abstract Recently, a strictly contractive
More informationJournal of Convex Analysis (accepted for publication) A HYBRID PROJECTION PROXIMAL POINT ALGORITHM. M. V. Solodov and B. F.
Journal of Convex Analysis (accepted for publication) A HYBRID PROJECTION PROXIMAL POINT ALGORITHM M. V. Solodov and B. F. Svaiter January 27, 1997 (Revised August 24, 1998) ABSTRACT We propose a modification
More informationGeneralized ADMM with Optimal Indefinite Proximal Term for Linearly Constrained Convex Optimization
Generalized ADMM with Optimal Indefinite Proximal Term for Linearly Constrained Convex Optimization Fan Jiang 1 Zhongming Wu Xingju Cai 3 Abstract. We consider the generalized alternating direction method
More informationSparse and Low-Rank Matrix Decomposition Via Alternating Direction Method
Sparse and Low-Rank Matrix Decomposition Via Alternating Direction Method Xiaoming Yuan a, 1 and Junfeng Yang b, 2 a Department of Mathematics, Hong Kong Baptist University, Hong Kong, China b Department
More informationSTRONG CONVERGENCE OF AN ITERATIVE METHOD FOR VARIATIONAL INEQUALITY PROBLEMS AND FIXED POINT PROBLEMS
ARCHIVUM MATHEMATICUM (BRNO) Tomus 45 (2009), 147 158 STRONG CONVERGENCE OF AN ITERATIVE METHOD FOR VARIATIONAL INEQUALITY PROBLEMS AND FIXED POINT PROBLEMS Xiaolong Qin 1, Shin Min Kang 1, Yongfu Su 2,
More informationConvergence theorems for a finite family. of nonspreading and nonexpansive. multivalued mappings and equilibrium. problems with application
Theoretical Mathematics & Applications, vol.3, no.3, 2013, 49-61 ISSN: 1792-9687 (print), 1792-9709 (online) Scienpress Ltd, 2013 Convergence theorems for a finite family of nonspreading and nonexpansive
More informationFrank-Wolfe Method. Ryan Tibshirani Convex Optimization
Frank-Wolfe Method Ryan Tibshirani Convex Optimization 10-725 Last time: ADMM For the problem min x,z f(x) + g(z) subject to Ax + Bz = c we form augmented Lagrangian (scaled form): L ρ (x, z, w) = f(x)
More informationStructural and Multidisciplinary Optimization. P. Duysinx and P. Tossings
Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be
More informationOptimization for Learning and Big Data
Optimization for Learning and Big Data Donald Goldfarb Department of IEOR Columbia University Department of Mathematics Distinguished Lecture Series May 17-19, 2016. Lecture 1. First-Order Methods for
More informationA Proximal Method for Identifying Active Manifolds
A Proximal Method for Identifying Active Manifolds W.L. Hare April 18, 2006 Abstract The minimization of an objective function over a constraint set can often be simplified if the active manifold of the
More informationSPARSE SIGNAL RESTORATION. 1. Introduction
SPARSE SIGNAL RESTORATION IVAN W. SELESNICK 1. Introduction These notes describe an approach for the restoration of degraded signals using sparsity. This approach, which has become quite popular, is useful
More informationA GENERAL FRAMEWORK FOR A CLASS OF FIRST ORDER PRIMAL-DUAL ALGORITHMS FOR TV MINIMIZATION
A GENERAL FRAMEWORK FOR A CLASS OF FIRST ORDER PRIMAL-DUAL ALGORITHMS FOR TV MINIMIZATION ERNIE ESSER XIAOQUN ZHANG TONY CHAN Abstract. We generalize the primal-dual hybrid gradient (PDHG) algorithm proposed
More informationMonotone variational inequalities, generalized equilibrium problems and fixed point methods
Wang Fixed Point Theory and Applications 2014, 2014:236 R E S E A R C H Open Access Monotone variational inequalities, generalized equilibrium problems and fixed point methods Shenghua Wang * * Correspondence:
More informationAn Infeasible Interior Proximal Method for Convex Programming Problems with Linear Constraints 1
An Infeasible Interior Proximal Method for Convex Programming Problems with Linear Constraints 1 Nobuo Yamashita 2, Christian Kanzow 3, Tomoyui Morimoto 2, and Masao Fuushima 2 2 Department of Applied
More informationA General Framework for a Class of Primal-Dual Algorithms for TV Minimization
A General Framework for a Class of Primal-Dual Algorithms for TV Minimization Ernie Esser UCLA 1 Outline A Model Convex Minimization Problem Main Idea Behind the Primal Dual Hybrid Gradient (PDHG) Method
More informationOn nonexpansive and accretive operators in Banach spaces
Available online at www.isr-publications.com/jnsa J. Nonlinear Sci. Appl., 10 (2017), 3437 3446 Research Article Journal Homepage: www.tjnsa.com - www.isr-publications.com/jnsa On nonexpansive and accretive
More informationMath 273a: Optimization Overview of First-Order Optimization Algorithms
Math 273a: Optimization Overview of First-Order Optimization Algorithms Wotao Yin Department of Mathematics, UCLA online discussions on piazza.com 1 / 9 Typical flow of numerical optimization Optimization
More informationRelaxed linearized algorithms for faster X-ray CT image reconstruction
Relaxed linearized algorithms for faster X-ray CT image reconstruction Hung Nien and Jeffrey A. Fessler University of Michigan, Ann Arbor The 13th Fully 3D Meeting June 2, 2015 1/20 Statistical image reconstruction
More informationMath 273a: Optimization Subgradient Methods
Math 273a: Optimization Subgradient Methods Instructor: Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com Nonsmooth convex function Recall: For ˉx R n, f(ˉx) := {g R
More informationYou should be able to...
Lecture Outline Gradient Projection Algorithm Constant Step Length, Varying Step Length, Diminishing Step Length Complexity Issues Gradient Projection With Exploration Projection Solving QPs: active set
More informationOn an iterative algorithm for variational inequalities in. Banach space
MATHEMATICAL COMMUNICATIONS 95 Math. Commun. 16(2011), 95 104. On an iterative algorithm for variational inequalities in Banach spaces Yonghong Yao 1, Muhammad Aslam Noor 2,, Khalida Inayat Noor 3 and
More information1 Introduction and preliminaries
Proximal Methods for a Class of Relaxed Nonlinear Variational Inclusions Abdellatif Moudafi Université des Antilles et de la Guyane, Grimaag B.P. 7209, 97275 Schoelcher, Martinique abdellatif.moudafi@martinique.univ-ag.fr
More informationA NEW ITERATIVE METHOD FOR THE SPLIT COMMON FIXED POINT PROBLEM IN HILBERT SPACES. Fenghui Wang
A NEW ITERATIVE METHOD FOR THE SPLIT COMMON FIXED POINT PROBLEM IN HILBERT SPACES Fenghui Wang Department of Mathematics, Luoyang Normal University, Luoyang 470, P.R. China E-mail: wfenghui@63.com ABSTRACT.
More informationWEAK CONVERGENCE OF RESOLVENTS OF MAXIMAL MONOTONE OPERATORS AND MOSCO CONVERGENCE
Fixed Point Theory, Volume 6, No. 1, 2005, 59-69 http://www.math.ubbcluj.ro/ nodeacj/sfptcj.htm WEAK CONVERGENCE OF RESOLVENTS OF MAXIMAL MONOTONE OPERATORS AND MOSCO CONVERGENCE YASUNORI KIMURA Department
More informationConvergence Theorems of Approximate Proximal Point Algorithm for Zeroes of Maximal Monotone Operators in Hilbert Spaces 1
Int. Journal of Math. Analysis, Vol. 1, 2007, no. 4, 175-186 Convergence Theorems of Approximate Proximal Point Algorithm for Zeroes of Maximal Monotone Operators in Hilbert Spaces 1 Haiyun Zhou Institute
More informationAn inexact subgradient algorithm for Equilibrium Problems
Volume 30, N. 1, pp. 91 107, 2011 Copyright 2011 SBMAC ISSN 0101-8205 www.scielo.br/cam An inexact subgradient algorithm for Equilibrium Problems PAULO SANTOS 1 and SUSANA SCHEIMBERG 2 1 DM, UFPI, Teresina,
More informationACCELERATED FIRST-ORDER PRIMAL-DUAL PROXIMAL METHODS FOR LINEARLY CONSTRAINED COMPOSITE CONVEX PROGRAMMING
ACCELERATED FIRST-ORDER PRIMAL-DUAL PROXIMAL METHODS FOR LINEARLY CONSTRAINED COMPOSITE CONVEX PROGRAMMING YANGYANG XU Abstract. Motivated by big data applications, first-order methods have been extremely
More informationALTERNATING DIRECTION METHOD OF MULTIPLIERS WITH VARIABLE STEP SIZES
ALTERNATING DIRECTION METHOD OF MULTIPLIERS WITH VARIABLE STEP SIZES Abstract. The alternating direction method of multipliers (ADMM) is a flexible method to solve a large class of convex minimization
More informationarxiv: v1 [math.oc] 23 May 2017
A DERANDOMIZED ALGORITHM FOR RP-ADMM WITH SYMMETRIC GAUSS-SEIDEL METHOD JINCHAO XU, KAILAI XU, AND YINYU YE arxiv:1705.08389v1 [math.oc] 23 May 2017 Abstract. For multi-block alternating direction method
More informationOn the order of the operators in the Douglas Rachford algorithm
On the order of the operators in the Douglas Rachford algorithm Heinz H. Bauschke and Walaa M. Moursi June 11, 2015 Abstract The Douglas Rachford algorithm is a popular method for finding zeros of sums
More informationBeyond Heuristics: Applying Alternating Direction Method of Multipliers in Nonconvex Territory
Beyond Heuristics: Applying Alternating Direction Method of Multipliers in Nonconvex Territory Xin Liu(4Ð) State Key Laboratory of Scientific and Engineering Computing Institute of Computational Mathematics
More informationA double projection method for solving variational inequalities without monotonicity
A double projection method for solving variational inequalities without monotonicity Minglu Ye Yiran He Accepted by Computational Optimization and Applications, DOI: 10.1007/s10589-014-9659-7,Apr 05, 2014
More informationStrong Convergence Theorem by a Hybrid Extragradient-like Approximation Method for Variational Inequalities and Fixed Point Problems
Strong Convergence Theorem by a Hybrid Extragradient-like Approximation Method for Variational Inequalities and Fixed Point Problems Lu-Chuan Ceng 1, Nicolas Hadjisavvas 2 and Ngai-Ching Wong 3 Abstract.
More informationDistributed Inexact Newton-type Pursuit for Non-convex Sparse Learning
Distributed Inexact Newton-type Pursuit for Non-convex Sparse Learning Bo Liu Department of Computer Science, Rutgers Univeristy Xiao-Tong Yuan BDAT Lab, Nanjing University of Information Science and Technology
More informationA projection algorithm for strictly monotone linear complementarity problems.
A projection algorithm for strictly monotone linear complementarity problems. Erik Zawadzki Department of Computer Science epz@cs.cmu.edu Geoffrey J. Gordon Machine Learning Department ggordon@cs.cmu.edu
More informationA Bregman alternating direction method of multipliers for sparse probabilistic Boolean network problem
A Bregman alternating direction method of multipliers for sparse probabilistic Boolean network problem Kangkang Deng, Zheng Peng Abstract: The main task of genetic regulatory networks is to construct a
More informationA generalized forward-backward method for solving split equality quasi inclusion problems in Banach spaces
Available online at www.isr-publications.com/jnsa J. Nonlinear Sci. Appl., 10 (2017), 4890 4900 Research Article Journal Homepage: www.tjnsa.com - www.isr-publications.com/jnsa A generalized forward-backward
More informationAbout Split Proximal Algorithms for the Q-Lasso
Thai Journal of Mathematics Volume 5 (207) Number : 7 http://thaijmath.in.cmu.ac.th ISSN 686-0209 About Split Proximal Algorithms for the Q-Lasso Abdellatif Moudafi Aix Marseille Université, CNRS-L.S.I.S
More informationThe Trust Region Subproblem with Non-Intersecting Linear Constraints
The Trust Region Subproblem with Non-Intersecting Linear Constraints Samuel Burer Boshi Yang February 21, 2013 Abstract This paper studies an extended trust region subproblem (etrs in which the trust region
More informationHAIYUN ZHOU, RAVI P. AGARWAL, YEOL JE CHO, AND YONG SOO KIM
Georgian Mathematical Journal Volume 9 (2002), Number 3, 591 600 NONEXPANSIVE MAPPINGS AND ITERATIVE METHODS IN UNIFORMLY CONVEX BANACH SPACES HAIYUN ZHOU, RAVI P. AGARWAL, YEOL JE CHO, AND YONG SOO KIM
More informationMcMaster University. Advanced Optimization Laboratory. Title: A Proximal Method for Identifying Active Manifolds. Authors: Warren L.
McMaster University Advanced Optimization Laboratory Title: A Proximal Method for Identifying Active Manifolds Authors: Warren L. Hare AdvOl-Report No. 2006/07 April 2006, Hamilton, Ontario, Canada A Proximal
More informationCoordinate Update Algorithm Short Course Subgradients and Subgradient Methods
Coordinate Update Algorithm Short Course Subgradients and Subgradient Methods Instructor: Wotao Yin (UCLA Math) Summer 2016 1 / 30 Notation f : H R { } is a closed proper convex function domf := {x R n
More informationOptimization methods
Lecture notes 3 February 8, 016 1 Introduction Optimization methods In these notes we provide an overview of a selection of optimization methods. We focus on methods which rely on first-order information,
More informationIteration-complexity of first-order penalty methods for convex programming
Iteration-complexity of first-order penalty methods for convex programming Guanghui Lan Renato D.C. Monteiro July 24, 2008 Abstract This paper considers a special but broad class of convex programing CP)
More informationAN EIGENVALUE STUDY ON THE SUFFICIENT DESCENT PROPERTY OF A MODIFIED POLAK-RIBIÈRE-POLYAK CONJUGATE GRADIENT METHOD S.
Bull. Iranian Math. Soc. Vol. 40 (2014), No. 1, pp. 235 242 Online ISSN: 1735-8515 AN EIGENVALUE STUDY ON THE SUFFICIENT DESCENT PROPERTY OF A MODIFIED POLAK-RIBIÈRE-POLYAK CONJUGATE GRADIENT METHOD S.
More informationPrimal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization
Primal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization Roger Behling a, Clovis Gonzaga b and Gabriel Haeser c March 21, 2013 a Department
More informationViscosity approximation methods for the implicit midpoint rule of asymptotically nonexpansive mappings in Hilbert spaces
Available online at www.tjnsa.com J. Nonlinear Sci. Appl. 9 016, 4478 4488 Research Article Viscosity approximation methods for the implicit midpoint rule of asymptotically nonexpansive mappings in Hilbert
More informationVariational inequalities for fixed point problems of quasi-nonexpansive operators 1. Rafał Zalas 2
University of Zielona Góra Faculty of Mathematics, Computer Science and Econometrics Summary of the Ph.D. thesis Variational inequalities for fixed point problems of quasi-nonexpansive operators 1 by Rafał
More informationQuasi-Newton Methods
Quasi-Newton Methods Werner C. Rheinboldt These are excerpts of material relating to the boos [OR00 and [Rhe98 and of write-ups prepared for courses held at the University of Pittsburgh. Some further references
More informationPrimal-dual Subgradient Method for Convex Problems with Functional Constraints
Primal-dual Subgradient Method for Convex Problems with Functional Constraints Yurii Nesterov, CORE/INMA (UCL) Workshop on embedded optimization EMBOPT2014 September 9, 2014 (Lucca) Yu. Nesterov Primal-dual
More informationA hybrid splitting method for smoothing Tikhonov regularization problem
Zeng et al Journal of Inequalities and Applications 06) 06:68 DOI 086/s3660-06-098-8 R E S E A R C H Open Access A hybrid splitting method for smoothing Tikhonov regularization problem Yu-Hua Zeng,, Zheng
More informationPriority Programme 1962
Priority Programme 1962 An Example Comparing the Standard and Modified Augmented Lagrangian Methods Christian Kanzow, Daniel Steck Non-smooth and Complementarity-based Distributed Parameter Systems: Simulation
More informationAn inexact parallel splitting augmented Lagrangian method for large system of linear equations
An inexact parallel splitting augmented Lagrangian method for large system of linear equations Zheng Peng [1, ] DongHua Wu [] [1]. Mathematics Department of Nanjing University, Nanjing PR. China, 10093
More information