Iteration-complexity of a Jacobi-type non-euclidean ADMM for multi-block linearly constrained nonconvex programs

Transcription

1 Iteraton-complexty of a Jacob-type non-eucldean ADMM for mult-block lnearly constraned nonconvex programs Jefferson G. Melo Renato D.C. Montero May 13, 017 Abstract Ths paper establshes the teraton-complexty of a Jacob-type non-eucldean proxmal alternatng drecton method of multplers ADMM for solvng mult-block lnearly constraned nonconvex programs. The subproblems of ths ADMM varant can be solved n parallel and hence the method has great potental to solve large scale mult-block lnearly constraned nonconvex programs. Moreover, our analyss allows the Lagrange multpler to be updated wth a relaxaton parameter n the nterval 0,. 000 Mathematcs Subject Classfcaton: 47J, 49M7, 90C5, 90C6, 90C30, 90C60, 65K10. Key words: Jacob multblock ADMM, nonconvex program, teraton-complexty, frst-order methods, non-eucldean dstances. 1 Introducton Ths paper consders the followng lnearly constraned optmzaton problem { } mn f x : A x = b, x R n, = 1,..., p 1 where f : R n, ], = 1,..., p, are proper lower semcontnuous functons, A R d n, = 1,..., p, and b R d. Optmzaton problems such as 1 appear n many mportant applcatons such as dstrbuted matrx factorzaton, dstrbuted clusterng, sparse zero varance dscrmnant analyss, tensor decomposton, matrx completon, and asset allocaton see, e.g., [1, 6, 4, 39, 40, 4]. Recently, some varants of the alternatng drecton method of multplers ADMM have been successfully appled to solve some nstances of the prevous problem despte the lack of convexty. Insttuto de Matemátca e Estatístca, Unversdade Federal de Goás, Campus II- Caxa Postal 131, CEP , Goâna-GO, Brazl. E-mal: jefferson@ufg.br. The work of ths author was partally supported by CNPq Grants 40650/013-8, /014-0 and / School of Industral and Systems Engneerng, Georga Insttute of Technology, Atlanta, GA, emal: montero@sye.gatech.edu. The work of ths author was partally supported by NSF Grant CMMI

2 In ths paper we analyze the Jacob-type proxmal ADMM for solvng 1, whch recursvely computes a sequence {x k 1,, xk p, λ k } as x k = argmn x { L β x k 1 1,..., x k 1 1, x, x k 1 +1,..., xk 1 p λ k = λ k 1 θβ A x k b }, λ k 1 + dw x, x k 1, = 1,..., p, where β > 0 s a penalty parameter, θ > 0 s a relaxaton parameter, dw s a Bregman dstance, and L β x 1,..., x p, λ := f x λ, A x b + β A x b 3 s the augmented Lagrangan functon for problem 1. An mportant feature of ths ADMM varant s that the subproblems can be solved n parallel and hence the method has great potental to solve large scale mult-block lnearly constraned nonconvex programs. Under the assumpton that s full row rank and f p : R np R s a dfferentable functon whose gradent s Lpschtz contnuous, we establsh an Oρ teraton-complexty bound for the Jacob-type ADMM to obtan x 1,..., x p, λ, r 1,..., r p 1 satsfyng r f x 1,..., x p A λ, = 1,, p 1, 4 max { A x b, r 1,, r p 1, f p x p A pλ } ρ 5 where f denotes the lmtng subdfferental see for example [3, 34]. We brefly dscuss n ths paragraph the development of ADMM n the convex settng. The standard ADMM.e., where p =, w 0 for = 1,..., and x k s obtaned as above but wth x k 1 1 replaced by x k 1 was ntroduced n [7, 8] and ts complexty analyss was frst carred out n [31]. Snce then several papers have obtaned teraton-complexty results for varous ADMM varants see for example [, 9, 1, 16, 18, 3, 5, 11, 14, 5, 33]. Multblock ADMM varants have also been extensvely studed see for example [15, 19, 6, 7, 8, 17, 4, 37, 3]. In partcular, papers [17, 4, 37, 3] study the convergence and/or complexty of Jacob-type ADMM varants. Recently, there have been a lot of nterest on the study of ADMM varants for nonconvex problems see, e.g., [13, 0, 1,, 35, 36, 38, 41, 10, 30, 9]. Papers [13,, 35, 36, 38, 41] establsh convergence of the generated sequence to a statonary pont of 1 under condtons whch guarantee that a certan potental functon assocated wth the augmented Lagrangan 3 satsfes the Kurdyka-Lojasewcz property. However, these papers do not study the teraton complexty of the proxmal ADMM although ther theoretcal analyss are generally half-way towards accomplshng such goal. Paper [0] analyzes the convergence of varants of the ADMM for solvng nonconvex consensus and sharng problems and establshes the teraton complexty of ADMM for the consensus problem. Paper [1] studes the teraton-complexty of two lnearzed varants of the multblock proxmal ADMM appled to a more general problem than 1 where a couplng term s also present n ts objectve functon. Paper [10] studes the teraton-complexty of a proxmal ADMM for the two block optmzaton problem,.e., p =, and the relaxaton parameter θ s arbtrarly chosen n the nterval 0,, contrary to the prevous related lterature where ths parameter s consdered as one or at most

3 5 + 1/. Paper [30] analyzes the teraton-complexty of a mult-block proxmal ADMM va a general lnearzaton scheme. Fnally, whle the authors were n the process of fnalzng ths paper, they have learned of the recent paper [9] whch studes the asymptotc convergence of a Jacob-type lnearzed ADMM for solvng non-convex problems. The latter paper though does not deal wth the ssue of teraton-complexty and consders the case of θ = 1 only. Our paper s organzed as follows. Subsecton 1.1 contans some notaton and basc results used n the paper. Secton descrbes our assumptons and contans two subsectons. Subsecton.1 ntroduces the concept of dstance generatng functons and ts correspondng Bregman dstances consdered n ths paper, and formally states the non-eucldean Jacob-type ADMM. Secton. s devoted to the convergence rate analyss of the latter method. Our man convergence rate result s n ths subsecton Theorem.11. The appendx contans proofs of some results stated n the paper. 1.1 Notaton and basc results The doman of a functon f : R s, ] s the set dom f := {x R s : fx < + }. Moreover, f s sad to be proper f fx < for some x R s. Lemma 1.1. Let S R n p be a non-zero matrx and let σ + S of SS. Then, for every u R p, there holds denote the smallest postve egenvalue P S u 1 Su. σ + S We next recall some defntons and results of subdfferental calculus [3, 34]. Defnton 1.. Let h : R s, ] be a proper lower sem-contnuous functon. The Fréchet subdfferental of h at x dom h, denoted by ˆ hx, s the set of all elements u R s satsfyng hy hx u, y x lm nf 0. y x y x y x When x / dom h, we set ˆ hx =. The lmtng subdfferental of h at x dom h, denoted by hx, s defned as hx = {u R s : x k x, hx k hx, u k ˆ hx k, wth u k u}. A crtcal or statonary pont of h s a pont x dom h satsfyng 0 hx. The followng result presents some propertes of the lmtng subdfferental. Proposton 1.3. Let h : R s, ] be a proper lower sem-contnuous functon. a If x R s s a local mnmzer of h, then 0 hx; b If g : R s R s a contnuously dfferentable functon, then h + gx = hx + gx. 3

4 Jacob-type non-eucldean proxmal ADMM and ts convergence rate We start by recallng the defnton of crtcal ponts of 1. Defnton.1. An element x 1,..., x p, λ R n 1... R np R d s a crtcal pont of problem 1 f 0 f x A λ, = 1,..., p, A x = b. Under some mld condtons, t can be shown that f x 1,..., x p s a global mnmum of 1, then there exsts λ such that x 1,..., x p, λ s a crtcal pont of 1. The augmented Lagrangan assocated wth problem 1 and wth penalty parameter β > 0 s defned as L β x 1,..., x p, λ := f x λ, A x b + β A x b. 6 We assume that problem 1 satsfes the followng set of condtons: A0 The functons f, = 1,..., p 1, are proper lower semcontnuous; A1 0 and Im {b} ImA 1... Im 1 ; A f p : R np R s dfferentable wth gradent L p Lpschtz contnuous. A3 there exsts β 0 such that v β > where vβ := nf f x + β A x b x 1,...,x p β R..1 The non-eucldean proxmal Jacob ADMM In ths subsecton, we ntroduce a class of dstance generatng functons and ts correspondng Bregman dstances whch s sutable for our study. We also formally descrbe the non-eucldean proxmal Jacob ADMM for solvng problem 1. Defnton.. For gven set Z R s and scalars m M, we let D Z m, M denote the class of real-valued functons w whch are dfferentable on Z and satsfy wz wz wz, z z m z z z, z Z, 7 wz wz M z z z, z Z. 8 A functon w D Z m, M wth m 0 s referred to as a dstance generatng functon and ts assocated Bregman dstance dw : R s Z R s defned as dwz ; z := wz wz wz, z z z, z R s Z. 9 4

5 For every z Z, the functon dw ; z wll be denoted by dw z so that Clearly, dw z z = dwz ; z z, z R s Z. dw z z = dw z z = wz wz z, z Z, 10 We now state the non-eucldean proxmal Jacob ADMM based on the class of dstance generatng functons ntroduced n Defnton.. In ts statement and n some techncal results, we denote the block of varables x 1,..., x 1 smply by x < and the block of varables x +1,..., x p smply by x >. Hence, the whole vector x 1,..., x p can also be denoted as x <, x, x > when there s a need to emphasze the -th block. For convenence, we also extend the above for notaton for = 1 and = p. Hence, x <1, x 1, x >1 = x <p, x p, x >p = x 1,..., x p. Non-Eucldean Proxmal Jacob ADMM NEPJ-ADMM 0 Defne Z := dom f for = 1,..., p, and let β be as n A3. Let an ntal pont x 0 1,..., x0 p, λ 0 Z 1... Z p R d. Choose scalars α > 0, β β, M m > 0, = 1,..., p, and a stepsze parameter θ 0, such that δ := m 4 δ p := m p 4 p + α + γ θp + 1 σ + A p βp 1 α β max A l > 0, = 1,..., p 1 1 l p 1 + γ θp + 1L p + Mp βσ + > 0 11 where σ Ap resp., σ + denotes the smallest egenvalue resp., postve egenvalue of A p, and γ θ s gven by θ γ θ := 1 θ 1. 1 Set k = 1 and go to step 1. 1 For each = 1,..., p, choose w k D Z m, M and compute an optmal soluton x k Rn of { mn L β x k 1 x R n <, x, x k 1 >, λ k 1 + dw k x k 1 } x. 13 Set k k + 1, and go to step 1. [ ] λ k = λ k 1 θβ A x k b, 14 5

6 end Some comments about the NEPJ-ADMM are n order. Frst, t s always possble to choose the constants m,,...,p, suffcently large so as to guarantee that δ,,...,p, are strctly postve. Second, one of the man features of NEPJ-ADMM s that ts subproblems 13 are completely ndependent of one another. As a result, they can all be solved n parallel whch shows the potental of NEPJ-ADMM as a sutable ADMM varant to solve large nstance of 1. Thrd, as n the papers [10, 30], NEPJ-ADMM allows the choce of a relaxaton parameter θ 0,.. Convergence Rate Analyss of the NEPJ-ADMM Ths subsecton s dedcated to the convergence rate analyss of the NEPJ-ADMM. We frst present some techncal lemmas whch are useful to prove our man result Theorem.11. To smplfy the notaton, we denote by x k the vector x k 1,..., xk p generated by the NEPJ-ADMM. Lemma.3. Consder the sequence {x k, λ k } generated by the NEPJ-ADMM. For every k 1, defne ˆλ k := λ k 1 β A x k b 15 and where R k :=,j βa A j x k j + w k, = 1,..., p, 16 x k := x k x k 1, w k := w k x k w k x k 1, = 1,..., p. 17 Then, for every k 1, we have: where λ k := λ k λ k 1. 0 f x k A ˆλ k + R k = 1,..., p 18 [ ] 0 = A x k b + 1 θβ λk 19 Proof. The optmalty condtons see Proposton 1.3 for 13 mply that 0 f x k A λ k 1 β A x k + A j x k 1 j b + w k, = 1,..., p.,j Ths relaton combned wth 15 and 16 mmedately yeld 18. Relaton 19 follows drectly from 14. Next result presents a recursve relaton nvolvng the dsplacements λ k and λ k 1. 6

7 Lemma.4. Consder the sequence {x k, λ k } generated by the NEPJ-ADMM and defne Rp 0 = A pλ 0 f p x 0 p, λ 0 = 0. 0 Then, for every k 1, we have A p λ k = 1 θa p λ k 1 + θu k, 1 where u k := f k p + R k p, f k p := f k p x k p f k p x k 1 p, R k p := R k p R k 1 p k 1, λ k and R k p are as n Lemma.3. Proof. From 15 and 19, we obtan the followng relaton Usng ths relaton and 18 wth = p, we have λ k = 1 θλ k 1 + θˆλ k, k 1. A pλ k = 1 θa pλ k 1 + θ[ f p x k p + R k p], k 1. 3 Hence, n vew of, relaton 1 holds for every k. Now, note that 0 s equvalent to f p x 0 p + R 0 p = A pλ 0. Ths relaton combned wth and 3, both wth k = 1, yeld A p λ 1 = θa pλ 0 + θ [ f p y 1 + Rp 1 ] = θa pλ 0 + θ [ f p x 0 p + R 0 p + u 1] = θa pλ 0 + θa pλ 0 + θu 1 = θu 1. Hence, n vew of λ 0 = 0, relaton 1 also holds for k = 1. Next we consder an auxlary result to be used to compare consecutve terms of the sequence {L β x k, λ k }. See the comments mmedately before the NEPJ-ADMM about the notaton used hereafter. Lemma.5. For every y 0 = y 0 1,..., y0 p, y = y 1,..., y p dom f 1... dom f p, λ R d and =,..., p, we have L β y <, y, y 0 >, λ L β y <, y 0, y 0 >, λ = L β y 0 <, y, y 0 >, λ L β y 0 <, y 0, y 0 >, λ Proof. It s easy to see thatf the gradent of the functon 1 + β A y, A j y j. y < L β y <, y, y 0 > L β y <, y 0, y 0 > 4 s gven by β[a 1 A 1 ] A y 7

8 and ts Hessan equal to zero everywhere n dom f 1... dom f 1. Hence, the functon gven n 4 s affne. The concluson of the lemma now follows by notng that 1 [A 1 A 1 ] A y, y < = A y, A j y j. The next result compares consecutve terms of the sequence {L β x k, λ k }. Lemma.6. For every k 1, we have L β x k, λ k L β x k 1, λ k 1 1 j< p β A x k, A j x k j m xk + 1 θβ λk. Proof. Frst note that 13 together wth the fact that w k D Z m, M mply that L β x k 1 <, x k, x k 1 >, λ k 1 L β x k 1, λ k 1 m x k /, = 1,..., p. Hence, usng Lemma.5 wth y 0 = x k 1, y = x k and λ = λ k 1, we see that Hence L β x k <, x k, x k 1 >, λ k 1 L β x k <, x k 1, x k 1 >, λ k 1 L β x k, λ k 1 L β x k 1, λ k 1 = = L β x k 1 <, x k, x k 1 >, λ k 1 L β x k 1, λ k 1 + β m 1 xk + β A x k, A j x k j. [ L β x k <, x k, x k 1 m xk + β 1 A x k, A j x k j >, λ k 1 L β x k <, x k 1 ], x k 1 >, λ k 1 1 A x k, A j x k j. 5 = On the other hand, due to λ k = λ k λ k 1 and 14, we have L β x k, λ k L β x k, λ k 1 = λ k λ k 1, A x k b To conclude the proof, just add the last relaton and 5. = 1 βθ λk. Lemma.6 s essental to show that a certan sequence { ˆL k } assocated to {L β x k, λ k } s monotoncally decreasng. Ths sequence s defned as 8

9 ˆL k := L β x k, λ k + η k k 0, 6 where η 0 := m p η k := 4M p A pλ 0 f p x 0 p 7 m 4 xk + c 1 A p λ k k 1, 8 θ 1 c 1 := βθ1 θ 1 σ B Before establshng the monotoncty property of the sequence { ˆL k }, we frst present an upper bound on ˆL k ˆL k 1 n terms of some quanttes related to x k 1,..., xk p, and λ k. Lemma.7. For any k 1, there holds where ˆL k ˆL p 1 p + αβ A k 1 Θ k λ := 1 βθ λk + c 1 p 1β Ap Θ k p := α and λ 0 = 0, x 0 p = R 0 p/m p see Lemma.4. Proof. From Lemma.6 and defntons of ˆL k and Θ k λ, we obtan ˆL k ˆL k 1 1 j<<p m p 4 m x k + Θ k λ 4 + Θk p 30 A p λ k A p λ k 1 31 m p x k 4 p + x k 1 p 3 p 1 β A x k A j x k j + β A x k x k p x k p + x k 1 1 j<<p p 1 p 1 β A x k + β A j x k j p 1 m xk + Θ k λ 33 p 34 p 1 αβ + A x k p 1β + x k α p 35 m xk + Θ k λ m p x k 4 p + x k 1 p 36 p + αβ A m x k + Θ k λ + Θk p 37 where the nequaltes are due to Cauchy-Schwarz nequalty and by usng the relaton s 1 s ts 1 + 1/ts, s 1, s R for t = 1 and t = α, respectvely. 9

10 The next result compares Θ k λ wth u k, defned n 31 and, respectvely, and provdes an upper bound for both elements n terms of x k 1,..., xk p. Lemma.8. Consder Θ k λ as n 31 and uk as n. Then, Θ k λ γ θ βσ + u k γ θp + 1 βσ + p 1 β A pa j x k j + x k 1 j + L p + Mp x k p + x k 1 p. where γ θ s as n 1 and x 0 = 0, = 1,..., p 1, and x0 p = R 0 p/m p see Lemma.4. Proof. The proof of ths lemma s gven n Appendx A. The next proposton shows, n partcular, that the sequence { ˆL k } s decreasng and bounded below. Proposton.9. Let x 0 = 0, = 1,..., p 1, and x0 p = R 0 p/m p. Then, the followng statements hold: 38 a for every k 1, ˆL k ˆL k 1 δ x k + x k 1 ; b the sequence { ˆL k } gven n 6 satsfes ˆL k vβ for every k 0; c for every k 1, k δ x j + x j 1 ˆL 0 vβ where vβ and δ are as n A3 and 11, respectvely. Proof. a It follows from 30, Lemma.8, 11 and 3 that [ ˆL k ˆL p 1 m k 1 = [ m p 4 p + α + γ θp + 1 σ + A p 4 βp 1 α δ x k + x k 1, + γ θp + 1L p + M p βσ + ] x k + x k 1 β max A 1 j p 1 ] x k p + x k 1 p provng a. The proof of b s gven Appendx B. The proof of c follows mmedately from a and b. 10

11 Next proposton presents some convergence rate bounds for the dsplacements x k, = 1,..., p, and λ k n terms of some ntal parameters. Our man result wll follow easly from ths proposton, due to the fact that the resdual generated by x k, ˆλ k n order to satsfy the Lagrangan system.1 see Lemma.3 can be controlled by these dsplacements. Proposton.10. Let δ,,...,p, be as n 11 and defne δ λ := θγ θp + 1 σ + mn δ β A p max A l + L p + M 1 p l 1 l p 1 1 l p where L 0 := ˆL 0 vβ see 6 and A3. Then, for every k 1, we have {[ k ] } δ x j + x j 1 + δ λ λ j L 0 40 and there exsts j k such that x j L0 kδ, = 1,..., p, λ j Proof. It follows from Proposton.9c that k x j + x j 1 L 0 mn 1 p δ and that n order to prove 40, t suffces to show that k 39 L0 kδ λ λ j L 0 δ λ. 43 Then, n the remanng part of the proof we wll show that 43 holds. By rewrtng 31, we have [ λ k c1 = βθ A p λ k 1 A p λ k ] + Θ k λ k 1. Hence, due to λ 0 = 0 and Lemma.8, we obtan k λ j βθ k Θ j λ θγ θ θγ θp + 1 σ + σ + k u j + θγ θp + 1 σ + L p + M p θγ θp + 1 σ + β A p max A l 1 l p 1 β A p k k p 1 x j + x j 1 x j p + x j 1 p. max A l + L p + Mp 1 l p 1 L0 mn 1 p δ 11

12 where the fourth nequalty s due to 4. It s now to verfy that the prevous estmate and 39 mply 43, whch n turn mples 40. We now present the man convergence rate result for the NEPJ-ADMM. Its man concluson s that the NEPJ-ADMM generates an element x 1,..., x p, λ whch satsfes the optmalty condtons of Defnton.1 wthn an error of O1/ k. Theorem.11. Let L 0 := L β x 0, λ 0 vβ + η 0 where η 0 and vβ are as n 7 and A3, respectvely. Let ˆλ k and R k, = 1,..., p, be as n 15 and 16, respectvely. Consder δ, = 1,..., p, as n 11 and let δ λ be as n 39. Then, the followng statements hold: a L 0 0; b for every k 1, 0 f x k A ˆλ k + R k = 1,..., p, and there exsts j k such that R j β A A l + M L 0 k mn δ, = 1,..., p, l l=1,l 1 l p A p x j b 1 L0. βθ kδ λ Proof. a holds due to Proposton.9c. Lemma.3 shows that the frst statement of b holds. Now, t follows from 16, 19 and the fact that w k D Z m, M, = 1,..., p, that R k l=1,l A x k b = 1 βθ λk. β A A l x k l + M x k, = 1,..., p, Hence, to end the proof, just combne the above relatons wth 41. A Proof of Lemma.8 Let us frst prove the frst nequalty 38. Assumpton A1 clearly mples that λ k = βθ A x b Im. Hence, t follows from Lemma 1.1 that λ k = P Ap λ k 1 σ + Ap A p λ k. 1

13 Thus, n vew of 1 and 31, we have Θ k λ 1 βθσ + A p λ k + c 1 A p λ k A p λ k 1 1 = βθσ + + c 1 1 θa p λ k 1 + θu k c 1 A p λ k 1. Note that f θ = 1, then 9 mples that c 1 = 0 and the above nequalty proves the frst nequalty of the lemma. We wll now prove the frst nequalty of the lemma for the case n whch θ 1. The prevous nequalty together wth the relaton s 1 + s 1 + t s /t s whch holds for every s 1, s R m and t > 0 yeld Θ k 1 λ βθσ + + c 1 [ 1 = βθσ + + c 1 = [ 1 + tθ 1 A p λ k 1 + { 1 + tθ tθ 1 c 1 βθσ + [ tθ 1 ] c 1 ] A p λ k 1 + } ] θ u k c 1 t A p λ k 1 + A p λ k 1 1 βθσ + + c t 1 βθσ + + c t θ u k θ u k. Usng the above expresson wth t = 1 + 1/ θ 1 and notng that t > 0 n vew of the assumpton that θ 0,, we conclude that [ ] Θ k 1 λ βθσ + θ 1 1 θ 1 c 1 A p λ k βθσ + + c 1 θ 1 θ 1 u k = 1 θ 1 θ βθσ θ 1 1 θ 1 u k where the last equalty s due to 9. Hence, n vew of 1, the frst nequalty of the lemma s proved. We now prove the second nequalty n 38. Due to Rp 0 = M p x 0 p, wp k D R np m p, M p, assumpton A, and relaton, we obtan u k = f k p + Rp k p 1 L p x k p + p β A pa j x k j + x k 1 j + M p x k p + x k 1 p 1 p + 1 L p x k p + β A pa j x k j + x k 1 j + M p x k p + x k 1 p where the nequaltes follow from the trangle nequalty for norms, defnton of Rp k n, and the relaton l s l l s for s R, = 1,..., l. Hence the proof of Lemma.8 follows. B Proof of Lemma.9b Note that due to a, we just need to prove the statement of b for k 1. Hence, assume by contradcton that there exsts an ndex k 0 0 such that ˆL k0+1 < vβ. Snce by a, { ˆL k } s decreasng, we obtan j ˆL k 0 k vβ ˆL k vβ + j k 0 ˆL k0+1 vβ j > k 0, k=1 k=1 13

14 whch mples that lm j k=1 j ˆL k vβ =. On the other hand, t follows from 6, 14, 6 and A3 that and hence that ˆL k = L β x k, λ k + η k L β x k, λ k = f x k + β A x k b + 1 βθ λk, λ k λ k 1 vβ + 1 λ k λ k 1 + λ k λ k 1 vβ + 1 λ k λ k 1 βθ βθ j ˆL k vβ 1 βθ k=1 whch yelds the desred contradcton. References λ j λ 0 1 βθ λ0 j 1, [1] B. P. W. Ames and M. Hong. Alternatng drecton method of multplers for penalzed zero-varance dscrmnant analyss. Comput. Optm. Appl., 643:75 754, 016. [] Y. Cu, X. L, D. Sun, and K. C. Toh. On the convergence propertes of a majorzed ADMM for lnearly constraned convex optmzaton problems wth coupled objectve functons. J. Optm. Theory Appl., 1693: , 016. [3] W. Deng and W. Yn. On the global and lnear convergence of the generalzed alternatng drecton method of multplers. J. Sc. Comput., pages 1 8, 015. [4] We Deng, Mng-Jun La, Zhmn Peng, and Wotao Yn. Parallel mult-block admm wth o1/k convergence. Journal of Scentfc Computng, 71:71 736, 017. [5] E. X. Fang, B. He, H. Lu, and X. Yuan. Generalzed alternatng drecton method of multplers: new theoretcal nsghts and applcatons. Math. Prog. Comp., 7: , 015. [6] P. A. Forero, A. Cano, and G. B. Gannaks. Dstrbuted clusterng usng wreless sensor networks. IEEE J. Selected Topcs Sgnal Process., 54:707 74, 011. [7] D. Gabay and B. Mercer. A dual algorthm for the soluton of nonlnear varatonal problems va fnte element approxmaton. Comput. Math. Appl., :17 40, [8] R. Glownsk and A. Marroco. Sur l approxmaton, par éléments fns d ordre un, et la résoluton, par penalsaton-dualté, d une classe de problèmes de drchlet non lnéares [9] M. L. N. Gonçalves, J. G. Melo, and R. D. C. Montero. Extendng the ergodc convergence rate of the proxmal ADMM. Arxv preprnt: [10] M. L. N. Goncalves, J. G. Melo, and R. D. C. Montero. Convergence rate bounds for a proxmal ADMM wth over-relaxaton stepsze parameter for solvng nonconvex lnearly constraned problems. Arxv Preprnt: [11] M. L. N. Gonçalves, J. G. Melo, and R. D. C. Montero. Improved pontwse teraton-complexty of a regularzed ADMM and of a regularzed non-eucldean HPE framework. SIAM J. Optm., 71: ,

15 [1] Y. Gu, B. Jang, and H. Deren. A sem-proxmal-based strctly contractve Peaceman-Rachford splttng method. Arxv preprnt: [13] K. Guo, D. R. Han, and T. T. Wu. Convergence of alternatng drecton method for mnmzng sum of two nonconvex functons wth lnear constrants. Int. J. Comput. Math. DOI: / , 016. [14] W. W. Hager, M. Yashtn, and H. Zhang. An O1/k convergence rate for the varable stepsze Bregman operator splttng algorthm. SIAM J. Numer. Anal., 543: , 016. [15] D. Han and X. Yuan. A note on the alternatng drecton method of multplers. J. Optm. Theory Appl., 1551:7 38, 01. [16] B. He, F. Ma, and X. Yuan. On the step sze of symmetrc alternatng drectons method of multplers. Preprnt: [17] B. He, H-K. Xu, and X. Yuan. On the proxmal jacoban decomposton of alm for multple-block separable convex mnmzaton problems and ts relatonshp to admm. Journal of Scentfc Computng, 663: , 016. [18] B. He and X. Yuan. On the O1/n convergence rate of the Douglas-Rachford alternatng drecton method. SIAM Journal on Numer. Anal., 50: , 01. [19] M. Hong and Z.-Q. Luo. On the lnear convergence of the alternatng drecton method of multplers. Math. Programmng, 161: , 017. [0] M. Hong, Z.-Q. Luo, and M. Razavyayn. Convergence analyss of alternatng drecton method of multplers for a famly of nonconvex problems. SIAM J. Optm., 61: , 016. [1] B. Jang, T. Ln, S. Ma, and S. Zhang. Structured nonconvex and nonsmooth optmzaton: algorthms and teraton complexty analyss. Arxv Preprnt: [] G. L and T. K. Pong. Global convergence of splttng methods for nonconvex composte optmzaton. SIAM J. Optm., 54: , 015. [3] M. L and X. Yuan. The augmented lagrangan method wth full jacoban decomposton and logarthmcquadratc proxmal regularzaton for multple-block separable convex programmng. Preprnt, 015. [4] A.P. Lavas and N.D. Sdropoulos. Parallel algorthms for constraned tensor factorzaton va the alternatng drecton method of multplers. Arxv Preprnt: [5] T. Ln, S. Ma, and S. Zhang. An extragradent-based alternatng drecton method for convex mnmzaton. Found. Comput. Math., pages 1 5, 015. [6] T. Ln, S. Ma, and S. Zhang. On the global lnear convergence of the admm wth multblock varables. SIAM J. Optm., 53: , 015. [7] T. Ln, S. Ma, and S. Zhang. On the sublnear convergence rate of mult-block admm. J. Oper. Res. Chna, 33:51 74, 015. [8] T. Ln, S. Ma, and S. Zhang. Iteraton complexty analyss of mult-block admm for a famly of convex mnmzaton wthout strong convexty. J. Sc. Comput., 691:5 81, 016. [9] Q. Lu, X. Shen, and Y. Gu. Lnearzed admm for non-convex non-smooth optmzaton wth convergence analyss. Arxv Preprnt: [30] J. G. Melo and R. D. C. Montero. Iteraton-complexty of a lnearzed proxmal multblock admm class for lnearly constraned nonconvex optmzaton problems. Avalable on: [31] R. D. C. Montero and B. F Svater. Iteraton-complexty of block-decomposton algorthms and the alternatng drecton method of multplers. SIAM J. Optm., 31: ,

16 [3] B.S. Mordukhovch. Varatonal analyss and generalzed dfferentaton I: basc theory. Grundlehren der mathematschen Wssenschaften. Sprnger, Berln,, 006. [33] Y. Ouyang, Y. Chen, G. Lan, and E. Paslao Jr. An accelerated lnearzed alternatng drecton method of multplers. SIAM J. Imagng Sc., 81: , 015. [34] R. T. Rockafellar and R. J.-B. Wets. Varatonal analyss. Sprnger, Berln, [35] F. Wang, W. Cao, and Z. Xu. Convergence of mult-block Bregman ADMM for nonconvex composte problems. Arxv preprnt: [36] F. Wang, Z. Xu, and H. K. Xu. Convergence of Bregman alternatng drecton method wth multplers for nonconvex composte problems. Arxv preprnt: [37] H. Wang, A. Banerjee, and Z-Q. Luo. Parallel drecton method of multplers. Arxv Preprnt: [38] W. Wang, Y. Yn and J. Zeng. Global convergence of ADMM n nonconvex nonsmooth optmzaton. Arxv preprnt: [39] Z. Wen, X. Peng, X. Lu, X. Sun, and X. Bas. Asset allocaton under the Basel accord rsk measures. Arxv preprnt: [40] Y. Xu, W. Yn, Z. Wen, and Y. Zhang. An alternatng drecton algorthm for matrx completon wth nonnegatve factors. Fronters Math. Chna, 7: , 01. [41] L. Yang, T. K. Pong, and X. Chen. Alternatng drecton method of multplers for a class of nonconvex and nonsmooth problems wth applcatons to background/foreground extracton. SIAM J. Imagng Sc., 101:74 110, 017. [4] R. Zhang and J. T. Kwok. Asynchronous dstrbuted admm for consensus optmzaton. Proceedngs of the 31st Internatonal Conference on Machne Learnng,