Inexact Alternating Direction Method of Multipliers for Separable Convex Optimization

Transcription

1 Inexact Alternating Direction Method of Multipliers for Separable Convex Optimization Hongchao Zhang Department of Mathematics Center for Computation and Technology Louisiana State University U.S.-Mexico Workshop on Optimization and Applications January 8th, 2016

2 Problem Constrained Separable Convex Optimization min m f i (x i )+h i (x i ) s. t. i=1 m A i x i = b i=1 where m 2 and f i : R n i R is convex, Lipschitz continuously differentiable. h i is a simple proper closed convex function on R n i, but not necessarily smooth. To put constraint x i X i, let h i be the indicator function for closed convex set X i R n i. Many applications in image processing, statistical learning and compressive sensing, etc.

3 Motivation m = 2 Total variation image reconstruction which is equivalent to where min H(u)+φ(Bu) min H(u)+φ(w) s. t. Bu = w, H(u) = 1 2 Au f 2 with A large, dense and ill-conditioned φ(bu) = α u TV = α N i=1 ( u) i with α > 0 Augmented Lagrangian L ρ (u,w,b) = H(u)+φ(w)+ b,bu w + ρ 2 Bu w 2

4 Motivation m = 2 Augmented Lagrangian Method (Method of Multipliers) (u k+1,w k+1 ) = argmin u,w Lρ (u,w,b k ), b k+1 = b k +ρ(bu k+1 w k+1 ). Alternating direction method of multipliers (ADMM) u k+1 1 = argmin u 2 Au f 2 + ρ 2 Bu wk +ρ 1 b k 2, w k+1 = argmin w φ(w)+ ρ 2 Buk+1 w+ρ 1 b k 2, b k+1 = b k +ρ(bu k+1 w k+1 ).

5 Motivation m = 2 The u subproblem: u k+1 1 = argmin u 2 Au f 2 + ρ 2 Bu wk +ρ 1 b k 2 Bregman operator splitting (BOS) scheme: δ k > A T A u k+1 = argmin u δ k u u k +δ 1 k AT (Au k f) 2 +ρ Bu w k +ρ 1 b k 2 Bregman operator splitting with variable stepsize (BOSVS): Apply nonmonotone line search with initial { A(u ˆδ k u k 1 ) 2 } k = max k, u k u k 1 2, where k > 0 is a lower bound and is adaptively increased.

6 BOSVS Algorithm Parameters: τ, η > 1, β, C 0, ρ, δ min > 0, ξ,σ (0,1), δ 0 = 1. Initialization: w 1, u 1 and b 1. Set Q 1 = 0 and 1 = δ min. For k = 1,2,... Step 1. Set δ k = η jˆδ k where j 0 is the smallest integer such that Q k+1 C k 2 where Q k+1 := ξ k Q k +Ω k with Ω k := σ(δ k u k+1 u k 2 +ρ Bu k+1 w k 2 ) A(u k+1 u k ) 2, and { u k+1 = argmin δ k u u k +δ 1 u k A T (Au k f) 2 +ρ w k Bu + bk ρ 2}, 0 ξ k min{ξ,(1 k 1 ) 2 }. Step 2. If δ k > max{δ k 1, k } when k > 1, k+1 := τ k. { Step 3. w k+1 = argmin φ(w)+ ρ w 2 w Buk+1 +b k /ρ 2 + β 2 w wk 2}. Step 4. b k+1 = b k +ρ(bu k+1 w k+1 ). Step 5. If a stopping criterion is satisfied, stop. End For

7 Convergence of BOSVS Theorem. If the minimizer exists, the sequence x k = (u k,w k,b k ) generated by BOSVS converges to a solution x = (u,w,b ) satisfying the first-order optimality conditions H(u ) B T b = 0, b φ(w ), w = Bu. Moreover, the ergodic mean u K given by u K := 1 K K k=1 uk satifies φ(bu K )+H(u K ) min u {φ(bu)+h(u)} = O( 1 K ).

8 Numerical Experiments (BOSVS) Partially Parallel Imaging (PPI) with L = 8 coils Optimize 1 min u 2 L F p (s l u) f l 2 +α u TV, l=1 where F p is the undersampled Fourier transform, S l is the sensitivity map of the l-th channel, and α = Three data sets: * data 1 and data 2 uses a random Poisson mask (25% Fourier coefficients); * data 3 uses a radial mask (34% Fourier coefficients). Plot the error in the objective function versus the iteration number (ρ = 10 4 ).

9 Numerical Experiments (BOSVS) 2 BOS 1.5 ERG 1 1 BOSVS 0.5 BOS ERG BOSVS log 10 (objective error) log 10 (objective error) (a) log K 10 (b) log 10 K BOS ERG BOSVS log 10 (objective error) (c) log 10 K Figure: Plots of the objective error for data 1, data 2, and data 3. Least squares fit of y = ck p give p as 1.6, 1.2, and 0.9 for data 1, data 2, and data 3 respectively.

10 Separable Convex Optimization m 3 Optimize min m f i (x i )+h i (x i ) s. t. i=1 m A i x i = b i=1 where f i : R n i R is convex, Lipschitz continuously differentiable. h i is a simple proper closed convex function on R n i, but not necessarily smooth. We assume A T i A i is nonsingular and the solution set of the problem is nonempty.

11 Separable Convex Optimization m 3 Direct extention of ADMM: For i = 1,...,m x k+1 i = argminl ρ (x k+1 1,...,x k+1 i 1,x i,x k i+1,...,xk m ;λk ); End λ k+1 = λ k +ρ(ax k+1 b), where L ρ (x 1,...,x m ;λ) = f(x)+λ T (Ax b)+ ρ 2 Ax b 2, f = m i=1 f i +h i, Ax := m i=1 A ix i and ρ > 0 is a parameter. Practical efficiency has been observed in many recent applications. However, not necessarily converge! (Chen, He, Ye, Yuan 2013) Moreover, each subproblem is solved exactly, which could be very expensive, not practical or even impossible when no closed formula exists.

12 Separable Convex Optimization m 3 Literature Han and Yuan (2012): * Assume all f i strongly convex and ρ in a specific range * Each subproblem needs to be solved exactly. He, Tao, Xu and Yuan (2013) * Block Gaussian backward substitution is used to ensure convergence. * Each subproblem needs to be solved exactly. Hong and Luo (2013) λ k+1 = λ k +αρ(ax k+1 b), * Stepsize α needs to be sufficiently small. * f i and h i needs to satisfy certain local error bound conditions. * Linear convergence rate is achieved. Much more recent works: randomization, m 2 strongly convex assumption,...

13 Separable Convex Optimization m 3 Motivation Extend BOSVS to the multi-block case to solve the subproblems inexactly. Analogous to the standard Augmented Lagrangian Method (ALM), i.e. m = 1, solve the subproblems to the adaptive accuracy relative to the current KKT error. (Do not require summable error! Ex. Eckstein-Bertsekas 1992: x k i x i ηk with k ηk < ) Apply accelerated optimal gradient methods to solve the subproblems. Global convergence is guaranteed.

14 Separable Convex Optimization m 3 Notation Let us define H = diag(a T 2 A 2,A T 3 A 3,...,A T ma m ) and A T 2 A M = A T 3 A 2 A T 3 A A T m A 2 A T m A 3 A T m A m Note H is positive definite and M is nonsingular. Let us define Φ k i (u,ū,δ) = f i (ū)+ f i (ū) T (u ū)+ δ 2 u ū 2 +h i (u) + ρ 2 A iu b k i +λ k /ρ 2, where b k i = b j<i A j x k j j>i A j x k j.

15 Linearized ADMM with Gaussian backward substitution Parameters: ρ, θ 1, θ 2, θ 3 R ++ ; 0 < δ min << δ max, α (0,1), 0 < σ < 1 < η, {ǫ k } R + with k=1 ǫk <. Step 0: Initialize λ 1. For i = 1,...,m, set δ min,i = δ min, initialize x 1 i and set x 1 i = x 1 i = x 1 i. Let k = 1. Step 1: For i = 1,...,m Choose δi k = η j δi,0 k, where j 0 is the smallest integer such that f i (x k i )+ f i(x k i ), xk i x k i + (1 σ)δk i 2 x k i x k i 2 +ǫ k f i ( x k i ), where δ min δ k i,0 δmax and xk i = Argmin u R n i Φ k i (u,xk i,δk i ). Let x k+1 i = x k i and r i k = (1/δi k ) x k i x k i 2. If δi k > max{δ k 1 i,δ min,i } when k > 1, δ min,i := ηδ min,i. End Step 2: Let e k = θ 1 v k ṽ k +θ 2 m i=1 A i x k i b +θ 3 m i=1 rk i. If e k is sufficiently small, stop the algorithm. Step 3: Let x k+1 1 = x k 1 and λk+1 = λ k +αρ( m i=1 A i x k i b). Compute ṽ k+1 = ṽ k +αm T H( v k ṽ k ). (Backward Substitution) Let k := k +1 and go to Step 1. Figure: L-ADMM-G Algorithm

16 Global Convergence of L-ADMM-G Theorem. Let w k = ( x k 1, xk 2,..., xk n,λk ), w k = ( x k 1, xk 2,..., xk n,λk ) be the iterates generated by the L-ADM-G algorithm. Then, lim k wk = lim k wk = w, where w = (x 1,x 2,...,x,,λ ) is an optimal primal-dual solution pair. Proof. Denote E k = ρ ṽ k e 2 G + 1 ρ λk e 2 +α m i=1 (δk i x k i,e 2 ). We can show for large k, where τ k = c E k E k+1 +τ k, m x k i x k i 2 + c(ρ ṽ k v k 2 H + 1 ρ λk λ k 2 ) ĉǫ k. i=1

17 Inexact ADMM with Gaussian backward substitution Parameters: ρ, θ 1, θ 2, θ 3 R ++, α (0,1), {ǫ k } R + with k=1 ǫk <. Step 0: Initialize λ 1. For i = 1,...,m, initialize x 1 i and set x 1 i = x 1 i = x 1 i. Let e0 =, Γ 0 i = 0 and k = 1. Step 1: For i = 1,...,m End Use the Accelerated optimal Gradient (AOG) method to solve min u R n i L k i (u) inexactly. Step 2: Let e k = θ 1 v k ṽ k +θ 2 m i=1 A i x k i b +θ 3 m i=1 rk i. If e k is sufficiently small, stop the algorithm. Step 3: Let x k+1 1 = x k 1 and λk+1 = λ k +αρ( m i=1 A i x k i b). Compute ṽ k+1 = ṽ k +αm T H( v k ṽ k ). (Backward Substitution) Let k := k +1 and go to Step 1. Figure: I-ADMM-G Algorithm

18 Accelerated Optimal Gradient Method Notation For any h i and z i R n i, we define its proximal mapping prox hi (z i) = Arg min h 1 u R n i i(u)+ 2 zi u 2, and define L k i (u) := L k i (u) h i(u), where L k i (u) := L ρ ( x k 1,..., x k i 1,u, x k i+1,..., x k m;λ k ). We have if and only if x i,k = Arg min u R n i L k i (u) prox hi (x i,k L k i (x i,k)) x i,k = 0. Define ψ : R R + to be any function having the property that lim ψ(t) = 0 and ψ(t) = 0 if and only if t = 0. t 0

19 Accelerated Optimal Gradient Method Figure: Solve min u R n i L k i (u) inexactly Parameters: 0 σ < 1, ǫ k > 0, {ω l } R + with l=1 ωl <. Let r 0 i = 0, y 0 i = u 0 i = x k i, α1 = 1, l = 1 and flag = true. while (flag = true) end Choose δ l > 0 and α l (0,1) for l 2 such that f i (ȳi l)+ f i(ȳi l),yl i ȳi l (1 σ)δl + y l 2α l i ȳi l 2 +π l f i (yi l ), (II) where ȳ l i = (1 α l )y l 1 i +α l u l 1 i, yi l = (1 α l )y l 1 i +α l u l i, u l i = arg min f i (ȳ l δl u R n i ),u + i 2 u ul 1 i 2 + ρ 2 A iu b k i +λ k /ρ 2 +h i (u), and π l = ǫ k ω l /(2γ l ) with γ 1 = 1/δ 1 and γ l = γ l 1 /(1 α l ) for l 2. Let ri l = r l 1 i +δ l α l γ l u l i u l 1 i 2. If γ l Γ k 1 i and prox hi (y l i L k i (yl i )) yl i ψ(ek 1 ), set x k+1 i else l := l+1. = u l i, Γk i = γ l, x k i = y l i, rk i = r l i /Γk i and flag = false;

20 Accelerated Optimal Gradient Method If the Lipschitz constant ζ i of f i is known, we could simply set δ l = 1 2ζ i (1 σ) l and α l = 2 l+1 (0,1]. Comment: * We have (1 σ)δ l α l = (l +1)ζ i l > ζ i. Hence, the condition (II) in the AG algorithm will be satisfied. * In addition, we have γ l = 1 l δ 1 (1 α j ) j=2 1 = l(l+1) 2δ 1 and ξ l := δ l α l γ l = 1.

21 Accelerated Optimal Gradient Method When the Lipschitz constant of f i is unknown, let τ l = 1/(δ l 0η j ), where δ l 0 [δ min,δ max] and j 0 is the smallest integer such that with δ l = 2 τ l + and α l 1 = (0,1], (τ l ) 2 +4τ l Λ l 1 1+δ l Λl 1 the condition (II) in the AG algorithm is satisfied, where Λ l 1 = l 1 i=1 Comment: * Then, we have (1 σ)δ l α l = 1 σ θ l = (1 σ)δ l 0η j. Hence, the condition (II) in the AG algorithm will be satisfied as j. * In addition, we have 1 δ i. γ l = Λ l l 2 4(ηζ i /(1 σ)+δ max ) and ξ l := δ l α l γ l = 1.

22 Accelerated Optimal Gradient Method Lemma. Suppose the accelerated optimal gradient method with the previous line search rule is applied to solve the subproblem. Then we have ( yi l x i,k 2 1 u 0 i x i,k 2 +ǫ k ) l j=1 ωj ν i ρ γ l, where x i,k = Argmin Lk i (u) and ν i is the minimum eigenvalue of A T i A i. Comment: We have optimal convergence rate y l i x i,k 2 O( 1 l 2). The subproblem stopping condition γ l Γ k 1 i and prox hi (y l i L k i (yl i )) yl i ψ(ek 1 ), will be satisfied in finite number of steps.

23 Global Convergence of I-ADMM-G Theorem. Let w k = ( x k 1, xk 2,..., xk n,λ k ), w k = ( x k 1, xk 2,..., xk n,λ k ) be the iterates generated by the I-ADMM-G algorithm. Suppose Then, lim l γ l = ; ξ l := δ l α l γ l is constant ; lim k wk = lim k wk = w, where w = (x 1,x 2,...,x,,λ ) is an optimal primal-dual solution pair. Proof. Denote E k = ρ ṽ k e 2 G + 1 ρ λk e 2 +α m i=1 for large k, where E k E k+1 +τ k, x k i,e 2 Γ k i. We can show m 1 τ k = c Γ k i=1 i Ii k u l i,k u l 1 i,k 2 + c(ρ ṽ k v k 2 H+ 1 ρ λk λ k 2 ) α l=1 m ǫ k Γ k i=1 i Ii k ω l. l=1

24 Numerical Experiments An image deblurring problem for the Cameraman image Optimize 1 min u 2 Au b 2 +α u TV +β Φ T u 1, which is equivalent to 1 min u 2 Au b 2 +α w 1,2 +β z 1, s. t. Bu = w,φ T u = z where Φ is the wavelet transform, α = and β = size , 9 9 uniform blur and Gaussian noise SNR = 40 Relative accuracy for the subproblem: ψ(e) = min{0.1e,e 1.1 } Plot the relative objective function error versus the CPU time (ρ = ).

25 Numerical Experiments 10 0 Cameraman L ADMM G I ADMM G E ADMM G Objective Error cpu time (s) Plots of the relative objective error of the Cameraman image (size , 9 9 uniform blur and Gaussian noise SNR = 40)

26 Numerical Experiments Three Partially Parallel Imaging (PPI) problems Optimize 1 min u 2 Au b 2 +α u TV +β Φ T u 1, which is equivalent to 1 min u 2 Au b 2 +α w 1,2 +β z 1, s. t. Bu = w,φ T u = z where Φ is the wavelet transform, α = 10 5 and β = Relative accuracy for the subproblem: ψ(e) = min{0.1e,e 1.1 } Plot the relative objective function error versus the CPU time (ρ = 10 3 ).

27 Numerical Experiments 10 0 Data 1 L ADMM G I ADMM G E ADMM G 10 0 Data 2 L ADMM G I ADMM G E ADMM G Objective Error Objective Error (a) cpu time (s) (b) cpu time (s) 10 0 Data 3 L ADMM G I ADMM G E ADMM G Objective Error (c) cpu time (s) Figure: Plots of the relative objective error for data 1, 2 and 3.