A Parallel Block-Coordinate Approach for Primal-Dual Splitting with Arbitrary Random Block Selection

Transcription

1 EUSIPCO /19 A Parallel Block-Coordinate Approach for Primal-Dual Splitting with Arbitrary Random Block Selection Jean-Christophe Pesquet Laboratoire d Informatique Gaspard Monge - CNRS Univ. Paris-Est Marne-la-Vallée, France EUSIPCO 2015 Nice

2 EUSIPCO /19 In collaboration with A. Repetti E. Chouzenoux

3 EUSIPCO /19 Variational formulation OBJECTIVE FUNCTION: Find a solution to the convex optimization problem where minimize x H Φ ( x ) H: signal space (separable real Hilbert space), Φ Γ 0 (H): class of convex lower-semicontinuous functions from H to ], + ] with a nonempty domain.

4 EUSIPCO /19 Variational formulation OBJECTIVE FUNCTION: Find a solution to the convex optimization problem where minimize x H Φ ( x ) H: signal space (separable real Hilbert space), Φ Γ 0 (H): class of convex lower-semicontinuous functions from H to ], + ] with a nonempty domain. In the context of large scale problems, how to find an optimization algorithm able to deliver a reliable numerical solution in a reasonable time, with low memory requirement?

5 EUSIPCO /19 Fundamental Tools in Convex Analysis The inf-convolution of f: H ],+ ] and g: H ],+ ] is f g: H [,+ ]: x inf y H f(y)+g(x y). PARTICULAR CASE: f ι {0} = f, where, for C H, ( x H) ι C (x) = { 0 if x C, + otherwise.

6 EUSIPCO /19 Fundamental Tools in Convex Analysis The inf-convolution of f: H ],+ ] and g: H ],+ ] is f g: H [,+ ]: x inf y H f(y)+g(x y). The conjugate of f: H ],+ ] is f : H [,+ ] such that ( u H) f ( ) (u) = sup x u f(x). x H

7 EUSIPCO /19 Fundamental Tools in Convex Analysis The inf-convolution of f: H ],+ ] and g: H ],+ ] is f g: H [,+ ]: x inf y H f(y)+g(x y). The conjugate of f: H ],+ ] is f : H [,+ ] such that ( u H) f ( ) (u) = sup x u f(x). x H Let f Γ 0 (H). Let U : H H be a strongly positive self-adjoint linear operator. The proximity operator prox U f (x) of f at x H relative to the metric induced by U is the unique vector ŷ H such that f(ŷ)+ 1 2 ŷ x 2 U = inf f(y)+ 1 y x U(y x). y H 2

8 EUSIPCO /19 Parallel proximal primal-dual problem PRIMAL PROBLEM We want to minimize x H We want to h(x)+ (g k l k )(L k x). minimize h ( v 1 G 1,...,v q G q DUAL PROBLEM L k v ) k + gk (v k)+l k (v k). h: H R convex, µ-lipschitz differentiable function with µ ]0,+ [ g k Γ 0(G k ) with G k separable real Hilbert space l k Γ 0(G k ) ν k -strongly convex with ν k ]0,+ [ l k Γ 0(G) ν k -Lipschitz differentiable L k : H G k linear and bounded.

9 EUSIPCO /19 Parallel proximal primal-dual problem PRIMAL PROBLEM We want to minimize x H We want to DIFFICULTIES: h(x)+ (g k l k )(L k x). minimize h ( v 1 G 1,...,v q G q DUAL PROBLEM L k v ) k + gk (v k)+l k (v k). Large-size optimization problem Functions g k often nonsmooth (indicator functions of constraint sets, sparsity measures,...). Linear operators required by standard optimization methods (e.g. ADMM) difficult to invert due to the form of operators L k (e.g. weighted incidence matrices of graphs).

10 EUSIPCO /19 Parallel proximal primal-dual algorithm for n = 0,1,... s n x n W h(x n ) y n = s n W L k v k,n for k = 1,...,q ( u k,n prox U 1 k g (v k,n +U k Lk y n l k (v k,n) )) k v k,n+1 = v k,n +λ n (u k,n v k,n ) p n = s n W L k u k,n x n+1 = x n +λ n (p n x n ).

11 EUSIPCO /19 Parallel proximal primal-dual algorithm for n = 0,1,... s n x n W h(x n ) y n = s n W L k v k,n for k = 1,...,q ( u k,n prox U 1 k g (v k,n +U k Lk y n l k (v k,n) )) k v k,n+1 = v k,n +λ n (u k,n v k,n ) p n = s n W L k u k,n x n+1 = x n +λ n (p n x n ).

12 EUSIPCO /19 Parallel proximal primal-dual algorithm for n = 0,1,... s n x n W h(x n ) y n = s n W L k v k,n for k = 1,...,q ( u k,n prox U 1 k g (v k,n +U k Lk y n l k (v k,n) )) k v k,n+1 = v k,n +λ n (u k,n v k,n ) p n = s n W L k u k,n x n+1 = x n +λ n (p n x n ). where W: H H and ( k {1,...,q}) U k : G k G k strongly positive self-adjoint bounded linear operators such that { min µ 1 W 1,ν 1( 1 U 1/2 k L kw 1/2 2)} > 1/2 with ν = max{ν 1 U 1,...,ν q U q }.

13 EUSIPCO /19 Parallel proximal primal-dual algorithm for n = 0,1,... s n x n W h(x n ) y n = s n W L k v k,n for k = 1,...,q ( u k,n prox U 1 k g (v k,n +U k Lk y n l k (v k,n) )) k v k,n+1 = v k,n +λ n (u k,n v k,n ) p n = s n W L k u k,n x n+1 = x n +λ n (p n x n ). where ( n N) λ n ]0,1] such that inf n N λ n > 0.

14 EUSIPCO /19 Parallel proximal primal-dual algorithm for n = 0,1,... s n x n W h(x n ) y n = s n W L k v k,n for k = 1,...,q ( u k,n prox U 1 k g (v k,n +U k Lk y n l k (v k,n) )) k v k,n+1 = v k,n +λ n (u k,n v k,n ) p n = s n W L k u k,n x n+1 = x n +λ n (p n x n ). Assume that there exists x H such that 0 h(x)+ L k( g ( k l k) L ) kx. We have: x n x where x is a solution to the primal problem ( k {1,...,q}) v k,n v k where ( v k) 1 k q is a solution to the dual problem.

15 EUSIPCO /19 Proximal primal-dual algorithm ADVANTAGES: No linear operator inversion. Use of proximable or/and differentiable functions. Less restrictive convergence conditions than other primal-dual algorithms. DISADVANTAGES: At each iteration, all the dual variables are updated in parallel, it is necessary to update the full primal variable.

16 EUSIPCO /19 Proximal primal-dual algorithm BIBLIOGRAPHICAL REMARKS: Pioneering work in the 1950 s: Arrow-Hurwicz method. Methods based on Forward-Backward iteration type I: [Vu ][Condat ] (extensions of [Esser et al ][Chambolle and Pock ]) type II : [Combettes et al ] (extensions of [Loris and Verhoeven ][Chen et al ]) Methods based on Forward-Backward-Forward iteration [Combettes and Pesquet ] [Boţ and Hendrich ] Projection based methods... [Alotaibi et al ]

17 EUSIPCO /19 Improvement via block alternation Idea: split variable. x 1 H 1 x 2 H 2 g x H H = p j=1 H j x p H p H 1,...,H p are real separable Hilbert spaces

18 EUSIPCO /19 Improvement via block alternation Assumption: h is an additively block separable function. x 1 x 2 h x = h = p h j (x j ) j=1 x p ( j {1,...,p}) h j convex and µ j -Lipschitz differentiable with µ j ]0,+ [.

19 EUSIPCO /19 Block-coordinate strategy At each iteration n N, update only a subset of components ( Gauss-Seidel methods). ADVANTAGES: Reduced computational cost at each iteration. Reduced memory requirement. More flexibility.

20 EUSIPCO /19 Primal-dual problem PRIMAL PROBLEM minimize x 1 H 1,...,x p H p p )( h j (x j )+ (g p ) k l k L k,j x j j=1 j=1 ( j {1,...,p})( k {1,...,q}) H j and G k real separable Hilbert spaces h j: H j R convex, µ j-lipschitz differentiable, with µ j ]0,+ [ g k Γ 0(G k ) l k Γ 0(G k ) ν k -strongly convex, with ν k ]0,+ [ L k,j : H j G k is linear and bounded L k = { j {1,...,p} L k,j 0 }, and L j = { k {1,...,q} L k,j 0 }.

21 EUSIPCO /19 Primal-dual problem PRIMAL PROBLEM minimize x 1 H 1,...,x p H p p )( h j (x j )+ (g p ) k l k L k,j x j j=1 j=1 minimize v 1 G 1,...,v q G q p j=1 h j ( ) L k,j v k + DUAL PROBLEM (gk (v k)+l k (v k)) Assume that there exists (x 1,...,x p ) H 1... H p such that ( j {1,...,p}) 0 h j (x j )+ L k,j ( g k l k ) ( ) L k,j x j. OBJECTIVE: Let F and F be the sets of solutions to the primal and dual problems. Find an F F -valued random variable ( x, v).

22 EUSIPCO /19 Random block-coordinate proximal primal-dual algorithm For n = 0,1,... for j = 1,...,p η j,n = max { } ε p+k,n k L ( ( j ) ) s j,n = η j,n x j,n W j hj (x j,n )+a j,n ( y j,n = η j,n sj,n W j L k,j v ) k,n k L j for k = 1,...,q u k,n+1 = ε p+k,n (prox U 1 ( ) k (v ) ) g k,n +U k L k,j y j,n U k l k (v k,n )+c k,n +b k,n k j L k v k,n+1 = v k,n +λ nε p+k,n (u k,n v k,n ) for j = 1,...,p ( ) p j,n+1 = ε j,n s j,n W j L k,j v k,n k L j x j,n+1 = x j,n +λ nε j,n (p j,n x j,n ). where (ε n) n N binary variables signaling the blocks to be activated

23 EUSIPCO /19 Random block-coordinate proximal primal-dual algorithm For n = 0,1,... for j = 1,...,p η j,n = max { ε p+k,n k L } ( j ( ) ) s j,n = η j,n x j,n W j hj(xj,n)+a j,n ( y j,n = η j,n sj,n W j L k,j v ) k,n k L j for k = 1,...,q u k,n+1 = ε p+k,n (prox U 1 ( ) k (v ) ) g k,n +U k L k,j y j,n U k l k (v k,n )+c k,n +b k,n k j L k v k,n+1 = v k,n +λ nε p+k,n (u k,n v k,n ) for j = 1,...,p ( ) p j,n+1 = ε j,n s j,n W j L k,j v k,n k L j x j,n+1 = x j,n +λ nε j,n(p j,n x j,n). where (ε n) n N identically distributedd-valued random variables with D = {0,1} p+q {0} binary variables signaling the blocks to be activated x 0, (a n) n N, and (c n) n N H-valued random variables,v 0, (b n) n N, and (d n) n N G-valued random variables with G = G 1 G q (a n) n N, (b n) n N and (c n) n N: error terms

24 EUSIPCO /19 Random block-coordinate proximal primal-dual algorithm For n = 0,1,... for j = 1,...,p ηj,n = max { } εp+k,n k L ( j ) sj,n = ηj,n xj,n Wj( ) hj(xj,n)+aj,n yj,n = ηj,n( sj,n Wj L ) k,j vk,n k L j for k = 1,...,q ( uk,n+1 = εp+k,n (prox U 1 k g vk,n +Uk Lk,jyj,n Uk k j L k vk,n+1 = vk,n + λnεp+k,n(uk,n vk,n) for j = 1,...,p ( ) pj,n+1 = εj,n sj,n Wj L k,j vk,n k L j xj,n+1 = xj,n +λnεj,n(pj,n xj,n). where ( ) ) l k (vk,n)+ck,n +bk,n) (εn)n N binary variables signaling the blocks to be activated (an)n N, (bn)n N and (cn)n N: error terms ( j {1,...,p}) Wj: Hj Hj and ( k {1,...,q}) Uk: Gk Gk strongly positive self-adjoint preconditioning linear operators such that { min µ 1 W 1,ν 1( 1 with µ = max{µ1 W1,...,µp Wp } and ν = max{ν1 U1,...,νq Uq }. U 1/2 k LkW1/2 2)} > 1/2

25 EUSIPCO /19 Random block-coordinate proximal primal-dual algorithm For n = 0,1,... for j = 1,...,p η j,n = max { } ε p+k,n k L ( ( j ) ) s j,n = η j,n x j,n W j hj (x j,n )+a j,n ( y j,n = η j,n sj,n W j L k,j v ) k,n k L j for k = 1,...,q u k,n+1 = ε p+k,n (prox U 1 ( ) k (v ) ) g k,n +U k L k,j y j,n U k l k (v k,n )+c k,n +b k,n k j L k v k,n+1 = v k,n +λ nε p+k,n (u k,n v k,n ) for j = 1,...,p ( ) p j,n+1 = ε j,n s j,n W j L k,j v k,n k L j x j,n+1 = x j,n +λ nε j,n (p j,n x j,n ). where (ε n) n N binary variables signaling the blocks to be activated (a n) n N, (b n) n N and (c n) n N: error terms ( j {1,...,p}) W j: H j H j and ( k {1,...,q}) U k: G k G k strongly positive self-adjoint preconditioning linear operators ( n N) λ n ]0,1] such that inf n Nλ n > 0.

26 EUSIPCO /19 Random block-coordinate proximal primal-dual algorithm Let (Ω, F, P) be the underlying probability space. Set ( n N) X n = (x n,v n ) 0 n n. Assume that E( an 2 X n) < +, n N E( bn 2 X n) < +, and n N E( cn 2 X n) < + a.s. The variables(ε n) n N are identically distributed such that ( j {1,...,p}) P[ε j,0 = 1] > 0. For every n N, ε n and X n are independent. For every k {1,...,q} and n N, { } ε p+k,n = max εj,n j Lk. 1 j p (x n) n N converges weakly a.s. to an F-valued random variable. (v n) n N converges weakly a.s. to an F -valued random variable. Proof: Based on properties of quasi-fejér stochastic sequences [Combettes and Pesquet 2014].

27 EUSIPCO /19 Illustration of the random sampling strategy Variable selection ( n N) x 1,n activated when ε 1,n = 1 How choosing ( n N) the variable ε n = (ε 1,n,...,ε 6,n )? x 2,n activated when ε 2,n = 1 x 3,n activated when ε 3,n = 1 x 4,n activated when ε 4,n = 1 x 5,n activated when ε 5,n = 1 x 6,n activated when ε 6,n = 1

28 EUSIPCO /19 Illustration of the random sampling strategy Variable selection ( n N) x 1,n activated when ε 1,n = 1 x 2,n activated when ε 2,n = 1 How choosing ( n N) the variable ε n = (ε 1,n,...,ε 6,n )? P[ε n = (1,1,0,0,0,0)] = 0.1 x 3,n activated when ε 3,n = 1 x 4,n activated when ε 4,n = 1 x 5,n activated when ε 5,n = 1 x 6,n activated when ε 6,n = 1

29 EUSIPCO /19 Illustration of the random sampling strategy Variable selection ( n N) x 1,n activated when ε 1,n = 1 x 2,n activated when ε 2,n = 1 x 3,n activated when ε 3,n = 1 How choosing ( n N) the variable ε n = (ε 1,n,...,ε 6,n )? P[ε n = (1,1,0,0,0,0)] = 0.1 P[ε n = (1,0,1,0,0,0)] = 0.2 x 4,n activated when ε 4,n = 1 x 5,n activated when ε 5,n = 1 x 6,n activated when ε 6,n = 1

30 EUSIPCO /19 Illustration of the random sampling strategy Variable selection ( n N) x 1,n activated when ε 1,n = 1 x 2,n activated when ε 2,n = 1 x 3,n activated when ε 3,n = 1 x 4,n activated when ε 4,n = 1 How choosing ( n N) the variable ε n = (ε 1,n,...,ε 6,n )? P[ε n = (1,1,0,0,0,0)] = 0.1 P[ε n = (1,0,1,0,0,0)] = 0.2 P[ε n = (1,0,0,1,1,0)] = 0.2 x 5,n activated when ε 5,n = 1 x 6,n activated when ε 6,n = 1

31 EUSIPCO /19 Illustration of the random sampling strategy Variable selection ( n N) x 1,n activated when ε 1,n = 1 x 2,n activated when ε 2,n = 1 x 3,n activated when ε 3,n = 1 x 4,n activated when ε 4,n = 1 x 5,n activated when ε 5,n = 1 How choosing ( n N) the variable ε n = (ε 1,n,...,ε 6,n )? P[ε n = (1,1,0,0,0,0)] = 0.1 P[ε n = (1,0,1,0,0,0)] = 0.2 P[ε n = (1,0,0,1,1,0)] = 0.2 P[ε n = (0,1,1,1,1,1)] = 0.5 x 6,n activated when ε 6,n = 1

32 EUSIPCO /19 Application: 3D mesh denoising Original mesh x Observed mesh z Undirected nonreflexive graph OBJECTIVE: Estimate x = (x i ) 1 i M from noisy observations z = (z i ) 1 i M where, for every i {1,...,M}, x i R 3 is the vector of 3D coordinates of the i-th vertex of a mesh H = (R 3 ) M

33 EUSIPCO /19 Application: 3D mesh denoising OBJECTIVE: Estimate x = (x i ) 1 i M from noisy observations z = (z i ) 1 i M where, for every i {1,...,M}, x i R 3 is the vector of 3D coordinates of the i-th vertex of a mesh COST FUNCTION: Φ(x) = M ψ j (x j z j )+ι Cj (x j )+η j (x j x i ) i Nj 1,2 j=1 where ( j {1,...,M}), ψ j : R 3 R: l 2 l 1 Huber function robust data fidelity measure convex, Lipschitz differentiable function C j : nonempty convex subset of R 3 N j : neighborhood of j-th vertex (η j ) 1 j M : nonnegative regularization constants.

34 EUSIPCO /19 Application: 3D mesh denoising OBJECTIVE: Estimate x = (x i) 1 i M from noisy observations z = (z i) 1 i M where, for every i {1,...,M}, x i R 3 is the vector of 3D coordinates of thei-th vertex of a mesh COST FUNCTION: M Φ(x) = ψ j(x j z j)+ι Cj (x j)+η j (x j x i) i Nj 1,2 j=1 IMPLEMENTATION DETAILS: a block a vertex p = M, q = 2M ( j {1,...,M}) h j = ψ j( z j) ( k {1,...,M})( x (R 3 ) M ) g k (L k x) = (x k x i) i Nk 1,2 g M+k (L M+k x) = ι Ck (x k ) l k = ι {0}

35 EUSIPCO /19 Simulation results positions of the original mesh are corrupted through an additive i.i.d. zero-mean Gaussian mixture noise model. a limited number r of variables can be handled at each iteration, where p ε j,n = r p. j=1 mesh decomposed into p/r non-overlapping sets. Original mesh, M = Noisy mesh, MSE =

36 EUSIPCO /19 Simulation results Proposed reconstruction MSE = Laplacian smoothing MSE =

37 EUSIPCO /19 Complexity Time (s.) p/r Memory (Mb) dashed line: required memory continuous line: reconstruction time

38 EUSIPCO /19 Conclusion No linear operator inversion. Flexibility in the random activation of primal/dual components. Existing parallel proximal primal-dual algorithms recovered when p = 1 and ε n (1,...,1). Possibility to address other graph processing problems than denoising. [Couprie et al.,2013] Available extensions: asynchronous distributed algorithms (stochastic, primal-dual, proximal, defined on a hypergraph).

39 EUSIPCO /19 Some references P. L. Combettes and J.-C. Pesquet Proximal splitting methods in signal processing in Fixed-Point Algorithms for Inverse Problems in Science and Engineering, H. H. Bauschke, R. Burachik, P. L. Combettes, V. Elser, D. R. Luke, and H. Wolkowicz editors. Springer-Verlag, New York, pp , C. Couprie, L. Grady, L. Najman, J.-C. Pesquet, and H. Talbot Dual constrained TV-based regularization on graphs SIAM Journal on Imaging Sciences, vol. 6, no 3, pp , P. L. Combettes, L. Condat, J.-C. Pesquet, and B. C. Vũ A forward-backward view of some primal-dual optimization methods in image recovery IEEE International Conference on Image Processing (ICIP 2014), pp , Paris, France, Oct , P. Combettes and J.-C Pesquet Stochastic quasi-fejér block-coordinate fixed point iterations with random sweeping SIAM Journal on Optimization, vol. 25, no. 2, pp , July N. Komodakis and J.-C. Pesquet Playing with duality: An overview of recent primal-dual approaches for solving large-scale optimization problems to appear in IEEE Signal Processing Magazine, J.-C. Pesquet and A. Repetti A class of randomized primal-dual algorithms for distributed optimization to appear in Journal of Nonlinear and Convex Analysis, A. Repetti, E. Chouzenoux and J.-C. Pesquet A random block-coordinate primal-dual proximal algorithm with application to 3D mesh denoising IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2015), pp , South Brisbane, Australia, April 2015.