An Overview of Recent and Brand New Primal-Dual Methods for Solving Convex Optimization Problems

Transcription

1 PGMO 1/32 An Overview of Recent and Brand New Primal-Dual Methods for Solving Convex Optimization Problems Emilie Chouzenoux Laboratoire d Informatique Gaspard Monge - CNRS Univ. Paris-Est Marne-la-Vallée, France 28 Oct. 2015

2 PGMO 2/32 In collaboration with A. Repetti J.-C. Pesquet

3 PGMO 3/32 Problem statement GOAL: Find a solution to the optimization problem minimize x H f(x) where H is a real Hilbertian space, f: H ],+ ] is convex. In the context of large scale problems, how to find an optimization algorithm able to deliver a reliable numerical solution in a reasonable time, with low memory requirement?

4 PGMO 4/32 Fundamental tools in convex analysis

5 PGMO 5/32 Notation and definitions Let f: H ],+ ]. The domain of function f is domf = {x H f(x) < + } If domf, function f is said to be proper. Function f is convex if ( (x,y) H 2 )( λ [0,1]) f(λx+(1 λ)y) λf(x)+(1 λ)f(y). Function f is lower semi-continuous (lsc) on H if, for every x H, for every sequence (x ) N of H, x x liminff(x ) f(x).

6 PGMO 5/32 Notation and definitions Let f: H ],+ ]. The domain of function f is domf = {x H f(x) < + } If domf, function f is said to be proper. Function f is convex if ( (x,y) H 2 )( λ [0,1]) f(λx+(1 λ)y) λf(x)+(1 λ)f(y). Function f is lower semi-continuous (lsc) on H if, for every x H, for every sequence (x ) N of H, x x liminff(x ) f(x). In the remainder of this tal, all functions will be assumed to be proper, lsc and convex.

7 PGMO 6/32 Notation and definitions The set of strongly positive, self adjoint, linear operators from H to H is denoted by S + (H). Let U S + (H). The weighted norm induced by U is with the convention = Id. U = U,

8 PGMO 6/32 Notation and definitions The set of strongly positive, self adjoint, linear operators from H to H is denoted by S + (H). Let U S + (H). The weighted norm induced by U is with the convention = Id. U = U, Let f: H ],+ [. Function f is said β-lipschitz differentiable if it is differentiable over H and its gradient fulfills with β ]0,+ [. ( (x,y) H 2 ) f(x) f(y) β x y,

9 Introduction Fundamental tools in convex analysis Primal-dual proximal algorithms Application to 3D mesh denoising PGMO 7/32 Subdifferential The subdifferential of f: H ],+ ] at x is the set f(x) = {t H ( y H) f(y) f(x)+ t y x } An element t of f(x) is called a subgradient of f at x. f(y) f(x)+ y x t t f(x) x x y If f is differentiable at x H then f(x) = { f(x)}.

10 Introduction Fundamental tools in convex analysis Primal-dual proximal algorithms Application to 3D mesh denoising PGMO 8/32 Conjugate function The conjugate of f: H ],+ ] is f : H [,+ ] such that ( u H) f ( ) (u) = sup x u f(x). x H f(x) x u f(x) x f (u) x f (u) x u ORIGINS: A. C. Clairaut ( ), A.-M. Legendre ( ), W. Fenchel ( )

11 PGMO 9/32 Notation and definitions The inf-convolution between f : H ],+ ] and g : H ],+ ] is f g: H [,+ ] x inf y H f(y)+g(x y). The convolution between f : H ],+ ] and g : H ],+ ] is f g: H [,+ ] x f(y)g(x y)dy. H PARTICULAR CASE: f ι {0} = f, where, for C H, ( x H) ι C (x) = { 0 if x C, + elsewhere.

12 PGMO 10/32 Proximity operator CHARACTERIZATION OF PROXIMITY OPERATOR ( x H) ŷ = prox U,f (x) x ŷ U 1 f(ŷ). The proximity operator prox U,f (x) of f at x H relative to the metric induced by U is the unique vector ŷ H such that f(ŷ)+ 1 2 ŷ x 2 U = inf f(y)+ 1 y x U(y x). y H 2

13 PGMO 11/32 Primal-dual proximal algorithms

14 PGMO 12/32 Primal-dual problem PRIMAL PROBLEM minimize x H f(x)+g(lx)+h(x). f: H ],+ ], g: G ],+ ], h: H R, β-lipschitz differentiable with β ]0,+ [, H and G real Hilbert spaces, L: H G linear and bounded.

15 PGMO 12/32 Primal-dual problem PRIMAL PROBLEM minimize x H f(x)+g(lx)+h(x). f: H ],+ ], g: G ],+ ], h: H R, β-lipschitz differentiable with β ]0,+ [, H and G real Hilbert spaces, L: H G linear and bounded. minimize v G (f h ) ( L v ) +g (v). DUAL PROBLEM

16 PGMO 13/32 Primal-dual problem KKT CONDITIONS If a pair ( x, v) fulfills the Karush-Kuhn-Tucer conditions: L v h( x) f( x) and L x g ( v), then { x is a solution to the primal problem v is a solution to the dual problem. LAGRANGIAN FORMULATION The problem is equivalent to searching for a saddle point of the Lagrangian: ( (x,y,v) H G 2 ) L(x,y,v) = f(x)+h(x)+g(y)+ v Lx y.

17 PGMO 14/32 Lin with Lagrange duality SADDLE POINTS ( x,ŷ, v) is a saddle point of the Lagrange function L if ( (x,y,v) H G 2 ) L( x,ŷ,v) L( x,ŷ, v) L(x,y, v). Let H and G be two real Hilbert spaces. Let ( x,ŷ, v) H G 2. Assume that ri(domg) L(domf) or domg ri ( L(domf) ). ( x,ŷ, v) is a saddle point of the Lagrange function ( x, v) is a Kuhn-Tucer point and ŷ = L x. EQUIVALENCE

18 PGMO 15/32 Primal-dual proximal algorithm for = 0,1,... y = prox τf (x τ ( h(x )+c +L ) ) v +a u = prox σg ( v +σl(2y x ) ) +b (x +1,v +1 ) = (x,v )+λ ((y,u ) (x,v ))

19 PGMO 15/32 Primal-dual proximal algorithm for = 0,1,... y = prox τf (x τ ( h(x )+c +L ) ) v +a u = prox σg ( v +σl(2y x ) ) +b (x +1,v +1 ) = (x,v )+λ ((y,u ) (x,v )) (τ,σ) ]0,+ [ 2 such that τ 1 σ L 2 > β/2,

20 PGMO 15/32 Primal-dual proximal algorithm for = 0,1,... y = prox τf (x τ ( h(x )+c +L ) ) v +a u = prox σg ( v +σl(2y x ) ) +b (x +1,v +1 ) = (x,v )+λ ((y,u ) (x,v )) (τ,σ) ]0,+ [ 2 such that τ 1 σ L 2 > β/2, for every N, λ ]0,λ[, where λ = 2 β(τ 1 σ L 2 ) 1 /2 [1,2[,

21 PGMO 15/32 Primal-dual proximal algorithm for = 0,1,... y = prox τf (x τ ( h(x )+c +L ) ) v +a u = prox σg ( v +σl(2y x ) ) +b (x +1,v +1 ) = (x,v )+λ ((y,u ) (x,v )) (τ,σ) ]0,+ [ 2 such that τ 1 σ L 2 > β/2, for every N, λ ]0,λ[, where λ = 2 β(τ 1 σ L 2 ) 1 /2 [1,2[, a < +, N N b < +, and N c < +,

22 PGMO 15/32 Primal-dual proximal algorithm for = 0,1,... y = prox τf (x τ ( h(x )+c +L ) ) v +a u = prox σg ( v +σl(2y x ) ) +b (x +1,v +1 ) = (x,v )+λ ((y,u ) (x,v )) (τ,σ) ]0,+ [ 2 such that τ 1 σ L 2 > β/2, for every N, λ ]0,λ[, where λ = 2 β(τ 1 σ L 2 ) 1 /2 [1,2[, a < +, N N b < +, and N c < +, there exists x H such that 0 f(x)+ h(x)+l g ( Lx ), Then: x x where x is a solution to the primal problem, v v where v is a solution to the dual problem.

23 PGMO 16/32 Preconditioned primal-dual proximal algorithm for = 0,1,... ( y = prox W 1,f x W ( h(x )+c +L ) ) v +a ( u = prox U 1,g v +UL(2y x ) ) +b (x +1,v +1 ) = (x,v )+λ ((y,u ) (x,v )) W S + (H) and U S + (G) such that 1 U 1/2 LW 1/2 > β 2 W, for every N, λ ]0,1], and inf N λ > 0, a < +, N N b < +, and N c < +, there exists x H such that 0 f(x)+ h(x)+l g ( Lx ). Then: x x where x is a solution to the primal problem, v v where v is a solution to the dual problem.

24 PGMO 17/32 Parallel primal-dual proximal algorithm PRIMAL PROBLEM minimize x H f(x)+g(lx)+h(x).

25 PGMO 17/32 Parallel primal-dual proximal algorithm PRIMAL PROBLEM minimize x H q f(x)+ g n (L n x)+h(x). n=1 with, for all n {1,...,q}, g n : G n ],+ ], (G n ) 1 n q real Hilbert spaces such thatg = G 1 G q, andl n : H G n bounded linear operators.

26 PGMO 17/32 Parallel primal-dual proximal algorithm PRIMAL PROBLEM minimize x H q f(x)+ g n (L n x)+h(x). n=1 with, for all n {1,...,q}, g n : G n ],+ ], (G n ) 1 n q real Hilbert spaces such thatg = G 1 G q, andl n : H G n bounded linear operators. minimize (v (1),...,v (q) ) G (f h ) ( q L nv (n)) + n=1 DUAL PROBLEM q gn(v (n) ). n=1

27 PGMO 17/32 Parallel primal-dual proximal algorithm PRIMAL PROBLEM minimize x H q f(x)+ g n (L n x)+h(x). n=1 with, for all n {1,...,q}, g n : G n ],+ ], (G n ) 1 n q real Hilbert spaces such thatg = G 1 G q, andl n : H G n bounded linear operators. minimize (v (1),...,v (q) ) G (f h ) ( q L nv (n)) + n=1 DUAL PROBLEM q gn(v (n) ). The separability of g implies that prox g can be obtained by computing in parallel the proximal operators of functions (g n) 1 n q PARALLEL PRIMAL-DUAL SCHEMES. n=1

28 PGMO 18/32 Parallel primal-dual proximal algorithm for = 0,1,... ( y = prox W 1,f x W ( q h(x )+c + n=1 x +1 = x +λ (y x ) for n,...,q u (n) ( (n) = prox U 1 n,g v n +U n L n (2y x ) ) +b (n) v (n) +1 = v(n) +λ (u (n) v (n) ) L nv (n) ) ) +a

29 PGMO 19/32 Bibliographical remars Pioneering wor: Arrow-Hurwicz-Uzawa method for saddle point problems [Arrow et al ] [Nedić and Ozdaglar ] Methods based on Forward-Bacward iteration: type I : [Vu ][Condat ] (extensions of [Esser et al ][Chambolle and Poc ]) type II: [Combettes et al ] (extensions of [Loris and Verhoeven ][Chen et al ]) Methods based on Forward-Bacward-Forward iteration [Combettes and Pesquet ] [Boţ and Hendrich,2014] Projection based methods... [Alotaibi et al ]

30 PGMO 20/32 Random bloc coordinate strategy

31 PGMO 21/32 Acceleration via bloc alternation Idea: variable splitting.

32 PGMO 21/32 Acceleration via bloc alternation Idea: variable splitting. x (1) H 1 x (2) H 2 g x H H = p j=1 H j x (p) H p H 1,...,H p are separable real Hilbert spaces

33 PGMO 21/32 Acceleration via bloc alternation Simplifying assumption: f and h are bloc separable functions. x (1) x (2) f x = f = p f j (x (j) ) j=1 x (p)

34 PGMO 21/32 Acceleration via bloc alternation Simplifying assumption: f and h are bloc separable functions. x (1) x (2) h x = h = p h j (x (j) ) j=1 x (p) ( j {1,...,p}) h j β j -Lipschitz differentiable with β j ]0,+ [.

35 PGMO 22/32 Acceleration via bloc alternation For each iteration N, update a subset of components ( Gauss-Seidel methods). ADVANTAGES: Reduced computational cost per iteration, Reduced memory requirement, More flexibility.

36 PGMO 23/32 Primal-dual problem PRIMAL PROBLEM Let F the set of solutions to the problem p ( ) minimize f j (x (j) )+h j (x (j) ) x (1) H 1,...,x (p) H p j=1 ( j {1,...,p})( n {1,...,q}) H j and G n separable real Hilbert spaces, f j: H j ],+ ] + q ( p g n L n,j x (j)) n=1 h j: H j R β j-lipschitz differentiable, with β j ]0,+ [ g n: G n ],+ ] L n,j: H j G n linear and bounded, L n = { j {1,...,p} L n,j 0 }, et L j = { n {1,...,q} L n,j 0 }. j=1

37 PGMO 23/32 Primal-dual problem PRIMAL PROBLEM Let F the set of solutions to the problem p ( ) minimize f j (x (j) )+h j (x (j) ) x (1) H 1,...,x (p) H p j=1 + q ( p g n L n,j x (j)) Let F DUAL PROBLEM the set of solutions to the problem p ( q minimize (fj h j) L n,jv (n)) q ( ) + gn(v (n) ) v (1) G 1,...,v (q) G q j=1 n=1 n=1 n=1 j=1 Assume that there exists (x (1),...,x (p) ) H 1... H p such that q ( j {1,...,p}) 0 f j (x (j) )+ h j (x (j) )+ L ( n,j g n Ln,j x (j)). n=1 GOAL: Find an F F -valued random variable ( x, v).

38 PGMO 24/32 Random bloc coordinate primal-dual proximal algorithm For = 0,1,... for j,...,p ( (j) y = ε (j) for n,...,q ( u(n) = ε (p+n) v (n) +1 = v(n) ( (j) prox W 1 j,f j x W j( h j (x (j) )+c(j) + q n=1 L n,j v(n) )) +a (j) x (j) +1 = x(j) +λ ε (j) (y(j) x (j) ) prox U 1 n,g n +λ ε (p+n) ( v (n) p +U n j=1 (u (n) v (n) ), ) L n,j (2y (j) x (j) )) +b (n) ) (ε ) N is a vector of p+q Boolean variables randomly chosen at each iteration N in order to signal the primal and dual blocs to be updated.

39 PGMO 25/32 Random bloc coordinate primal-dual proximal algorithm Assume that The variables (ε ) N are independent and identically distributed, such that for every N, ( n {1,...,q}) P[ε (p+n) ] > 0, and ( j {1,...,p}) ε (j) { (p+n) = max ε n L j}, 1 n q

40 PGMO 25/32 Random bloc coordinate primal-dual proximal algorithm Assume that The variables (ε ) N are independent and identically distributed, such that for every N, ( n {1,...,q}) P[ε (p+n) ] > 0, and ( j {1,...,p}) ε (j) { (p+n) = max ε n L j}, 1 n q ( j {1,...,p}) W j S + (H j), ( n {1,...,q}) U n S + (G n), with ( p ) 1/2 1 q j=1 n=1 U1/2 n L n,jw 1/2 j 2 > 1 max{( Wj βj) 2 1 j p}, ( N) λ ]0,1] and inf N λ > 0, E a 2 < +, E b 2 < +, and E c 2 < + a.s. N N N (x ) N converges wealy a.s. to an F-valued random variable. (v ) N converges wealy a.s. to an F -valued random variable. Proof: Based on properties of quasi-fejér stochastic sequences [Combettes & Pesquet 2015].

41 PGMO 26/32 Illustration of the random sampling strategy Bloc selection ( N) x (1) activated when ε (1) How to choose ( N) the variable ε = (ε (1),...,ε(6) )? x (2) activated when ε (2) x (3) activated when ε (3) x (4) activated when ε (4) x (5) activated when ε (5) x (6) activated when ε (6)

42 PGMO 26/32 Illustration of the random sampling strategy Bloc selection ( N) x (1) activated when ε (1) x (2) activated when ε (2) x (3) activated when ε (3) How to choose ( N) the variable ε = (ε (1),...,ε(6) )? P[ε = (1,1,0,0,0,0)] = 0.1 x (4) activated when ε (4) x (5) activated when ε (5) x (6) activated when ε (6)

43 PGMO 26/32 Illustration of the random sampling strategy Bloc selection ( n N) x (1) activated when ε (1) x (2) activated when ε (2) x (3) activated when ε (3) How to choose ( N) the variable ε = (ε (1),...,ε(6) )? P[ε = (1,1,0,0,0,0)] = 0.1 P[ε = (1,0,1,0,0,0)] = 0.2 x (4) activated when ε (4) x (5) activated when ε (5) x (6) activated when ε (6)

44 PGMO 26/32 Illustration of the random sampling strategy Bloc selection ( N) x (1) activated when ε (1) x (2) activated when ε (2) x (3) activated when ε (3) x (4) activated when ε (4) How to choose ( N) the variable ε = (ε (1),...,ε(6) )? P[ε = (1,1,0,0,0,0)] = 0.1 P[ε = (1,0,1,0,0,0)] = 0.2 P[ε = (1,0,0,1,1,0)] = 0.2 x (5) activated when ε (5) x (6) activated when ε (6)

45 PGMO 26/32 Illustration of the random sampling strategy Bloc selection ( N) x (1) activated when ε (1) x (2) activated when ε (2) x (3) activated when ε (3) x (4) activated when ε (4) x (5) activated when ε (5) How to choose ( N) the variable ε = (ε (1),...,ε(6) )? P[ε = (1,1,0,0,0,0)] = 0.1 P[ε = (1,0,1,0,0,0)] = 0.2 P[ε = (1,0,0,1,1,0)] = 0.2 P[ε = (0,1,1,1,1,1)] = 0.5 x (6) activated when ε (6)

46 PGMO 27/32 Extension to distributed algorithms... OPTIMIZATION PROBLEM Find an element of the set F of the solutions to m minimize f i (x)+h i (x)+g i (M i x) x H i=1 with H,G 1,...,G m separable Hilbert spaces and, for all i {1,...,m}: h i β i -Lipschitz differentiable with β i ]0,+ [, M i : H G i non-zero linear and bounded operators.

47 PGMO 27/32 Extension to distributed algorithms... OPTIMIZATION PROBLEM Find an element of the set F of the solutions to m minimize f i (x)+h i (x)+g i (M i x) x H i=1 OPTIMIZATION PROBLEM Find (x 1,...,x m ) H m such that m minimize f i (x i )+h i (x i )+g i (M i x i ) (x i ) 1 i m Λ m i=1 with Λ m = { (x i ) 1 i m H m } x 1 =... = x m

48 PGMO 28/32 3D mesh denoising (ANR GRAPHSIP) Original mesh x Observed mesh z Undirected nonreflexive graph OBJECTIVE: Estimate x = (x i ) 1 i M from noisy observations z = (z i ) 1 i M where, for every i {1,...,M}, x i R 3 is the vector of 3D coordinates of the i-th vertex of a mesh H = (R 3 ) M

49 PGMO 28/32 3D mesh denoising (ANR GRAPHSIP) OBJECTIVE: Estimate x = (x i ) 1 i M from noisy observations z = (z i ) 1 i M where, for every i {1,...,M}, x i R 3 is the vector of 3D coordinates of the i-th vertex of a mesh COST FUNCTION: Φ(x) = M ψ j (x j z j )+ι Cj (x j )+η j (x j x i ) i Nj 1,2 j=1 where ( j {1,...,M}), ψ j : R 3 R: l 2 l 1 Huber function robust data fidelity measure convex, Lipschitz differentiable function C j : nonempty convex subset of R 3 N j : neighborhood of j-th vertex (η j ) 1 j M : nonnegative regularization constants.

50 PGMO 28/32 3D mesh denoising (ANR GRAPHSIP) OBJECTIVE: Estimate x = (x i) 1 i M from noisy observations z = (z i) 1 i M where, for every i {1,...,M}, x i R 3 is the vector of 3D coordinates of the i-th vertex of a mesh COST FUNCTION: M Φ(x) = ψ j(x j z j)+ι Cj (x j)+η j (x j x i) i Nj 1,2 j=1 IMPLEMENTATION DETAILS: a bloc a vertex p = q = M ( j {1,...,M}) h j = ψ j( z j) f j = ι Cj ( {1,...,M})( x R 3M ) g (L x) = (x x i) i N 1,2

51 PGMO 29/32 Simulation results positions of the original mesh are corrupted through an additive i.i.d. zero-mean Gaussian mixture noise model. a limited number r of variables can be handled at each iteration, where p ε j,n = r p. j=1 mesh decomposed into p/r non-overlapping sets. Original mesh, M Noisy mesh, MSE =

52 PGMO 29/32 Simulation results Proposed reconstruction MSE = Laplacian smoothing MSE =

53 PGMO 30/32 Complexity Time (s.) p/r Memory (Mb) dashed line: required memory continuous line: reconstruction time

54 PGMO 31/32 Conclusion Simple handling of Proven convergence and robustness to errors Primal-dual proximal algorithms Resolution of (non) smooth convex optimization problems contraints No inversion of linear operators Proven efficiency in some signal /image processing applications Bloccoordinate strategies Flexibility in the random selection of primal/dual components Distributed versions of the algorithms available

55 PGMO 32/32 Some references... F. Abboud, E. Chouzenoux, J.-C. Pesquet, J.-H. Chenot and L. Laborelli A Dual Bloc Coordinate Proximal Algorithm with Application to Deconvolution of Interlaced Video Sequences IEEE International Conference on Image Processing (ICIP 2015), 5 p., Québec, Canada, Sep G. Chierchia, N. Pustelni, B. Pesquet-Popescu and J.-C. Pesquet A Non-Local Structure Tensor Based Approach for Multicomponent Image Recovery Problems IEEE Transaction on Image Processing, 23(12), pp , Dec P. L. Combettes and J.-C. Pesquet Proximal Splitting methods in Signal Processing in Fixed-Point Algorithms for Inverse Problems in Science and Engineering, H. H. Bausche, R. Burachi, P. L. Combettes, V. Elser, D. R. Lue, and H. Wolowicz editors. Springer-Verlag, New Yor, pp , P. L. Combettes, L. Condat, J.-C. Pesquet, and B. C. Vũ A Forward-Bacward View of Some Primal-Dual Optimization Methods in Image Recovery IEEE International Conference on Image Processing (ICIP 2014), 5 p., Paris, France, Oct , P. Combettes and J.-C Pesquet Stochastic Quasi-Fejér Bloc-Coordinate Fixed Point Iterations with Random Sweeping SIAM Journal on Optimization, 25(2), pp , A. Florescu, E. Chouzenoux, J.-C. Pesquet, P. Ciuciu and S. Ciochina A Majorize-Minimize Memory Gradient Method for Complex-Valued Inverse Problems Signal Processing, 103, pp , A. Jeziersa, E. Chouzenoux, J.-C. Pesquet and H. Talbot A Convex Approach for Image Restoration with Exact Poisson-Gaussian Lielihood to appear in SIAM Journal in Imaging Sciences, N. Komodais and J.-C. Pesquet Playing with Duality: An Overview of Recent Primal-Dual Approaches for Solving Large-Scale Optimization Problems to appear in IEEE Signal Processing Magazine, J.-C. Pesquet and A. Repetti A Class of Randomized Primal-Dual Algorithms for Distributed Optimization to appear in Journal of Nonlinear and Convex Analysis, A. Repetti, E. Chouzenoux and J.-C. Pesquet A Random Bloc-Coordinate Primal-Dual Proximal Algorithm with Application to 3D Mesh Denoising IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2015), 5 p., Brisbane, Australia, Apr , 2015.