Solving DC Programs that Promote Group 1-Sparsity

Transcription

1 Solving DC Programs that Promote Group 1-Sparsity Ernie Esser Contains joint work with Xiaoqun Zhang, Yifei Lou and Jack Xin SIAM Conference on Imaging Science Hong Kong Baptist University May / 22

2 A General Convex Problem min u N F i=1 F i (A i u b i )+G(u) s.t. A i u = b i, i = N F +1,..., N (P) F i : R m i (, ], G : R m (, ] convex Can include indicator functions for convex sets dened by { 0 u C ι C (u) = otherwise 2 / 22

3 Algorithms Many primal dual methods can eectively solve (P) as long as the functions F i, G or their conjugates F i, G have simple proximal mappings or have Lipschitz continuous gradients: Alternating Direction Method of Multipliers (ADMM) Split Inexact Uzawa [Zhang, Burger, Osher 2010] Modied PDHG [Chambolle, Pock 2010], [He, Yuan 2012], [Esser, Zhang, Chan 2010] Bregman ADMM [Wang, Banerjee 2013] Accelerated Linearized ADMM [Ouyang, Chen, Lan, Pasiliao 2014] Bregman Operator Splitting (BOS) [Zhang, Burger, Bresson, Osher 2010] BOSVS [Chen, Hager, Yashtini, Ye, Zhang 2012] What if G is not convex? 3 / 22

4 Dierence of Convex, Majorization/Minimization Strategy Linearize the nonconvex part and possibly add and subtract a strongly convex term to dene G(u, u n ) G(u) such that G is strongly convex and G(u n, u n ) = G(u n ) Then solve a sequence of convex problems u n+1 = arg min u N F i=1 s.t. A i u = b i, F i (A i u b i ) + G(u, u n ) i = N F + 1,..., N 4 / 22

5 Nonconvex Penalties to Promote Group 1-Sparsity An interesting class of applications are problems requiring group 1-sparsity such as x 1 µ min x 2 b [ ] A 1... A N. x N s.t. x i 0, x i 0 1 One eective strategy is to replace x i 0 1 with γ( x i 1 x i 2 ) or γ( x i 1 x i 2 ) and use the DC majorization/minimization strategy. Details in A Method for Finding Structured Sparse Solutions to Non-negative Least Squares Problems with Applications, with Yifei Lou and Jack Xin, / 22

6 A Binary Group 1-Sparsity Model Many applications additionally require the elements of x i to be 0 or 1. A dierence of convex model for this is to constrain each x i to the unit simplex x S := {x : x 0, x i 1 = 1} i = 1,..., N and minimize a concave quadratic γ x 2. Some imaging applications of binary labeling models: Image segmentation Nonlocal patch-based image inpainting Linear unmixing for hyperspectral images Point matching for image registration Phase Unwrapping 6 / 22

7 DC versus Direct Application of Convex Optimization Methods The dierence of convex approach can guarantee limit points are stationary points. If the convex subproblem is expensive, solving a sequence of them may cost too much. Direct application of convex optimization methods such as modied PDHG to nonconvex problems lacks convergence guarantees in general but works well for some applications. 7 / 22

8 Modied PDHG To simplify notation, let F i = ι {0} for i = N F + 1,..., N and consider N min F i (A i u b i ) + G(u) u F i (A i u b i ) = F i i=1 (A i u b i ) = sup p i, A i u b i F i (p i ) p i Find saddle point of min u sup F (p)+ p, Au b +G(u) by iterating p u k+1 = arg min u G(u) A T p k, u + 1 2α u uk 2 M 1 p k+1 = arg min p F (p) + p, A(2u k+1 u k ) b + 1 2δ p pk 2 where αδ 1 AMA T (Can also derive via linearized ADMM or SIU) 8 / 22

9 Modied PDHG for Binary Labeling Problems Suppose we want u to be 1-sparse and binary. Let G(u) = ι S (u) γ u 2 and let M = γ 0 2αγ I γ 0 (0, 1) ) Then u k+1 1 = Π S (( )(u k αma T p k ) 1 γ 0 The only change in the nonconvex version is to lengthen u k αma T p k slightly before projecting onto the simplex. This empirically works if γ 0 is small and γ is large. If not, iterates can oscillate. More algorithm details are in Nonlocal Patch-Based Imaging Inpainting Through Minimization of a Sparsity Promoting Nonconvex Functional with Xiaoqun Zhang (preprint). 9 / 22

10 Nonconvex Proximal Point Interpretation Let M = I and denote the normal cone of S by N S (u) = {s : s, v u 0 for all v S}. Then ( ) ( 0 F ) [ (p k+1 ) + Au k+1 b 1 ] ( ) 0 N S (u k+1 ) A T p k+1 2γu k+1 + δ I A p k+1 p k A T u k+1 u k where αδ 1 A T A. See [He, Yuan 2012] for the convex case. 1 α 10 / 22

11 Example: Phase Unwrapping d = (h mod 2π) + noise Assuming h is smooth, recover it from d. Example application: Interferometric Synthetic Aperture Radar for generating elevation maps (see [Richards 2007]) Measure wrapped phase at two nearby SAR apertures Obtain wrapped phase dierence by wrapping dierence of wrapped phases Use trigonometry to interpret as measurements of relative height modulo 2πa for some factor a 11 / 22

12 Numerical Experiment: Face Unwrapping Original image Original image as surface Wrapped image missing stripe With noise (SNR = 16.9) 12 / 22

13 Proposed Model unknown u = (x, y, s) x - denoised wrapped image y - unwrapped image s - 3 labels per edge X i - selects ith label from s s 0, X 1 s + X 2 s + X 3 s = 1 want s ij {0, 1} D - discrete gradient X 1 s - 1 if no jump X 2 s - 1 if jump by 2πa X 3 s - 1 if jump by 2πa X d - crops to where data dened X D - where dierences dened X 0 - reference no jump region min λ X 1s β x,y,s 2 Dy 2 + ι S (s) γ s 2 X d x d 2 ɛ Dy (π η)a X D Dy = X D Dx + 2πa(X 3 s X 2 s) X 0 y = X 0 x 13 / 22

14 Parameters 1 γ = 200, γ 0 =.002, β =.1, λ = 100, αδ < AMA T Let M = I I M 3 I, M 3 = γ 0 (For the convex case, let γ = 0 and dene M 3 independently.) ɛ and η for the constraints X d x d 2 ɛ and Dy (π η)a reect assumptions about the noise and sampling respectively. 2αγ 14 / 22

15 Primal Iterates x k+1 = x k + α( X T d pk 3 D T XD T pk 4 X T 0 pk 5 ) y k+1 = y k + α( D T p k 2 + D T XD T pk 4 + X T 0 pk 5 ) ) s k+1 1 = Π S (( )(s k αm 3 X T 1 1 γ pk 1 + 2πaαM 3 (X T 2 X T 3 )p k 4 ) 0 15 / 22

16 Dual Iterates p k+1 1 = Π 1(p k 1 + δx 1 (2s k+1 s k ) δ) p k+1 2 = p k 2 + δd(2y k+1 y k ) δπ a(π η) ( ) p k 2 + δd(2y k+1 y k ) β + δ Note this combines β 2 z 2 + ι (π η)a(z) by observing it is the conjugate of the Huber norm. p k+1 3 = p k 3 + δ(x d (2x k+1 x k ) d) Π 2 ɛδ(p k 3 + δ(x d (2x k+1 x k ) d)) p k+1 4 = p 4 δ(x D D(2y k+1 y k 2x k+1 + x k ) + 2πa(X 2 X 3 )(2s k+1 s k )) p k+1 5 = p k 5 δ(x 0 (2y k+1 y k 2x k+1 + x k )) 16 / 22

17 Unwrapped Images Convex, no noise Convex, with noise Nonconvex, no noise Nonconvex, with noise 17 / 22

18 Recovered 1D Slice Convex, no noise Convex, with noise Nonconvex, no noise Nonconvex, with noise 18 / 22

19 Recovery Error Convex, no noise Convex, with noise Nonconvex, no noise Nonconvex, with noise 19 / 22

20 Estimated Jump Set Convex, no noise Convex, with noise Nonconvex, no noise Nonconvex, with noise 20 / 22

21 Sorted 'no jump' Labels Convex, no noise Convex, with noise Nonconvex, no noise Nonconvex, with noise 21 / 22

22 Conclusions and Open Questions Modied PDHG can empirically nd good local minima for some nonconvex problems where the nonconvexity is of the form of ι S (s) γ s 2 (maximizing the l 2 norm of s over a unit simplex). An application to 2D phase unwrapping was demonstrated. Are there conditions on the parameters that can guarantee at least that limit points are stationary points? 22 / 22