A FORWARD-BACKWARD VIEW OF SOME PRIMAL-DUAL OPTIMIZATION METHODS IN IMAGE RECOVERY

Transcription

1 A FORWARD-BACKWARD VIEW OF SOME PRIMAL-DUAL OPTIMIZATION METHODS IN IMAGE RECOVERY P. L. Combettes, 1 L. Condat, 2 J.-C. Pesquet, 3 and B. C. Vũ 4 1 Sorbonne Universités UPMC Univ. Paris 06 Laboratoire Jacques-Louis Lions, Paris, France 2 University of Grenoble Alpes, GIPSA-lab, St Martin d Hères, France 3 Université Paris-Est, LIGM, UMR CNRS 8049, Marne-la-Vallée, France 4 LCSL, Istituto Italiano di Tecnoloia and MIT, Genova, Italy ABSTRACT A wide array of imae recovery problems can be abstracted into the problem of minimizin a sum of composite convex functions in a Hilbert space. To solve such problems, primal-dual proximal approaches have been developed which provide efficient solutions to lare-scale optimization problems. The objective of this paper is to show that a number of existin alorithms can be derived from a eneral form of the forward-backward alorithm applied in a suitable product space. Our approach also allows us to develop useful extensions of existin alorithms by introducin a variable metric. An illustration to imae restoration is provided. Index Terms convex optimization, duality, parallel computin, proximal alorithm, variational methods, imae recovery. 1. INTRODUCTION Many imae recovery problems can be formulated in Hilbert spaces H andg i 1 i m as structured optimization problems of the form minimize il ix, 1 where, for every i {1,...,m}, i is a proper lower semicontinuous convex function from G i to ],+ ] and L i is a bounded linear operator from H to G i. For example, the functions i L i 1 i m may model data fidelity terms, smooth or nonsmooth measures of reularity, or hard constraints on the solution. In recent years, many alorithms have been developed to solve such a problem by takin advantae of recent advances in convex optimization, especially in the development of proximal tools see [12, 29] and the references therein. In imae processin, however, solvin such a problem still poses a number of conceptual and numerical challenes. First of all, one often looks for methods which have the ability to split the problem by activatin each of the functions throuh elementary processin steps which can be computed in parallel. This makes it possible to reduce the complexity of the oriinal problem and to benefit from existin parallel computin architectures. Secondly, it is often useful to desin alorithms which can exploit, in a flexible manner, the structure of the problem. In particular, some of the functions may be Lipschitz differentiable in which case they should be exploited throuh their radient rather than throuh their proximity operator, which is usually harder to This work was supported by the CNRS MASTODONS project rant 2013 MesureHD. implement examples of proximity operators with closed-form expression can be found in [6, 12]. In some problems, the functions i 1 i m can be expressed as the infimal convolution of simpler functions see [9] and the references therein. Last but not least, in imae recovery, the operators L i 1 i m may be of very lare size so that their inversions are costly e.., in reconstruction problems. Findin alorithms which do not require to perform inversions of these operators is thus of paramount importance. Note that all the existin convex optimization alorithms do not have these desirable properties. For example, the Alternatin Direction Method of Multipliers ADMM [18, 17, 20] requires a strinent assumption of invertibility of the involved linear operator. Parallel versions of ADMM [28] and related Parallel Proximal Alorithm PPXA [11, 25] usually necessitate a linear inversion to be performed at each iteration. Also, early primal-dual alorithms [4, 5, 7, 10, 16, 21] did not make it possible to handle smooth functions throuh their radients. Only recently, have primal-dual methods been proposed with this feature. Such work was initiated in [13] in the line of [4] and subsequent developments can be found in [2, 3, 8, 9, 15, 27, 30]. As will be seen in the present paper, another advantae of these approaches is that they can be coupled with variable metric strateies which can potentially accelerate their converence. In Section 2, we provide some backround on convex analysis and monotone operator theory. In Section 3, we introduce a eneral form of the forward-backward alorithm which uses a variable metric. This alorithm is employed in Section 4 to develop a versatile family of primal-dual proximal methods. Several particular instances of this framework are discussed. Finally, we provide illustratin numerical results in Section NOTATION AND BACKGROUND Monotone operator theory [1] provides a both insihtful and eleant framework for dealin with convex optimization problems and developin new solution alorithms that could not be devised usin purely variational tools. We summarize a number of related concepts that will be needed. Throuhout, H, G, and G i 1 i m are real Hilbert spaces. We denote the scalar product of a Hilbert space by and the associated norm by. The symbol denotes weak converence, 1 and Id denotes the identity operator. We denote by BH,G the space of bounded linear operators from H to G, we set SH = 1 In a finite-dimensional space, weak converence is equivalent to stron converence.

2 { } L BH,H L = L, where L denotes the adjoint of L. The Loewner partial orderin on SH is denoted by. For every α [0,+ [, we set P αh = { U SH U αid }, and we denote by U the square root of U P αh. Moreover, for every U P αh and α > 0, we define the norm x U = Ux x. We denote byg = G 1 G m the Hilbert direct sum of the Hilbert spacesg i 1 i m, i.e., their product space equipped with the scalar product : x,y m xi yi where x = xi 1 i m andy = y i 1 i m denote eneric elements ing. Let A: H 2 H be a set-valued operator. We denote by raa = { x,u H H u Ax } the raph of A, by zera = { x H 0 Ax } the set of zeros of A, and by rana = { u H x H u Ax } its rane. The inverse of A is A 1 : H 2 H : u { x H u Ax }, and the resolvent of A is J A = Id+A 1. Moreover, A is monotone if x,y H H u,v Ax Ay x y u v 0, 2 and maximally monotone if it is monotone and there exists no monotone operator B: H 2 H such that raa rab and A B. An operator B: H H isβ-cocoercive for some β ]0,+ [ if x H y H x y Bx By β Bx By 2. 3 The conjuate of a function f: H ],+ ] is f : H [,+ ] : u sup x u fx, 4 and the infimal convolution of f with: H ],+ ] is f : H [,+ ] : x inf fy+x y. 5 y H The class of lower semicontinuous convex functions f: H ],+ ] such that domf = { x H fx < + } is denoted by Γ 0H. If f Γ 0H, then f Γ 0H and the subdifferential of f is the maximally monotone operator f: H 2 H x { u H y H y x u +fx fy }. 6 LetU P αh for some α ]0,+ [. The proximity operator of f Γ 0H relative to the metric induced byu is [22, Section XV.4] prox U f : H H: x armin fy+ 1 y H 2 x y 2 U. 7 When U = Id, we retrieve the standard definition of the proximity operator [1, 24]. Let C be a nonempty subset of H. The indicator function of C is defined on H as { 0, if x C; ι C: x 8 +, if x / C. Finally,l 1 +N denotes the set of summable sequences in[0,+ [. 3. A GENERAL FORM OF FORWARD-BACKWARD ALGORITHM Optimization problems can often be reduced to findin a zero of a sum of two maximally monotone operators A and B actin on H. When B is cocoercive see 3, a useful alorithm to solve this problem is the forward-backward alorithm, which can be formulated in a eneral form involvin a variable metric as shown in the next result. Theorem 3.1 Letα ]0,+ [, let β ]0,+ [, leta: H 2 H be maximally monotone, and let B: H H be cocoercive. Let η n n N l 1 +N, and let V n n N be a sequence in P αh such that { sup n N V n < + n N 1+η nv n+1 V n 9 and Vn 1/2 BVn 1/2 is β-cocoercive. Let λ n n N be a sequence in ]0,1] such that inf n N λ n > 0 and let γ n n N be a sequence in ]0,2β[ such thatinf n N γ n > 0 andsup n N γ n < 2β. Letx 0 H, and let a n n N and b n n N be absolutely summable sequences in H. Suppose that Z = zera+b, and set n N yn = x n γ nv nbx n +b n Thenx n x for some x Z. x n+1 = x n +λ n JγnV nay n+a n x n. 10 At iteration n, variables a n and b n model numerical errors possibly arisin when applyin J γnvna or B. Note also that, if B is µ-cocoercive with µ ]0, + [, one can choose β = µsup n N V n 1, which allows us to retrieve [14, Theorem 4.1]. In the next section, we shall see how a judicious use of this result allows us to derive a variety of flexible convex optimization alorithms. 4. A VARIABLE METRIC PRIMAL-DUAL METHOD 4.1. Formulation A wide array of optimization problems encountered in imae processin are instances of the followin one, which was first investiated in [13] and can be viewed as a more structured version of the minimization problem in 1: Problem 4.1 Let z H, let m be a strictly positive inteer, let f Γ 0H, and let h: H R be convex and differentiable with a Lipschitzian radient. For every i {1,...,m}, let r i G i, let i Γ 0G i, let l i Γ 0G i be stronly convex, 2 and suppose that 0 L i BH,G i. Suppose that z ran f + Consider the problem minimize fx+ and the dual problem L i i l il i r i+ h. 11 i l il ix r i+hx x z, 12 minimize f h z v 1 G 1,...,v m G m + L iv i i v i+l iv i+ v i r i For every i {1,...,m}, l i is ν 1 i -stronly convex with ν i ]0,+ [ if and only if l i is ν i-lipschitz differentiable [1, Theorem 18.15].

3 Note that in the special case when l i = ι {0}, i l i reduces to i in 12. Let us now examine how Problem 4.1 can be reformulated from the standpoint of monotone operators. To this end, let us define Γ 0G, l Γ 0G and L BH,G by : v iv i, l: v l iv i and L: x L 1x,...,L mx. 14 Let us now introduce the product spacek = H G and the operators A: K 2 K and x,v fx z +L v Lx+ v+r 15 B: K K x,v hx, l v. 16 The operator A can be shown to be maximally monotone,whereas B is cocoercive. A key observation in this context is that, if there exists x,v K such that x,v zera+b, then x,v is a pair of primal-dual solutions to Problem 4.1 [13]. This connection with the construction for a zero of A + B makes it possible to apply a forward-backward alorithm as discussed in Section 3, by usin a linear operator V n BK,K to chane the metric at each iteration n. Dependin on the form of this operator various alorithms can be obtained A first class of primal-dual alorithms Let α ]0,+ [, let U n n N be a sequence in P αh such that n N U n+1 U n. For every i {1,...,m}, let U i,n n N be a sequence inp αg i such that n NU i,n+1 U i,n. A first possible choice for V n n N is iven by n N V 1 n : x,v Un 1 x L v, Lx+Ũ 1 n v 17 where Ũ n: G G: v 1,...,v m U 1,nv 1,...,U m,nv m. 18 The followin result constitutes a direct extension of [14, Example 6.4]: Proposition 4.2 Let x 0 H, and let a n n N and c n n N be absolutely summable sequences in H. For every i {1,...,m}, let v i,0 G i, let b i,n n N and d i,n n N be absolutely summable sequences in G i. For every n N, let µ n ]0,+ [ be a Lipschitz constant of Un 1/2 h Un 1/2 and, for every i {1,...,m}, let ν i,n ]0,+ [ be a Lipschitz constant of U 1/2 i,n l i U 1/2 i,n. Let λ n n N be a sequence in]0,1] such thatinf n N λ n > 0. For every n N, set m δ n = 1/2 U i,nl i Un 2 1, 19 and suppose that inf n N δ n 1+δ nmax{µ n,ν 1,n,...,ν m,n} > Set For n = 0,1,... p n = prox U 1 n f +c n z +a n y n = 2p n x n x n+1 = x n +λ np n x n For i = 1,...,m q i,n = prox U 1 i,n i d i,n r i +b i,n x n U n m L iv i,n + hx n v i,n+1 = v i,n +λ nq i,n v i,n. v i,n +U i,n Liy n l iv i,n 21 Then x n n N converes weakly to a solution to 12, for every i {1,...,m} v i,n n N converes weakly to some v i G i, and v 1,...,v m is a solution to 13. In the special case when U n τ Id with τ ]0,+ [ and, for every i {1,...,m}, U i,n σ iid with σ i ]0,+ [, we recover the parallel alorithm proposed in [30]. Variants of this alorithm where, for everyi {1,...,m},l i = ι {0} are also investiated in [15]. In this case, less restrictive assumptions on the choice of τ,σ 1,...,σ m can be made. Note that this alorithm itself can be viewed as a eneralization of the alorithm which constitutes the main topic of [5, 16, 21] desinated by some authors as PDHG. A preconditioned version of this alorithm was proposed in [26] correspondin to the case when m = 1, n N U n and U 1,n are constant matrices, and no error term is taken into account. Alorithm 21 when, for every n N, λ n 1, U n and U i,n 1 i m are diaonal matrices, h = 0, and i {1,...,m} l i = ι {0} appears also to be closely related to the adaptive method in [19] A second class of primal-dual alorithms Let α ]0,+ [, let U n n N be a sequence in P αh such that n NU n+1 U n. For everyi {1,...,m}, letu i,n n N be a sequence in P αg i such that n N U i,n+1 U i,n. A second possible choice forv n n N is iven by the followin diaonal form: n N V 1 n : x,v U 1 n x,ũ 1 n LU nl v 22 where Ũn is iven by 18. The followin result can then be deduced from Theorem 3.1. Its proof is skipped due to the lack of space. Proposition 4.3 Let x 0 H, and let c n n N be an absolutely summable sequence in H. For every i {1,...,m}, let v i,0 G i, let b i,n n N and d i,n n N be absolutely summable sequences in G i. For every n N, let µ n ]0,+ [ be a Lipschitz constant of Un 1/2 h Un 1/2 and, for everyi {1,...,m}, letν i,n ]0,+ [ be a Lipschitz constant ofu 1/2 i,n l i U 1/2 i,n. Letλn n N be a sequence in]0,1] such that inf n N λ n > 0. For every n N, set and suppose that inf n N ζ n = 1 U i,nl i Un 2 23 ζ n max{ζ nµ n,ν 1,n,...,ν m,n} >

4 a b Fi. 2. Normalized norm of the error on the iterate vs computation time in seconds for Experiment 1 blue, dash dot line and Experiment 2 red, continuous line. c Fi. 1. Oriinal imae x a, noisy imae w 1 SNR = 5.87 db b, blurred imaew 2 SNR =16.63 db c, and restored imae x SNR =21.61 db d. Set For n = 0,1,... s n = x n U n hx n+c n z y n = s n U m n L iv i,n For i = 1,...,m q i,n = prox U 1 i,n i d v i,n +U i,n Liy n l iv i,n d i,n r i +b i,n v i,n+1 = v i,n +λ nq i,n v i,n. p n = s n U n m L iq i,n x n+1 = x n +λ np n x n. 25 Assume thatf = 0. Thenx n n N converes weakly to a solution to 12, for every i {1,...,m}v i,n n N converes weakly to some v i G i, and v 1,...,v m is a solution to 13. The alorithm proposed in [23, 8] is a special case of the previous one, in the absence of errors, when m = 1, H and G 1 are finite dimensional spaces, l 1 = ι {0}, U n τ Id with τ ]0,+ [, U 1,n σid with σ ]0,+ [, and no relaxation λ n 1 or a constant one λ n κ < 1 is performed. 5. APPLICATION TO IMAGE RESTORATION We illustrate the flexibility of the proposed primal-dual alorithms on an imae recovery example. Two observed imaes w 1 and w 2 of the same scene x R N N = are available see Fi. 1a- c. The first one is corrupted with a noise with a varianceθ 2 1 = 576, while the second one has been deraded by a linear operator H R N N 7 7 uniform blur and a noise with variance θ 2 2 = 25. The noise components are mutually statistically independent, additive, zero-mean, white, and Gaussian distributed. Note that this kind of multivariate restoration problem is encountered in some push-broom satellite imain systems. An estimate x ofxis computed as a solution to 12 wherem = 2, z = 0, r 1 = 0, r 2 = 0, h = 1 w w 2 θ1 2 θ2 H 2, = ι [0,255] N, 2 = κ 1,2, 27 f = 0, l 1 = l 2 = ι {0} 28 where the second function in 27 denotes the l 1,2-norm and κ ]0,+ [. In addition, L 1 = Id and L 2 = [G 1,G 2 ] where G 1 R N N and G N N 2 are horizontal and vertical discrete radient operators. Function 1 introduces some a priori constraint on the rane values in the taret imae, while function 2 L 2 corresponds to a classical total variation reularization. The minimization problem is solved numerically by usin Alorithm 25 with λ n 1. In a first experiment, standard choices of the alorithm parameters are made by settin U n τ Id, U 1,n σ 1Id, and U 2,n = σ 2Id with τ,σ 1,σ 2 ]0,+ [ 3. In a second experiment, a more sophisticated choice of the metric is made. The operators U n n N, U 1,n n N and U 2,n n N are still chosen diaonal and constant in order to facilitate the implementation of the alorithm, but the diaonal values are optimized in an empirical manner. A similar stratey was applied in [26] in the case of Alorithm 21. The reularization parameter κ has been set so as to et the hihest value of the resultin sinal-to-noise ratio SNR. The restored imae is displayed in Fi. 1d. Fi. 2 shows the converence profile of the alorithm. We plot the evolution of the normalized Euclidean distance in lo scale between the iterates and x in terms of computational time Matlab R2011b codes runnin on a sinle-core Intel i7-2620m CPU@2.7 GHz with 8 GB of RAM. An approximation of x obtained after 5000 iterations is used. This result illustrates the fact that an appropriate choice of the metric may be beneficial in terms of speed of converence.

5 6. REFERENCES [1] H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces. New York: Spriner, [2] S. R. Becker and P. L. Combettes, An alorithm for splittin parallel sums of linearly composed monotone operators, with applications to sinal recovery, Nonlinear Convex Anal., vol. 15, no. 1, pp , Jan [3] R. I. Boţ and C. Hendrich, Converence analysis for a primal-dual monotone + skew splittin alorithm with applications to total variation minimization, J. Math. Imain Vision, 2013, accepted rabot/publications/jour13-18.pdf. [4] L. M. Briceño-Arias and P. L. Combettes, A monotone + skew splittin model for composite monotone inclusions in duality, SIAM J. Optim., vol. 21, no. 4, pp , Oct [5] A. Chambolle and T. Pock, A first-order primal-dual alorithm for convex problems with applications to imain, J. Math. Imain Vision, vol. 40, no. 1, pp , [6] C. Chaux, P. L. Combettes, J.-C. Pesquet, and V. R. Wajs, A variational formulation for frame-based inverse problems, Inverse Problems, vol. 23, no. 4, pp , Jun [7] G. Chen and M. Teboulle, A proximal-based decomposition method for convex minimization problems, Math. Proram., vol. 64, pp , [8] P. Chen, J. Huan, and X. Zhan, A primal-dual fixed point alorithm for convex separable minimization with applications to imae restoration, Inverse Problems, vol. 29, no. 2, 2013, doi: / /29/2/ [9] P. L. Combettes, Systems of structured monotone inclusions: duality, alorithms, and applications, SIAM J. Optim., vol. 23, no. 4, pp , Dec [10] P. L. Combettes, D. Dũn, and B. C. Vũ, Dualization of sinal recovery problems, Set-Valued Var. Anal., vol. 18, pp , Dec [11] P. L. Combettes and J.-C. Pesquet, A proximal decomposition method for solvin convex variational inverse problems, Inverse Problems, vol. 24, no. 6, Dec [12], Proximal splittin methods in sinal processin, in Fixed-Point Alorithms for Inverse Problems in Science and Enineerin, H. H. Bauschke, R. S. Burachik, P. L. Combettes, V. Elser, D. R. Luke, and H. Wolkowicz, Eds. New York: Spriner-Verla, 2011, pp [13], Primal-dual splittin alorithm for solvin inclusions with mixtures of composite, Lipschitzian, and parallel-sum type monotone operators, Set-Valued Var. Anal., vol. 20, no. 2, pp , June [14] P. L. Combettes and B. C. Vũ, Variable metric forwardbackward splittin with applications to monotone inclusions in duality, Optimization, 2012, published online DOI: / [15] L. Condat, A primal-dual splittin method for convex optimization involvin Lipschitzian, proximable and linear composite terms, J. Optim. Theory Appl., vol. 158, no. 2, pp , Au [16] E. Esser, X. Zhan, and T. Chan, A eneral framework for a class of first order primal-dual alorithms for convex optimization in imain science, SIAM J. Imain Sci., vol. 3, no. 4, pp , [17] M. A. T. Fiueiredo and R. D. Nowak, Deconvolution of Poissonian imaes usin variable splittin and aumented Laranian optimization, in IEEE Work. on Stat. Si. Proc., Cardiff, United Kindom, Au Sept , pp. x-x+4. [18] M. Fortin and R. Glowinski, Eds., Aumented Laranian Methods: Applications to the Numerical Solution of Boundary- Value Problems. Amsterdam: North-Holland: Elsevier Science Ltd, [19] T. Goldstein, E. Esser, and R. Baraniuk, Adaptive primal-dual hybrid radient methods for saddle-point problems, 2013, [20] T. Goldstein and S. Osher, The split Breman method for l 1- reularized problems, SIAM J. Imain Sci., vol. 2, pp , [21] B. He and X. Yuan, Converence analysis of primal-dual alorithms for a saddle-point problem: from contraction perspective, SIAM J. Imain Sci., vol. 5, no. 1, pp , [22] J.-B. Hiriart-Urruty and C. Lemaréchal, Convex Analysis and Minimization Alorithms, Part II : Advanced Theory and Bundle Methods. New York: Spriner-Verla, [23] I. Loris and C. Verhoeven, On a eneralization of the iterative soft-thresholdin alorithm for the case of non-separable penalty, Inverse Problems, vol. 27, no. 12, p , [24] J.-J. Moreau, Proximité et dualité dans un espace hilbertien, Bull. Soc. Math. France, vol. 93, pp , [25] J.-C. Pesquet and N. Pustelnik, A parallel inertial proximal optimization method, Pac. J. Optim., vol. 8, no. 2, pp , Apr [26] T. Pock and A. Chambolle, Diaonal preconditionin for first order primal-dual alorithms in convex optimization, in Proc. IEEE Int. Conf. Comput. Vis., Barcelona, Spain, Nov , pp [27] A. Repetti, E. Chouzenoux, and J.-C. Pesquet, A penalized weihted least squares approach for restorin data corrupted with sinal-dependent noise, in Proc. Eur. Si. and Imae Proc. Conference, Bucharest, Romania, Au. 2012, pp [28] S. Setzer, G. Steidl, and T. Teuber, Deblurrin Poissonian imaes by split Breman techniques, J. Visual Communication and Imae Representation, vol. 21, no. 3, pp , Apr [29] S. Sra, S. Nowozin, and S. J. Wriht, Optimization for Machine Learnin. Cambride, MA: MIT Press, [30] B. C. Vũ, A splittin alorithm for dual monotone inclusions involvin cocoercive operators, Adv. Comput. Math., vol. 38, no. 3, pp , Apr