Scaled gradient projection methods in image deblurring and denoising

Transcription

1 Scaled gradient projection methods in image deblurring and denoising Mario Bertero 1 Patrizia Boccacci 1 Silvia Bonettini 2 Riccardo Zanella 3 Luca Zanni 3 1 Dipartmento di Matematica, Università di Genova 2 Dipartimento di Matematica, Università di Ferrara 3 Dipartimento di Matematica, Università di Modena e Reggio Emilia Conference on Applied Inverse Problems, Vienna July Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

2 Outline 1 Examples of Imaging problems 2 Optimization problem 3 Gradient methods and step-length selections 4 Scaled Gradient Projection (SGP) Method 5 Test results 6 Conclusions and Future Works Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

3 Image Deblurring example Image acquisition model: y = Hx + b + n, where: y R n observed image, H R n n blurring operator, b R n background radiation, n R n unknown noise. Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

4 Image Deblurring example Image acquisition model: y = Hx + b + n, where: y R n observed image, H R n n blurring operator, b R n background radiation, n R n unknown noise. Goal: Find an approximation of the true image x R n Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

5 Image Deblurring example Image acquisition model: where: y R n observed image, H R n n blurring operator, b R n n R n y = Hx + b + n, background radiation, unknown noise. Goal: Find an approximation of the true image x R n Maximum Likelihood Approach (and early stopping) min L y (x) sub. to x Ω Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

6 Image Denoising example Image acquisition model: y = x + n, where: y R n n R n observed image, unknown noise. Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

7 Image Denoising example Image acquisition model: y = x + n, where: y R n n R n observed image, unknown noise. Goal: Remove noise from y R n, while preserving some features Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

8 Image Denoising example Image acquisition model: y = x + n, where: y R n n R n observed image, unknown noise. Goal: Remove noise from y R n, while preserving some features Regularized Approach where J R (x) is (for example): x 2 2 Tikhonov, x 1 sparsity inducing, x Total Variation. Ω min J (0) y (x) + µj R (x) sub. to x Ω Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

9 Problem setting Both examples lead to: Constrained optimization problem min f (x) sub. to x Ω Ω f (x) is a convex and closed set is countinuously differentiable in Ω Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

10 Why gradient type methods? Gradient methods are first order optimization methods. Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

11 Why gradient type methods? Gradient methods are first order optimization methods. pros Simplicity of implementation first order iterative method Low memory requirements suitable to face high dimensional problems Ability to provide medium-accurate solutions Semiconvergence from numerical practice Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

12 Why gradient type methods? Gradient methods are first order optimization methods. pros Simplicity of implementation first order iterative method Low memory requirements suitable to face high dimensional problems Ability to provide medium-accurate solutions Semiconvergence from numerical practice cons Low convergence rate hundreds or thousands of iterations Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

13 The Barzilai-Borwein (BB) step-length selection rules Consider the gradient method: x (k+1) = x (k) α k g (k) k = 0, 1,..., with g(x) = f (x). Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

14 The Barzilai-Borwein (BB) step-length selection rules Consider the gradient method: x (k+1) = x (k) α k g (k) k = 0, 1,..., with g(x) = f (x). Problem: How the step-length α k > 0 can be chosen to improve the convergence rate? Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

15 The Barzilai-Borwein (BB) step-length selection rules Consider the gradient method: with g(x) = f (x). Solution: x (k+1) = x (k) α k g (k) k = 0, 1,..., Regard the matrix B(α k ) = (α k I) 1 as an approximation of the Hessian 2 f (x (k) ) Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

16 The Barzilai-Borwein (BB) step-length selection rules Consider the gradient method: x (k+1) = x (k) α k g (k) k = 0, 1,..., with g(x) = f (x). Solution: Regard the matrix B(α k ) = (α k I) 1 as an approximation of the Hessian 2 f (x (k) ) Determine α k by forcing a quasi-newton property on B(α k ): α k BB1 = argmin α R B(α)s(k 1) z (k 1) or α k BB2 = argmin α R s(k 1) B(α) 1 z (k 1), where s (k 1) = ( x (k) x (k 1)) and z (k 1) = (g (k) g (k 1) ). Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

17 The BB step-length selection rules (cont.) It follows that: α BB1 k = s(k 1)T s (k 1) or α BB2 k = s(k 1)T z (k 1) s (k 1)T z (k 1) z (k 1)T z (k 1) where s (k 1) = ( x (k) x (k 1)) and z (k 1) = (g (k) g (k 1) ). Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

18 The BB step-length selection rules (cont.) It follows that: α BB1 k = s(k 1)T s (k 1) or α BB2 k = s(k 1)T z (k 1) s (k 1)T z (k 1) z (k 1)T z (k 1) where s (k 1) = ( x (k) x (k 1)) and z (k 1) = (g (k) g (k 1) ). Remarkable improvements in comparison with the steepest descent method are observed: [Barzilai-Borwein, IMA J. Num. Anal. 1988] [Raydan, IMA J. Num. Anal. 1993] [Friedlander et al., SIAM J. Num. Anal. 1999] [Raydan, SIAM J. Optim. 1997] [Fletcher, Tech. Rep. 207, 2001] [Dai-Liao, IMA J. Num. Anal. 2002] Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

19 Effective use of the BB rules Further improvements are obtained by using adaptive alternations of the two BB rules; for example: α k = αk BB2 if αk BB2 /αk BB1 < τ, α k = αk BB1 otherwise, Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

20 Effective use of the BB rules Further improvements are obtained by using adaptive alternations of the two BB rules; for example: α k = αk BB2 if αk BB2 /αk BB1 < τ, α k = αk BB1 otherwise, Many suggestions for the alternation are available: [Dai, Optim., 2003] [Dai-Fletcher, Math. Prog. 2005] [Serafini et al., Opt. Meth. Soft. 2005] [Dai et al., IMA J. Num. Anal. 2006] [Zhuo et al., Comput. Opt. Appl., 2006 ] [Frassoldati et al., J. Ind. Manag. Opt. 2008] Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

21 The BB step-lengths and Scaled Gradient Methods Consider the scaled gradient method: x (k+1) = x (k) α k D k g (k) k = 0, 1,..., where D k is the symmetric positive definite scaling matrix. Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

22 The BB step-lengths and Scaled Gradient Methods Consider the scaled gradient method: x (k+1) = x (k) α k D k g (k) k = 0, 1,..., where D k is the symmetric positive definite scaling matrix. By forcing the quasi-newton properties on B(α k ) = (α k D k ) 1 we have α BB1 k = s(k 1)T D 1 k D 1 k s (k 1)T D 1 k z (k 1) α k BB2 = and s (k 1) s(k 1)T D k z (k 1) z (k 1)T D k D k z (k 1), where s (k 1) = ( x (k) x (k 1)) and z (k 1) = (g (k) g (k 1) ). Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

23 Scaled Gradient Projection (SGP) method: basic notations [Bonettini et al., Inv. Prob. 2009] Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

24 Scaled Gradient Projection (SGP) method: basic notations [Bonettini et al., Inv. Prob. 2009] Scaling matrix: D k D L = {D s.p.d. R n n D L, D 1 L}, L > 1, if D k is diagonal, the requirement leads to: L 1 (D k ) ii L. Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

25 Scaled Gradient Projection (SGP) method: basic notations [Bonettini et al., Inv. Prob. 2009] Scaling matrix: D k D L = {D s.p.d. R n n D L, D 1 L}, L > 1, if D k is diagonal, the requirement leads to: Projection operator: L 1 (D k ) ii L. P Ω,D (x) argmin y Ω x y D, where x D = x T Dx. Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

26 Scaled Gradient Projection (SGP) method Given 0 < α min < α max, β, γ (0, 1) line-search parameters, and fix a positive integer M. Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

27 Scaled Gradient Projection (SGP) method Given 0 < α min < α max, β, γ (0, 1) line-search parameters, and fix a positive integer M. 1. Initialization. Set x (0) Ω, D 0 D L, α 0 [α min, α max ] Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

28 Scaled Gradient Projection (SGP) method Given 0 < α min < α max, β, γ (0, 1) line-search parameters, and fix a positive integer M. 1. Initialization. Set x (0) Ω, D 0 D L, α 0 [α min, α max ] For k = 0, 1, 2,... end Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

29 Scaled Gradient Projection (SGP) method Given 0 < α min < α max, β, γ (0, 1) line-search parameters, and fix a positive integer M. 1. Initialization. Set x (0) Ω, D 0 D L, α 0 [α min, α max ] For k = 0, 1, 2, Projection. y (k) = P Ω,D 1(x (k) α k D k f (x (k) )); k If y (k) = x (k) then stop. end Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

30 Scaled Gradient Projection (SGP) method Given 0 < α min < α max, β, γ (0, 1) line-search parameters, and fix a positive integer M. 1. Initialization. Set x (0) Ω, D 0 D L, α 0 [α min, α max ] For k = 0, 1, 2, Projection. y (k) = P Ω,D 1(x (k) α k D k f (x (k) )); k If y (k) = x (k) then stop. 3. Descent direction. d (k) = y (k) x (k). end Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

31 Scaled Gradient Projection (SGP) method Given 0 < α min < α max, β, γ (0, 1) line-search parameters, and fix a positive integer M. 1. Initialization. Set x (0) Ω, D 0 D L, α 0 [α min, α max ] For k = 0, 1, 2, Projection. y (k) = P Ω,D 1(x (k) α k D k f (x (k) )); k If y (k) = x (k) then stop. 3. Descent direction. d (k) = y (k) x (k). 3. Line-search. Set λ k = 1 and f = max 0 j min{k,m 1} f (x (k j) ) While f (x (k) + λ k d (k) ) > f + γλ k f (x (k) ) T d (k) λ k = βλ k end. Set x (k+1) = x (k) + λ k d (k). end Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

32 Scaled Gradient Projection (SGP) method Given 0 < α min < α max, β, γ (0, 1) line-search parameters, and fix a positive integer M. 1. Initialization. Set x (0) Ω, D 0 D L, α 0 [α min, α max ] For k = 0, 1, 2, Projection. y (k) = P Ω,D 1(x (k) α k D k f (x (k) )); k If y (k) = x (k) then stop. 3. Descent direction. d (k) = y (k) x (k). 3. Line-search. Set λ k = 1 and f = max 0 j min{k,m 1} f (x (k j) ) While f (x (k) + λ k d (k) ) > f + γλ k f (x (k) ) T d (k) λ k = βλ k end. Set x (k+1) = x (k) + λ k d (k). 4. Update. Define D k+1 and α k+1 [α min, α max ]. end Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

33 SGP acceleration techniques The acceleration technique involves: Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

34 SGP acceleration techniques The acceleration technique involves: selection of the step-length α k : general algorithm Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

35 SGP acceleration techniques The acceleration technique involves: selection of the step-length α k : general algorithm definition of the scaling matrix D k : problem dependent (see the experiment section) Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

36 SGP step-length selection Let α min = 10 3, α max = 10 5, M α = 3, τ = 0.5 Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

37 SGP step-length selection Let α min = 10 3, α max = 10 5, M α = 3, τ = 0.5 if s (k 1)T D 1 k z (k 1) 0 if s (k 1)T D k z (k 1) 0 αk BB1 = α max αk BB2 = α max else else α = s(k 1)T D 1 D 1 s (k 1) (k 1) T k k D k z (k 1) end s (k 1)T D 1 z (k 1)T D k D k z (k 1) αk BB1 = min{α max, max{α min, α}} αk BB2 = min{α max, max{α min, α}} k z (k 1) α = s end Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

38 SGP step-length selection Let α min = 10 3, α max = 10 5, M α = 3, τ = 0.5 if s (k 1)T D 1 k z (k 1) 0 if s (k 1)T D k z (k 1) 0 αk BB1 = α max αk BB2 = α max else else α = s(k 1)T D 1 D 1 s (k 1) (k 1) T k k D k z (k 1) end s (k 1)T D 1 z (k 1)T D k D k z (k 1) αk BB1 = min{α max, max{α min, α}} αk BB2 = min{α max, max{α min, α}} k z (k 1) α = s end if αk BB2 /αk BB1 < τ α k = min{αk j BB2, j = 0,..., M α 1} τ = τ 0.9 else α k = αk BB1 τ = τ 1.1 end Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

39 Convergence of SGP min f (x) sub. to x Ω (1) Ω f (x) is a convex and closed set is countinuously differentiable in Ω Theorem Assume that the level set Ω 0 = {x Ω : f (x) f (x (0) )} is bounded. Every accumulation point of the sequence {x (k) } generated by the algorithm SGP is a stationary point of (1). Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

40 Image Deblurring: Poisson noise Object Blurred Noisy image Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

41 Image Deblurring: Poisson noise Object Blurred Noisy image f (x) = D KL (Hx + b, y) = n i=1 ( n j=1 H ijx j + b i y i y i log Ω = {x R n x i 0, i = 1,..., n} P n ) j=1 H ij x j +b i y i A suited reconstruction is obtained by early stopping the SGP iterations. Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

42 Image Deblurring: Poisson noise (II) Algorithms: SGP Adaptive selection of α k, scaling matrix D k = min L, max diag(x (k) ), L 1. EM Richardson-Lucy or Expectation Maximization algorithm. EM_MATLAB deconvlucy function, Matlab image toolbox. WMRNSD Weighted Minimum Residual Norm Steepest Descent [Bardsley-Nagy]. Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

43 Image Deblurring: Poisson noise (II) Algorithms: SGP Adaptive selection of α k, scaling matrix D k = min L, max diag(x (k) ), L 1. EM Richardson-Lucy or Expectation Maximization algorithm. EM_MATLAB deconvlucy function, Matlab image toolbox. WMRNSD Weighted Minimum Residual Norm Steepest Descent [Bardsley-Nagy]. Algorithm it. number l 2 rel. err. time [s] SGP EM EM_MATLAB WMRNSD Test environment: Matlab on an AMD Opteron Dual Core 2.4 GHz processor. Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

44 Image Deblurring: SGP reconstruction Object Blurred Noisy image SGP reconstruction Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

45 Image Denoising: Poisson noise Object Blurred Noisy image Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

46 Image Denoising: Poisson noise Object Blurred Noisy image f (x) = D KL (x, y) + β TV(x) Ω = {x R n x i η, i = 1,..., n} Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

47 Image Denoising: Poisson noise (II) Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

48 Image Denoising: Poisson noise (II) Algorithms: SGP Adaptive selection of α k, scaling matrix D k = x (k) / (1 + βv ), with f (x (k) )) = V U, V i 0 and U i 0. [Lanteri et al., Inv. Prob. 2002] GP Adaptive selection of α k, scaling matrix D k = I. GP-BB Only α k BB1, scaling matrix D k = I. Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

49 Image Denoising: Poisson noise (II) Algorithms: SGP Adaptive selection of α k, scaling matrix D k = x (k) / (1 + βv ), with f (x (k) )) = V U, V i 0 and U i 0. [Lanteri et al., Inv. Prob. 2002] GP Adaptive selection of α k, scaling matrix D k = I. GP-BB Only α k BB1, scaling matrix D k = I. Algorithm it. number l 2 rel. err. time [s] SGP GP GP-BB Test environment: Matlab on an AMD Opteron Dual Core 2.4 GHz processor. [Zanella et al., Inv. Prob. 2009] Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

50 Image Denoising: SGP reconstruction Object Noisy image SGP reconstruction Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

51 An application in medical imaging Object Noisy image SGP reconstruction Image size: , parameters: β = 0.3. Noisy image relative error: 17.9%. Reconstructed image relative error: 2.9%. Computational time: seconds (Matlab on an AMD Opteron Dual Core 2.4 GHz processor). Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

52 GPU Implementation: Deblurring CPU GPU N = n n it. l 2 rel. err. time [s] it. l 2 rel. err. time [s] Speedup C implementation: C-CUDA implementation: Microsoft Visual Studio 2005, AMD Athlon X2 Dual-Core at 3.11GHz. CUDA 2.0, NVIDIA GTX 280, AMD Athlon X2 Dual-Core at 3.11GHz. [Ruggiero et al., J. Global Optim. 2009] Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

53 GPU Implementation: Denoising CPU GPU N = n n it. l 2 rel. err. time [s] it. l 2 rel. err. time [s] Speedup C implementation: C-CUDA implementation: Microsoft Visual Studio 2005, AMD Athlon X2 Dual-Core at 3.11GHz. CUDA 2.0, NVIDIA GTX 280, AMD Athlon X2 Dual-Core at 3.11GHz. [Serafini et al., ParCo 2009] Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

54 Other examples of applications Least-squares minimization: F. Benvenuto: Iterative methods for constrained and regularized least-square problems, M20, 23 July, 15:15-17:15, C2 Sparsity constraints: C. De Mol: Iterative Algorithms for Sparse Recovery, M19, 21 July, 15:15-17:15, C2 Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

55 Conclusions and Future Works Conclusions: by exploiting both the scaling matrix and the Barzilai-Borwein step-length rules, the SGP Method is able to achieve a satisfactory reconstruction in a reasonable time. easy to implement remarkable results in massively parallel architectures (GPU). Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26

56 Conclusions and Future Works Conclusions: by exploiting both the scaling matrix and the Barzilai-Borwein step-length rules, the SGP Method is able to achieve a satisfactory reconstruction in a reasonable time. easy to implement remarkable results in massively parallel architectures (GPU). Works in progress: comparative analysis TV image reconstruction [S. Wright, M39, 23 July, 10:30-12:30, D] Duality-based algorithms [Zhu-Wright, COAP 2008] Primal-dual approach [Zhu-Chan, CAM Rep. UCLA 2008], [Lee-Wright, 2008] Regularized deblurring [G. Landi, M39, 23 July, 10:30-12:30, D] Quasi-Newton approaches [Landi-Loli-Piccolomini, Num. Alg. 2008] Zanella (UniMoRe) Gradient projection methods in imaging AIP / 26