Generalized greedy algorithms.

Transcription

1 Generalized greedy algorithms. François-Xavier Dupé & Sandrine Anthoine LIF & I2M Aix-Marseille Université - CNRS - Ecole Centrale Marseille, Marseille ANR Greta Séminaire Parisien des Mathématiques Appliquées à l'imagerie, 05/01/2017

2 Sparse approximation Sparsity is good, Saturn (original) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

3 Sparse approximation Sparsity is good, Saturn (original) Saturn (coecients) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

4 Sparse approximation Sparsity is good, Saturn (original) Saturn (2.6% of the coecients) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

5 Sparse approximation Classically Find the best k-sparse minimizer of y on the dictionary Φ, min y x R Φx 2 n 2 s.t. x 0 k (A) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

6 Sparse approximation Classically Find the best k-sparse minimizer of y on the dictionary Φ, More generally Find the best k-sparse minimizer, min y x R Φx 2 n 2 s.t. x 0 k (A) min f(x) s.t. x 0 k (P) x H F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

7 Sparse approximation Classically Find the best k-sparse minimizer of y on the dictionary Φ, More generally Find the best k-sparse minimizer, min y x R Φx 2 n 2 s.t. x 0 k (A) min f(x) s.t. x 0 k (P) x H Today Find a k-sparse zero of an operator T : H H (e.g. T = f), { x Find x H s.t 0 k 0 = T(x) (Q) H: Hilbert space. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

8 Literature How to solve these problems? F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

9 Literature How to solve these problems? Linear setting (A), at least four families of methods, convex relaxation; MP, OMP; CoSaMP, SP; IHT, HTP. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

10 Literature How to solve these problems? Linear setting (A), at least four families of methods, convex relaxation; MP, OMP; CoSaMP, SP; IHT, HTP. How to generalize? Convergence? F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

11 Literature Generalization for (P), OMP [Zhang 2011], GraSP [Bahmani et. al. 2013], IHT, HTP [Yuan et. al. 2013]; F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

12 Literature Generalization for (P), OMP [Zhang 2011], GraSP [Bahmani et. al. 2013], IHT, HTP [Yuan et. al. 2013]; Today's talk: Goal solve (P) or/and (Q), Approach greedy - theoretically grounded, - inspired by CoSaMP [Needell, Tropp, 2009] and GraSP [Bahmani et. al., 2013]. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

13 Today's topic 1 CoSaMP and its generalizations 2 The Restricted Diagonal Property 3 Three other generalizations 4 Poisson Noise Removal F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

14 Today's topic 1 CoSaMP and its generalizations CoSaMP and its guarantees GraSP and its guarantees GCoSaMP and its guarantees 2 The Restricted Diagonal Property 3 Three other generalizations Generalized Subspace Pursuit Generalized Hard Thresholding Pursuit Generalized Iterative Hard Thresholding 4 Poisson Noise Removal Moreau-Yosida regularization Experiments F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

15 Optimization problem Let T : H H be an operator k the expected sparsity. We wish to solve Find x H such that T(x) = 0 and x 0 k. (Q) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

16 Optimization problem Let T : H H be an operator k the expected sparsity. We wish to solve Find x H such that T(x) = 0 and x 0 k. (Q) Special case: T = f (Q): nd a critical point of f that is k-sparse. Related problem: nd a minimizer of f among the k-sparse vectors. Related algorithms: T(x) = x ( Φx y 2 2): CoSaMP, SP... T(x) = f(x), f convex: GraSP, GOMP... F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

17 CoSaMP [Needell, Tropp, 2009] Algorithm Require: y, Φ, k. Initialization: x 0 = 0. For t = 0 to N 1, g = Φ (Φx t y), G = supp(g 2k ), S = G supp(x t ), z = Φ S y, T = supp(z k ). x t+1 = z k. Output : x N. Goal: min Φx y 2 x 2 s. t. x 0 k (select new directions) (set extended support) (solve on extended support) (set support) (approximately solve on the support) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

18 CoSaMP [Needell, Tropp, 2009] Algorithm Require: y, Φ, k. Initialization: x 0 = 0. For t = 0 to N 1, g = Φ (Φx t y), G = supp(g 2k ), S = G supp(x t ), z = argmin Φx y 2 2, {x/ supp(x) S} T = supp(z k ). x t+1 = z k. Output : x N. Goal: min Φx y 2 x 2 s. t. x 0 k (select new directions) (set extended support) (solve on extended support) (set support) (approximately solve on the support) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

19 CoSaMP [Needell, Tropp, 2009] Algorithm Require: y, Φ, k. Initialization: x 0 = 0. For t = 0 to N 1, G = supp([φ (Φx t y)] 2k ), Goal: min Φx y 2 x 2 s. t. x 0 k (select new directions) S = G supp(x t ), z = argmin Φx y 2 2, {x/ supp(x) S} T = supp(z k ). x t+1 = z k. Output : x N. (set extended support) (solve on extended support) (set support) (approximately solve on the support) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

20 CoSaMP guarantees Restricted Isometry Property Goal: min Φx y 2 x 2 s. t. x 0 k Φ is has the Restricted Isometry Property with constant δ k if and only if x s.t. card(supp(x)) k, (1 δ k ) x Φx (1 + δ k ) x F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

21 CoSaMP guarantees Restricted Isometry Property Goal: min Φx y 2 x 2 s. t. x 0 k Φ is has the Restricted Isometry Property with constant δ k if and only if x s.t. card(supp(x)) k, (1 δ k ) x Φx (1 + δ k ) x Consider y = Φu + e Theorem (CoSaMP error bound) If Φ is has the RIP with constant δ 4k 0.1, then at iteration t, x t veries x t u 1 u + 20ν, 2t with ν the incompressible error: u u k u k u k 1 + e 2. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

22 Gradient Support Pursuit [Bahmani et. al., 2013] CoSaMP Require: f, k. Initialization: x 0 = 0. For t = 0 to N 1, G = supp([φ (Φx t y)] 2k ), S = G supp(x t ), z = argmin Φx y 2 2, {x/ supp(x) S} T = supp(z k ). x t+1 = z k. Output : x N. Goal: min f(x) s. t. x x 0 k (select new directions) (set extended support) (solve on extended support) (set support) (approximately solve on the support) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

23 Gradient Support Pursuit [Bahmani et. al., 2013] Goal: min f(x) s. t. x x 0 k Gradient Support Pursuit (GraSP) Require: f, k. Initialization: x 0 = 0. For t = 0 to N 1, G = supp([ f(x)] 2k ), S = G supp(x t ), z argmin {x/ supp(x) S} f(x), T = supp(z k ). x t+1 = z k. Output : x N. (select new directions) (set extended support) (solve on extended support) (set support) (approximately solve on the support) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

24 GraSP guarantees Goal: min f(x) s. t. x x 0 k Stable Restricted Hessian f has a Stable Restricted Hessian with constant µ k if and only if A k (x) B k (x) µ k, x s.t. card(supp(x)) k, where { } y, H f (x)y A k (x) = sup y 2 card(supp(x) supp(y)) k, 2 { } y, H f (x)y B k (x) = inf y 2 card(supp(x) supp(y)) k. 2 F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

25 GraSP guarantees Goal: min f(x) s. t. x x 0 k Theorem (GraSP error bound) f has a Stable Restricted Hessian with constant µ 4k and there exists ɛ > 0 such that B 4k (u) > ɛ u, then at iteration t, x t veries x t u 1 2 t u + C ɛ f(u ) 3k. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

26 GraSP guarantees Goal: min f(x) s. t. x x 0 k Theorem (GraSP error bound) f has a Stable Restricted Hessian with constant µ 4k and there exists ɛ > 0 such that B 4k (u) > ɛ u, then at iteration t, x t veries x t u 1 2 t u + C ɛ f(u ) 3k. Error bound for f only once dierentiable. Notion of convexity restricted to k-sparse vectors. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

27 Generalized CoSaMP Goal: Find x H s. t. T(x) = 0 and x 0 k Gradient Support Pursuit (GraSP) Require: T, k. Initialization: x 0 = 0. For t = 0 to N 1, G = supp([ f(x)] 2k ), S = G supp(x t ), (select new directions) (set extended support) z argmin {x/ supp(x) S} f(x), (solve on extended support) T = supp(z k ). (set support) x t+1 = z k. Output : x N. (approximately solve on the support) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

28 Generalized CoSaMP Goal: Find x H s. t. T(x) = 0 and x 0 k GCoSaMP Require: T, k. Initialization: x 0 = 0. For t = 0 to N 1, G = supp([t(x)] 2k ), S = G supp(x t ), { supp(z) S z s. t. T(z) S = 0. T = supp(z k ). x t+1 = z k. Output : x N. (select new directions) (set extended support), (solve on extended support) (set support) (approximately solve on the support) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

29 Uniform Restricted Diagonal Property H H, D 1 = D : x i d i x i e i s.t. x Dx x. Denition (Uniform Restricted Diagonal Property) T is said to have the Uniform Restricted Diagonal Property (URDP) of order k if there exists α k > 0 and a diagonal operator D k in D 1 such that (x, y) H 2, card(supp(x) supp(y)) k T(x) T(y) D k (x y) α k x y. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

30 Uniform Restricted Diagonal Property H H, D 1 = D : x i d i x i e i s.t. x Dx x. Denition (Restricted Diagonal Property) T is said to have the Restricted Diagonal Property (RDP) of order k if there exists α k > 0 such that for all subsets S of N of cardinal at most k, there exists a diagonal operator D S in D 1 such that (x, y) H 2, supp(x) S supp(y) S } T(x) T(y) D S (x y) α k x y. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

31 Generalized CoSaMP Goal: Find x H s. t. T(x) = 0 and x 0 k Theorem (Generalized CoSaMP error bound) Denote by x any k-sparse vector and α C = If there exists ρ > 0 such that ρt has the Restricted Diagonal Property of order 4k with α 4k α C then at iteration t, x t veries x t x 1 2 t x + 12ρ T(x ) 3k. (1) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

32 Today's topic 1 CoSaMP and its generalizations CoSaMP and its guarantees GraSP and its guarantees GCoSaMP and its guarantees 2 The Restricted Diagonal Property 3 Three other generalizations Generalized Subspace Pursuit Generalized Hard Thresholding Pursuit Generalized Iterative Hard Thresholding 4 Poisson Noise Removal Moreau-Yosida regularization Experiments F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

33 The Restricted Diagonal Property and RIP Denition (Restricted Diagonal Property) T has RDP of order k : α k > 0, S N, with S k, D S D 1 such that supp(x) S & supp(y) S T(x) T(y) D S (x y) α k x y. Assume that T has RDP of order 2k, then if x 0 k and y 0 k: T(x) T(y) (1 α 2k ) x y (injectivity) F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

34 The Restricted Diagonal Property and RIP Denition (Restricted Diagonal Property) T has RDP of order k : α k > 0, S N, with S k, D S D 1 such that supp(x) S & supp(y) S T(x) T(y) D S (x y) α k x y. Assume that T has RDP of order 2k, then if x 0 k and y 0 k: T(x) T(y) (1 α 2k ) x y (injectivity) (1 α 2k ) x y T(x) T(y) (D + α 2k ) x y (D = D 2k or sup D S if exists). F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

35 The Restricted Diagonal Property and RIP Denition (Restricted Diagonal Property) T has RDP of order k : α k > 0, S N, with S k, D S D 1 such that supp(x) S & supp(y) S T(x) T(y) D S (x y) α k x y. Assume that T has RDP of order 2k, then if x 0 k and y 0 k: T(x) T(y) (1 α 2k ) x y (injectivity) (1 α 2k ) x y T(x) T(y) (D + α 2k ) x y (D = D 2k or sup D S if exists). T(x) = Φ (Φx z) Φ is RIP. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

36 Uniform Restricted Diagonal Property: characterization Theorem βt is URDP of order k for D, with α k < 1 and β > 0 (m, L) such that 0 < m and 0 D 2 m2 L 2 < 1 and supp(x) supp(y) k { T(x) T(y) L x y T(x) T(y), D(x y) m x y 2. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

37 Uniform Restricted Diagonal Property: characterization Theorem βt is URDP of order k for D, with α k < 1 and β > 0 (m, L) such that 0 < m and 0 D 2 m2 L 2 < 1 and supp(x) supp(y) k { T(x) T(y) L x y T(x) T(y), D(x y) m x y 2. L-Lipschitz property on sparse elements. D = I: monotone operator. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

38 Uniform Restricted Diagonal Property: characterization Theorem β f is URDP of order k for D, with α k < 1 and β > 0 (m, L) such that 0 < m and 0 D 2 m2 L 2 < 1 and supp(x) supp(y) k { f(x) f(y) L x y f(x) f(y), D(x y) m x y 2. L-Lipschitz property on sparse elements. D = I: monotone operator. If T = f L-Lipschitz property: Restricted Strong Smoothness. if D = I: Restricted Strong Convexity recovers the conditions of [Bahmani et. al., 2013]. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

39 Today's topic 1 CoSaMP and its generalizations CoSaMP and its guarantees GraSP and its guarantees GCoSaMP and its guarantees 2 The Restricted Diagonal Property 3 Three other generalizations Generalized Subspace Pursuit Generalized Hard Thresholding Pursuit Generalized Iterative Hard Thresholding 4 Poisson Noise Removal Moreau-Yosida regularization Experiments F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

40 Generalized Subspace Pursuit GCoSaMP Require: T, k. GSP Require: T, k. Initialization: x 0 = 0. Initialization: x 0 = 0. For t = 0 to N 1, G = supp([t(x)] 2k ), For t = 0 to N 1, G = supp([t(x)] k ), S = G supp(x t ), S = G supp(x t ), z s. t. { supp(z) S T(z) S = 0. T = supp(z k ). x t+1 = z k. Output : x N., { supp(z) S z s. t., T(z) S = 0. T = supp(z k ). { supp(x x t+1 s. t. t+1 ) T T(x t+1 ) S = 0. Output : x N.. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

41 x is k-sparse. α S is the real root of x 3 + x 2 + 7x 1. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38 Generalized Subspace Pursuit Theorem (Generalized CoSaMP error bound) If ρt has the Restricted Diagonal Property of order 4k with α 4k α C = then at iteration t of GCoSaMP, x t veries x t x 1 2 t x + 12ρ T(x ) 3k. Theorem (Generalized Subspace Pursuit error bound) If ρt has the Restricted Diagonal Property of order 3k with α 3k α S then at iteration t of GSP, x t veries x t x 1 2 x + 12ρ T(x ) t 2k.

42 Generalized Hard Thresholding Pursuit GCoSaMP GHTP Require: T, k. Require: T, k, η. Initialization: x 0 = 0. Initialization: x 0 = 0. For t = 0 to N 1, For t = 0 to N 1, G = supp([t(x)] 2k ), G = supp([t(x)] k ), S = G supp(x t ), S = G supp(x t ), z s. t. { supp(z) S T(z) S = 0. T = supp(z k ). x t+1 = z k. Output : x N., z = [ (I ηt)(x t ) ] S, T = supp(z k ). { supp(x x t+1 s. t. t+1 ) T T(x t+1 ) S = 0. Output : x N.. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

43 Generalized Hard Thresholding Pursuit Theorem (Generalized CoSaMP error bound) If ρt has the Restricted Diagonal Property of order 4k with α 4k α C = then at iteration t of GCoSaMP, x t veries x t x 1 2 t x + 12ρ T(x ) 3k. Theorem (Generalized HTP error bound) If T has the Uniform Restricted Diagonal Property of order 2k with D 2k = I, α 2k α H and 3 4 < η < 5 4, then at iteration t of GHTP, xt veries x t x 1 2 x + 2 (1+2η)(1 α 2k)+4 T(x t (1 α 2k ) ) 2 2k. (2) x is k-sparse. α H = F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

44 Generalized Iterative Hard Thresholding GCoSaMP Require: T, k. Initialization: x 0 = 0. For t = 0 to N 1, G = supp([t(x)] 2k ), S = G supp(x t ), { supp(z) S z s. t. T(z) S = 0. T = supp(z k ). x t+1 = z k. Output : x N., GIHT Require: T, k, η. Initialization: x 0 = 0. For t = 0 to N 1, z = (I ηt)(x t ), T = supp(z k ). x t+1 = z k. Output : x N. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

45 Generalized Iterative Hard Thresholding Theorem (Generalized CoSaMP error bound) If ρt has the Restricted Diagonal Property of order 4k with α 4k α C = then at iteration t of GCoSaMP, x t veries x t x 1 2 t x + 12ρ T(x ) 3k. Theorem (Generalized IHT error bound) If T has the Uniform Restricted Diagonal Property of order 2k with D 2k = I, 3 4 < η < 5 4 and α 2k α η then at iteration t of GIHT, x t veries x t x 1 2 t x + 4η T(x ) 3k. x is k-sparse. α η = 1 4 η 1 4(1+ η 1 ). F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

46 About these error bounds For all four algorithms Guaranteed convergence to unique solution if it exists, at an exponential rate. Incompressible error of the form T (x) k. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

47 About these error bounds For all four algorithms Guaranteed convergence to unique solution if it exists, at an exponential rate. Incompressible error of the form T (x) k. For GCoSaMP and GSP Invariance to scaling of the algorithm, no parameters to set. Guarantee in the RDP case (no monotonicity of T or convexity of f required). F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

48 About these error bounds For all four algorithms Guaranteed convergence to unique solution if it exists, at an exponential rate. Incompressible error of the form T (x) k. For GCoSaMP and GSP Invariance to scaling of the algorithm, no parameters to set. Guarantee in the RDP case (no monotonicity of T or convexity of f required). For GHTP and GIHT Requires careful setting of the step η w.r.t to T. Guarantee only in the URDP case with D = I (monotonicity of T or convexity of f required). F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

49 Today's topic 1 CoSaMP and its generalizations CoSaMP and its guarantees GraSP and its guarantees GCoSaMP and its guarantees 2 The Restricted Diagonal Property 3 Three other generalizations Generalized Subspace Pursuit Generalized Hard Thresholding Pursuit Generalized Iterative Hard Thresholding 4 Poisson Noise Removal Moreau-Yosida regularization Experiments F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

50 Forward model The observed data y is a Poisson noise corrupted version of x, y P(x). Sparsity assumption x = Φα with α 0 k and k << n. with y the observation (e.g. a n n image), x the true image, Φ R n d a dictionary (redundant if d > n), α coecients to be found (x = Φα). F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

51 Regularized Sparse Poisson denoising Writing F y (x) = log P (y x), one naturally seeks to Minimize the neg-log-likelihood under a sparsity constraint F y : x R n min F y(φ(α)) s. t. α 0 k. α R d n fy(x[i]), i with i=1 y[i] log(ξ) + ξ if y[i] > 0 and ξ > 0, fy(ξ) i = ξ if y[i] = 0 and ξ 0, + otherwise. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

52 Regularized Sparse Poisson denoising Writing F y (x) = log P (y x), one naturally seeks to Minimize the neg-log-likelihood under a sparsity constraint F y : x R n Issues : min F y(φ(α)) s. t. α 0 k. α R d n fy(x[i]), i with i=1 y[i] log(ξ) + ξ if y[i] > 0 and ξ > 0, fy(ξ) i = ξ if y[i] = 0 and ξ 0, + otherwise. no full domain, no tolerance if a pixel is removed (set to 0), non-lipschitz gradient. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

53 Regularized Sparse Poisson denoising Instead, we propose Min. a regularized neg-log-likelihood under a sparsity constraint which is equivalent to Using GCoSaMP/GSP with min M λ,f y (Φ(α)) s. t. α 0 k. α R d T = 1 νλ Φ (I prox νλfy ) Φ M λ,f is the Moreau-Yosida regularization of f with parameter λ. prox f is the proximal operator. ν is the frame bound for Φ. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

54 Experiments We compare using GSP, GCoSaMP, GHTP or GIHT to solve min M λ,f y Φ(α) s. t. α 0 k, α R d F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

55 Experiments We compare using GSP, GCoSaMP, GHTP or GIHT to solve min M λ,f y Φ(α) s. t. α 0 k, α R d to Subspace Pursuit (SP):[Dai and Milenkovic, 2009] min Φ(α) y 2 α R d 2 s. t. α 0 k, l 1 -relaxation: using Forward-Backward-Forward primal-dual algorithm [Combettes et. al. 2012] to solve min F y Φ(α) + γ α 1, α R d SAFIR: adaptation of BM3D [Boulanger et al., 2010]. MSVST: variance-stabilizing method [Zhang et al., 2008]. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

56 Comparing greedy l 0 and l 1 results (a) Original (b) Noisy (c) GSP (d) GHTP (e) GCoSaMP Figure : Cameraman, maximal intensity 30, undecimated wavelet transform. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

57 Comparing greedy l 0 and l 1 results (c) GSP (d) GHTP (e) GCoSaMP (f) GIHT (g) l 1 method Figure : Cameraman, maximal intensity 30, undecimated wavelet transform. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

58 Inuence of λ in M λ, Original Noisy λ = 10, MAE=0.81 λ = 0.1, MAE=0.74 Maximal Intensity: 10, k: 1500, Φ: cycle-spinning wavelet transform. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

59 Results (a) Original (b) Noisy (c) GSP (d) SP (e) l 1 -relaxation Figure : NGC 2997 Galaxy image. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

60 Results (c) GSP (d) SP (e) l 1 -relaxation (f) SAFIR (g) MSVST Figure : NGC 2997 Galaxy image. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

61 Numerical results Sparse Cameraman Galaxy MAE SSIM MAE SSIM Noisy GSP SP SAFIR MSVST l 1-relaxation Table : Comparison of denoising methods on a sparse version of Cameraman (k/n = 0.15) and the NGC 2997 Galaxy. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

62 Take away messages We have presented a general greedy optimization algorithms, with theoretical guarantees; applied it to a Moreau-Yosida regularization of Poisson likelihood; shown encouraging results for Poisson denoising. Perspectives includes application in other known settings such as dictionary learning, additional regularization... gain understanding on the behavior of GCoSaMP, GSP and co. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

63 Thanks for your attention. Any questions? F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

64 Moreau-Yosida regularization Denition (Lemaréchal et al, 1997) Let f : R d R {+ } be a proper, convex and lower semi-continuous function. The Moreau-Yosida regularization of f with parameter λ is: M λ,f : R d R, [ s inf 1 x R d 2λ s x 2 + f(x) ] F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

65 Moreau-Yosida regularization Denition (Lemaréchal et al, 1997) Let f : R d R {+ } be a proper, convex and lower semi-continuous function. The Moreau-Yosida regularization of f with parameter λ is: Remarks: M λ,f : R d R, [ s inf 1 x R d 2λ s x 2 + f(x) ] The gradient of M λ,f is linked with proximal operator. λ regulates the similarity between f and M λ,f. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

66 Gradient for the Moreau-Yosida regularization of Poisson likelihood Proposition (Combettes and Pesquet, 2007) Let Φ is a tight frame (i.e. ν > 0, such that Φ Φ = νi), then the gradient of the Moreau-Yosida regularization of F y Φ is: M λ,fy Φ(x) = 1 νλ Φ (I prox νλfy ) Φ with prox νλfy (x)[i] = x[i] νλ + x[i] νλ 2 + 4νλy[i] 2 F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

67 Optimization problem with, Φ the dictionary, min M λ,f y Φ(α) s. t. α 0 k (P), α R d λ the regularization parameter, k sought sparsity. F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38

68 Optimization problem with, Φ the dictionary, min M λ,f y Φ(α) s. t. α 0 k (P), α R d λ the regularization parameter, k sought sparsity. The bias between the solution of (P) and the non-regularized problem, under mild condition, is O( λ). [Mahey et Tao, 1993] F-X Dupé, S. Anthoine (LIF, I2M) Generalized greedy algorithms. 05/01/ / 38