Basis Pursuit Denoising and the Dantzig Selector

Transcription

1 BPDN and DS p. 1/16 Basis Pursuit Denoising and the Dantzig Selector West Coast Optimization Meeting University of Washington Seattle, WA, April 28 29, 2007 Michael Friedlander and Michael Saunders Dept of Computer Science Dept of Management Sci & Eng University of British Columbia Stanford University Vancouver, BC V6K 2C6 Stanford, CA

2 BPDN and DS p. 2/16 Abstract Many imaging and compressed sensing applications seek sparse solutions to under-detered least-squares problems. The Lasso and Basis Pursuit Denoising (BPDN) approaches of bounding the 1-norm of the solution have led to several computational algorithms.

3 BPDN and DS p. 2/16 Abstract Many imaging and compressed sensing applications seek sparse solutions to under-detered least-squares problems. The Lasso and Basis Pursuit Denoising (BPDN) approaches of bounding the 1-norm of the solution have led to several computational algorithms. PDCO uses an interior method to handle general linear constraints and bounds. Homotopy, LARS, OMP, and STOMP are specialized active-set methods for handling the implicit bounds. l1 ls and GPSR are further recent entries in the l1-regularized least-squares competition, both based on bound-constrained optimization.

4 BPDN and DS p. 2/16 Abstract Many imaging and compressed sensing applications seek sparse solutions to under-detered least-squares problems. The Lasso and Basis Pursuit Denoising (BPDN) approaches of bounding the 1-norm of the solution have led to several computational algorithms. PDCO uses an interior method to handle general linear constraints and bounds. Homotopy, LARS, OMP, and STOMP are specialized active-set methods for handling the implicit bounds. l1 ls and GPSR are further recent entries in the l1-regularized least-squares competition, both based on bound-constrained optimization. The Dantzig Selector of Candes and Tao is promising in its production of sparse solutions using only linear programg. Again, interior or active-set (simplex) methods may be used. We compare the BPDN and DS approaches via their dual problems and some numerical examples.

5 BPDN and DS p. 3/16 Sparse x Lasso(t) Tibshirani 1996 x 1 2 b Ax 2 2 s.t. x 1 t A = explicit

6 BPDN and DS p. 3/16 Sparse x Lasso(t) Tibshirani 1996 x 1 2 b Ax 2 2 s.t. x 1 t A = explicit Basis Pursuit Chen, Donoho & S 2001 x x 1 s.t. Ax = b A = fast operator

7 BPDN and DS p. 3/16 Sparse x Lasso(t) Tibshirani 1996 x 1 2 b Ax 2 2 s.t. x 1 t A = explicit BPDN(λ) Chen, Donoho & S 2001 x 1 2 b Ax λ x 1 A = fast operator

8 BPDN and DS p. 4/16 BP and BPDN Algorithms OMP Davis, Mallat et al 1997 Greedy BPDN-interior Chen, Donoho & S, 1998 Interior, CG PDSCO, PDCO S 1997, 2002 Interior, LSQR BCR Sardy, Bruce & Tseng 2000 Orthogonal blocks Homotopy Osborne et al 2000 Active-set, all λ LARS Efron, Hastie et al 2004 Active-set, all λ STOMP Donoho, Tsaig et al 2006 Double greedy l1 ls Kim, Koh et al 2007 Primal barrier, PCG GPSR Figueiredo, Nowak & Wright 2007 Gradient-projection

9 BPDN and DS p. 5/16 Basis Pursuit Denoising (BPDN) Chen, Donoho and S 1998 Pure LS x,r 1 2 r 2 2 s.t. r = b Ax

10 BPDN and DS p. 5/16 Basis Pursuit Denoising (BPDN) Chen, Donoho and S 1998 Pure LS x,r 1 2 r 2 2 s.t. r = b Ax If x not unique, need regularization

11 BPDN and DS p. 5/16 Basis Pursuit Denoising (BPDN) Chen, Donoho and S 1998 Pure LS x,r 1 2 r 2 2 s.t. r = b Ax If x not unique, need regularization BPDN(λ) x,r λ x r 2 2 s.t. r = b Ax

12 BPDN and DS p. 5/16 Basis Pursuit Denoising (BPDN) Chen, Donoho and S 1998 Pure LS x,r 1 2 r 2 2 s.t. r = b Ax If x not unique, need regularization BPDN(λ) x,r λ x r 2 2 s.t. r = b Ax smaller x 1 bigger r 2

13 BPDN and DS p. 6/16 The Dantzig Selector (DS) Candès and Tao 2007 Pure LS A T r = 0, r = b Ax

14 BPDN and DS p. 6/16 The Dantzig Selector (DS) Candès and Tao 2007 Pure LS A T r = 0, r = b Ax Plausible regularization x,r x 1 s.t. A T r = 0, r = b Ax

15 BPDN and DS p. 6/16 The Dantzig Selector (DS) Candès and Tao 2007 Pure LS A T r = 0, r = b Ax Plausible regularization x,r x 1 s.t. A T r = 0, r = b Ax DS(λ) x,r x 1 s.t. A T r λ, r = b Ax

16 BPDN and DS p. 6/16 The Dantzig Selector (DS) Candès and Tao 2007 Pure LS A T r = 0, r = b Ax Plausible regularization x,r x 1 s.t. A T r = 0, r = b Ax DS(λ) x,r x 1 s.t. A T r λ, r = b Ax smaller x 1 bigger A T r

17 BPDN and DS p. 7/16 BP Denoising and the Dantzig Selector Dual problems BPDN(λ) x,r λ x r 2 2 s.t. r = b Ax QP

18 BPDN and DS p. 7/16 BP Denoising and the Dantzig Selector Dual problems BPDN(λ) x,r λ x r 2 2 s.t. r = b Ax QP DS(λ) x,r x 1 s.t. A T r λ, r = b Ax LP

19 BPDN and DS p. 7/16 BP Denoising and the Dantzig Selector Dual problems BPDN(λ) x,r λ x r 2 2 s.t. r = b Ax QP DS(λ) x,r x 1 s.t. A T r λ, r = b Ax LP BPdual(λ) r b T r r 2 2 s.t. A T r λ

20 BPDN and DS p. 7/16 BP Denoising and the Dantzig Selector Dual problems BPDN(λ) x,r λ x r 2 2 s.t. r = b Ax QP DS(λ) x,r x 1 s.t. A T r λ, r = b Ax LP BPdual(λ) r b T r r 2 2 s.t. A T r λ DSdual(λ) r,z b T r + λ z 1 s.t. A T r λ, r = Az

21 BPDN and DS p. 8/16 BPDN(λ) implementation Chen, Donoho & S 1998 v,w,r s.t. λ1 T (v + w) rt r [ ] A A v + r = b, v, w 0 w 2007: Apply PDCO (Matlab primal-dual interior method)

22 BPDN and DS p. 8/16 BPDN(λ) implementation Chen, Donoho & S 1998 v,w,r s.t. λ1 T (v + w) rt r [ ] A A v + r = b, v, w 0 w 2007: Apply PDCO (Matlab primal-dual interior method) Dense A in test problems Dense Cholesky Double-handling of A [ A ] A D 1 D 2 AT A T could be coded as A(D 1 + D 2 )A T

23 BPDN and DS p. 9/16 DS(λ) implementation Candès and Tao 2007 x,u 1 T u s.t. u x u, λ1 A T (b Ax) λ1, Apply l1dantzig pd (Matlab primal-dual interior method) Romberg 2005 Part of l 1 -magic Candes 2006

24 BPDN and DS p. 9/16 DS(λ) implementation Candès and Tao 2007 x,u 1 T u s.t. u x u, λ1 A T (b Ax) λ1, Apply l1dantzig pd (Matlab primal-dual interior method) Romberg 2005 Part of l 1 -magic Candes 2006 Dense A in test problems Dense Cholesky on A T (AD 1 A T )A + D 2 (much bigger) + D 2

25 BPDN and DS p. 10/16 Two other DS LP implementations Introduce s = A T r DS1 Interior v,w,s s.t. 1 T (v + w) [ ] A T A A T A I w v = A T b, v, w 0, s λ s

26 BPDN and DS p. 10/16 Two other DS LP implementations Introduce s = A T r DS1 Interior v,w,s s.t. 1 T (v + w) [ ] A T A A T A I w v = A T b, v, w 0, s λ s DS2 Interior, Simplex v,w,r,s 1T (v + w) s.t. A A I A T I v w rs = b, v, w 0, s λ 0

27 A, b depend on dimensions m, n, T Test data rand ( state,0); % initialize generators randn( state,0); x = zeros(n,1); % random +/-1 signal q = randperm(m); x(q(1:t)) = sign(randn(t,1)); [A,R] = qr(randn(n,m),0); A = A ; % m x n measurement mtx sigma = 0.005; b = A*x + sigma*randn(m,1); % noisy observations A dense AA T = I T components x j ±1 For example m = 500 n = 2000 T = 80 λ = 3e-3 BPDN and DS p. 11/16

28 DS vs BPDN DS with interior methods vs BPDN interior and greedy DS2 pdco DS2 cplex barrier DS1 pdco DS l1magic BPDN pdco BPDN greedy 800 secs m (n > 4m) BPDN and DS p. 12/16

29 BPDN and DS p. 13/16 CPLEX dual simplex on DS2 with loose and tight tols sizes tol = 0.1 tol = m n T itns time S itns time S Too many simplex iterations, too many x j 0

30 More small values more simplex iterations and more time per iteration BPDN and DS p. 14/16 CPLEX dual simplex on DS2 Large and small x j 1.0 Dual simplex, tol = 0.1 Dual simplex, tol =

31 BPDN and DS p. 15/16 References E. Candès, l1 -magic, E. Candès and T. Tao, The Dantzig selector: Statistical estimation when p n, Annals of Statistics, to appear (2007). S. S. Chen, D. L. Donoho, and M. A. Saunders, Atomic decomposition by basis pursuit, SIAM Review, 43 (2001). B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, Least angle regression, Ann. Statist., 32 (2004). M. R. Osborne, B. Presnell, and B. A. Turlach, The Lasso and its dual, J. of Computational and Graphical Statistics, 9 (2000). M. R. Osborne, B. Presnell, and B. A. Turlach, A new approach to variable selection in least squares problems, IMA J. of Numerical Analysis, 20 (2000). M. A. Saunders, PDCO. Matlab software for convex optimization, R. Tibshirani, Regression shrinkage and selection via the lasso, J. Roy. Statist. Soc. Ser. B, 58 (1996). Y. Tsaig, Sparse Solution of Underdetered Linear Systems: Algorithms and Applications, PhD thesis, Stanford University, 2007.

32 BPDN and DS p. 16/16 Many thanks Michael

33 BPDN and DS p. 16/16 Many thanks Michael and Jim, Terry, Paul