Reconstruction of Block-Sparse Signals by Using an l 2/p -Regularized Least-Squares Algorithm

Transcription

1 Reconstruction of Block-Sparse Signals by Using an l 2/p -Regularized Least-Squares Algorithm Jeevan K. Pant, Wu-Sheng Lu, and Andreas Antoniou University of Victoria May 21, 2012 Compressive Sensing 1/23 University of Victoria

2 Outline Compressive Sensing Compressive Sensing 2/23 University of Victoria

3 Outline Compressive Sensing Signal Recovery by Using l 1 or l p Minimization Compressive Sensing 2/23 University of Victoria

4 Outline Compressive Sensing Signal Recovery by Using l 1 or l p Minimization Recovery of Block-Sparse Signals Compressive Sensing 2/23 University of Victoria

5 Outline Compressive Sensing Signal Recovery by Using l 1 or l p Minimization Recovery of Block-Sparse Signals Block-Sparse Signal Recovery by Using l 2/p Minimization Compressive Sensing 2/23 University of Victoria

6 Outline Compressive Sensing Signal Recovery by Using l 1 or l p Minimization Recovery of Block-Sparse Signals Block-Sparse Signal Recovery by Using l 2/p Minimization Performance Evaluation Compressive Sensing 2/23 University of Victoria

7 Outline Compressive Sensing Signal Recovery by Using l 1 or l p Minimization Recovery of Block-Sparse Signals Block-Sparse Signal Recovery by Using l 2/p Minimization Performance Evaluation Conclusions Compressive Sensing 2/23 University of Victoria

8 Compressive Sensing A signal x(n) of length N is K-sparse if it contains K nonzero components with K N. Compressive Sensing 3/23 University of Victoria

9 Compressive Sensing A signal x(n) of length N is K-sparse if it contains K nonzero components with K N. A signal is near K-sparse if it contains K significant components. Compressive Sensing 3/23 University of Victoria

10 Compressive Sensing A signal x(n) of length N is K-sparse if it contains K nonzero components with K N. A signal is near K-sparse if it contains K significant components. Example: an image with near sparse wavelet coefficients: An image of An equivalent Wavelet coefficients Lena 1-D signal Compressive Sensing 3/23 University of Victoria

11 Compressive Sensing, cont d Compressive sensing (CS) is a data acquisition process whereby a sparse signal x(n) represented by a vector x of length N is determined using a small number of projections represented by a matrix Φ of dimension M N. Compressive Sensing 4/23 University of Victoria

12 Compressive Sensing, cont d Compressive sensing (CS) is a data acquisition process whereby a sparse signal x(n) represented by a vector x of length N is determined using a small number of projections represented by a matrix Φ of dimension M N. In such a process, measurement vector y and signal vector x are interrelated by the equation y = Φ x = 16 measurements projection matrix 4-sparse signal of size of length 30 Compressive Sensing 4/23 University of Victoria

13 Signal Recovery by Using l 1 or l p Minimization The inverse problem of recovering signal x from measurement y such that Φ x = y M N N 1 M 1 is an ill-posed problem. Compressive Sensing 5/23 University of Victoria

14 Signal Recovery by Using l 1 or l p Minimization The inverse problem of recovering signal x from measurement y such that Φ x = y M N N 1 M 1 is an ill-posed problem. l 2 minimization often fails to yield a sparse x, i.e., a signal obtained as x = arg x is often not sparse. min x 2 subject to Φx = y Compressive Sensing 5/23 University of Victoria

15 Signal Recovery by Using l 1 or l p Minimization The inverse problem of recovering signal x from measurement y such that Φ x = y M N N 1 M 1 is an ill-posed problem. l 2 minimization often fails to yield a sparse x, i.e., a signal obtained as x = arg x is often not sparse. min x 2 subject to Φx = y A sparse x can be recovered using l 1 minimization as x = arg x min x 1 subject to Φx = y Compressive Sensing 5/23 University of Victoria

16 Signal Recovery by Using l 1 or l p Minimization, cont d Recently, l p minimization based algorithms have been shown to recover sparse signals using fewer measurements. In these algorithms, the signal is recovered by using the optimization problem where p < 1. minimize x subject to x p p = N i=1 x i p Φx = y Compressive Sensing 6/23 University of Victoria

17 Signal Recovery by Using l 1 or l p Minimization, cont d Recently, l p minimization based algorithms have been shown to recover sparse signals using fewer measurements. In these algorithms, the signal is recovered by using the optimization problem where p < 1. minimize x subject to x p p = N i=1 x i p Φx = y Note that the objective function x p p in the above problem is nonconvex and nondifferentiable. Compressive Sensing 6/23 University of Victoria

18 Signal Recovery by Using l 1 or l p Minimization, cont d Recently, l p minimization based algorithms have been shown to recover sparse signals using fewer measurements. In these algorithms, the signal is recovered by using the optimization problem where p < 1. minimize x subject to x p p = N i=1 x i p Φx = y Note that the objective function x p p in the above problem is nonconvex and nondifferentiable. Despite this, it has been shown in the literature that if the above problem is solved with sufficient care, improved reconstruction performance can be achieved. Compressive Sensing 6/23 University of Victoria

19 Signal Recovery by Using l 1 or l p Minimization, cont d Example: N = 256, K = 35, M = 100. Compressive Sensing 7/23 University of Victoria

20 Recovery of Block-Sparse Signals Let N, d, and N/d be positive integers such that d < N and N/d < N. Compressive Sensing 8/23 University of Victoria

21 Recovery of Block-Sparse Signals Let N, d, and N/d be positive integers such that d < N and N/d < N. A signal x of length N can be divided into N/d blocks as x = [ x 1 x 2 x N/d ] T where x i = [ x (i 1)d+1 x (i 1)d+1 x (i 1)d+d ] T for i = 1, 2,..., N/d. Compressive Sensing 8/23 University of Victoria

22 Recovery of Block-Sparse Signals Let N, d, and N/d be positive integers such that d < N and N/d < N. A signal x of length N can be divided into N/d blocks as x = [ x 1 x 2 x N/d ] T where x i = [ x (i 1)d+1 x (i 1)d+1 x (i 1)d+d ] T for i = 1, 2,..., N/d. Signal x is said to be K-block sparse if it has K nonzero blocks with K N/d. Compressive Sensing 8/23 University of Victoria

23 Recovery of Block-Sparse Signals Let N, d, and N/d be positive integers such that d < N and N/d < N. A signal x of length N can be divided into N/d blocks as x = [ x 1 x 2 x N/d ] T where x i = [ x (i 1)d+1 x (i 1)d+1 x (i 1)d+d ] T for i = 1, 2,..., N/d. Signal x is said to be K-block sparse if it has K nonzero blocks with K N/d. Note that the definition of K-sparse in the conventional CS is the special case of K-block sparse with d = 1. Compressive Sensing 8/23 University of Victoria

24 Recovery of Block-Sparse Signals, cont d Block-sparsity naturally arises in various signals such as speech signals, multiband signals, and some images. Compressive Sensing 9/23 University of Victoria

25 Recovery of Block-Sparse Signals, cont d Block-sparsity naturally arises in various signals such as speech signals, multiband signals, and some images. Speech signal: Compressive Sensing 9/23 University of Victoria

26 Recovery of Block-Sparse Signals, cont d Block-sparsity naturally arises in various signals such as speech signals, multiband signals, and some images. Speech signal: Multiband spectrum: Compressive Sensing 9/23 University of Victoria

27 Recovery of Block-Sparse Signals, cont d An image of Jupiter: An image of Jupiter An equivalent 1-D signal The 1-D signal from n = to Compressive Sensing 10/23 University of Victoria

28 Recovery of Block-Sparse Signals, cont d The block sparsity of a signal can be measured using the l 2/0 -pseudonorm which is given by N/d x 2/0 = ( x i 2 ) 0 i=1 Compressive Sensing 11/23 University of Victoria

29 Recovery of Block-Sparse Signals, cont d The block sparsity of a signal can be measured using the l 2/0 -pseudonorm which is given by N/d x 2/0 = ( x i 2 ) 0 The value of function x 2/0 is equal to the number of blocks of x which have all-nonzero components. i=1 Compressive Sensing 11/23 University of Victoria

30 Recovery of Block-Sparse Signals, cont d The block sparsity of a signal can be measured using the l 2/0 -pseudonorm which is given by N/d x 2/0 = ( x i 2 ) 0 The value of function x 2/0 is equal to the number of blocks of x which have all-nonzero components. A block-sparse signal can therefore be recovered by solving the optimization problem minimize x subject to i=1 x 2/0 Φx = y Compressive Sensing 11/23 University of Victoria

31 Recovery of Block-Sparse Signals, cont d The block sparsity of a signal can be measured using the l 2/0 -pseudonorm which is given by N/d x 2/0 = ( x i 2 ) 0 The value of function x 2/0 is equal to the number of blocks of x which have all-nonzero components. A block-sparse signal can therefore be recovered by solving the optimization problem minimize x subject to i=1 x 2/0 Φx = y Unfortunately, this problem is nonconvex with combinatorial complexity. Compressive Sensing 11/23 University of Victoria

32 Recovery of Block-Sparse Signals, cont d A practical method for recovering a block sparse signal is to solve the problem minimize x 2/1 x subject to Φx = y where N/d x 2/1 = x i 2 i=1 Compressive Sensing 12/23 University of Victoria

33 Recovery of Block-Sparse Signals, cont d A practical method for recovering a block sparse signal is to solve the problem minimize x 2/1 x subject to Φx = y where N/d x 2/1 = x i 2 i=1 Note that function x 2/1 is the l 1 norm of the vector [ x1 2 x 2 2 x N/d 2 ] T, which essentially gives a measure of the inter-block sparsity of x. Compressive Sensing 12/23 University of Victoria

34 Recovery of Block-Sparse Signals, cont d A practical method for recovering a block sparse signal is to solve the problem minimize x 2/1 x subject to Φx = y where N/d x 2/1 = x i 2 i=1 Note that function x 2/1 is the l 1 norm of the vector [ x1 2 x 2 2 x N/d 2 ] T, which essentially gives a measure of the inter-block sparsity of x. The above problem is a convex programming problem which can be solved using a semidefinite-programming or a second-order cone-programming (SOCP) solver. Compressive Sensing 12/23 University of Victoria

35 Recovery of Block-Sparse Signals, cont d Example: N = 512, d = 8, K = 5, M = 100. Compressive Sensing 13/23 University of Victoria

36 Block-Sparse Signal Recovery by Using l 2/p Minimization We propose reconstructing a block-sparse signal x from measurement y by solving the l 2/p -regularized least-squares problem minimize x with p < 1 for a small ɛ where F ɛ (x) = 1 2 Φx y λ x p 2/p,ɛ (P) N/d x p 2/p,ɛ = ( xi ɛ 2) p/2 i=1 Compressive Sensing 14/23 University of Victoria

37 Block-Sparse Signal Recovery by Using l 2/p Minimization We propose reconstructing a block-sparse signal x from measurement y by solving the l 2/p -regularized least-squares problem minimize x F ɛ (x) = 1 2 Φx y λ x p 2/p,ɛ (P) with p < 1 for a small ɛ where N/d x p 2/p,ɛ = ( xi ɛ 2) p/2 i=1 Note that lim ɛ 0 x p 2/p,ɛ = x p 2/p lim p 0 x p 2/p = x 2/0 Compressive Sensing 14/23 University of Victoria

38 Block-Sparse Signal Recovery by Using l 2/p Minimization, cont d Good signal reconstruction performance is expected when problem P on slide 14 is solved with a sufficiently small ɛ. Compressive Sensing 15/23 University of Victoria

39 Block-Sparse Signal Recovery by Using l 2/p Minimization, cont d Good signal reconstruction performance is expected when problem P on slide 14 is solved with a sufficiently small ɛ. However, for small ɛ the objective function F ɛ (x) becomes highly nonconvex and nearly nondifferentiable. Compressive Sensing 15/23 University of Victoria

40 Block-Sparse Signal Recovery by Using l 2/p Minimization, cont d Good signal reconstruction performance is expected when problem P on slide 14 is solved with a sufficiently small ɛ. However, for small ɛ the objective function F ɛ (x) becomes highly nonconvex and nearly nondifferentiable. The larger the ɛ, the easier the optimization of F ɛ (x). Compressive Sensing 15/23 University of Victoria

41 Block-Sparse Signal Recovery by Using l 2/p Minimization, cont d Good signal reconstruction performance is expected when problem P on slide 14 is solved with a sufficiently small ɛ. However, for small ɛ the objective function F ɛ (x) becomes highly nonconvex and nearly nondifferentiable. The larger the ɛ, the easier the optimization of F ɛ (x). Therefore, we propose to solve problem P on slide 14 by using the following sequential optimization procedure: Choose a sufficiently large value of ɛ and solve problem P using Fletcher-Reeves conjugate-gradient (CG) algorithm. Set the solution to x. Reduce the value of ɛ, use x as an initializer, and solve problem P again. Repeat this procedure until problem P is solved for a sufficiently small value of ɛ. Output the final solution and stop. Compressive Sensing 15/23 University of Victoria

42 Block-Sparse Signal Recovery by Using l 2/p Minimization, cont d In the kth iteration of Fletcher-Reeves CG algorithm, iterate x k is updated as x k+1 = x k + α k d k where d k = g k + β k 1 d k 1 β k 1 = g k 2 2 g k Compressive Sensing 16/23 University of Victoria

43 Block-Sparse Signal Recovery by Using l 2/p Minimization, cont d In the kth iteration of Fletcher-Reeves CG algorithm, iterate x k is updated as x k+1 = x k + α k d k where d k = g k + β k 1 d k 1 β k 1 = g k 2 2 g k Given x k and d k, the step size α k is obtained by solving the optimization problem minimize α f (α) = F ɛ (x k + αd k ) Compressive Sensing 16/23 University of Victoria

44 Block-Sparse Signal Recovery by Using l 2/p Minimization, cont d By setting the first derivative of f (α) to zero, we get where α = G(α) d T k Φ T (Φx k y) + λ p G(α) = N/d Φd k λ p γ i γ i = ( x i + α d i 22 + ɛ 2 ) p/2 1 i=1 N/d γ i ( x Tki d ) ki i=1 ( d T ki d ki ) Compressive Sensing 17/23 University of Victoria

45 Block-Sparse Signal Recovery by Using l 2/p Minimization, cont d By setting the first derivative of f (α) to zero, we get where α = G(α) d T k Φ T (Φx k y) + λ p G(α) = N/d Φd k λ p γ i γ i = ( x i + α d i 22 + ɛ 2 ) p/2 1 i=1 N/d γ i ( x Tki d ) ki i=1 ( d T ki d ki ) In the above equations, x ki and d ki are the ith blocks of vectors x k and d k, respectively. Compressive Sensing 17/23 University of Victoria

46 Block-Sparse Signal Recovery by Using l 2/p Minimization, cont d Step size α k is determined by using the recursive relation α l+1 = G(α l ) for l = 1, 2,... Compressive Sensing 18/23 University of Victoria

47 Block-Sparse Signal Recovery by Using l 2/p Minimization, cont d Step size α k is determined by using the recursive relation α l+1 = G(α l ) for l = 1, 2,... According to Banach s fixed-point theorem, if dg(α)/dα < 1 then function G(α) is a contraction mapping, i.e., G(α 1 ) G(α 2 ) η α 1 α 2 with η < 1 and, as a consequence, the above recursion converges to a solution. Compressive Sensing 18/23 University of Victoria

48 Block-Sparse Signal Recovery by Using l 2/p Minimization, cont d Step size α k is determined by using the recursive relation α l+1 = G(α l ) for l = 1, 2,... According to Banach s fixed-point theorem, if dg(α)/dα < 1 then function G(α) is a contraction mapping, i.e., G(α 1 ) G(α 2 ) η α 1 α 2 with η < 1 and, as a consequence, the above recursion converges to a solution. Extensive experimental results have shown that function G(α) for function f (α) is, in practice, a contraction mapping. Compressive Sensing 18/23 University of Victoria

49 Performance Evaluation Number of perfectly recovered instances with N = 512, M = 100, and d = 8 over 100 runs. l 2/p -RLS: Proposed l 2/p -Regularized Least-Squares l 2/1 -SOCP: l 2/1 Second-Order Cone-Programming (Eldar and Mishali, 2009) BOMP: Block Orthogonal Matching Pursuit (Eldar et. al., 2010) Compressive Sensing 19/23 University of Victoria

50 Performance Evaluation, cont d Average CPU time with M = N/2, K = M/2.5d, and d = 8 over 100 runs. l 2/p -RLS: Proposed l 2/p -Regularized Least-Squares l 2/1 -SOCP: l 2/1 Second-Order Cone-Programming (Eldar and Mishali, 2009) BOMP: Block Orthogonal Matching Pursuit (Eldar et. al., 2010) Compressive Sensing 20/23 University of Victoria

51 Performance Evaluation, cont d Example: N = 512, d = 8, K = 9, M = 100. Compressive Sensing 21/23 University of Victoria

52 Conclusions Compressive sensing is an effective sampling technique for sparse signals. Compressive Sensing 22/23 University of Victoria

53 Conclusions Compressive sensing is an effective sampling technique for sparse signals. l 1 -minimization and l p -minimization with p < 1 work well for the reconstruction of sparse signals. Compressive Sensing 22/23 University of Victoria

54 Conclusions Compressive sensing is an effective sampling technique for sparse signals. l 1 -minimization and l p -minimization with p < 1 work well for the reconstruction of sparse signals. l 2/1 -minimization offers improved reconstruction performance for block-sparse signals. Compressive Sensing 22/23 University of Victoria

55 Conclusions Compressive sensing is an effective sampling technique for sparse signals. l 1 -minimization and l p -minimization with p < 1 work well for the reconstruction of sparse signals. l 2/1 -minimization offers improved reconstruction performance for block-sparse signals. The proposed l 2/p -regularized least-squares algorithm offers improved reconstruction performance for block-sparse signals relative to the l 2/1 -SOCP and BOMP algorithms. Compressive Sensing 22/23 University of Victoria

56 Thank you for your attention. This presentation can be downloaded from: andreas/rlectures/iscas2012-jeevan-pres.pdf Compressive Sensing 23/23 University of Victoria