On combinatorial approaches to compressed sensing

Transcription

1 On combinatorial approaches to compresse sensing Abolreza Abolhosseini Moghaam an Hayer Raha Department of Electrical an Computer Engineering, Michigan State University, East Lansing, MI, U.S. Abstract In this paper, we look at combinatorial algorithms for Compresse Sensing from a ifferent perspective. We show that certain combinatorial solvers are in fact recursive implementations of convex relaxation methos for solving compresse sensing, uner the assumption of sparsity for the projection matrix. We exten the notion of sparse binary projection matrices to sparse real-value ones. We prove that, contrary to their binary counterparts, this class of sparse-real matrices has the Restricte Isometry Property. Finally, we generalize the voting mechanism (employe in combinatorial algorithms) to notions of isolation/alignment an present the require solver for real-value sparse projection matrices base on such isolation/alignment mechanisms. Keywors: Compresse sensing, combinatorial approaches I. INTRODUCTION The theory of Compresse Sensing (CS)[1][2] eals with fining the unique sparse (or compressible) signal x R from a limite number of linear samples y = Px where P R an m < n. Roughly speaking, there are three approaches to fining the solution x : convex relaxation methos [2][3][4], greey algorithms [5] an combinatorial approaches [6][7][8][11]. Convex relaxation methos fin the unique sparsest solution to the uner-etermine system of linear equations y = Px by solving a convex optimization problem in the form of: arg min x s. t. y = Px (1) where p = 1. Although being tractable an emaning the fewest number of samples/equations (m) to guarantee perfect recovery, solving such linear program generally coul be very complex (e.g. O(n ) for Basis Pursuit [3]). On the other extreme, combinatorial algorithms have much less complexities (compare to convex relaxation methos) but at the cost of higher sample/equation requirements. The low computational costs of combinatorial algorithms stem from the fact that these approaches are base on binary sparse projection matrices P an light voting-like ecoing stages. Although there have been some works (e.g.[8]) highlighting connections between convex relaxation methos an combinatorial algorithms, in this paper we present a ifferent perspective onto such link. We show that, a typical combinatorial ecoer to CS is in fact a simple recursive l minimizer in case of a binary projection matrix. Then, we answer the following question: what will happen if non-zero entries of the binary projection matrix P are replace by (specifically Gaussian) ranom variables? We prove that, such moification provies us RIP-2 [2] in case of sparse projection matrices, a property which binary sparse projection matrices o not possess [10] for m = O(k log n/k ) where k is the sparsity of the signal x ( k = x = #i: x 0 ). Unfortunately, this moification (going from a sparse binary matrix to a sparse real-value one) implies that the voting mechanism of combinatorial approaches [6][7] will no longer work in case of such new projection matrices. Consequently, we present concepts of alignment an isolation for generic (real-value) sparse projection matrices. We show that the voting mechanism is in fact a special implementation of those concepts in case of binary projections. Finally, we present necessary an sufficient conitions uner which the new voting mechanism (alignment/isolation) is guarantee to fin the sparsest solution. This paper is structure as follows: In section II, we show how a basic combinatorial algorithm coul be viewe as a recursive l minimizer an exten the notion of using sparse binary projections to sparse real-value matrices. In section III, we prove that RIP-2 hols for such new class of projection matrices. Concepts of alignment/isolation an conitions for their equivalency are presente in Section IV. Simulation results are brought in Section V an Section VI conclues this paper. II. A GENERAL FRAMEWORK FOR CS COMBINATORIAL APPROACHES First, we introuce the notation use in this paper. For q N, efine[q] {1,2,, q}. For a projection matrix P R an for a row inex i [m], efine ω = j: P, 0. Hence, ω is the set of column inices where the i-th row of P is nonzero. We generalize such notation for any set of rows i [m] as: ω = ω. The set of row inices where the i -th column of P are non-zero, is Ω, i.e. Ω = {j: P, 0}. Similarly, one can generalize this notation for any set of columns i [n] by: Ω = Ω. A zero vector of length l is enote by 0. Throughout this paper, we consier the class of combinatorial algorithms in their most basic form. Hence, our iscussion exclues message passing algorithms [9] an we focus only on those algorithms which estimate signal values at each coorinate only once (e.g. [6]). Algorithm 1 shows how a typical combinatorial approach to CS operates for solving (1) uner the zero pseuo norm p = 0. It can be easily verifie that Algorithm 2 coul be perceive as the recursive formulation of Algorithm 1 when p is set to zero in the thir line of Algorithm 2.

2 Inputs : P R an y R Output : x R 1) y = y, z = ; 2) While z [n] 3) Fin l [m] such that x = 1 4) Let x enotes estimate of values x 5) For θ = ω : y y P, x, z z ω 6) En Algorithm 1. A typical combinatorial algorithm for CS. The set of coefficients which their values are etermine, is enote by z. Inputs : P R, θ an y R Output : x R 1) x 0 2) Fin l [m] an θ = ω θ such that x = 1 3) x = arg min {} ξ subject to y = P, ξ 4) x / Δ(P, θ/θ, y Px) Algorithm 2. Δ(P, θ, y) the recursive formulation of Algorithm 1. Initializing with θ = [n] an after fining θ such that x = 1 (fining these subsets is iscusse in Section IV), the algorithm recursively fins θ [n]/ [] θ such that x = 1 where θ enotes the value of θ (line 2 in Algorithm 2) in the j -th level of the stack. This process continues until all non-zero coefficients are recovere. Note that, for a vector ξ with a single non-zero entry ( ξ = 1) an z = Pξ, the solutions to (P1) for p = 0 an p = 1 are equal by the triangle inequality: (P1) ξ = arg min ξ s. t. z = Pξ (2) Also note that for any signal x an two isjoint subsets of a [n] an a = [n]/a one has: x + x = x both for p = 0 an p = 1 (see line 3 an 4 in Algorithm 2). Combining this observation with the equality of solutions of (P1) for p = 0 an p = 1, one can conclue that in Algorithm 2, the solutions for p = 0 an p = 1 are exactly the same. Hence, Algorithm 1 coul be recast as a recursive l minimizer. The success of the aforementione recursive algorithm epens on: (a) the ability of step 3 in Algorithm 1 to fin isolate non-zeros an (b) the quality of the projection matrix P in recoverability. More specifically, the first conition emans that the ecoer at the i-th iteration coul truly fin θ = ω, a subset of signal coefficients, containing only one non-zero ( x = 1 ). Meanwhile, the secon requirement states that, at least one isolate non-zero must exist throughout the recovery process an hence such process shall not be halte before the perfect recovery. It has been shown [7] that certain sparse matrices, for instance the ajacency matrices of high quality expaner graphs provie the secon conition. In the rest of this paper, we assume that the employe sparse projection matrix has the secon conition, meaning that if isolate non-zeros are correctly ientifie in step 3 of Algorithm 1, then the recovery process will not halt. To that en an to make our analysis simple, we aapt the ajacency matrices of ranom left regular expaner graphs [7]. That is, each column of P is non-zero in exactly = O(log ) ranom inices. However, we exten the notion of using binary sparse projection matrices to ranom (real value) sparse projections. More specifically, we simply substitute the non-zero entries of a sparse binary P (i.e. entries with values of one) with a zeromean Gaussian ranom variable with variance of 1/. We show that by such simple moification, RIP-2 will hol true in case of the propose P, the property which binary sparse projection matrices o not possess [10] (for m = k log n/k). Having the guarantee of recoverability, now our concern will be to satisfy the first conition, i.e. correctly fining isolate non-zeros. The most common approach for fining isolate non-zeros (in case of a binary sparse projection matrix) is to apply a voting mechanism (see [6][7]). More specifically, assume that there exists a set of sample inices of l [m] such that all of them vote the same value, i.e.: p, q l: y = y an they span the same single coefficient (i.e. ω = i). Then typical combinatorial algorithms infer that x equals to that vote an x = 0 for j ω /i. Such conclusion might imply certain presumptions about the unerlying signal. For example, it has been assume in [11] that no two subsetsθ, θ [n] exist such that x x or other assumptions in some other methos such as non-negativity of the signal an so on. Consequently, the applicability of the aforementione approaches will be restricte to signals satisfying corresponing conitions. On the other han, ue to the non-binary nature of the propose sparse projection matrix P, the mechanism of voting is not applicable in case of P. Consequently, we introuce the notions of alignment an isolation in section IV an show that the metho of voting is a special case of such notions in case of binary matrices. Furthermore, we present necessary an sufficient conitions for etecting isolate non-zeros. Before that, we prove that the RIP property hols for the propose sparse-real matrices; whereas it has been shown that RIP oes not hol for the corresponing sparse-binary matrices [10]. III. RESTRICTED ISOMETRY PROPERTY A matrix P R has RIP-2 of orer k N [2], if for every k-sparse signal x R, there exists a constant δ such that: (1 δ ) x Px (1 + δ ) x In this section, we present the proof sketch that certain sparse real-value projection matrices have RIP-2. Theorem 1. Assume P R has the following properties: (a) each column is non-zero in exactly = O(log ) ranom inices an (b) the non-zeros entries are zero mean Gaussian ranom variables with variance of 1/. Then for any x R with k = x = O(n) an sufficiently large m = O(k log ) e(), there exists constant c (ε) such that: Pr( Px x ε x ) 2e () (3) Proof: We follow simple an elegant approaches of [12][13] to show (3) hols. Recall that non-zero entries of P are i.i.

3 N(0,1/). Hence E(y ) = x E(P, ) = 0. Also by using the fact that non-zero entries of P are inepenent, it can be easily shown that: x E(y ) = x E(P, ) = (4) Consequently for E( y ) = E( y ) = E(y ) E( y ) = x = c x : where c is the number of times that i [n] occurs in ω. Recall that, all columns of P are non-zero in exactly row inices. Hence, for all i [n]we have c =. Consequently: E( y ) = x. Now we focus on fining how much y coul eviate from x. Here, we erive bouns for the lower an upper tails separately. Note that (see [13]): y ~x / N(0,1). Hence: y ~ x χ (1) where χ (1) is a chi-square with one egree of freeom. Since non-zero entries of P are inepenent, y an y are inepenent for i j. Consequently the moment generating function for y is: M(t, y ) = Ee = 1 2x t / for t < 1/2. Applying the metho of generating functions an then the Markov s inequality yiels: Pr( y (1 + ε) x ) g(t) = Ee = 1 e () e () For simplicity an without loss of generality assume x is a unit norm vector. Thus for all i, we have x x = 1 an consequently: (5) (6) (8) g(t) 1 2t e () (9) To get the tightest bouns, one nees to set the erivate of g(t) with respect to t to zero. Since t < 1/2, only the solution of t = woul be of importance. Plugging this value () into (9) an then (8) yiels: Pr( y (1 + ε) x ) e () m e(1 + ε) (7) (10) It is straightforwar to verify that, uner the assumption of m e () (recall = O log an m = O k log ): Pr( y (1 + ε) x ) e ( ) (11) where exp{ m matrix from a Gaussian ensemble[12]. } is the boun for a ense projection Similarly for fining the boun for the lower tail, one can follow the same approach to get: Pr( y (1 + ε) x ) Ee e () t = e () 1 + 2x / (12) for the unit-norm vector x an t < 1/2. Define S = {i: x 0}. Hence S is the set of inices where the signal x is nonzero. Then efine the constant δ (note that this constant is ifferent from the RIP constant δ ) as follow: δ = inf{ x } i S (13) In wors, if x 0 then x δ. Note that, since k = x = O(n) an P (when its non-zero entries are replace with one) correspons to a (k, β) expaner graph (β < 1), hence (a) there exists inices Ξ where Ξ [m], (1 β)k ξ = Ξ k an (b) i Ξ: δ x 1. Thus ξ is in the orer of O(m). Consequently: Where: Letting t = Pr( y (1 ε) x ) f(t) (14) f(t) = e () 1 δt / () < aroun ε = 0 yiels: (15) an calculating Taylor series of f(t) f(t) = δξ 4 ε + (4 12δξ 2δ ξ + δ ξ ) 32 + O(ε ) It can be verifie that, when: then: 2 ξ ε < δ (16) f(t) 2 exp 1 2 δξ 4 ϵ (17) up to orer O(ε ). Note that in the asymptotic case when k, constraint (16) emans that δ > 0 which by efinition of (13) is trivially true. Assume m = O k log =

4 C log an = O log = C log. Then combining (17) with (14) yiels: Where: Pr( y (1 ε) x ) e (18) C (1 β)c δε 4C (19) Thus, uner the assumption of k = O(n) (or equivalently = O log = O(1)), we woul have: C = O(1). Having Theorem 1, one can apply Theorem 5.2 in [12] to prove that the sparse ranom projection matrix P (with the aforementione conitions) has RIP-2. Consequently, BP [3] or other convex relaxation approaches [4] coul be utilize for the propose projection matrix to have a robust an stable recovery in the presence of noise an in case of compressible (as oppose to exactly sparse) signals. In the next section, we fin conitions for Algorithm 1 to guarantee perfect recovery in case of exactly sparse signals. IV. NECESSARY AND SUFFICIENT CONDITIONS FOR FINDING ISOLATED NON-ZEROS As before, assume P R is a matrix where each of its columns is non-zero in exactly = O(log ) ranom inices. Also assume that, non-zero entries of P are zero-mean Gaussian ranom variables with variance of 1/. If non-zero entries of P are replace with one, then the resulting binary matrix woul correspon to the ajacency matrix of a high quality expaner graph (with high probability) [7]. Since the analysis of [7] is not base on values of P, therefore the recoverability hols for P. Consequently in here, we only iscuss lines 3 an 4 in Algorithm 1. We show, how one can fin subsets of signal coefficients containing only one nonzero, by examining compressive samples. To that en, we nee to efine two basic concepts. We say x is isolate in sample inices of l [m] if x = 1, x 0. In other wors, x is the only non-zero signal coefficient (among inices ω ) spanne by samples y. Clearly, since P :, is nonzero only in inices Ω, therefore l Ω. For q [m] we say y is aligne with the j-th column of P if: i q: P, 0 an α 0: y = αp,. Note that, if P is binary (i.e. P, is an all one vector), then the efinition of alignment reas as there exists a set of samples (y ), all voting the same value of α. This is exactly the voting mechanism, employe in combinatorial algorithms. This verifies our claim that the voting mechanism is in fact an alignment in the special case of binary matrices. Assume that x 0 is isolate in sample inices of l Ω [m]. Then: y = P, x = x P, (20) In wors, if x is isolate in sample inices of l then y will be aligne with the i -th column of P. Thus isolation is a necessary conition for alignment. Now, if isolation is a sufficient conition for alignment as well, then isolate nonzero coefficients in the signal coul be foun by looking for alignment of samples among the columns of P. For now assume that isolation an alignment are equivalent. Then step 3 an 4 in Algorithm 1 coul be implemente as follows: to see whether x is isolate in a set of unknown samples, one might consier P Ω, an looks for q Ω such that y = αp,. If there exists any q with those conitions, then (20) coul be applie to infer x = α. In the rest of this section, we ientify the unerlying conitions uner which isolation is a sufficient conition for alignment. Here, it is important to highlight the impact of the size of the set l Ω [m] in our analysis. Since, in an extreme case when l = 1 in (20), there always exists a meaningless alignment. Recall that, each column of P is non-zero in exactly = O(log n/k) entries. In [7], it has been shown that in case of a sparse binary projection matrix base on the ajacency matrix of a left -regular ranom graph, there exists at least τ 1 + /2 samples which span only one (an the same) non-zero in all stages of ecoing. Hence, in here we have l = τ > /2 = O(log n/k) an ignore alignments with size smaller than /2 in the ecoer. As before, assume y is aligne with the i-th column of P, where l > /2. Since by efinition y = P, x an y > 0, there are two possible cases here: (a) x = 1 or (b) x > 1. In the first case, since y is non-zero in all entries ( y = l ) an for all j ω /i we have P, < l (this comes from the expaner property of P) thus one can conclue that, x is the only non-zero among inices of ω an its value can be easily compute by (20). In wors, in the first case, isolation is a sufficient an necessary conition for alignment. Unfortunately, a similar argument generally oes not hol true for the secon case (x > 1) for P. Nevertheless, here we show how one can moify P such that, the secon case woul never happen with a very high probability. To that en, we follow these steps (a) for any subset of samples l [m] with l 1 + = O(log ) first we fin Q = maxx, the maximum number of non-zeros which will be spanne (with high probability) by samples l. (b) Then we moify the projection matrix, such that every Q columns of P, woul be inepenent. Finally by using elementary arguments, one can show that it is impossible to have x > 1 an y to be aligne with the i-th column of P. Define the binary ranom variable X as: X = 1 if x 0 an i ω (21) 0 otherwise It can be easily verifie that Q = max X.Now since the probability that x 0 is inepenent from the probability of the event that i ω, thus: Pr(X = 1) = k m l n 1 m (22)

5 Assuming k 1 an to simplify erivations, here use the approximation: 1/(m + 1) 1/m. Using the inequality 1 + α e, one can show: μ = E[Q] = n Pr(X = 1) k(1 exp( l/m)). Recalling that = O log = c log, /2 < l an m = O log = c k log, we have: μ k 1 k n (23) Here we woul like to note that μ is extremely small. In fact, it can be easily shown that for a constant α 1 an uner the assumption of n > c log /c log 1 : we get μ α. Nevertheless, here we assume that μ = α = O(1) for a properly chosen value of α. Now it only remains to boun the eviation of Q from μ. Fortunately, since Q is the sum of Bernoulli ranom variables, there exists goo concentration inequalities for it. For instance, by using Chernoff boun for Poisson trials, it can be easily shown that for log(1 + δ) > 2 (or δ 6.389) we have: Pr(Q μ + δβ log n). Now if the spark [14] of P, is more then Q, then it is impossible to have y aligne with the i-th column of P an at the same time x > 1. This can be prove by contraiction. More specifically, assume 1 < x < Q an y is aligne with the i -th column of P ( y = P, x = αp, ). Define a new vector x where x = x α an x = x for i ω /j. Then P, x = 0 an x Q which contraicts the presumption that the spark of P, is more than Q. In summary, as long as every Q = O(log n) columns of P, for every possible subset l [m] with l O log are inepenent, then isolation woul be a sufficient an necessary conition for alignment. Unfortunately, this requirement oes not hol for a projection matrix constructe from a ranom left regular bi-partite graph, since it is quite possible to have two columns with inices u an v in P, such that, both of them are non-zero exactly in one (an the same) row inex j. Clearly, these two columns in P, are linearly epenent which reuces the spark of P, to a minimal of one. This problem coul be resolve by using Algorithm 3 to moify P. Inputs : P R an Q Output : P For i = 1 to n o T = 0 For j Ω o l {Q(j 1) + 1,, Qj}: T ~N 0, En P :, = T En Algorithm 3 Increasing the spark of sub-matrices of P. The output matrix P has the following properties: (a) P R (b) Ω = j: P, 0 = Ω {j(q 1) + 1, j(q 1) + 2,, jq} (c) Let ω = j: P, 0. Then i {j(q 1) + 1, j(q 1) + 2,, jq} we have ω = ω. Thus for instance, if x is isolate in sample inices l, then the same coefficient x shall be isolate in samples y where y = P x an b = {j(q 1) + 1, j(q 1) + 2,, jq}. Consequently, the ecoing algorithm an all the analysis we presente in case of the projection matrix P stay the same in case of the new projection matrix P. However, P has a feature that P oes not have an that is: for any subset l [m] with l = O(), P, woul have a spark of at least Q. This can be shown by the following argument. If u, v ω then (by properties of P ) either Ω Ω = (meaning those two columns are orthogonal) or Ω Ω Q. Now since the non-zero entries of P are populate ranomly (~ N(0, )), thus every Q columns of P woul be inepenent. Note that, this property is gaine at the cost of increasing the sample requirement from O(k log(n/k)) to O(Qk log(n/k)) = O(k log( n) log(n/k)). Amittely our analysis coul be improve furthermore an one can come up with tighter bouns for Q. Our simulation results show that, in practice an optimal number of samples is require for perfect recovery. V. SIMULATION RESULTS In this section, we present simulation results valiating our claims/proofs in this paper. First, the joint performance of the propose combinatorial algorithm an sparse real-value projection matrix for recovering sparse signals is investigate. To that en, for a fixe signal length of n = 1000, ten ifferent levels of m/n an seven levels of k/m, we fin probability of exact recovery as functions of those parameters. For each configuration of m/n an k/m, one hunre simulations are performe as follows: (a) we select uniformly at ranom k locations among [n] (b) In selecte inices, the test signal x woul be populate accoring to a normal istribution. (c) We generate P, a sparse real-value projection matrix base on assumptions Section II an Algorithm 4, with parameters = 3 an Q = 2. () The compressive samples y = Px are then given to Algorithm 1 to compute x. Figure 1, shows the ratio of perfect recovery ( x = x ) among one hunre simulations for each configuration of m/n an k/m. Figure 1. The probability of exact recovery of with signal of length n = 1000 as a function of k/m an m/n.

6 To evaluate performance of introuce projection matrices in the presence of noise, we consiere a signal x with length n = 2000 where k = 150 of its entries are selecte uniformly at ranom an their values are uniformly chosen in the range of [ 0.5,0.5]. To assess, how the propose projection matrix woul act in conjunction with a robust solver such as BP, we generate two matrices P, P R where m = 600, P is a sparse real-value projection matrix with parameters = 3 an Q = 2 an P is a ense ranom matrix with Gaussian entries. Corresponingly, two sets of compressive samples were generate an contaminate by the same noise ε, i.e. y = Px + ε an y = P x + ε an fe into BP. We chose ε to be a Gaussian noise (ε~n(0, σ )) an consiere ten ifferent levels for σ. For each variance level, we performe one hunre simulations an recore the maximum l norm of recovery error ( x x ). The results are presente in Figure 2. As clearly shown in that figure, BP shows very similar stabilities uner both types of (ense Gaussian an the propose) projection matrices. Figure 2. l 2 norm of recovery error in a noisy setting. VI. CONCLUSION In this paper, we show how certain classes of combinatorial algorithms are linke to convex relaxation methos for CS. We consiere the scenario when non-zero entries of a sparse binary projection matrix are replace with ranom Gaussian numbers an show how the notion of voting (employe for sparse binary projections) extens to isolation/alignment. The RIP-2 is prove for this new class of projections. Sufficient an necessary conitions for the equivalency of isolation an alignment are presente. REFERENCES [1] Davi Donoho, Compresse sensing. IEEE Trans. on Information Theory, 52(4), pp , April [2] E. Canès, J. Romberg, an T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. on IT, 52(2) pp , February 2006 [3] S.S. Chen, D. L. Donoho, Michael, A. Sauners, Atomic ecomposition by Basis Pursuit, Siam J. of Scientific Comp, pp , [4] M. A. T. Figueireo, R. D. Nowak, an S. J. Wright, Graient projection for sparse reconstruction: Application to compresse sensing an other inverse problems. IEEE J. of Selecte Topics in Signal Processing: Special Issue on Convex Optimization Methos for Signal Processing, 1(4), pp , [5] D. Neeell, J. A. Tropp, CoSaMP: Iterative signal recovery from incomplete an inaccurate samples. (Preprint, 2008) [6] R. Berine an P. Inyk, Sparse recovery using sparse ranom matrices. (Preprint, 2008) [7] S. Jafarpour, W. Xu, B. Hassibi, an R. Calerbank, Efficient compresse sensing using high-quality expaner graphs, IEEE Trans Info Theory, 2009 [8] R. Berine, A. C. Gilbert, P. Inyk, H. Karloff, an M. J. Strauss, Combining geometry an combinatorics: A unifie approach to sparse signal recovery. (Preprint, 2008). [9] D.L. Donoho, A. Maleki, an A. Montanari. Message passing algorithms for compresse sensing, Proc. Natl Aca. Sci., [10] Venkat Chanar, A negative result concerning explicit matrices with the restricte isometry property. (Preprint, 2008) [11] S. Sarvotham, D. Baron, an R. Baraniuk, Suocoes-Fast measurement an reconstruction of sparse signals. IEEE ISIT [12] R. Baraniuk, M. Davenport, R. DeVore, an M. Wakin, A simple proof of the restricte isometry property for ranom matrices. Constructive Approximation, 28(3), pp , December [13] D. Achlioptas, Database-frienly ranom projections, 20th Annual Symposium on Principles of Database Systems, 2001, pp [14]