AKRON: An Algorithm for Approximating Sparse Kernel Reconstruction

Transcription

1 : An Algorithm for Aroximating Sarse Kernel Reconstruction Gregory Ditzler Det. of Electrical and Comuter Engineering The University of Arizona Tucson, AZ 8572 USA Nidhal Carla Bouaynaya Det. of Electrical and Comuter Engineering Rowan University Glassboro, NJ 828 USA Roman Shterenberg Det. of Mathematics University of Alabama at Birmingham Birmingham, AL USA Abstract Exact reconstruction of a sarse signal for an under-determined linear system using the l -measure is, in general, an NP-hard roblem. The most oular aroach is to relax the l -otimization roblem to an l -aroximation. However, the strength of this convex aroximation relies uon rigid roerties on the system, which are not verifiable in ractice. Greedy algorithms have been roosed in the ast to seed u the otimization of the l roblem, but their comutational efficiency comes at the exense of a larger error. In an effort to control error and comlexity, this aer goes beyond the l -aroximation by growing neighborhoods of the l -solution that moves towards the otimal solution. The size of the neighborhood is tunable deending on the comutational resources. The roosed algorithm, termed Aroximate Kernel RecONstruction (), yields significantly smaller errors than current greedy methods with a controllable comutational cost. By construction, the error of is smaller than or to equal the l -solution. enjoys all the error bounds of l under the restricted isometry roerty condition. We benchmarked on simulated data from several under-determined systems, and the results show that can significantly imrove the reconstruction error with slightly more comutational cost than solving the l roblem directly Introduction Many engineering roblems are formulated as inverse roblems, which is where the number of arameters () greatly exceeds the number of measurements (n) available. Examles include: Prerint submitted to Elsevier Setember 4, 27

2 source estimation of electroencehalograhic (EEG) and magnetoencehalograhic (MEG) data [, 2], reverse-engineering of genetic regulatory networks from high-throughut gene exression data [3, 4], magnetic resonance imaging [5], information theory and communication engineering [6], and electromagnetics and antenna design [7]. These inverse roblems, known as large small n, ose a challenge, because of the non-identifiability of a solution. Additional constraints or rior knowledge are needed to solve such under-determined systems. In many cases, such as inference of genetic regulatory networks [3, 4], we are interested in the sarsest solution. The objective is then to recover the sarsest signal, x C, from a measurement matrix, A C n, and observed vector y C n such that y = Ax, where n. In a noisy setting, the roblem is formulated as y = Ax + e, where e is a vector of measurement noise with a bounded variance, i.e., e 2 ɛ. Without loss of generality, it is assumed that A is full-rank; otherwise, the observations would be redundant. Finding the sarsest solution amounts to solving the following otimization roblem: x = argmin { x : Ax = y}, () where x denotes the l -measure of vector x, i.e., the number of non-zero elements of x. Observe that l is not a roer norm and that is why we refer to it as a measure although, by abuse of notation, we may also write l -norm. Unfortunately, () is in general an NP-hard combinatorial roblem since it involves finding the number and ositions of the zeros in a -dimensional sace [8]. The field of comressive sensing (CS) addresses this roblem by solving the underdetermined system with a unique sarsest solution under secific conditions on the system. The l -norm objective in () can be relaxed to the l -norm, solving the following convex otimization roblem: x = argmin { x : Ax = y}. (2) This convex relaxation makes the roblem more tractable; however, in general, the solutions of () and (2) are not equivalent. CS theory has shown that, if A satisfies the null sace roerty (NSP) or the restricted isometry roerty (RIP), then the l roblem yields the otimal l solution [8]. Unfortunately, these conditions are not verifiable in ractice. In articular, one cannot check if the obtained l -solution is the sarsest solution or not! Through examles and simulations, we show that, in general, the l -solution may be far from the l -otimal solution. Hence, it is crucial to develo greedy algorithms that achieve a balance between comutational comlexity and reconstruction error. 2. Related Work Recent efforts focused on greedy algorithms to infer a sarse solution. In articular, a family of Hard Thresholding (HT) algorithms have been suggested in [9], which makes an initial guess for the suort of x and then rojects the measurements y onto this suort. An iterative version called Iterative Hard Thresholding (IHT) udates the residual and estimates a new x at every iteration until the stoing criterion is satisfied. Another version of lower comlexity er iteration, called Matching Pursuit (MP), has been suggested. The Orthogonal Matching Pursuit () [] is an iterative greedy algorithm that selects at each ste the column which is most correlated with 2

3 32 the current residuals, and estimates the nonzero entries in the vector x with a comutational com- 33 lexity O(k log ). s comutational imrovements, however, come at the cost of increased 34 reconstruction error. Comressive Samling Matching Pursuit (CoSaMP) [] combines the a- 35 roaches of and HT in a two-stage greedy algorithm that aims to imrove the reconstruction 36 error of. Unfortunately, these methods sacrifice accuracy of the reconstruction for the runtime 37 as they aroximate the l -norm by other cost functions. Recently,, or smoothed l, has been 38 roosed as a fast algorithm to directly aroximate the l solution []. Candés et al. roosed an 39 iterative re-weighted l minimization algorithm that has theoretical guarantees that is can imrove 4 the l solution [2]. 4 In our revious work, we resented Kernel RecONstruction (KRON), a greedy algorithm, that achieves an exact solution to (), without exhaustively searching C 42 [3]. In KRON, finding the sarsest solution amounts to solving ( ( s= n) linear equations. All 43 s) otential solutions have at 44 least s zeros. The sarsest solution is guaranteed to be one of them. The comutational comlexity 45 of KRON is O( s ). KRON yields the otimal sarsest solution (zero reconstruction error) at a high 46 comutational cost. 47 Against this background, we seek to develo an aroach for aroximating () yielding re- 48 construction errors lower than l -norm, and other aroaches such as and CoSaMP, and at 49 the same time having comarable, or at least controllable, comutational cost Aroximate Kernel Reconstruction Central Idea behind 52 In this section, we motivate the central idea behind, given general linear algebra 53 knowledge about the under-determined system. First, we know that the system Ax = y always 54 admits solutions with s = ( n) zeros because s is the dimension of the Kernel of A; hence 55 the name Kernel RecONstruction (KRON) in [3]. KRON distributes s zeros among the entries 56 then searches for all the solutions with exactly s zeros. The sarsest solution is guaranteed to be among these ( 57 s) solutions. However, we do not know in advance which one it will be. KRON tries all ossible ( 58 s) solutions then chooses the sarsest. Notice that no conditions are imosed on 59 the matrix A; that is, KRON recovers the otimal sarsest solution whether the RIP condition is 6 satisfied or not. The central issue with KRON is that it becomes comutationally rohibitive when is large and s in the order of. Therefore, we roose to reduce the number of enu merations that need to be erformed in KRON. To achieve this, the central idea behind is 63 to use the standard l -aroximation to guess the locations of the s zeros that will result in the 64 sarsest solution. Finding s correct zero locations is sufficient to find the otimal sarsest solution. 65 This idea can also be viewed as a erturbation of the l -aroximation to make it closer to the 66 l -norm. Formally, we define a δ-neighborhood of the l -aroximation that allows to 67 find sarser solutions and reduce the reconstruction error. The size of the neighborhood is tunable 68 deending on the comutational ower available, and vary from (l -aroximation) to n (KRON, 69 i.e., erfect reconstruction). In articular, when the l -aroximation is otimal (RIP conditions 7 satisfied), is also otimal, but when the l -solution is subotimal, results in a 7 better (i.e., sarser) solution with smaller recovery error The Noiseless Case begins by solving the l convex otimization roblem in (2). Denote the solution by x. In general, x is different from the desired l -solution. However, since l is the closest convex 3

4 78 75 norm to l, we can use x to find the locations of s zeros, which would corresond to the s-smallest 76 magnitudes in x. The central idea behind s -neighborhood solution is as follows: () 77 find the indices (Q) with the s-smallest magnitudes of the l solution, (2) set these indices to zero, then (3) re-solve the system Ax = y. Call this solution x δ=. The following roosition bounds 79 the error between the l -solution and the (δ = )-neighborhood solution x δ=. Proosition. Let x denote the l -solution of the under-determined roblem in (2). Without loss of generality, we assume that A C n is full-rank, and call s = n. Let { x (i ),..., x (i s ) } be the set of the s-smallest magnitudes of x. Then, we have x x δ= 2 s C A max { x (i ),, x (i s ) }, (3) where C A is a constant that deends only on the matrix A: C A = ( + A Q 2 A Q 2 ), where A Q is the (n n) sub-matrix of A obtained by removing the s columns indexed by {i,, i s }, and A Q C n s is the comlement matrix, i.e., the matrix that contains only the columns corresonding to these s-smallest elements. Proof. Denote by A Q C n n the reduced matrix, where the columns corresonding to the indices of the s-smallest elements in x were removed. Notice that A Q is invertible because A is fullrank. Let A Q C n s be the comlement matrix, i.e., the matrix that contains only the columns corresonding to the s-smallest elements {i,, i s }. We adot similar notations for x Q C n and x Q Cs. We have Observe that since, by construction, x δ= Q From Eqs. (5) and (6), we have Therefore, Using norm inequalities, we obtain A x = A x δ= = y (4) A Q x Q + A Q x Q = A x δ=. (5) =, we have that A Q x δ= Q = A x δ=. (6) A Q ( x δ= Q x Q ) = A Q ˆx Q. x δ= Q x Q = A Q A Q x Q. On the other hand, we have by triangle inequality x δ= Q x Q 2 A Q 2 A Q 2 x Q 2. (7) x x δ= 2 x Q x δ= Q 2 + x Q x δ= Q 2. 4

5 89 But, by construction, x δ= Q =, hence x x δ= 2 x Q x δ= Q 2 + x Q 2 x ( Q 2 + A Q ) 2 A Q 2 (8) where the inequality in (8) follows from (7). By using the fact that x Q 2 s max { x (i ),, x (i s ) }, we obtain the desired result with C A = ( + A 9 Q A Q ). The interretation of the bound in Proosition is quite intuitive: if the s-smallest elements 9 of the l -aroximation are all small, then x 92 δ= will be close to the l -solution. However, if at least one of these s-smallest elements is large, then the obtained x 93 δ= will be far from the l - 94 aroximation. In other words, the s-smallest elements of the l -aroximation are all small if l is a good aroximation of l ; In that case, the solution x 95 δ= will be close to this good 96 l solution. On the other hand, having a large element among the s-smallest indices indicates that 97 l -solution is not a good estimate of the otimal sarsest solution, and hence solution 98 will be sarser and further from l solution. The following theorem derives an error bound of the recovery. We first need to recall the restricted isometry roerty (RIP). The restricted isometry constant α k of a matrix A C n is the smallest number such that ( α k ) x 2 2 Ax 2 2 ( + α k ) x for all k-sarse x. A matrix A is said to satisfy the RIP of order k with constant α k if α k (, ). Theorem. Assume that A C n satisfies the RIP of order 3k with α 3k <, and let s = n. 3 Let x (i ),, x (i s ) be the s-smallest magnitude elements of the l -solution x. Then, letting x be the otimal l -solution, we have x x δ= 2 s C A max { x (i ),, x (i s ) } + C σ k(x ) k, (9) 2 where σ k (x) = inf z {z C : z k} z x is the best k-term aroximation error of the vector x in l, C is a constant that deends on α 3k [8], and C A is a constant that deends on the matrix A as defined in Proosition. Proof. From [8], we have, under the RIP condition, for all x, x x 2 C σ k(x) k, () 5

6 ( ) with C = 2 γ+ γ 2 + γ, γ = +α3k 2( α 3k [8]. From the triangle inequality, we have ) x x δ= 2 x δ= x 2 + x x 2 s C A max { x (i ),, x (i s ) } + C σ k(x ) k, () 3 4 where, in (), the first inequality follows from Proosition and the second inequality follows from () in [8]. 5 δ-neighborhoods of l in quest of sarser solutions: The solution given by x δ= is sarser 6 than the l -aroximation; however, x δ= may still be far from the otimal l solution in (). We can imrove uon x δ= by finding sarser solutions that increase the neighborhood of the 8 l -aroximation as follows: consider the (s + δ)-smallest elements of x, where the true s zeros, that lead to the otimal sarsest solution, may be located. Then, consider all ossible ( ) s+δ 9 s combinations of setting s elements to zero. For each of these s zero locations, re-solves the system. The sarsity of the solution is recorded for each combination and the sarsest solution, x ( δ, is returned by. It is imortant to note that is highly arallelizable once the s+δ ) 3 s combinations are known. We state that x 4 δ is the otimal solution within the δ-neighborhood of the l -aroximation. 5 If the l -aroximation is close to the l -solution, then we can find the otimal solution within 6 a small δ-neighborhood. To observe this result, let x be the otimal l -solution, and assume it 7 is unique and has K > s zeros and that x is close to x. These assumtions imly that the 8 corresonding K entries of x are close to zero. By choosing δ = K s, will surely 9 find the otimal solution. It may aear that if K s, then δ and significant comutational 2 resources are needed. However, if x is close to x, then even utting the smallest s elements to 2 zero will lead to the exact solution by uniqueness. Therefore, we suggest a ste-wise aroach. 22 First, consider δ =. We may find the otimal sarsest solution (i.e., number of zeros s) right 23 away by uniqueness of the sarsest solution and the fact that l -aroximation may be close (for 24 the secific system at hand) to the otimal l -solution. Otherwise, we increase δ [, n] deending 25 on the available comutational ower. The entire rocess can be reeated for growing values of 26 δ =,..., δ max n, where δ max is set deending on the comutational ower available or until 27 we reach a sufficiently sarse solution. Note that comutational comlexity of is of the 28 order of O(s δ ), and when δ = n, reduces to the erfect reconstruction KRON algorithm 29 in [3]. 3 State-of-the-art aroaches, such as and CoSaMP, require that the sarsity level be sec- 3 ified in advance, which is, in general, imossible to correctly guess., on the other hand, 32 starts with an educated initial estimate of the level of sarsity, given by the s smallest elements 33 in the l solution. This initial estimate has well-known theoretical roerties in the comressive 34 sensing literature [8]. We derived a theoretical bound between (with δ = ) and the oti- 35 mal solution. We then imrove uon this estimate by exloring higher δ-neighborhoods in quest of 36 sarser solutions. In articular, the arameter δ controls the tradeoff between sarsity and comu- 37 tational comlexity. The sensitivity of this free arameter δ is evaluated in Section 4. The seudo 38 code for is detailed in Algorithm. Examle : To understand the algorithm and illustrate the imortance of the δ- neighborhoods, we resent a simle numerical examle. Consider the following randomly gen- 6

7 erated noiseless system A = y = ( ) T ; The otimal l -norm sarsest solution is given by x = ( ) T (2) The l solution, which solves (2), is given by x = ( ) T 39 Clearly, the l -solution is not as sarse as the otimal solution and has incorrect zero locations. We 4 have n = 3, = 5 and thus s = 2. If we choose δ = the considers the s+δ = 3-smallest 4 magnitudes of x, which are located at indices, 2 and 4. We set s = 2 locations to zero among these 3 indices. We consider all ( ) ( s+δ s = 3 ) 42 2 = 3 combinations of two zeros in indices, 2 and 4 of 43 x. The combination of indices and 2 set to zero leads to the sarsest otimal solution x in (2). 44 Thus, in this case, the l -norm solution is sub-otimal; but by considering a δ = -neighborhood 45 of this aroximation, is able to exactly recover the sarsest otimal l -solution The Noisy Case can be reformulated to coe with noise (oi) in a system given by y = Ax+e, where e is the noise vector. The noisy case is formulated as the following otimization roblem x ɛ = argmin{ x : Ax y 2 ɛ}, (3) 47 where ɛ is the fixed error bound. In the noisy case, the sarsest solution can be sarser than the 48 solution of the homologous noiseless system. Intuitively, the noise allows a margin of error 49 where more entries can be set to zero as long as the constraint in (3) is satisfied. 5 Following this logic of noise enhances sarsity, one may start by solving the noiseless case 5 using the algorithm in Section 3.2, and then further set more elements to zero while ensuring the 52 constraint in (3) is satisfied. This aroach is romising if the corresonding noiseless system 53 Ax = y admits a sarse solution, i.e., with a number of zeros K s. However, in general, 54 the air (A, y), corresonding to the noisy system in (3), may not admit sarse solutions to the 55 noiseless homologue Ax = y. In this case, the l -aroximation of the noiseless system will be 56 very far from the otimal l solution, and small erturbations (small δ) of the l -solution may not 57 be sufficient to aroximate the otimal solution with desired accuracy. 58 Instead, consider the corresonding l aroximation of the noisy system x ɛ = argmin{ x : 59 Ax y 2 ɛ}. oi starts from x ɛ and considers a δ-neighborhood to obtain the locations 6 of the s + δ smallest magnitude elements of x ɛ. For all ossible combinations of s zeros among 6 these s + δ locations, we obtain the corresonding solutions to the noisy system with constraint Ax y 2 ɛ. Call these solutions x q δ, where q =,, ( ) s+δ s. We can further make these 63 7

8 satisfied. This can be done, for instance, by utting to zero all elements in x q δ such that xq δ (i) ɛ 65 A F, where A F is the Frobenius norm of A. A more direct aroach would be to ut the 66 smallest elements to zero one at a time then check if the constraint in (3) is satisfied. On the other 67 hand, oi is not guaranteed to lead to the exact otimal l solution for large neighborhoods 68 because the exact value of y for which one attains the sarsest solution is not known. The following 69 numerical examle illustrates the remise behind oi. 7 Examle 2: Let A and x be as rovided in Examle, and consider the noisy system y = 7 Ax +e, where e = (.,.7,.28) T is a samle noise vector that was randomly generated 72 from a Gaussian distribution. Observe that the ower of the noise is about 2% the ower of the 73 signal. The starting oint for oi is the l solution of the noisy system, which is given by 74 x ɛ = (,.46,,,.752) T, where ɛ is the norm of the noise vector e. Clearly this solution is 75 far from the otimal solution in terms of zero locations and norm. oi with δ = 2 yields 76 the solution (,,,.34,.23) T. Although this solution is not exact, oi is able to 77 correctly recover the locations of the non-zero entries in x. Furthermore, the reconstruction error 78 of oi is significantly lower than that of noisy l. The following section demonstrates the 79 effectiveness of and oi on a variety of different under-determined linear systems Exeriments was benchmarked against [4], CoSaMP [], [], Iterative Re-weighted l Minimization (IRLM) [2], and the l -norm using CVX [5]. The data sets are designed as follows: A s entries are randomly samled and x has k indices samled from N (, ), but all other entries are set to zero. The matrix A is full rank. We assess the erformance of the different algorithms using two criteria: The reconstruction error defined as err(x) = x x 2 / x 2, where x is the solution from an algorithm, and x is the otimal solution. However, this error does not always cature the correct location of zeros. For instance, assume that x = [,, ] t and the solutions to two different algorithms are given by x = [5, 5, 5] t and x 2 = [,, 2] t. Then, we have err(x ) < err(x 2 ) although, intuitively, x 2 is closer to the desired solution. To cature the similarity of the zero locations, we use the Jaccard index comuted as Z Z, where Z Z Z and Z are the set of indices of the non-zero entries in the true solution and those returned by the algorithm, resectively, and denotes the cardinality of a set. The sarsity level arameter for and CoSaMP is set to k, which should give them an advantage at error evaluation. All results resented were averaged over 5 simulations of linear systems. Finally, reroducibles for these exeriments are ublicly available. 4.. Exeriment #: Fix Swee n The first simulation examines the imact of the number of measurements n with fixed dimension = 2 and fixed sarsity k = 8 (here k denotes the number of nonzero elements). Intuitively, the reconstruction errors should decrease as n increases. Figures (a)-(d) show the reconstruction errors and Jaccard stability indices for the noiseless and noisy cases. First, we observe that the reconstruction error of KRON dros to zero once n =, which is exactly what is exected, as KRON leads erfect reconstruction for n > k + [3]. Among all the other aroaches,, htt://github.com/gditzler// 8

9 with δ = 3 in these exeriments, rovides the smallest error and largest Jaccard index. is even able to outerform and IRLM. Furthermore, outerforms the other algorithms in the noisy scenarios as shown in Figures (c) and (d) Exeriment #2: Swee fix n The second simulation shows the imact, on the reconstruction error, when the dimension varies from 5,..., 25 and k =.5, corresonding to a fixed 95% sarsity level. The reconstruction errors (Figure 2), and Jaccard indices (Figure 3) are comuted for each of the different values of n. used a neighborhood of δ = 3. The sarsity threshold for and CoSaMP was set to 9. When n (i.e., n =.,.2), the l error is large and, although with a small neighborhood, is able to imrove uon the l aroximation and rovide nearly zero reconstruction error with high zero-location stability (for n.2). Furthermore, not only does lead to a low reconstruction error, but it is also able to identify the locations of the zeros as shown in the Jaccard stability figures. When n increases (here n =.3,.4), the l -aroximation is quite good in the sense that the l solution is very close to the otimal sarsest solution (both in terms of reconstruction error and stability). has a similar erformance as l, but with an even smaller reconstruction error. It is also worth noting that the erformance of CoSaMP also imroved as n increases Runtime Analysis The comutational run-times for all algorithms in the different scenario cases are shown in Figure 4. In this simulation, is fixed at 2 and the value of n is swet from to 7. As exected, the comutational comlexity of decreases because s becomes smaller, which decreases the number of combinations that need to be evaluated. Finally, the run-times can be imroved further by distributing the for loo in Figure across more cores. In our ongoing work, we are looking into GPU imlementation of the combinations involved in to further reduce the comutational time. 5. Conclusions In this aer, we resented and oi to address the issue of obtaining a tradeoff between reconstruction error and comutational comlexity for recovering sarse signals from under-determined linear systems. The l -norm solution may have a large or small error deending on the algebraic roerties of the system at hand. Greedy algorithms, such as and CoSaMP, run quickly but lead to a high error. starts with the l -aroximation and builds δ-neighborhoods, where a sarser solution can be found. These δ-neighborhoods can grow until reaching the otimal sarsest l -solution. δ is tunable deending on the available comutational resources. Our simulation results showed that, for n, the l -aroximation can have a high error and can bring this error down substantially with a comarable run-time (in seconds). The user can control the arameter δ deending on a tradeoff between the desired sarsity/otimality and the comutational ower available. Acknowledgements This work was suorted by the National Science Foundation under Award Numbers CCF and ACI

10 References [] P. Georgieva, N. Bouaynaya, F. Silva, L. Mihaylova, L. Jain, A beamformer-article filter framework for localization of correlated eeg sources, IEEE Journal of Biomedical and Health Informatics (25). [2] G. Ditzler, J. C. Morrison, Y. Lan, G. Rosen, Fizzy: Feature selection for metagenomics, BMC Bioinformatics 6 (25). [3] J. Khan, N. Bouaynaya, H. Fathallah-Shaykh, Tracking of time-varying genomic regulatory networks with a LASSO-kalman smoother, EURASIP Journal on Bioinformatics and Systems Biology 24 (24). [4] B. Bayar, N. Bouaynaya, R. Shterenberg, SMURC: High-dimension small-samle multivariate regression with covariance estimation, IEEE Journal of Biomedical and Health Informatics (26). To aear. [5] E. J. Candès, J. K. Romberg, T. Tao, Stable signal recovery from incomlete and inaccurate measurements, Communications on Pure and Alied Mathematics 59 (26) [6] J. Meng, W. Yin, Y. Li, N. Nguyen, Z. Han, Comressive sensing based high-resolution channel estimation for ofdm system, IEEE Journal on Selected Toics in Signal Processing 6 (22) [7] A. Massa, P. Rocca, G. Oliveri, Comressive sensing in electromagnetics - a review, IEEE Antennas and Proagation Magazine 57 (25) [8] M. Fornasier, H. Rauhut, Comressive Sensing, Handbook of Mathematical Methods in Imaging, 2. [9] T. Blumensath, M. E. Davis, Iterative hard thresholding for comressed sensing, Alied and Comutational Harmonic Analysis 27 (29) [] D. Needell, J. A. Tro, Cosam: iterative signal recovery from incomlete and inaccurate samles, Alied and Comutational Harmonic Analysis 26 (29) [] H. Mohimani, M. Babaie-Zadeh, C. Jutten, A fast aroach for overcomlete sarse decomosition based on smoothed l norm, IEEE Transactions on Signal Processing 57 (29) [2] E. Candés, M. B. Wakin, S. Boyd, Enhancing sarsity by reweighted l minimization, Journal of Fourier Analysis and Alications 4 (28) [3] B. Bayar, N. Bouaynaya, R. Shterenberg, Kernel reconstruction: an exact greedy algorithm for comressive sensing, in: International Worksho on Genomic Signal Processing and Statistics. [4] J. A. Tro, A. C. Gilbert, Signal recovery from random measurements via orthogonal matching ursuit, IEEE Transactions on Information Theory 53 (27) [5] M. Grant, S. Boyd, CVX: Matlab software for discilined convex rogramming, version 2., htt: //cvxr.com/cvx/, 24.

11 Algorithm seudo-code Inut: A, y, δ [n] Outut: x : x = argmin { x : Ax = y} 2: s = Ker(A) 3: Choose Bs s+δ to be the combinations of the smallest s + δ magnitudes in x // Parallel Loo 4: for q =,..., Bs s+δ do 5: Q = []\Bs s+δ (q) 6: x δ = 7: x δ = A Q y // Udate only the indices in Q 8: sar q = x δ 9: end for : q min = min{sar q : q B s+δ : Q = []\B s+δ s (q min ) 2: x = 3: x Q = A Q y s } reconstruction error.5 KRON CoSaMP IRM n (a) Error of noiseless system stability KRON CoSaMP IRM n (b) Stability of noiseless system reconstruction error.5 oi CoSaMP IRM n (c) Error of noisy system stability oi CoSaMP IRM n (d) Stability of noisy system Figure : Performance evaluation (using reconstruction error and Jaccard stability index) of with δ = 3, l -aroximation, CoSaMP,, IRLM and on synthetic data sets with fixed dimension = 2 and increasing number of measurements n. In the noisy systems, the error variance ɛ =.5. CoSaMP and were given the correct number of non-zero elements. reconstruction error 2.5 CoSam IRM reconstruction error CoSam IRM reconstruction error CoSam IRM reconstruction error CoSam IRM (a) n =. (b) n =.2 (c) n =.3 (d) n =.4 Figure 2: Reconstruction error of with δ = 3, l -aroximation, CoSaMP,, IRLM and on synthetic data sets of increasing dimensionality. k =.5 was fixed in all exeriments, corresonding to 95% sarsity level. KRON has been omitted due the the comutational comlexity mentioned in Section.

12 stability CoSam IRM stability CoSam IRM stability CoSam IRM stability CoSam IRM (a) n =. (b) n =.2 (c) n =.3 (d) n =.4 Figure 3: Jaccard stability index for with δ = 3, l -aroximation, CoSaMP,, IRLM and on synthetic data sets of increasing dimensionality and fixed k =.5. evaluation time CoSam IRM n Figure 4: Run-time in seconds for CoSaMP,, with δ = 3, and l -aroximation on synthetic data sets for a fixed = 2 and an increasing n. 2