arxiv: v1 [math.oc] 19 Oct 2017

Transcription

1 UNIFIED SUFFICIENT CONDITIONS FOR UNIFORM RECOVERY OF SPARSE SIGNALS VIA NONCONVEX MINIMIZATIONS HOANG TRAN AND CLAYTON WEBSTER arxiv: v1 [math.oc] 19 Oct 2017 Abstract. Nonconvex minimizationsgenerally produce solutions thatare closerto the l 0 penalty than widely utilized l 1 norm, and it is widely accepted that they are able to enhance the sparsity of the solutions. However, the theory verifying that nonconvex penalties are as good as or better than l 1 minimization in exact, uniform recovery of sparse signals has not been available beyond a few specific cases. In this paper, we aim to fill this gap by establishing a theoretical recovery guarantee through unified null space properties that encompass most of the currently proposed nonconvex functionals in the literature. These conditions are less demanding than or identical to the standard null space property, the necessary and sufficient condition for l 1 minimization. 1. Introduction. This paper is concerned with the reconstruction of a sparse signal x R N from relatively few observed data y R m. More precisely, we recover the unknown vector x R N from the system Az = y, (1.1) given the matrix A R m N (m N), and using linear measurements y = Ax. In general, the system (1.1) is underdetermined and has infinitely many solutions. With the additional acknowledgement that the unknown signal is sparse, which is the case in several contexts such as compressed sensing [2, 6], statistics [14], and uncertainty quantification [7, 19, 4], we search for the sparsest solution of (1.1) only. It is natural to reconstruct x via the l 0 minimization problem min z R N z 0, subject to Az = y, (1.2) where z 0 is the number of nonzero components in z. However, since the locations of the nonzero components are not available, solving (1.2) directly requires a combinatorial search and is unrealistic in general. An alternative and popular approach is basis pursuit or l 1 minimization, which consists in finding the minimizer of the problem min z R N z 1, subject to Az = y. (1.3) The convex optimization problem (1.3) is an efficient relaxation for (1.2) and often produces sparse solutions. The sparse recovery property of l 1 minimization has been well-developed. It is known from the compressed sensing literature that if A possesses certain properties, such as the null space property and restricted isometry property, problem (1.2) and its convex relaxation (1.3) are equivalent [11]. Although the l 1 minimization technique has been used in a wide variety of problems, it is not able to reconstruct the sparsest solutions in many applications. As such, several nonconvex regularizations have been applied to improve the recovery performance. These penalties are generally closer to the l 0 penalty than the l 1 norm, Department of Computational and Applied Mathematics, Oak Ridge National Laboratory, 1 Bethel Valley Road, P.O. Box 2008, Oak Ridge TN tranha@ornl.gov. Department of Mathematics, University of Tennessee, 1403 Circle Drive, Knoxville, TN and Department of Computational and Applied Mathematics, Oak Ridge National Laboratory, 1 Bethel Valley Road, P.O. Box 2008, Oak Ridge TN webstercg@ornl.gov. 1

2 thus it is reasonable to expect that nonconvex minimizations can further enhance the sparsity of the solutions. The most commonly used nonconvex penalty is probably l p with 0 < p < 1 [3, 10], which interpolates between l 0 and l 1. Other well-known nonconvex methods in the literature include Smoothly Clipped Absolute Deviation (SCAD) [9], capped l 1 [28, 21] and transformed l 1 [17]. All of these penalties are separable, in the sense that they are the sum of the penalty functions applied to each individual component of the vector. Recently, many non-separable regularizations are also considered, for instance, iterative support detection [22], l 1 l 2 [8, 24, 23], two-levell 1 [15]andsortedl 1 [16]. Whilenonconvexpenaltiesaregenerallymorechallenging to minimize, they have been shown to reconstruct the sparse target signals from significantly fewer measurements in many computational test problems. Yet, to date, the theoretical evidence that nonconvex penalties are superior to (or at least, not worse than) l 1 optimization in uniform reconstruction of sparse signals has not been fully developed. For some regularizations such as l p (0 < p < 1) [10], iterative support detection [22], recovery guarantees are available, established via variants of restricted isometry property (RIP), and proved to be less restrictive than that of l 1. However, such properties are missing for many others including SCAD, capped l 1 and sorted l 1, to the best of our knowledge. On the other hand, while available for l 1 l 2 [24, 23], two-level l 1 [15] and transformed l 1 [27], the current recovery guarantees are more demanding or not directly comparable to that of l 1. This point can be elucidated by an examination of the null space property. For any v = (v 1,...,v N ) R N and S {1,...,N}, denote by S the complement of S in {1,...,N} and #(S) the cardinality of S, then the null space property of the measurement matrix A, introduced in [5], can be stated as follows. Definition 1.1 (Null spaceproperty). For the matrix A R m N, the null space property is given by: { } ker(a)\{0} v R N : v S 1 < v S 1, S {1,...,N} with #(S) s. (NSP) It is well-known that (NSP) is the necessary and sufficient condition for the successful reconstruction using l 1 minimization [5, 11], thus, stronger recovery properties, which are desirable and expected for nonconvex minimizations, would essentially allow ker(a) to be contained in a larger set than that in (NSP). The arguments in [24, 23, 15] are, however, at odds with this observation. Therein, the RIP were developed so that { } ker(a)\{0} v R N : v S 1 < v H S 1, S: #(S) s, H: #(H) = N/2, { } and ker(a)\{0} v R N : v S 1 + v S 2 + v S 2 < v S 1, S: #(S) s, respectively for two-level l 1 and l 1 l 2 penalties. As then ker(a) was restricted to strictly smaller sets than as in l 1 case, the acquired RIPs were inevitably more demanding. In this paper, we seek to establish theoretical, generalized results which verify that nonconvex minimizations are superior to or at least as good as l 1 in uniform recovery of sparse vectors. More specifically, we consider the nonconvex optimization problem in general form minimize R(z) subject to Az = Ax, (P R ) z R N 2

3 and, under some mild assumptions of R (which are applicable to most nonconvex penalties considered in the literature), and derive unified sufficient conditions for the exact reconstruction via (P R ). These conditions are based on several variants of the null space property, which are less demanding or identical to the standard conditions required by the l 1 norm. The main achievements of this paper are summarized below (for the precise definition of separable, concave and symmetric notions, we refer to Section 2): (i) For the regularizations that are separable and concave, the necessary and sufficient condition for the exact recovery can be generalized as ker(a)\{0} { } v R N : R(v S ) < R(v S ), S with #(S) s. (gnsp) This represents an extension of several similar results for specific penalties [10, 27]. The above null space property, however, is not automatically less restrictive than the standard (NSP). We verify that this is only the case if the regularization is also symmetric. Penalties that can be treated in this setting include l p, SCAD, transformed l 1, capped l 1. (ii) For the regularizations that are concave, symmetric, but non-separable(such as l 1 l 2, two-level l 1, sorted l 1 ), (NSP) is sufficient for the exact recovery. However, in many cases, a slightly weaker, yet improved variant of (NSP) is enough, i.e., ker(a)\{0} { } v R N : v S 1 v S 1, S with #(S) s. (insp) Our work provides, for the first time, the theoretical requirements for uniform recovery for many nonconvex penalties. For several others, the present unified conditions remarkably improve the existing theoretical results Related works. This paper examines the recovery of sparse signals by virtue of global minimizers of nonconvex problems. These are the best solutions one can acquire via the considered nonconvex regularizations, regardless of the numerical procedures used to realize them. Therefore, our results serve as a benchmark for the performance of concrete algorithms. Often in practice, one can only obtain the local minimizers of nonconvex problems. The theoretical recovery properties via local minimizers, attached to specific numerical schemes, has also gained considerable attention in the literature. In [28, 29], a multi-stage convex relaxation scheme was developed for solving problems with nonconvex objective functions, with a focus on capped-l 1. Theoretical error bound was established for fixed designs showing that the local minimum solution obtained by this procedure is superior to the global solution of the standard l 1. In [25], l 1 l 2 minimization was proved not worse than l 1, based on a difference of convex function algorithm (DCA), which is an iterative procedure and returns l 1 solution in the first step. Generalized conditions for nonconvex penalties were also established in [9, 17], under which three desirable properties of the regularizations - unbiasedness, sparsity, and continuity - are fulfilled. Therein, the properties of the local minimizers and the sufficient conditions for a vector to be the local minimizer were analyzed with a unified approach, and specified for SCAD and transformed l 1 (referred to as SICA in [17]). We remark that this framework applies to separable penalties only. 3

4 Finally, sorted l 1 is a nonconvexmethod recently introduced in [16]. This method generalizes several nonconvex approaches, including iterative hard thresholding, twolevel l 1, truncated l 1, and small magnitude penalized (SMAP) Organization. The remainder of this paper is organized as follows. In Section 2, we describe a general nonconvex minimization problem, some examples, and provide the necessary background results. In Section 3, we derive the conditions for the exact reconstruction via separable penalties. The recovery conditions for nonseparable penalties are established in Section 4. Finally, concluding remarks are given in Section General nonconvex minimization problem. Throughout this paper, we denote U = [0, ) N, and for any z = (z 1,...,z N ) R N, let z = ( z 1,..., z N ) with z [1]... z [N] the components of z in decreasing order. If z has nonnegative components, i.e., z U, we simply write z [1]... z [N]. We call a vector z R N : equal-height if all nonzero coordinates of z have the same magnitude; and s-sparse if it has at most s nonzero coefficients. Also, e j is the standard basis vector with a 1 in the j-th coordinate and 0 s elsewhere. The j-th coordinate of a vector z R N is often denoted simply by z j, but for convenience, we also use the notation (z) j. Recall the nonconvex optimization problem of interest is given by minimize R(z) subject to Az = Ax. (P R ) z R N We define the following theoretical properties of the penalty R described in (P R ). Definition 2.1. Let R be a mapping from R N to [0, ) satisfying R(z 1,...,z N ) = R( z 1,..., z N ), for all z = (z 1,...,z N ) R N. R is called separable on R N if there exist functions r j : R [0, ), j {1,...,N} such that for every z R N, R can be represented as R(z) = N r j (z j ). (2.1) R is called symmetric on R N if for every z R N and every permutation (π(1),...,π(n)) of (1,...,N): R(z π(1),...,z π(n) ) = R(z). (2.2) R is called concave on U if for every z,z U and 0 λ 1: R(λz +(1 λ)z ) λr(z)+(1 λ)r(z ). (2.3) R is called increasing on U if for every z,z U, z z then Here, z z means z j z j, 1 j N. R(z) R(z ). (2.4) 4

5 l l 1 l capped l two level l 1 Fig The contour maps of four different nonconvex functionals. The theory for uniform recovery developed herein is applicable for all symmetric, concave, and(for some stronger results) separable penalties. In particular, the variants of the null space property for the following regularizations can be derived from our general framework. Example 2.2. Several nonconvex optimization problems and the corresponding penalties: 1. l p norm with 0 < p < 1: R lp (z) = z p p, [3, 10]. 2. SCAD: R SCAD (z) = N r SCAD(z j ), [9]. a 1 z j, if z j < a 1, Here, r SCAD (z j ) = a1 zj 2 2a 1a 2 z j +a 3 1 2(a 2 a 1) if a 1 z j a 2, a 1a 2+a if z j > a Transformed l 1 : R tl1 (z) = N ρ a(z j ), [17]. Here, ρ a (z j ) = (a+1) zj a+ z j, z j R with a (0, ). 4. Capped l 1 : R cl1 (z) = N min{ z j,α}, [28]. 5. l 1 l 2 : R l1 l 2 (z) = z 1 z 2, [8, 24]. 6. Two-level l 1 : R 2l1 (z) = ρ z j + z j, [15]. j J(z) j J(z) c Here, 0 ρ < 1 and J(z) is the set of largest components of z j. 5

6 7. Sorted l 1 : R sl1 (z) = β 1 z [1] +...+β N z [N], [16]. Here, 0 β 1... β N and z [1]... z [N] are the components of z ranked in decreasing order. We illustrate several contour maps, from Example 2.2, in Figure 2, and remark that the following penalties are separable: l p ; SCAD; transformed l 1 ; and capped l 1. Note that the l p regularization defined here is slightly different from those commonly found in the literature, i.e., p. This definition is merely for ease of presentation, as the optimization problem (P R ) remains unchanged whether p p or p is used, while p p is a separable penalty in the strict sense of (2.1). The next proposition establishes the concavity and symmetry of all functions listed in the above example. Proposition 2.3. All functions R listed in Example 2.2 are symmetric on R N and concave on U. Proof. It is easy to see that all penalty functions in Example 2.2 are symmetric. We will verify their concavity one by one. Note that for separable regularizations (R SCAD, R tl1 and R cl1 ), it is enough to show the desired property for the univariate penalty applied on each component of the vector. 1. R lp : This regularization is separable. The univariate penalty r lp (z j ) = z p j is concave in (0, ) since r l p (z j ) = p(p 1)z p 2 j < 0, z j > 0. Thus, R tl1 is concave. 2. R SCAD : This regularization is separable. The univariate penalty r SCAD (z j ) is continuously differentiable with a 1, if 0 < z j < a 1, r SCAD (z a j) = 1(a 2 z j) a 2 a 1 if a 1 z j a 2, 0 if z j > a 2. Since r SCAD is decreasing in (0, ), r SCAD is concave. 3. R tl1 : This regularization is separable. The univariate penalty r tl1 (z j ) = (a+1)z j a+z j = a + 1 a+a2 a+z j is concave in (0, ) since r tl 1 (z j ) = 2(a+a2 ) (a+z j) 3 < 0, z j > 0. Thus, R tl1 is concave. 4. R cl1 : Thisregularizationisseparable. Theconcavityoftheunivariatepenalty function r cl1 (z j ) = min{ z j,α} can be shown as follows. Let z j,z j 0 and 0 λ 1, we have λz j λmin{z j,α}, and (1 λ)z j (1 λ)min{z j,α}, thus λz j +(1 λ)z j λmin{z j,α}+(1 λ)min{z j,α}. (2.5) Also, it can be seen that α = λα+(1 λ)α λmin{z j,α}+(1 λ)min{z j,α}. (2.6) Combining (2.5) and (2.6) gives r cl1 (λz j + (1 λ)z j ) λr cl 1 (z j ) + (1 λ)r cl1 (z j ). Thus, R cl 1 is concave. 5. R l1 l 2 : z z 2 is convex, thus R l1 l 2 is concave. 6. R 2l1 : This regularization is a specific case of sorted l 1 regularization. Its concavity can be deduced from that of sorted l 1 penalty function (see below). 7. R sl1 : Let z,z U and 0 λ 1. Let (π(1),...,π(n)) be a permutation of (1,...,N) such that (λz +(1 λ)z ) π(1) (λz +(1 λ)z ) π(2)... (λz +(1 λ)z ) π(n). 6

7 Applying the rearrangement inequality, [13, p. 262], there follows R sl1 (λz +(1 λ)z ) = N β j (λz +(1 λ)z ) π(j) (2.7) N N N = λ β j z π(j) +(1 λ) β j z π(j) λ N β j z [j] +(1 λ) β j z [j] = λr sl1 (z)+(1 λ)r sl1 (z ), yielding that R sl1 is concave on U. Our analysis does not cover non-concave or non-symmetric regularizations. However, unlike their concave and symmetric counterparts, (NSP) is not sufficient to guarantee the uniform and sparse reconstruction with these penalties in general. Therefore, non-concave or non-symmetric regularizations are not necessarily better than l 1 minimization in exact, uniform recovery, in the sense that for some sampling matrices, l 1 can successfully recover all s-sparse vectors, while a non-concave or non-symmetric penalty fails to do so. A well-known example of non-concave penalties which are less efficient than l 1 is l p with p > 1. Non-symmetric regularizations, on the other hand, do not always recover the sparsest vectors due to their preference for some components over others. An example of a non-symmetric penalties is given below. Example 2.4. Consider the underdetermined system Az = Ax, where the matrix A R 4 5 is defined as A = As ker(a) = ( t, t,2t/3,2t/3,3t/4) satisfies (NSP) with N = 5 and s = 2, one can successfully reconstruct all 2-sparse vectors using l 1 minimization. Consider the weighted l 1 regularization 9. R wl1 (z) = 4 z 1 +3 z 2 + z 3 + z 4 + z 5, which is normally perceived as a convex penalty but also concave according to Definition 2.1. R wl1 is not symmetric, and we can see that not every 2-sparse vector can be recovered using this penalty, especially those whose first two entries are nonzero. For instance, if x = (1,1,0,0,0), all solutions to Az = Ax can be represented as z = (1 t,1 t,2t/3,2t/3,3t/4). Among these, the solution that minimizes R wl1 (z) is (0,0,2/3,2/3,3/4), different from x and not the sparsest one. However, it is worth noting that weighted l 1 minimization, with appropriate choices of weights can be a very efficient approach that outperforms l 1, in the case when a priori knowledge of (the structure of) the support set is available, see, e.g., [12, 26, 4, 1] Properties of penalty functions. Next, we prove some necessary supporting results for the penalty functions of interest. Lemma 2.5. Let R be a map from R N to [0, ). If R is concave on U, then R is increasing on U. 7

8 Proof. Assume that R is not increasing on U. There exist z,z U,z z such that R(z) < R(z ). For some α > 0, there holds R(z ) > R(z)+α. Let us define z = 1 R(z ) R(z) α (R(z )z (R(z)+α)z ), (2.8) it is easy to see that z U. Rearranging (2.8) gives z = R(z)+α and by the concavity of R, R(z ) z + R(z ) R(z) α R(z ) R(z) R(z)+α R(z ) R(z )+ R(z ) R(z) α R(z R( z) = R(z)+α+ R(z ) R(z) α ) R(z R( z). ) This implies R( z) < 0, a contradiction. We conclude that R is increasing on U. Remark 2.6. If R : R N [0, ) satisfies R(z 1,...,z N ) = R( z 1,..., z N ), z = (z 1,...,z N ) R N and is increasing on U, then R(z) R(z ) for all z,z R N with z j z j, 1 j N. (2.9) R is therefore increasing in the whole space R N in the sense of (2.9). We will use both terms increasing on U and increasing on R N interchangeably in the sequel. Lemma 2.7. Let R be a map from R N to [0, ) satisfying R(z 1,...,z N ) = R( z 1,..., z N ), z = (z 1,...,z N ) R N. If R is separable on R N and concave on U, then R is subadditive on R N : R(z +z ) R(z)+R(z ), z,z R N. z, Proof. Since R is separable, it is enough to show each of r j is subadditive, i.e., for all j {1,...,N} and z j,z j R, r j (z j +z j ) r j(z j )+r j (z j ). (2.10) First, r j is concave on [0, ) for all j {1,...,N}. Indeed, let z j,z j [0, ) and 0 λ 1, the concavity of R on U yields r j (λz j +(1 λ)z j )+ r j (0) = R(λz j e j +(1 λ)z j e j) j {1,...,N}\{j} λr(z j e j )+(1 λ)r(z je j ) = λr j (z j )+(1 λ)r j (z j)+ j {1,...,N}\{j} r j (0). Also, from Lemma 2.5, R is increasing on U, thus, r j is increasing on [0, ). Now, the subadditivity (2.10) holds for all z j,z j > 0, since ( z j r j (z j ) = r j z j +z j ( r j (z j ) = r j z j z j +z j ) 0+ z j z j +z j (z j +z j ) z j z j +z j r j (0)+ z j z j +z j r j (z j +z j ), ) 0+ z j z j +z j (z j +z j ) z j z j +z j r j (0)+ z j z j +z j r j (z j +z j ). Summing theaboveinequalitiessidebysidegivesr j (z j )+r j (z j ) r j(0)+r j (z j +z j ) r j (z j +z j ). 8

9 (2.10) can be extended to all z j,z j R as follows. If at least one of z j and z j is zero, then (2.10) is trivially true. Otherwise, from the subadditivity on (0, ) and the increasing property of r j, we have r j (z j )+r j (z j ) = r j( z j )+r j ( z j ) r j( z j + z j ) r j( z j +z j ) = r j(z j +z j ), as desired. Remark 2.8. In Lemma 2.7, the separable condition is necessary to guarantee the subadditive property of R. Indeed, consider the penalty function R l1 l 2 : R 2 [0, ), corresponding to l 1 l 2 regularization. R l1 l 2 is not separable and also not subadditive, as one has 2 2 = R l1 l 2 (1,1) > R l1 l 2 (1,0)+R l1 l 2 (0,1) = 0. To establish the generalized conditions for successful sparse recovery, especially in the non-separable case, we employ the concept of majorization. This notion, defined below, makes precise and rigorous the idea that the components of a vector are more (or less) equal than those of another. Definition 2.9 (Majorization, see [18]). For z,z U, z is said to be majorized by z, denoted by z z, if n z [j] n z [j], n = 1,...,N 1, N z [j] = N (2.11) z [j]. Given condition (2.11), we also say z majorizes z and denote z z. As a simple example of majorization, we have ( 1 6, 1 6, 1 6, 1 6, 1 6, 1 ) ( 1 6 4, 1 4, 1 4, 1 ) ( 3 4,0,0 8, 1 4, 1 4, 1 ) 8,0,0 ( 1 2, 1 ) ( 2 2,0,0,0,0 3, 1 ) 3,0,0,0,0 (1,0,0,0,0,0). Loosely speaking, a sparse vector tends to majorize a dense one with the same l 1 norm. On the other hand, a sparse-promoting penalty function should have small values at sparse signals and larger values at dense signals. One may think that a penalty function which reverses the order of majorization would promote sparsity, in particular, outperform l 1. We will show in the next sections that this intuition is indeed correct, but first, let us verify that all symmetric and concave penalty functions considered are order-reversing 1. Proposition Let R be a function from R N to[0, ) satisfying R(z 1,...,z N ) = R( z 1,..., z N ), z = (z 1,...,z N ) R N. If R is symmetric on R N and concave on U, then R reverses the order of majorization: R(z) R(z ) for all z,z U with z z. (2.12) 1 Functions that reverse the order of majorization are often referred to as Schur-concave functions, see, e.g., [18]. 9

10 Proof. First, by [18, Lemma 2.B.1], for any z,z U with z z, z can be derived from z by successive applications of a finite number of linear transformation of the form T( z) = ( z 1,..., z j 1,λ z j +(1 λ) z k, z j+1,..., z k 1,λ z k +(1 λ) z j, z k+1,..., z N ), where λ [0,1] and 1 j k N. Each transformation only changes two coordinates and creates a new vector that is majorized by the current one, thus, it is enough to prove (2.12) for the case z differs from z in at most two components. Since R is symmetric, we can assume z = (z 1,z 2,z 3,...,z N), and z = (λz 1 +(1 λ)z 2,λz 2 +(1 λ)z 1,z 3,...,z N). The concavity of R gives as desired. R(z) = R(λ(z 1,z 2,z 3,...,z N )+(1 λ)(z 2,z 1,z 3,...,z N )) λr(z 1,z 2,z 3,...,z N )+(1 λ)r(z 2,z 1,z 3,...,z N ) = R(z ), 3. Exact recovery of sparse signals via separable penalties. In this section, we investigate the condition for the exact recovery of sparse signals assuming the concave penalty is also separable. First, a general necessary and sufficient condition expressed in terms of null space property of the measurement matrix is established for concave and separable regularizations. This extends several similar results for specific penalties, e.g., l p [10], weighted l 1 [20], and transformed l 1 [27]. We then prove that if, in addition, the penalty function is symmetric, the present condition is always less demanding than that for l 1 minimization. Theorem 3.1. Let A be an m N real matrix. Consider the problem (P R ), where R is a function from R N to [0, ) satisfying R(z 1,...,z N ) = R( z 1,..., z N ), z = (z 1,...,z N ) R N, separable on R N, concave on U and R(0) = 0. Then every s- sparse vector x R N is the unique solution to (P R ) if and only if the generalized null space property (gnsp) is satisfied. Proof. First, assume that every s-sparse vector x R N is the unique solution to (P R ). For any v ker(a) \ {0} and any S [N] with #(S) s, v S is the unique minimizer of R(z) subject to Az = Av S. Observe that A( v S ) = Av S and v S v S (since v 0), we have R(v S ) < R(v S ), implying (gnsp). Conversely, assume that (gnsp) holds. Let x R N be an s-sparse vector whose support set is S. Any other solution to Az = Ax can be represented as z = x+v with v ker(a)\{0}. We have by the separable property of R and Lemma 2.7 R(x+v) = R(x+v)+R(0) = R(x+v S )+R(v S ) R(x) R(v S )+R(v S ) > R(x), (3.1) thus x is the unique solution to (P R ). Remark 3.2. (gnsp) is actually a necessary condition for the exact recovery of every s-sparse vector even when the separable and concave property on R is removed, thus applicable to a very general class of penalty functions. For this condition to become sufficient, the separable assumption on R is critical. In estimate (3.1), we utilize this assumption in two ways: splitting R(x+v) into R(x+v S )+R(v S ), and bounding R(x + v S ) R(x) R(v S ) via the subadditivity of R. Note that without 10

11 the separable property, a concave penalty may not be subadditive (see Remark 2.8). In this case, the analysis for the sufficient condition follows a significantly different and more complicated path (see Section 4). We note that the general condition (gnsp) for concave and separable penalties is not necessarily less demanding than (NSP), as exemplified by weighted l 1 minimization in Example 2.4. However, this will hold true if, in addition, the penalty function is symmetric. Our following result verifies that separable, concave, and symmetric regularizations are superior to l 1 minimization in sparse, uniform recovery. Proposition 3.3. Let N > 1, s N with 1 s < N/2. Assume R is a function from R N to [0, ) satisfying R(z 1,...,z N ) = R( z 1,..., z N ), z = (z 1,...,z N ) R N, separable and symmetric on R N, and concave on U. Also, R(0) = 0 and R(z) > 0, z 0. Then { } v R N : v S 1 < v S 1, S {1,...,N} with #(S) s { } v R N : R(v S ) < R(v S ), S {1,...,N} with #(S) s. Consequently, (gnsp) is less demanding than (NSP). Proof. Let v R N satisfy v S 1 < v S 1, S {1,...,N} with #(S) s. We denote by S the set of indices of s largest components of v (in magnitude), then v S 1 < v S 1. Since R is concave on U, by Lemma 2.5, R is increasing on U. It is enough to prove that R(v S ) < R(v S ). Let α = v S 1 v S 1 > 0 and j be an index in S, we define ṽ = v S + αe j. Since R is separable and R(z) > 0, z 0, one has R(ṽ) = R(v S )+R(αe j ) > R(v S ). (3.2) On the other hand, ṽ 1 = v S 1 and, by the definition of S, any nonzero component of ṽ (with the possible exception of the j-th one) is larger than any component of v S. This yields ṽ v S. Applying Proposition 2.10, there follows R(ṽ) R(v S ). Combining with (3.2), the proposition is concluded. 4. Exact recovery of sparse signals via non-separable penalties. In this section, we prove that concave and symmetric regularizations are superior to l 1 in sparse, uniform recovery, without the separable assumption. This setting applies to non-separable penalties such as two-level l 1, sorted l 1, and l 1 l 2. Our main result is given below. Theorem 4.1. Let N > 1, s N with 1 s < N/2, and A be an m N real matrix. Consider the problem (P R ), where R is a function from R N to [0, ) satisfying R(z 1,...,z N ) = R( z 1,..., z N ), z = (z 1,...,z N ) R N, symmetric on R N, and concave on U. i) If R(z 1,...,z s,z s+1,0,...,0) > R(z 1,...,z s,0,...,0), z 1,...,z s+1 > 0, (R 1 ) then every s-sparse vector x R N is the unique solution to (P R ) provided that the null space property (NSP) is satisfied. In this sense, (P R ) is at least as good as l 1 -minimization. ii) If R(z 1,...,z s 1,z s,z s+1,0,...,0) > R(z 1,...,z s 1,z s +z s+1,0,...,0), 11 z 1,...,z s+1 > 0, (R 2 )

12 then every s-sparse vector x R N (except equal-height vectors) is the unique solution to (P R ) provided that the improved null space property (insp) is satisfied. The recovery guarantee of (P R ) therefore is better than that of l 1 - minimization. The proof of this theorem is rather lengthy and is therefore relegated to Section 4.1. We can observe from this proof that the concavity and symmetry of the penalty function R is enough to guarantee every s-sparse vector is a solution to (P R ). For the exact recovery, we also need these solutions to be unique. Such uniqueness could be derived assuming R is strictly concave, but several regularizations do not satisfy this property. Rather, we only require strict concavity (or strictly increasing property) in one direction and locally at s-sparse vectors, reflected in (R 1 ) and (R 2 ). These mild conditions can be validated easily for the considered non-separable penalties; see Proposition 4.2. We note that (R 1 ) is weaker than (R 2 ). On the other hand, (insp) cannot guarantee the exact recovery of equal-height, s- sparse vectors with symmetric penalties in general. Indeed, it is possible that ker(a) contains equal-height, 2s-sparse vectors, for example, 2s e j, in which case the recovery problem of an equal-height, s-sparse vector, say z = s ze j, would essentially have at least another solution, namely z = 2s j=s+1 ze j, as R(z) = R(z ). We therefore exclude the reconstruction of equal-height vectors under (insp). Proposition 4.2. i) The following methods: two-level l 1, sorted l 1 with β s+1 > 0 are at least as good as l 1 -minimization in recovering sparse vectors in the sense that these methods exactly reconstruct all s-sparse vectors under the null space property (NSP). ii) The following methods: l 1 l 2, sorted l 1 with β s+1 > β s are provably superior to l 1 -minimization in recovering sparse vectors in the sense that these methods exactly reconstruct all s-sparse (except equal-height) vectors under the improved null space property (insp). Proof. In this proof, for convenience, we often drop the zero components when denoting vectors in R N, for instance, (z 1,...,z s,0,...,0) with z i 0, 1 i s and s < N is simply represented as (z 1,...,z s ). Applying Proposition 2.3 and Theorem 4.1, we only need to show that: i) R 2l1 and R sl1 with β s+1 > 0 satisfy (R 1 ) and ii) R l1 l 2 and R sl1 with β s+1 > β s satisfy (R 2 ). 1. R l1 l 2 : For z 1,...,z s+1 > 0, ( ) z z2 s+1 +z s+1 = thus, z s+1 both sides yields (R 2 ). 2. R 2l1 : We have ( z z2 s+1 +z s+1 z z2 s > z zs 2 )( z z2 s+1 z s+1 z z2 s+1 > z z2 s. Adding z z s in R 2l1 (z 1,...,z s+1 ) R 2l1 (z 1,...,z s )+ρz s+1 > R 2l1 (z 1,...,z s ) for all z 1,...,z s+1 > 0 and (R 1 ) is deduced. 12 ),

13 3. R sl1 : Let us define z = (z 1,...,z s+1,0,...,0) R N and assume z s and z s+1 are the T-th and t-th largest components of z, i.e., z s = z [T], z s+1 = z [t]. Consider β s+1 > 0, we assume t < s + 1 (the other case t = s + 1 is trivial). For any j with t j s, β j z [j] β j z [j+1]. At j = s+1, we estimate β s+1 z [s+1] > 0. There follows s+1 t 1 R sl1 (z 1,...,z s+1 ) = β j z [j] > β j z [j] + s β j z [j+1] = R sl1 (z 1,...,z s ), giving (R 1 ). Next, consider β s+1 > β s. Without loss of generality, assume t > T and also t < s+1 (the below argument also applies to t = s+1 with minor changes). At j = t, weestimateβ t z [t] β T z [t]. Forallj with t+1 j s+1, β j z [j] β j 1 z [j]. In particular, at j = s+1, the strict inequality holds, i.e., β s+1 z [s+1] > β s z [s+1]. Combining these facts and applying rearrangement inequality yield j=t s+1 t 1 R sl1 (z 1,...,z s+1 ) = β j z [j] > β j z [j] +β T z [t] + t 1 = β j z [j] + j T s+1 j=t+1 We obtain (R 2 ) and complete the proof. s+1 j=t+1 β j 1 z [j] β j 1 z [j] +β T (z [T] +z [t] ) R sl1 (z 1,...,z s 1,z s +z s+1 ). We conclude this section by pointing out that Theorem 4.1 does not have an analog in the fixed support scenario. More specifically, it may be tempting to think that (NSP) can be proved to be a sufficient condition for nonconvex minimizations by the same mechanism as in l 1 case, i.e., combining all conditions for the recovery of vectors supported on fixed sets S of cardinality s, i.e., } ker(a)\{0} {v R N : v S 1 < v S 1, (4.1) see[11, Section4.1]. However,wecanshowthatunlikel 1, (4.1)doesnotguaranteethe successful recovery of vectors supported on S with concave and symmetric penalties. Indeed, consider the underdetermined system Az = Ax, where the matrix A R 4 5 is defined as A = As ker(a) = ( t, t,3t/2,t/2,t/10) satisfies (4.1) with S = {1,2}, all sparse signals supported on S can be exactly recovered with l 1 penalty. Consider l 1 l 2 regularization, let x = (1,1,0,0,0) supported on S, then any solution to Az = Ax can be represented as (1 t,1 t,3t/2,t/2,t/10). Among those, the unique minimizer of R l1 l 2 (z) is z=(0,0,3/2,1/2,1/10), which is different from x and not the sparsest solution. In this case, l 1 succeeds in recovering the sparsest signal while l 1 l 2 does not. 13.

14 4.1. Proof of Theorem 4.1. First, since R is concaveon U, R is also increasing on R N ; see Lemma 2.5 and Remark 2.6. In what follows we denote K 1 := {v R N : v S 1 < v S 1, S {1,...,N} with #(S) s}, K 2 := {v R N \{0} : v S 1 v S 1, S {1,...,N} with #(S) s}. Recall that if A satisfies (NSP) or correspondingly (insp), then ker(a)\{0} K 1 or ker(a) \{0} K 2 respectively. Let x be a fixed s-sparse vector in R N, then z = x solves Az = Ax and any other solution of this system can be written as z = x+v with some v ker(a)\{0} K 1 2. We will show in the next part that: if (R 1 ) holds then R(x+v) > R(x), v K 1 ; and if (R 2 ) holds and x is not an equal-height vector then R(x+v) > R(x), v K 2, thus x is the unique solution to (P R ) assuming either: (i) (R 1 ) and (NSP); or (ii) (R 2 ) and (insp) and x is not an equal-height vector. Since R is symmetric, without loss of generality, we assume x = (x 1,...,x s,0,...,0), where x i 0, 1 i s. (4.2) For every v K 1 2, there exists ṽ K 1 2 that R(x+ṽ) R(x+v), and the s first components of ṽ are nonpositive, i.e., ṽ i 0, 1 i s. Indeed, let v = (v 1,...,v N ) be a vector in K 1 2. For any 1 i s, if v i > 0, we flip the sign of v i, i.e., replacing v i by v i. Denoting the newly formed vector by ṽ, then ṽ K 1 2 and (x+ṽ) i = x i v i x i +v i = (x+v) i, for every i where the sign is flipped, while (x +ṽ) i = (x+v) i, for other i. By the increasing property of R, ṽ satisfies R(x+ṽ) R(x+v). The first s components of ṽ are nonpositive by definition. It is therefore enough to consider v K 1 2 represented as v = ( a 1,..., a s,b 1,...,b t,0,...,0), (4.3) where {a i }, {b j } are correspondingly nonnegative and positive sequences. We denote by E(v) the multiset {a i : i 1,s} {b j : j 1,t}, by U(v) the multiset containing s largest elements in E(v) and L(v) := E(v)\U(v). Let b(v) := minu(v), and σ(v) and λ(v) be the sum of all elements in U(v) and L(v) respectively. Then v K 1 (or v K 2 correspondingly) if and only if σ(v)< λ(v) (σ(v) λ(v) and v 0 respectively). Also, note that t s+1 for all v K 1, and t s for all v K 2 14

15 with t = s occurring only if v is an equal-height vector. Below we consider two cases. Case 1: Assume (R 1 ) and v K 1 as in (4.3). We will show there exists v K 1 that v =( a 1,..., a s,b,...,b,b }{{} T,0,...,0), for some T s+1, 0 < b T b b(v), T 1 and R(x+ v) R(x+v). First, we replace all components b j in U(v) (which satisfy b j b) by b and subtract a total amount b j U(v) (b j b) from b j s in L(v). It is possible to form an elementwise nonnegative vector (referred to as v ) with this step, since Indeed, one has it gives a i U(v) a i + a i L(v) b j U(v) b j U(v) a i sb b j < a i L(v) (b j b) < a i U(v) a i + b j L(v) a i + b j L(v) yielding (4.4). We remark that v K 1, since σ(v )= σ(v) b j U(v) b j U(v) b j b j. (4.4) b. Combining with σ(v)< λ(v), a i U(v) (b j b) < λ(v) b j U(v) a i + b j U(v) b+ (b j b) = λ(v ). b j L(v) Ontheotherhand, bytheconstruction,themagnitudesofcoordinatesofx+v areless thanorequaltothatofx+v. TheincreasingpropertyofR givesr(x+v ) R(x+v). Now, representing v as v = ( a 1,..., a s,b 1,...,b t,0,...,0), we observe σ(v ) sb, as v has at least s components whose magnitudes are not less than b. Thus, t b j λ(v ) > σ(v ) sb and there exist T s+1 and b T (0,b] such that (T 1)b+b T = t. We define b j v = ( a 1,..., a s,b,...,b,b }{{} T,0,...,0). T 1 One has U( v) = U(v ), which implies σ( v) = σ(v ) and λ( v) = λ(v ). Then, v K 1 can be deduced from the fact that v K 1. As b j b 1 j t, it is easy to see x+v x+ v. From Proposition 2.10, there follows R(x+ v) R(x+v ). We proceed to prove R(x) < R(x+ v). Let a = s i=1 a i/s. If a = 0, the assertion can be deduced easily from the increasing property of R and (R 1 ). Let us consider a > 0. Since v K 1, (T 1)b+b T > sa. There exists 0 < κ < 1 such that (T 1)κb+κb T > sa > (T 1)κb. 15 b j,

16 We write (T 1)κb+κb T = sa+a = a a s +a, then 0 < a < κb T. Also, note that a > κb, as T 1 s. Denoting z 1 = ( x 1 a 1,..., x s a s,κb,...,κb,κb }{{} T,0,...,0), T 1 z 2 = ( x 1 a 1,..., x s a s,a,...,a,a,0,...,0), }{{} s z 3 = ( x 1 a 1,..., x s a s,a 1,...,a s,a,0,...,0), z 4 = ( x 1 a 1 +a 1,..., x s a s +a s,a,0,...,0), x = (x 1,...,x s,a,0,...,0), (4.5) there holds z 1 z 2 z 3 z 4. Applying Proposition 2.10 yields On the other hand, x+ v z 1 and z 4 x, thus We have from (R 1 ) that R(z 1 ) R(z 2 ) R(z 3 ) R(z 4 ). (4.6) R(x+ v) R(z 1 ) and R(z 4 ) R(x ). (4.7) R(x ) > R(x). (4.8) Combining (4.6) (4.8) gives R(x + v) > R(x), as desired. Case 2: Assume (R 2 ) and v K 2 as in (4.3) and x is not an equal-height vector. Following the arguments in Case 1, there exists v K 2 that v =( a 1,..., a s,b,...,b,b }{{} T,0,...,0), for some T s+1, 0 b T < b b(v), T 1 and R(x+ v) R(x+v). Note that here b T [0,b), implying #(supp( v)) 2s (rather than b T (0,b] and #(supp( v)) 2s+1 as in previous case). First, if #(supp( v)) = 2s, then v must be an equal-height vector: We denote v = ( b,..., b,b,...,b,0,...,0), }{{}}{{} s s x+ v = (x 1 b,...,x s b,b,...,b,0,...,0). }{{} s z 5 = ( x 1 b +b,..., x s b +b,0,...,0). Since x is not an equal-height vector, x i b = 0 for some 1 i s. Proposition 2.10 and assumption (R 2 ) give R(z 5 ) < R(x+ v). It is easy to see x z 5, therefore, R(x) R(z 5 ), and we arrive at R(x) < R(x+ v). Otherwise, if #(supp( v)) 2s+1, then (T 1)b+b T > sb. Let us again denote a = s i=1 a i/s and consider a > 0. Since v K 2, (T 1)b+b T sa, and we can find 0 < κ 1 that (T 1)κb+κb T sa > sκb. 16

17 We write (T 1)κb+κb T = sa+a, then a 0 and a > κb. Denoting z 1,z 2,z 3,z 4 and x as in (4.5), similarly to Case 1, there holds R(x) R(x ) R(z 4 ) R(z 3 ) R(z 2 ) R(z 1 ) R(x+ v) due to z 1 z 2 z 3 z 4, z 4 x x and z 1 x + v. We show that the strict inequality must occur somewhere in the chain. If a > 0, we have from (R 2 ) that R(x) < R(x ). Otherwise, if a = 0, then #(supp(z 1 )) s+1 and #(supp(z 4 )) s. Applying Proposition 2.10 and assumption (R 2 ) gives R(z 4 ) < R(z 1 ). This concludes the proof. 5. Concluding remarks. In this effort, we establish theoretical, generalized sufficient conditions for the uniform recovery of sparse signals via concave and symmetric regularizations. These conditions are all proved less restrictive than the standard null space property for l 1 minimization, thus verifying that concave and symmetric penalties are better than or at least as good as l 1 in enhancing the sparsity of the solutions. Our work unifies and improves existing NSP-based conditions developed for many specific penalties, and also provides first theoretical recovery guarantees in some cases. Extending the present results to the more practical scenarios, which allows measurement errors and compressible (i.e., only close to sparse) signals is the next logical step. In particular, an important open question is: are concave and symmetric regularizations still provably better than l 1 in uniform recovery, when taking noise and sparsity defect into account? Also, the general sufficient conditions for non-separable penalties, established herein from that of l 1, may be suboptimal for specific penalties. It will be interesting to investigate the specialized and optimized conditions in such cases. Finally, while the advantage of nonconvex minimizations over l 1 in terms of null space property is obvious, how this advantage reflects itself in sample complexity is unclear to us and a topic for future work. REFERENCES [1] Adcock, B.: Infinite-dimensional compressed sensing and function interpolation. Found Comput Math (2017) [2] Candès, E., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inform. Theory 52(1), (2006) [3] Chartrand, R.: Exact reconstruction of sparse signals via nonconvex minimization. IEEE Signal Processing Letters 14(10), (2007) [4] Chkifa, A., Dexter, N., Tran, H., Webster, C.: Polynomial approximation via compressed sensing of high-dimensional functions on lower sets. Math. Comp. (2017) [5] Cohen, A., Dahmen, W., DeVore, R.: Compressed sensing and best k-term approximation. J. Amer. Math. Soc. 22, (2009) [6] Donoho, D.L.: Compressed sensing. IEEE Trans. Inform. Theory 52(4), (2006) [7] Doostan, A., Owhadi, H.: A non-adapted sparse approximation of pdes with stochastic inputs. Journal of Computational Physics 230, (2011) [8] Esser, E., Lou, Y., Xin, J.: A method for finding structured sparse solutions to nonnegative least squares problems with applications. SIAM J. Imaging Sci. 6(4), (2013) [9] Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association 96(456), (2001) [10] Foucart, S., Lai, M.J.: Sparsestsolutions ofunderdetermined linearsystemsvial q-minimization for 0 < q 1. Appl. Comput. Harmon. Anal. pp (2009) [11] Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing. Applied and Numerical Harmonic Analysis. Birkhäuser (2013) [12] Friedlander, M., Mansour, H., Saab, R., Yilmaz, O.: Recovering compressively sampled signals 17

18 using partial support information. IEEE Transactions on Information Theory 58(2), (2012) [13] Hardy, G., Littlewood, J., Pólya, G.: Inequalities, 2nd edn. Cambridge University Press, London and New York (1952) [14] Hastie, T., Tibshirani, R., Wainwright, M.: Statistical Learning with Sparsity, Monographs on Statistics and Applied Probability, vol Taylor & Francis Group, LLC (2015) [15] Huang, X., Liu, Y., Shi, L., Van Huffel, S., Suykens, J.: Two-level l 1 minimization for compressed sensing. Signal Processing 108( ) (2015) [16] Huang, X., Shi, L., Yan, M.: Nonconvex sorted l 1 minimization for sparse approximation. Journal of Operations Research Society of China 3( ) (2015) [17] Lv, J., Fan, Y.: A unified approach to model selection and sparse recovery using regularized least squares. The Annals of Statistic 37(6A), (2009) [18] Marshall, A., Olkin, I., Arnold, B.: Inequalities: Theory of Majorization and Its Applications, 2nd edn. Springer Series in Statistics. Springer (2011) [19] Rauhut, H., Schwab, C.: Compressive sensing Petrov-Galerkin approximation of high dimensional parametric operator equations. Math. Comp. 86, (2017) [20] Rauhut, H., Ward, R.: Interpolation via weighted l 1 -minimization. Applied and Computational Harmonic Analysis 40(2), (2016) [21] Shen, X., Pan, W., Zhu, Y.: Likelihood-based selection and sharp parameter estimation. J Am Stat Assoc. 107(497), (2012) [22] Wang, Y., Yin, W.: Sparse signal reconstruction via iterative support detection. SIAM J. Imaging Sci. 3(3), (2010) [23] Yan, L., Shin, Y., Xiu, D.: Sparse approximation using l 1 l 2 minimization and its applications to stochastic collocation. SIAM J. Sci. Comput. 39(1), A229 A254 (2017) [24] Yin, P., Lou, Y., He, Q., Xin, J.: Minimization of l 1 2 for compressed sensing. SIAM J. Sci. Comput. 37(1), A536 A563 (2015) [25] Yin, P., Xin, J.: Iterative l 1 minimization for non-convex compressed sensing. submitted (2016) [26] Yu, X., Baek, S.: Sufficient conditions on stable recovery of sparse signals with partial support information. IEEE Signal Processing Letters 20(5), (2013) [27] Zhang, S., Xin, J.: Minimization of transformed l 1 penalty: Theory, difference of convex function algorithm, and robust application in compressed sensing. arxiv: v3[cs.it] (2016) [28] Zhang, T.: Multi-stage convex relaxation for learning with sparse regularization. In: D. Koller, D. Schuurmans, Y. Bengio, L. Bottou (eds.) Advances in Neural Information Processing Systems 21, pp Curran Associates, Inc. (2009) [29] Zhang, T.: Analysis of multi-stage convex relaxation for sparse regularization. Journal of Machine Learning Research 11, (2010) 18