LATELY, there has been a lot of fuss about sparse approximation. Just Relax: Convex Programming Methods for Identifying Sparse Signals in Noise

Transcription

1 1030 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 3, MARCH 2006 Just Relax: Convex Programming Methods for Identifying Sparse Signals in Noise Joel A. Tropp, Student Member, IEEE Abstract This paper studies a difficult and fundamental problem that arises throughout electrical engineering, applied mathematics, and statistics. Suppose that one forms a short linear combination of elementary signals drawn from a large, fixed collection. Given an observation of the linear combination that has been contaminated with additive noise, the goal is to identify which elementary signals participated and to approximate their coefficients. Although many algorithms have been proposed, there is little theory which guarantees that these algorithms can accurately and efficiently solve the problem. This paper studies a method called convex relaxation, which attempts to recover the ideal sparse signal by solving a convex program. This approach is powerful because the optimization can be completed in polynomial time with standard scientific software. The paper provides general conditions which ensure that convex relaxation succeeds. As evidence of the broad impact of these results, the paper describes how convex relaxation can be used for several concrete signal recovery problems. It also describes applications to channel coding, linear regression, and numerical analysis. Index Terms Algorithms, approximation methods, basis pursuit, convex program, linear regression, optimization methods, orthogonal matching pursuit, sparse representations. I. INTRODUCTION LATELY, there has been a lot of fuss about sparse approximation. This class of problems has two defining characteristics. 1) An input signal is approximated by a linear combination of elementary signals. In many modern applications, the elementary signals are drawn from a large, linearly dependent collection. 2) A preference for sparse linear combinations is imposed by penalizing nonzero coefficients. The most common penalty is the number of elementary signals that participate in the approximation. Sparse approximation problems arise throughout electrical engineering, statistics, and applied mathematics. One of the most common applications is to compress audio [1], images [2], and video [3]. Sparsity criteria also arise in linear regression [4], deconvolution [5], signal modeling [6], preconditioning [7], machine learning [8], denoising [9], and regularization [10]. Manuscript received June 7, 2004; revised February 1, This work was supported by the National Science Foundation Graduate Fellowship. The author was with the Institute for Computational Engineering and Sciences (ICES), The University of Texas at Austin, Austin, TX USA. He is now with the Department of Mathematics, The University of Michigan at Ann Arbor, Ann Arbor, MI USA ( jtropp@umich.edu). Communicated by V. V. Vaishampayan, Associate Editor At Large. Digital Object Identifier /TIT A. The Model Problem In this work, we will concentrate on the model problem of identifying a sparse linear combination of elementary signals that has been contaminated with additive noise. The literature on inverse problems tends to assume that the noise is an arbitrary vector of bounded norm, while the signal processing literature usually models the noise statistically; we will consider both possibilities. To be precise, suppose we measure a signal of the form where is a known matrix with unit-norm columns, is a sparse coefficient vector (i.e., few components are nonzero), and is an unknown noise vector. Given the signal, our goal is to approximate the coefficient vector. In particular, it is essential that we correctly identify the nonzero components of the coefficient vector because they determine which columns of the matrix participate in the signal. Initially, linear algebra seems to preclude a solution whenever has a nontrivial null space, we face an ill-posed inverse problem. Even worse, the sparsity of the coefficient vector introduces a combinatorial aspect to the problem. Nevertheless, if the optimal coefficient vector is sufficiently sparse, it turns out that we can accurately and efficiently approximate given the noisy observation. B. Convex Relaxation The literature contains many types of algorithms for approaching the model problem, including brute force [4, Sec ], nonlinear programming [11], and greedy pursuit [12] [14]. In this paper, we concentrate on a powerful method called convex relaxation. Although this technique was introduced over 30 years ago in [15], the theoretical justifications are still shaky. This paper attempts to lay a more solid foundation. Let us explain the intuition behind convex relaxation methods. Suppose we are given a signal of the form (1) along with a bound on the norm of the noise vector, say. At first, it is tempting to look for the sparsest coefficient vector that generates a signal within distance of the input. This idea can be phrased as a mathematical program (1) subject to (2) where the quasi-norm counts the number of nonzero components in its argument. To solve (2) directly, one must sift through all possible disbursements of the nonzero components in. This method is intractable because the search space is exponentially large [12], [16] /$ IEEE

2 TROPP: CONVEX PROGRAMMING METHODS FOR IDENTIFYING SPARSE SIGNALS IN NOISE 1031 To surmount this obstacle, one might replace the quasinorm with the norm to obtain a convex optimization problem subject to - where the tolerance is related to the error bound. Intuitively, the norm is the convex function closest to the quasi-norm, so this substitution is referred to as convex relaxation. One hopes that the solution to the relaxation will yield a good approximation of the ideal coefficient vector. The advantage of the new formulation is that it can be solved in polynomial time with standard scientific software [17]. We have found that it is more natural to study the closely related convex program As before, one can interpret the norm as a convex relaxation of the quasi-norm. So the parameter manages a tradeoff between the approximation error and the sparsity of the coefficient vector. This optimization problem can also be solved efficiently, and one expects that the minimizer will approximate the ideal coefficient vector. Appendix I offers an excursus on the history of convex relaxation methods for identifying sparse signals. C. Contributions The primary contribution of this paper is to justify the intuition that convex relaxation can solve the model problem. Our two major theorems describe the behavior of solutions to ( - ) and solutions to ( - ). We apply this theory to several concrete instances of the model problem. Let us summarize our results on the performance of convex relaxation. Suppose that we measure a signal of the form (1) and that we solve one of the convex programs with an appropriate choice of or to obtain a minimizer. Theorems 8 and 14 demonstrate that the vector is close to the ideal coefficient vector ; the support of (i.e., the indices of its nonzero components) is a subset of the support of ; moreover, the minimizer is unique. In words, the solution to the convex relaxation identifies every sufficiently large component of, and it never mistakenly identifies a column of that did not participate in the signal. As other authors have written, It seems quite surprising that any result of this kind is possible [18, Sec. 6]. The conditions under which the convex relaxations solve the model problem are geometric in nature. Section III-C describes the primary factors influencing their success. Small sets of the columns from should be well conditioned. The noise vector should be weakly correlated with all the columns of. These properties can be difficult to check in general. This paper relies on a simple approach based on the coherence parameter of, which measures the cosine of the minimal angle between - a pair of columns. It may be possible to improve these results using techniques from Banach space geometry. As an application of the theory, we will see that convex relaxation can be used to solve three versions of the model problem. If the coherence parameter is small both convex programs can identify a sufficiently sparse signal corrupted by an arbitrary vector of bounded norm (Sections IV-C and V-B); the program ( - ) can identify a sparse signal in additive white Gaussian noise (Section IV-D); the program ( - ) can identify a sparse signal in uniform noise of bounded norm (Section V-C). In addition, Section IV-E shows that ( - ) can be used for subset selection, a sparse approximation problem that arises in statistics. Section V-D demonstrates that ( - ) can solve another sparse approximation problem from numerical analysis. This paper is among the first to analyze the behavior of convex relaxation when noise is present. Prior theoretical work has focused on a special case of the model problem where the noise vector. See, for example, [14], [19] [23]. Although these results are beautiful, they simply do not apply to real-world signal processing problems. We expect that our work will have a significant practical impact because so many applications require sparse approximation in the presence of noise. As an example, Dossal and Mallat have used our results to study the performance of convex relaxation for a problem in seismics [24]. After the first draft [25] of this paper had been completed, it came to the author s attention that several other researchers were preparing papers on the performance of convex relaxation in the presence of noise [18], [26], [27]. These manuscripts are significantly different from the present work, and they also deserve attention. Turn to Section VI-B for comparisons and contrasts. D. Channel Coding It may be illuminating to view the model problem in the context of channel coding. The Gaussian channel allows us to send one real number during each transmission, but this number is corrupted by the addition of a zero-mean Gaussian random variable. Shannon s channel coding theorem [28, Ch. 10] shows that the capacity of the Gaussian channel can be achieved (asymptotically) by grouping the transmissions into long blocks and using a random code whose size is exponential in the block length. A major practical problem with this approach is that decoding requires an exponentially large lookup table. In contrast, we could also construct a large code by forming sparse linear combinations of vectors from a fixed codebook. Both the choice of vectors and the nonzero coefficients carry information. The vector length is analogous with the block length. When we transmit a codeword through the channel, it is contaminated by a zero-mean, white Gaussian random vector. To identify the codeword, it is necessary to solve the model problem. Therefore, any sparse approximation algorithm such as convex relaxation can potentially be used for decoding. To see that this coding scheme is practical, one must show that the codewords carry almost as much information as the channel permits. One must also demonstrate that the decoding algorithm can reliably recover the noisy codewords.

3 1032 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 3, MARCH 2006 We believe that channel coding is a novel application of sparse approximation algorithms. In certain settings, these algorithms may provide a very competitive approach. Sections IV-D and V-C present some simple examples that take a first step in this direction. See [29] for some more discussion of these ideas. E. Outline The paper continues with notation and background material in Section II. Section III develops the fundamental lemmata that undergird our major results. The two subsequent sections provide the major theoretical results for ( - ) and ( - ). Both these sections describe several specific applications in signal recovery, approximation theory, statistics, and numerical analysis. The body of the paper concludes with Section VI, which surveys extensions and related work. The appendices present several additional topics and some supporting analysis. II. BACKGROUND This section furnishes the mathematical mise en scène. We have adopted an abstract point of view that facilitates our treatment of convex relaxation methods. Most of the material here may be found in any book on matrix theory or functional analysis, such as [30] [32]. A. The Dictionary We will work in the finite-dimensional, complex innerproduct space, which will be called the signal space. 1 The usual Hermitian inner product for will be written as, and we will denote the corresponding norm by.adictio- nary for the signal space is a finite collection of unit-norm elementary signals. The elementary signals are called atoms, and each is denoted by, where the parameter is drawn from an index set. The indices may have an interpretation, such as the time frequency or time scale localization of an atom, or they may simply be labels without an underlying metaphysics. The whole dictionary structure is written as The letter will denote the number of atoms in the dictionary. It is evident that, where returns the cardinality of a finite set. B. The Coherence Parameter A summary parameter of the dictionary is the coherence, which is defined as the maximum absolute inner product between two distinct atoms [12], [19] When the coherence is small, the atoms look very different from each other, which makes them easy to distinguish. It is common that the coherence satisfies. We say informally that a dictionary is incoherent when we judge that the coherence 1 Modifications for real signal spaces should be transparent, but the apotheosis to infinite dimensions may take additional effort. (3) parameter is small. An incoherent dictionary may contain far more atoms than an orthonormal basis (i.e., ) [14, Sec. II-D]. The literature also contains a generalization of the coherence parameter called the cumulative coherence function [14], [21]. For each natural number, this function is defined as This function will often provide better estimates than the coherence parameter. For clarity of exposition, we only present results in terms of the coherence parameter. Parallel results using cumulative coherence are not hard to develop [25]. It is essential to be aware that coherence is not fundamental to sparse approximation. Rather, it offers a quick way to check the hypotheses of our theorems. It is possible to provide much stronger results using more sophisticated techniques. C. Coefficient Space Every linear combination of atoms is parameterized by a list of coefficients. We may collect them into a coefficient vector, which formally belongs to the linear space 2. The standard basis for this space is given by the vectors whose coordinate projections are identically zero, except for one unit coordinate. The th standard basis vector will be denoted. Given a coefficient vector, the expressions and both represent its th coordinate. We will alternate between them, depending on what is most typographically felicitous. The support of a coefficient vector is the set of indices at which it is nonzero Suppose that. Without notice, we may embed short coefficient vectors from into by extending them with zeros. Likewise, we may restrict long coefficient vectors from to their support. Both transubstantiations will be natural in context. D. Sparsity and Diversity The sparsity of a coefficient vector is the number of places where it equals zero. The complementary notion, diversity, counts the number of places where the coefficient vector does not equal zero. Diversity is calculated with the quasi-norm For any positive number,define with the convention that. As one might expect, there is an intimate connection between the definitions (5) and (6). Indeed,. It is well known that 2 In case this notation is unfamiliar, is the collection of functions from to. It is equipped with the usual addition and scalar multiplication to form a linear space. (4) (5) (6)

4 TROPP: CONVEX PROGRAMMING METHODS FOR IDENTIFYING SPARSE SIGNALS IN NOISE 1033 the function (6) is convex if and only if, in which case it describes the norm. From this vantage, one can see that the norm is the convex function closest to the quasi-norm (subject to the same normalization on the zero vector and the standard basis vectors). For a more rigorous approach to this idea, see [33, Proposition 2.1] and also [21]. E. Dictionary Matrices Let us define a matrix, called the dictionary synthesis matrix, that maps coefficient vectors to signals. Formally via The matrix describes the action of this linear transformation in the standard bases of the underlying vector spaces. Therefore, the columns of are the atoms. The conjugate transpose of is called the dictionary analysis matrix, and it maps each signal to a coefficient vector that lists the inner products between signal and atoms H. Operator Norms The operator norm of a matrix is given by An immediate consequence is the upper norm bound Suppose that and. Then we have the identity We also have the following lower norm bound, which is proved in [25, Sec. 3.3]. Proposition 1: For every matrix If has full row rank, equality holds in (8). When is invertible, this result implies (7) (8) The rows of are atoms, conjugate-transposed. The symbol matrix. denotes the range (i.e., column span) of a F. Subdictionaries A subdictionary is a linearly independent collection of atoms. We will exploit the linear independence repeatedly. If the atoms in a subdictionary are indexed by the set, then we define a synthesis matrix and an analysis matrix. These matrices are entirely analogous with the dictionary synthesis and analysis matrices. We will frequently use the fact that the synthesis matrix has full column rank. Let index a subdictionary. The Gram matrix of the subdictionary is given by. This matrix is Hermitian, it has a unit diagonal, and it is invertible. The pseudoinverse of the synthesis matrix is denoted by, and it may be calculated using the formula. The matrix will denote the orthogonal projector onto the span of the subdictionary. This projector may be expressed using the pseudoinverse:. On occasion, the distinguished index set is instead of. In this case, the synthesis matrix is written as, and the orthogonal projector is denoted. G. Signals, Approximations, and Coefficients Let be a subdictionary, and let be a fixed input signal. This signal has a unique best approximation using the atoms in, which is determined by. Note that the approximation is orthogonal to the residual. There is a unique coefficient vector supported on that synthesizes the approximation:. We may calculate that. III. FUNDAMENTAL LEMMATA Fix an input signal and a positive parameter.define the convex function In this section, we study the minimizers of this function. These results are important to us because (L) is the objective function of the convex program ( - ), and it is essentially the Lagrangian function of ( - ). By analyzing the behavior of (L), we can understand the performance of convex relaxation methods. The major lemma of this section provides a sufficient condition which ensures that the minimizer of (L) is supported on a given index set. To develop this result, we first characterize the (unique) minimizer of (L) when it is restricted to coefficient vectors supported on. We use this characterization to obtain a condition under which every perturbation away from the restricted minimizer must increase the value of the objective function. When this condition is in force, the global minimizer of (L) must be supported on. A. Convex Analysis The proof relies on standard results from convex analysis. As it is usually presented, this subject addresses the properties of real-valued convex functions defined on real vector spaces. Nevertheless, it is possible to transport these results to the complex (L)

5 1034 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 3, MARCH 2006 setting by defining an appropriate real-linear structure on the complex vector space. In this subsection, therefore, we use the bilinear inner product instead of the usual sesquilinear (i.e., Hermitian) inner product. Note that both inner products generate the norm. Suppose that is a convex function from a real-linear innerproduct space to. The gradient of the function at the point is defined as the usual (Fréchet) derivative, computed with respect to the real-linear inner product. Even when the gradient does not exist, we may define the subdifferential of at a point for every The elements of the subdifferential are called subgradients.if possesses a gradient at, the unique subgradient is the gradient. That is, The subdifferential of a sum is the (Minkowski) sum of the subdifferentials. Finally, if is a closed, proper convex function then is a global minimizer of if and only if. The standard reference for this material is [34]. Remark 2: The subdifferential of a convex function provides a dual description of the function in terms of its supporting hyperplanes. In consequence, the appearance of the subdifferential in our proof is analogous with the familiar technique of studying the dual of a convex program. B. Restricted Minimizers First, we must characterize the minimizers of the objective function (L). Fuchs developed the following result in the real setting using essentially the same method [23]. Lemma 3: Suppose that indexes a linearly independent collection of atoms, and let minimize the objective function (L) over all coefficient vectors supported on. A necessary and sufficient condition on such a minimizer is that where the vector is drawn from. Moreover, the minimizer is unique. Proof: Suppose that. Then the vectors and are orthogonal to each other. Apply the Pythagorean Theorem to see that minimizing (L) over coefficient vectors supported on is equivalent to minimizing the function (9) (10) over coefficient vectors from. Recall that has full column rank. It follows that the quadratic term in (10) is strictly convex, and so the whole function must also be strictly convex. Therefore, its minimizer is unique. The function is convex and unconstrained, so is a necessary and sufficient condition for the coefficient vector to minimize. The gradient of the first term of at equals. From the additivity of subdifferentials, it follows that for some vector drawn from the subdifferential.we premultiply this relation by to reach Apply the fact that to reach the conclusion. Now we identify the subdifferential of the end, define the signum function as for for norm. To that One may extend the signum function to vectors by applying it to each component. Proposition 4: Let be a complex vector. The complex vector lies in the subdifferential if and only if whenever, and whenever. Indeed, unless, in which case. We omit the proof. At last, we may develop bounds on how much a solution to the restricted problem varies from the desired solution. Corollary 5 (Upper Bounds): Suppose that indexes a subdictionary, and let minimize the function (L) over all coefficient vectors supported on. The following bounds are in force: (11) (12) Proof: We begin with the necessary and sufficient condition (13) where. To obtain (11), we take the norm of (13) and apply the upper norm bound Proposition 4 shows that, which proves the result. To develop the second bound (12), we premultiply (13) by the matrix and compute the norm As before,. Finally, we apply the identity (7) to switch from the norm to the norm.

6 TROPP: CONVEX PROGRAMMING METHODS FOR IDENTIFYING SPARSE SIGNALS IN NOISE 1035 C. The Correlation Condition Now, we develop a condition which ensures that the global minimizer of (L) is supported inside a given index set. This result is the soul of the analysis. Lemma 6 (Correlation Condition): Assume that indexes a linearly independent collection of atoms. Let minimize the function (L) over all coefficient vectors supported on. Suppose that where is determined by (9). It follows that is the unique global minimizer of (L). In particular, the condition guarantees that is the unique global minimizer of (L). In this work, we will concentrate on the second sufficient condition because it is easier to work with. Nevertheless, the first condition is significantly more powerful. Note that either sufficient condition becomes worthless when the bracket on its right-hand side is negative. We typically abbreviate the bracket in the second condition as The notation stands for exact recovery coefficient, which reflects the fact that is a sufficient condition for several different algorithms to recover the optimal representation of an exactly sparse signal [14]. Roughly, measures how distinct the atoms in are from the remaining atoms. Observe that if the index set satisfies, then the sufficient condition of the lemma will hold as soon as becomes large enough. But if is too large, then. Given a nonzero signal,define the function Then, any coefficient vector that minimizes the function (L) must be supported inside. One might wonder whether the condition is really necessary to develop results of this type. The answer is a qualified affirmative. Appendix II offers a partial converse of Lemma 6. D. Proof of the Correlation Condition Lemma We now establish Lemma 6. The argument presented here is different from the original proof in the technical report [25]. It relies on a perturbation technique that can be traced to the independent works [18], [25]. Proof of Lemma 6: Let be the unique minimizer of the objective function (L) over coefficient vectors supported on. In particular, the value of the objective function increases if we change any coordinate of indexed in. We will develop a condition which guarantees that the value of the objective function also increases when we change any other component of. Since (L) is convex, these two facts will imply that is the global minimizer. Choose an index not listed in, and let be a nonzero scalar. We must develop a condition which ensures that where is the th standard basis vector. To that end, expand the left-hand side of this relation to obtain Next, simplify the first bracket by expanding the norms and canceling like terms. Simplify the second bracket by recognizing that since the two vectors have disjoint supports. Hence, and place the convention that. This quantity measures the maximum correlation between the signal and any atom in the dictionary. To interpret the left-hand side of the sufficient conditions in the lemma, observe that Therefore, the lemma is strongest when the magnitude of the residual and its maximum correlation with the dictionary are both small. If the dictionary is not exponentially large, a generic vector is weakly correlated with the dictionary on account of measure concentration phenomena. Since the atoms are normalized, the maximum correlation never exceeds one. This fact yields a (much weaker) result that depends only on the magnitude of the residual. Corollary 7: Let index a subdictionary, and let be an input signal. Suppose that the residual vector satisfies Add and subtract in the left-hand side of the inner product, and use linearity to split the inner product into two pieces. We reach We will bound the right-hand side below. To that end, observe that the first term is strictly positive, so we may discard it. Then invoke the lower triangle inequality, and use the linearity of the inner products to draw out (14) It remains to rewrite (14) in a favorable manner. First, identify in the last term on the right-hand side of (14). Next, let us examine the second term. Lemma 3 characterizes the difference. Introduce this characterization, and identify the pseudoinverse of to discover that (15)

7 1036 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 3, MARCH 2006 where bound (14) is determined by (9). Substitute (15) into the (16) Our final goal is to ensure that the left-hand side of (16) is strictly positive. This situation occurs whenever the bracket is nonnegative. Therefore, we need We require this expression to hold for every index that does not belong to. Optimizing each side over in yields the more restrictive condition Since is orthogonal to the atoms listed in, the lefthand side does not change if we maximize over all from. Therefore, We conclude that the relation is a sufficient condition for every perturbation away from to increase the objective function. Since (L) is convex, it follows that is the unique global minimizer of (L). In particular, since, it is also sufficient that This completes the argument. IV. PENALIZATION Suppose that is an input signal. In this section, we study applications of the convex program The parameter controls the tradeoff between the error in approximating the input signal and the sparsity of the approximation. We begin with a theorem that provides significant information about the solution to ( - ). Afterward, this theorem is applied to several important problems. As a first example, we show that the convex program can identify the sparsest representation of an exactly sparse signal. Our second example shows that ( - ) can be used to recover a sparse signal contaminated by an unknown noise vector of bounded norm, which is a well-known inverse problem. We will also see that a statistical model for the noise vector allows us to provide more refined results. In particular, we will discover that ( - ) is quite effective for recovering sparse signals corrupted with Gaussian noise. The section concludes with an application of the convex program to the subset selection problem from statistics. The reader should also be aware that convex programs of the form ( - ) have been proposed for numerous other applications. Geophysicists have long used them for deconvolution [5], [35]. Certain support vector machines, which arise in machine learning, can be reduced to this form [8]. Chen, - Donoho, and Saunders have applied ( - ) to denoise signals [9], and Fuchs has put it forth for several other signal processing problems, e.g., in [36], [37]. Daubechies, Defrise, and De Mol have suggested a related convex program to regularize ill-posed linear inverse problems [10]. Most intriguing, perhaps, Olshausen and Field have argued that the mammalian visual cortex may solve similar optimization problems to produce sparse representations of images [38]. Even before we begin our analysis, we can make a few immediate remarks about ( - ). Observe that, as the parameter tends to zero, the solution to the convex program approaches a point of minimal norm in the affine space. It can be shown that, except when belongs to a set of signals with Lebesgue measure zero in, no point of the affine space has fewer than nonzero components [14, Proposition 4.1]. So the minimizer of ( - ) is generically nonsparse when is small. In contrast, as approaches infinity, the solution tends toward the zero vector. It is also worth noting that ( - ) has an analytical solution whenever is unitary. One simply computes the orthogonal expansion of the signal with respect to the columns of and applies the soft thresholding operator to each coefficient. More precisely, where if otherwise. This result will be familiar to anyone who has studied the process of shrinking empirical wavelet coefficients to denoise functions [39], [40]. A. Performance of ( - ) Our major theorem on the behavior of ( - ) simply collects the lemmata from the last section. Theorem 8: Let index a linearly independent collection of atoms for which. Suppose that is an input signal whose best approximation over satisfies the correlation condition Let solve the convex program ( - ) with parameter. We may conclude the following. The support of is contained in, and the distance between and the optimal coefficient vector satisfies In particular, contains every index in for which Moreover, the minimizer is unique. In words, we approximate the input signal over, and we suppose that the remaining atoms are weakly correlated with the residual. It follows then that the minimizer of the convex program involves only atoms in and that this minimizer is not far from the coefficient vector that synthesizes the best approximation of the signal over.

8 TROPP: CONVEX PROGRAMMING METHODS FOR IDENTIFYING SPARSE SIGNALS IN NOISE 1037 To check the hypotheses of this theorem, one must carefully choose the index set and leverage information about the relationship between the signal and its approximation over. Our examples will show how to apply the theorem in several specific cases. At this point, we can also state a simpler version of this theorem that uses the coherence parameter to estimate some of the key quantities. One advantage of this formulation is that the index set plays a smaller role. Corollary 9: Suppose that, and assume that contains no more than indices. Suppose that is an input signal whose best approximation over satisfies the correlation condition Let solve ( - ) with parameter. We may conclude the following. The support of is contained in, and the distance between and the optimal coefficient vector satisfies In particular, contains every index in for which Moreover, the minimizer is unique. Proof: The corollary follows directly from the theorem when we apply the coherence bounds of Appendix III. This corollary takes an especially pleasing form if we assume that. In that case, the right-hand side of the correlation condition is no smaller than. Meanwhile, the coefficient vectors and differ in norm by no more than. Note that, when is unitary, the coherence parameter. Corollary 9 allows us to conclude that the solution to the convex program identifies precisely those atoms whose inner products against the signal exceed in magnitude. Their coefficients are reduced in magnitude by at most. This description matches the performance of the soft thresholding operator. B. Example: Identifying Sparse Signals As a simple application of the foregoing theorem, we offer a new proof that one can recover an exactly sparse signal by solving the convex program ( - ). Fuchs has already established this result in the real case [22]. Corollary 10 (Fuchs): Assume that indexes a linearly independent collection of atoms for which. Choose an arbitrary coefficient vector supported on, and fix an input signal. Let denote the unique minimizer of ( - ) with parameter. We may conclude the following. There is a positive number for which implies that. In the limit as,wehave. Proof: First, note that the best approximation of the signal over satisfies and that the corresponding coefficient vector. Therefore,. The Correlation Condition is in force for every positive, so Theorem 8 implies that the minimizer of the program ( - ) must be supported inside. Moreover, the distance from to the optimal coefficient vector satisfies It is immediate that as the parameter. Finally, observe that contains every index in provided that Note that the left-hand side of this inequality furnishes an explicit value for. As a further corollary, we obtain a familiar result for Basis Pursuit that was developed independently in [14], [23], generalizing the work in [19] [22]. We will require this corollary in the sequel. Corollary 11 (Fuchs, Tropp): Assume that. Let be supported on, and fix a signal. Then is the unique solution to subject to (BP) Proof: As, the solutions to ( - ) approach a point of minimum norm in the affine space. Since is the limit of these minimizers, it follows that must also solve (BP). The uniqueness claim seems to require a separate argument and the assumption that is strictly positive. See [23], [41] for two different proofs. C. Example: Identifying Sparse Signals in Noise This subsection shows how ( - ) can be applied to the inverse problem of recovering a sparse signal contaminated with an arbitrary noise vector of bounded norm. Let us begin with a model for our ideal signals. We will use this model for examples throughout the paper. Fix a -coherent dictionary containing atoms in a real signal space of dimension, and place the coherence bound. Each ideal signal is a linear combination of atoms with coefficients of. To formalize things, select to be any nonempty index set containing atoms. Let be a coefficient vector supported on, and let the nonzero entries of equal. Then each ideal signal takes the form. To correctly recover one of these signals, it is necessary to determine the support set of the optimal coefficient vector as well as the signs of the nonzero coefficients. Observe that the total number of ideal signals is since the choice of atoms and the choice of coefficients both carry information. The coherence estimate in Proposition 26 allows us to establish that the power of each ideal signal satisfies

9 1038 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 3, MARCH 2006 Similarly, no ideal signal has power less than. In this example, we will form input signals by contaminating the ideal signals with an arbitrary noise vector of bounded norm, say. Therefore, we measure a signal of the form We would like to apply Corollary 9 to find circumstances in which the minimizer of ( - ) identifies the ideal signal. That is,. Observe that this is a concrete example of the model problem from the Introduction. First, let us determine when the Correlation Condition holds for. According to the Pythagorean Theorem Therefore,. Each atom has unit norm, so Referring to the paragraph after Corollary 9, we see that the correlation condition is in force if we select. Invoking Corollary 9, we discover that the minimizer of ( - ) with parameter is supported inside and also that Meanwhile, we may calculate that The coherence bound in Proposition 26 delivers (17) (18) In consequence of the triangle inequality and the bounds (17) and (18) ideal signals provided that the noise level. For the supremal value of, the SNR lies between and db. The total number of ideal signals is just over, so each one encodes 68 bits of information. On the other hand, if we have a statistical model for the noise, we can obtain good performance even when the noise level is substantially higher. In the next subsection, we will describe an example of this stunning phenomenon. D. Example: The Gaussian Channel A striking application of penalization is to recover a sparse signal contaminated with Gaussian noise. The fundamental reason this method succeeds is that, with high probability, the noise is weakly correlated with every atom in the dictionary (provided that the number of atoms is subexponential in the dimension ). Note that Fuchs has developed qualitative results for this type of problem in his conference publication [26]; see Section VI-B for some additional details on his work. Let us continue with the ideal signals described in the last subsection. This time, the ideal signals are corrupted by adding a zero-mean Gaussian random vector with covariance matrix. We measure the signal and we wish to identify. In other words, a codeword is sent through a Gaussian channel, and we must decode the transmission. We will approach this problem by solving the convex program ( - ) with an appropriate choice of to obtain a minimizer. One may establish the following facts about this signal model and the performance of convex relaxation. For each ideal signal, the SNR satisfies SNR The capacity of a Gaussian channel [28, Ch. 10] at this SNR is no greater than It follows that provided that For our ideal signals, the number of bits per transmission equals In conclusion, the convex program ( - ) can always recover the ideal signals provided that the norm of the noise is less than. At this noise level, the signal-to-noise ratio (SNR) is no greater than The probability that convex relaxation correctly identifies the ideal signal exceeds SNR Similarly, the SNR is no smaller than. Note that the SNR grows linearly with the number of atoms in the signal. Let us consider some specific numbers. Suppose that we are working in a signal space with dimension, that the dictionary contains atoms, that the coherence, and that the sparsity level. We can recover any of the It follows that the failure probability decays exponentially as the noise power drops to zero. To establish the first item, we compare the power of the ideal signals against the power of the noise, which is. To determine the likelihood of success, we must find the probability (over the noise) that the Correlation Condition is in force so that we can

10 TROPP: CONVEX PROGRAMMING METHODS FOR IDENTIFYING SPARSE SIGNALS IN NOISE 1039 invoke Corollary 9. Then, we must ensure that the noise does not corrupt the coefficients enough to obscure their signs. The difficult calculations are consigned to Appendix IV-A. We return to the same example from the last subsection. Suppose that we are working in a signal space with dimension, that the dictionary contains atoms, that the coherence, and that the sparsity level is. Assume that the noise level, which is about as large as we can reliably handle. In this case, a good choice for the parameter is. With these selections, the SNR is between 7.17 and db, and the channel capacity does not exceed bits per transmission. Meanwhile, we are sending about bits per transmission, and the probability of perfect recovery exceeds 95% over the noise. Although there is now a small failure probability, let us emphasize that the SNR in this example is over 10 db lower than in the last example. This improvement is possible because we have accounted for the direction of the noise. Let us be honest. This example still does not inspire much confidence in our coding scheme: the theoretical rate we have established is nowhere near the actual capacity of the channel. The shortcoming, however, is not intrinsic to the coding scheme. To obtain rates close to capacity, it is necessary to send linear combinations whose length is on the order of the signal dimension. Experiments indicate that convex relaxation can indeed recover linear combinations of this length. But the analysis to support these empirical results requires tools much more sophisticated than the coherence parameter. We hope to pursue these ideas in a later work. E. Example: Subset Selection Statisticians often wish to predict the value of one random variable using a linear combination of other random variables. At the same time, they must negotiate a compromise between the number of variables involved and the mean-squared prediction error to avoid overfitting. The problem of determining the correct variables is called subset selection, and it was probably the first type of sparse approximation to be studied in depth. As Miller laments, statisticians have made limited theoretical progress due to numerous complications that arise in the stochastic setting [4, Prefaces]. We will consider a deterministic version of subset selection that manages a simple tradeoff between the squared approximation error and the number of atoms that participate. Let be an arbitrary input signal. Suppose is a threshold that quantifies how much improvement in the approximation error is necessary before we admit an additional term into the approximation. We may state the formal problem (Subset) Were the support of fixed, then (Subset) would be a least-squares problem. Selecting the optimal support, however, is a combinatorial nightmare. In fact, if the dictionary is unrestricted, it must be NP-hard to solve (Subset) in consequence of results from [12], [16]. The statistics literature contains dozens of algorithmic approaches to subset selection, which [4] describes in detail. A method that has recently become popular is the lasso, which replaces the difficult subset selection problem with a convex relaxation of the form ( - ) in hope that the solutions will be related [42]. Our example provides a rigorous justification that this approach can succeed. If we have some basic information about the solution to (Subset), then we may approximate this solution using ( - ) with an appropriate choice of. We will invoke Theorem 8 to show that the solution to the convex relaxation has the desired properties. To do so, we require a theorem about the behavior of solutions to the subset selection problem. The proof appears in Appendix V. Theorem 12: Fix an input signal, and choose a threshold. Suppose that the coefficient vector solves the subset selection problem, and set. For,wehave. For,wehave. In consequence of this theorem, any solution to the subset selection problem satisfies the Correlation Condition with, provided that is chosen so that Applying Theorem 8 yields the following result. Corollary 13 (Relaxed Subset Selection): Fix an input signal. Suppose that the vector solves (Subset) with threshold ; the set satisfies ; and the coefficient vector solves ( - ) with parameter. Then it follows that the relaxation never selects a nonoptimal atom since the solution to the relaxation is nearly optimal since In particular, contains every index for which Moreover, is the unique solution to ( - ). In words, if any solution to the subset selection problem satisfies a condition on the Exact Recovery Coefficient, then the solution to the convex relaxation ( - ) for an appropriately chosen parameter will identify every significant atom in that solution to (Subset) and it will never involve any atom that does not belong in that optimal solution. It is true that, in the present form, the hypotheses of this corollary may be difficult to verify. Invoking Corollary 9 instead of Theorem 8 would yield a more practical result involving the coherence parameter. Nevertheless, this result would still involve a strong assumption on the sparsity of an optimal solution to the subset selection problem. It is not clear that one could verify this hypothesis in practice, so it may be better to view these results as philosophical support for the practice of convex relaxation.

11 1040 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 3, MARCH 2006 For an incoherent dictionary, one could develop a converse result of the following shape. Suppose that the solution to the convex relaxation ( - ) is sufficiently sparse, has large enough coefficients, and yields a good approximation of the input signal. Then the solution to the relaxation must approximate an optimal solution to (Subset). As this paper was being completed, it came to the author s attention that Gribonval et al. have developed some results of this type [43]. Their theory should complement the present work nicely. Theorem 14: Let index a linearly independent collection of atoms for which, and fix an input signal. Select an error tolerance no smaller than Let solve the convex program ( - ) with tolerance. We may conclude the following. The support of is contained in, and the distance between and the optimal coefficient vector satisfies V. ERROR-CONSTRAINED MINIMIZATION In particular, contains every index from for which Suppose that is an input signal. In this section, we study applications of the convex program. subject to - Minimizing the norm of the coefficients promotes sparsity, while the parameter controls how much approximation error we are willing to tolerate. We will begin with a theorem that yields significant information about the solution to ( - ). Then, we will tour several different applications of this optimization problem. As a first example, we will see that it can recover a sparse signal that has been contaminated with an arbitrary noise vector of bounded norm. Afterward, we show that a statistical model for the bounded noise allows us to sharpen our analysis significantly. Third, we describe an application to a sparse approximation problem that arises in numerical analysis. The literature does not contain many papers that apply the convex program ( - ). Indeed, the other optimization problem ( - ) is probably more useful. Nevertheless, there is one notable study [18] of the theoretical performance of ( - ). This article will be discussed in more detail in Section VI-B. Let us conclude this introduction by mentioning some of the basic properties of ( - ). As the parameter approaches zero, the solutions will approach a point of minimal norm in the affine space. On the other hand, as soon as exceeds, the unique solution to ( - ) is the zero vector. A. Performance of ( - ) The following theorem describes the behavior of a minimizer of ( - ). In particular, it provides conditions under which the support of the minimizer is contained in a specific index set. We reserve the proof until Section V-E. Moreover, the minimizer is unique. In words, if the parameter is chosen somewhat larger than the error in the optimal approximation, then the solution to ( - ) identifies every significant atom in and it never picks an incorrect atom. Note that the manuscript [18] refers to the first conclusion as a support result and the second conclusion as a stability result. To invoke the theorem, it is necessary to choose the parameter carefully. Our examples will show how this can be accomplished in several specific cases. Using the coherence parameter, it is possible to state a somewhat simpler result. Corollary 15: Suppose that, and assume that lists no more than atoms. Suppose that is an input signal, and choose the error tolerance no smaller than Let solve ( - ) with tolerance. We may conclude that. Furthermore, we have Proof: The corollary follows directly from the theorem when we apply the coherence bounds from Appendix III. This corollary takes a most satisfying form under the assumption that. In that case, if we choose the tolerance then we may conclude that. B. Example: Identifying Sparse Signals in Noise (Redux) In Section IV-C, we showed that ( - ) can be used to solve the inverse problem of recovering sparse signals corrupted with arbitrary noise of bounded magnitude. This example will demonstrate that ( - ) can be applied to the same problem.