An Overview of Compressed Sensing

Transcription

1 An Overview of Compressed Sensing Nathan Schneider November 18, 2009 Abstract In a large number of applications, the system will be designed to sample at a rate equal to at least the frequency bandwidth of the signal class under study. In a similar fashion, Compressed Sensing (CS) utilizes knowledge of the signal s sparsity, as opposed to bandwidth, to determine a sampling rate. For naturally sparse signals, this can allow a sampling rate significantly below the ubiquitous Nyquist rate. However, this savings in measurement cost is not free, for there is a corresponding increase in the computational cost of reconstruction. Perfect reconstruction cannot be realized through a linear combination of a reconstruction kernel (such as a sinc function) weighted by the sample values. Instead, it is realized through the solution of a convex optimization problem. For this exact reconstruction to be possible, the method of data collection is of significance. The collection waveforms, grouped together as rows of the sensing matrix, must satisfy the Restricted Isometry Property (RIP). RIP is a restriction on the coherence of columns within the sensing matrix and is the key ingredient to any proof within the CS domain. Interestingly, it has been found that many types of random matrices satisfy the RIP with high probability. This facet has drawn comparisons to coding theory and the channel coding theorem. In fact, one leader in the field suggests that the concepts of CS may be used as a universal encoding strategy [11, 12]. Similar to coding theory, the search for deterministic encoding schemes (sensing matrices) has taken hold because real-time storage and retrieval requirements make the use of random sensing matrices impractical for a large number of applications. Key results in this area have been achieved through a weakening of the RIP [6]. Primarily, the randomness in the sensing matrix is removed and instead placed on the signal model. Perfect reconstruction of all signals of a given sparsity is no longer guaranteed; however, if a signal satisfies the sparsity requirement it is reconstructable with high probability. As an extension of CS, research has begun to focus on other a priori knowledge that may be readily available. The primary assumption utilized assumes structure in where the signal coefficients are large (in the basis where the signal is sparse). 1 Motivations Compressed sensing represents a paradigm shift in concept that occurred near the middle of the current decade. While in the midst of the digital revolution, researchers began to ask a fundamentally different question than that of the Shannon sampling theorem [21]. This question centered around the sparsity of the signal of interest and not necessarily the bandwidth of the signal. For example, what if the support in the frequency domain of a signal class is not well contained within 1

2 a connected set but yet is sparse (such as multiple tones being present due to different transmitters). The Shannon sampling theorem implies that the signal must be sampled at a rate of at least the total width of the support in the frequency domain. However, the theory of CS implies that significantly lower sampling rates may be utilized if the frequency support of the signal class is sparse. Reduced sampling rates are extremely useful since, in many applications, sampling at the Nyquist rate may be prohibitively expensive or may be infeasible with current hardware capabilities. Specifically, one of the key constraints on a system designer is that of the Analog to Digital Converter (ADC). Currently, state of the art ADCs sample at approximately 1 GHz with eight to ten bit resolution [13]. This implies that the implementation of a wideband solution requires a large number of parallel downconversion / sampling systems. This requirement can be prohibitive in applications where space and power constraints are critical. 2 Origins and Key Concepts Compressed sensing appears to have formed through collaboration between various California based professors: Emmanuel Candes, Justin Romberg (Candes student at the time), Terence Tao (Fields Medal recipient, 2006), and David Donoho. The first publications occurred approximately four to five years ago in the IEEE Transactions on Information Theory [9, 15]. The key questions to which these researchers began to provide answers are as follows. 1. If compression is performed after measurement, why take so many measurements? 2. In applications where the Shannon Nyquist sampling rate cannot be achieved, under what conditions is reconstruction possible? A simple illustrative example of the first question is found in image formation. One is hardpressed to find a handheld digital camera without megabyte resolution (in pixels). However, images formed by these cameras are typically compressed to kilobytes under the various JPEG standards. Richard Baraniak applied the concepts of CS to this problem and showed that it was indeed possible to form images from fewer measurements than the image itself. In [2], he describes a single pixel camera where 1,600 measurements are taken of a soccer ball. Afterwards, the 64x64 (4096 pixel) image is reconstructed from these measurements. 2.1 Sparsity Assumption CS begins with the assumption that a signal has a sparse representation in some known orthonormal basis. This requirement is not as restrictive as it may seem since such knowledge is currently available (in some cases) from past research in an area known as Computational Harmonic Analysis [12]. For example, it is well known that images may be represented sparsely in a discrete cosine basis as well as a wavelet basis. This fact is exploited in the JPEG standards [4]. As another example, a linear combination of a small number of sinusoids affords a sparse representation in the Fourier basis (representable by the frequency, phase, and magnitude of each tone). More generally, the Fourier basis will likely provide a sparse representation for smoother signals whereas a wavelet basis will likely provide a sparse representation for piecewise smooth signals with finite discontinuities. 2

3 Let the basis of sparse representation be denoted by the N N matrix Ψ where each column is an N 1 vector of the basis. In a very general fashion, it is assumed that the observed phenomenon, f, may be written as a linear combination of the columns of Ψ 1 : f = Ψ 1 x. In order to solve the inverse problem and determine x, a set of linear measurements {y m } M m=1 is taken under the measurement waveforms denoted by {θ m } M m=1, which are the rows of the matrix Θ. This may be written in standard inner product notation. y m = θ m, f = θ m, Ψ 1 x It is important to note that the development thus far assumes that all signals are already digitized. For example, the assumption is made that Ψ is a matrix as opposed to being a set of continuoustime waveforms. Extensions to cases where this assumption is not made is in [16] where sampling of analog signals is considered. However, in keeping with the original literature on the topic, the same assumption of digitization will be made throughout this overview. Thus, the measurements may be grouped together in matrix form with the M N matrix Φ being defined as Φ = ΘΨ 1. y = ΘΨ 1 x = Φx Thus, without loss of generality, the CS formulation provides conditions on the matrix Φ that allow exact reconstruction of x given that x has no more than K non-zero entries (i.e. x is K-sparse). This is without loss of generality because of the formulation just described: one may solve for the actual sensing matrix Θ = ΦΨ given a design of the matrix Φ and a knowledge of the sparsity basis Ψ. 2.2 Significance Prior to stating the key theorems of CS, it is appropriate to explain the significance of the results. Perhaps the greatest achievement of compressed sensing is that it gives conditions under which an under-determined system of equations may be solved exactly through the solution of a convex optimization problem. This quite general, widely applicable result is not intuitive and deserves some comparison to the more common overdetermined case. Consider the typical linear regression problem: find the vector ˆx that provides the minimum error given a known M N measurement matrix Φ and observed measurements y. ˆx = arg min Φx y x p In the overdetermined case (M > N), with p = 2, the solution to this problem may be found analytically and is well known as the least squares solution. ˆx = ( Φ T Φ ) 1 Φ T y Now consider the underdetermined case (M < N) that is of interest to the CS community. The measurement matrix Φ is called wide and Φ T Φ is N N but of rank less than or equal to M (and thus not invertible). Obviously, the least squares solution no longer holds. In fact, there are now an 3

4 infinite number of x such that Φx = y. This implies that the desired x must be selected according to some a priori criteria given the problem domain. Compressed sensing provides conditions under which the unique, sparse x may be recovered from the measurements y. This recovery is performed by solving a convex optimization problem. 2.3 Method of Reconstruction Ideally, one desires the following solution when x is known to be K-sparse. ˆx = arg min x x 0, subject to Φx = y (1) In words, the minimization is over the number of non-zero coefficients of x. Unfortunately, solving this problem as stated is known to be numerically unstable and requires checking all possible configurations of x with K non-zero entries [2]. In any problem where N is large, this becomes infeasible quickly. For example, if N = 100 and K = 10, then the number of combinations that must be checked is in the trillions. In order to present an optimiziation problem that is solvable in polynomial time, a convex relation of equation 1 is given by the following problem statement, where p [1, ). where ˆx = arg min x x p, subject to Φx = y x p = ( N i=1 x i p ) 1 p Compressed sensing relies on a version of this convex relaxation (p = 1) that has been proven to yield the sparse, correct solution in noise free data. This guarantee is subject to the enforcement of requirements on the measurement matrix Φ. The method of reconstruction can now be stated. ˆx = arg min x x 1, subject to Φx = y (2) In the case of noisy data, the measurement model includes an additive noise term: y = Φx + η. In this case, setting p = 1 is typically referred to as robust statistics due to the l 1 norm s ability to prevent large outliers from having a significant negative impact on the solution [22]. Hence, the seemingly innocuous convex relaxation not only yields a problem that is solvable in polynomial time, but also provides nearly optimal solutions in many cases. This will be quantified in the discussion below. In the presence of noise, the problem statement changes slightly in the constraint, where the value of ɛ is chosen in accordance with the noise power. ˆx = arg min x x 1, subject to Φx y 2 ɛ (3) 4

5 3 Fundamental Theorems and Bounds To ensure recovery of the true vector x, there are two requirements. The first requirement is that x is K-sparse. The second requirement is on the columns of the sensing matrix Φ. This requirement is best expressed in terms of the isometry constant δ S, S {1, 2,..., N} of the sensing matrix Φ. The isometry constant is defined as the smallest constant such that the following inequalities hold for all S-sparse vectors x [7, 13]. (1 δ S ) x 2 2 Φx 2 2 (1 + δ S) x 2 2 (4) Any matrix Φ that satisfies equation 4 is said to satisfy the Restricted Isometry Property (RIP) with constant δ S. More succinctly, the matrix is said to be 2S-RIP if the constant δ 2S is satisfactory for reconstruction of an S-sparse signal. The next major section addresses which matrices satisfy this requirement with high probability. For a K-sparse vector x, the hypothesis for the recovery theorems below are stated in terms of δ 2K. A largest value of δ 2K will be given such that recovery is possible to within a given error. Thus, the restriction on Φ is that the user may select any of its 2K columns and these columns must satisy equation 4 for any x R 2K. For a different viewpoint on this isometry requirement, it may be restated in terms of distance preservation between K-sparse vectors x 1 and x 2 under the linear transformation Φ. (1 δ 2K ) x 1 x Φx 1 Φx (1 + δ 2K) x 1 x Therefore, good measurement matrices have the characteristic that they preserve distances between any 2K-sparse vectors and satisfy equation 4 for large values of S. 3.1 Reconstruction Theorems Both claims below compare the quality of solution to that of an oracle which knows the K largest magnitude entries of x. Let x K denote this K-sparse representation of x formed by keeping the K largest magnitude components of x. The error in representation found by solving the convex relaxation is given relative to the error in representation by x K. Since there are no assumptions or requirements that x be K-sparse, these results are quite general. The following claims are stated and proved in [7, 10, 13]. In both cases it is assumed that δ 2K < and that the method of reconstruction is given by equations 2 or Noiseless Recovery. If δ 2K < 2 1, then the solution ˆx to equation 2 satisfies both ˆx x 1 C 0 x x K 1 (5) x x K ˆx x 2 C 1 0 K Note that if x is indeed K-sparse, then recovery is exact. Also, an upper bound for C 0 is given below in terms of δ 2K. 5

6 2. Robust Recovery in Noise. If δ 2K < 2 1 and the noise power is bounded, η 2 ɛ, then the solution ˆx to equation 3 satisfies x x K ˆx x 2 C 1 0 K + C 1 ɛ (6) where C 0 is the same as above and an upper bound for C 1 is also given below in terms of δ 2K. The above statements would be of little value if the constants (C 0, C 1 ) were large and the sparsity of x was small. As it turns out though, upper bounds for these constants provide surprising small amplification when values of δ 2K are small. However, as δ 2K approaches 2 1, these constants may grow in an unbounded fashion. They are given by the following expressions [7] and shown below in figure 1. Let Then α = δ 2K 1 δ 2K, ρ = C ρ 1 ρ C 1 2 α 1 ρ 2δ2K 1 δ 2K The results stated in equations 5 and 6 deserve some comments. First, the recovery of a K- sparse signal from an underdetermined system of equations may be performed exactly by solving a convex optimization problem. This result holds for any K-sparse signal given that the sensing matrix Φ is 2K-RIP. Therefore, one obtains both the locations of the non-zero elements of x (the support) as well as the values of x at these locations. Of significance, this recovery of the support is robust to noise as well. From the noisy version, if a non-zero component of the K-sparse vector x is larger than C 1 ɛ, then its location (index) in x will be correctly recovered. This is important because, in some applications, the support of x may be more important than the values of x. For example, a radar desires to know the location of its targets. Second, the effect of noise on the recovery is directly proportional to its power. This is intuitively satisfying as well as practically useful. For example, a system designer may now predict performance of this algorithm in an online, standalone system when the SNR of operation is approximately known. Third, the actual sparsity of x must not be known since the theorems make no assumption of sparsity. In other words, no knowledge of signal coefficient rate of decay is required because the convex programs being solved in equations 2-3 adapt naturally to it [12]. Lastly, none of these results are useful if one cannot find a sensing matrix Φ that satisfies the 2K-RIP. This is the topic of the next section and will mostly be deferred until then. However, a sneak preview will be given here. Obviously, one desires to take as few measurements as possible in order to ensure a certain reconstruction capability. The number of measurements is the number of rows of Φ, namely M. If N, the number of columns of Φ, is fixed by the length of x, then taking more measurements makes it (7) 6

7 Figure 1: Variation of Bounding Constants (C 0, C 1 ) with the Isometry Constant δ 2K more likely that it is possible to find a set of N columns that are 2K-RIP. In other words, the number of matrices (or more appropriately, types of matrices) that are 2K-RIP increases with the number of measurements. Thus, the problem becomes that of finding a lower bound on M where one may generate a matrix that satisfies the 2K-RIP with high probability. For certain random matrices, it is shown that this bound is given by equation 8 for some constant C that varies by the type of random matrix [3, 12]. Of significance, results from n-widths and approximation theory are cited to state that to within a constant, one cannot do any better given the method of reconstruction. In this sense, these matrices are optimal. ( ) N M C 2K log (8) 2K In many practical systems, it is desired that Φ is not random. For example, in some of the original works, the authors were interested in applying CS concepts to the field of Magnetic Resonance Imaging (MRI) [8, 9, 10]. In this case, Φ is a partial Discrete Fourier Transform (DFT) matrix where any M rows are selected from the N N complete DFT matrix. Then for Φ to be 2K-RIP with high probability, equation 9 has been proved; however, it is suspected to hold for log(n) as well. M C 2K (log(n)) 4 (9) 7

8 3.2 Information Theoretic Bounds on Measurement Requirement There have been multiple publications on information theoretic bounds related to compressed sensing and sensing networks [5, 18, 23]. This section will be concerned primarily with the approach of [20] where the author compares the measurement system to an Additive White Gaussian Noise (AWGN) channel in order to find a lower bound on the number of measurements required for perfect reconstruction. In other words, given the rate distortion function of the source (R), the distortion level using a given decoder (E), and the SNR of the measurement process, how many measurements are required to perfectly reconstruct a K-sparse x? It is shown that reconstruction is possible when equation 10 is satisfied, where the result is asymptotic in that N goes to infinity. The key item to note is that the number of measurements is inversely proportional to the capacity of a Gaussian channel. M N 2R(E) log(1 + SNR) In the same sense as the Channel Coding Theorem, this proof methodology does not provide a method of reconstruction or a practical encoding / decoding scheme (in addition to being asymptotic). For this reason, the authors lean on previous coding theory results by viewing the measurement matrix Φ as an encoding matrix. The suggestion is made to utilize Low Density Parity Check Codes (LDPC) as an encoding / sensing scheme. 4 Restricted Isometry Property The definition of the Restricted Isometry Property (RIP) was given in equation 4. The first subsection is concerned with how to generate these sensing matrices with high probability. This facet of the theory is important because the computational cost of verifying that a matrix satisfies the 2K-RIP property is NP-hard (and thus impractical in any non-trivial problem). Specifically, random matrices are considered because they are known to be optimal in some fashion (recall equation 8 and the discussion surrounding it). The second subsection is concerned with a weakening of RIP that allows one to reconstruct most K-sparse signals (instead of all K-sparse signals). This alteration treats the measurement matrix as fixed and allows the signal model to vary within a probabilistic framework. 4.1 Random Matrices Above, it was stated that there are certain types of random matrices that satisfy the 2K-RIP with high probability. This facet of CS allows one to draw parallels with the channel coding theorem. In this theme, the sensing matrix Φ is viewed as a random encoding scheme. However, one key difference between the two constructs is that CS is a non-asymptotic result. In other words, the length of x, N, does not need to tend to infinity in order to prove the results. Rather, the goal is to find methods that generate valid sensing matrices for a given length N. This leads to the random matrix construction, which is the only known method to approach the smallest number of measurements required given l 1 reconstruction [3]. (10) 8

9 A random matrix is formed by producing every entry according to a given probability distribution. Some example distributions that generate 2K-RIP matrices with high probability (when M satisfies equation 8) are listed below [3]. Φ is formed by each entry φ ij being drawn independently from a zero-mean Normal distribution with variance 1 N. ( φij N 0, 1 ) N Φ is formed from independent realizations of a Bernoulli ( 1 2) random variable. { φ ij U 1 } 1, N N Φ is formed from independent realizations of a random variable that is ideal for producing sparse matrices (good for database applications). P r[φ ij = 3/N] = 1 6 P r[φ ij = 3/N] = 1 6 P r[φ ij = 0] = 2 3 Unfortunately, there is a drawback to utilizing a random matrix in practice in that one must store all of its entries. This can be impractical in large applications with limited on-board storage. 4.2 Deterministic Constructions There are a few practical issues with utilizing random matrices for sensing: on-board storage of the sensing matrix, implementation in sampling circuitry, and verifying that the matrix does indeed satisfy the RIP after it has been constructed. Thus, despite the fact that certain random matrices satisfy the RIP with high probability [3, 9], it is still desirable for deterministic sensing matrices to be used in practice. Deterministic waveforms afford formulaic construction during run-time [1, 6]. As such, there have been multiple attempts in the literature to construct deterministic sensing matrices. A few examples are given in [1, 6, 14, 19]. Of these attempts, the approach by Calderbank et al. [1, 6] appears to be the most promising and useful to this author. In this work, the authors provide multiple examples from the literature of attempts to construct deterministic sensing matrices. Also included is the number of measurements required, robustness to noise, and complexity of reconstruction. Most efforts use concepts from the theory of linear codes with the primary results centering around a weakening of the RIP to consider average reconstruction capabilities (i.e. RIP is focused on worst case and ensures that all K-sparse signals may be reconstructed given enough measurements). There are a few key items provided in the formulation of Calderbank et al. [6]. First, there are conditions that are simple to check to determine if the sensing matrix can recover all but 9

10 an exponentially small fraction of K-sparse signals. Second, the entries of the sensing matrix can typically be computed on the fly (chirps are an example of a waveform that is considered). Third, there are recovery algorithms available that are less computationally expensive than l 1 minimization. As a result, the authors show that partial Fourier matrices will reconstruct a K-sparse signal (with high probability) if the number of measurements, M, satisfies equation 9 with (log(n)) 4 replaced by log(n). This is a significant reduction in the number of measurements required and explains why DFT matrices work well in practice. Additionally, this satisfies the intuition and suspicions of Candes et al. [13]. 5 Extensions of Compressed Sensing and Future Directions 5.1 Model Based Compressed Sensing Model based extensions of CS utilize variations on the fundamental concepts from CS for their proofs; however, more assumptions are allowed on signal structure. Primarily, structural relationships between large coefficients of x are exploited to decrease the number of measurements required. For example, Eldar and Mishali consider the assumption that the large coefficients of x appear in blocks [17]. This notion is generalized by Baraniuk et al. [4]. In this work, no assumption is made that the large coefficients appear in blocks; however, it is assumed that the large coefficients of x live on a rooted, connected tree structure. In this case, it is shown that recovery is possible with on the order of K measurements through alterations of CS recovery algorithms. Since images represented in a wavelet basis tend to satisfy this constraint, the assumption has merit. 5.2 Compressed Measurements in Decision Systems Because the goal of many applications will not be that of reconstructing the signal (i.e. any detection based application), the pertinent question becomes that of utilizing the compressed measurements without having to actually perform the reconstruction. The solution of this problem will allow insight into linking the measurement system with the controlling, decision based system. References [1] Lorne Applebaum, Stephen Howard, Stephen Searle, and Robert Calderbank. Chirp sensing codes: Deterministic compressed sensing measurements for fast recovery. Applied and Computational Harmonic Analysis, 26(2): , March [2] Richard Baraniuk. Compressive sensing. IEEE Signal Processing Magazine, 24(4): , July [3] Richard Baraniuk, Mark Davenport, Ronald DeVore, and Michael Wakin. A simple proof of the restricted isometry property for random matrices. Constructive Approximation, 28(3): , Dec

11 [4] Richard G. Baraniuk, Volkan Cevher, Marco F. Duarte, and Chinmay Hegde. Model-based compressive sensing. Preprint, [5] Dror Baron, Marco Duarte, Shriram Sarvotham, Michael Wakin, and Richard Baraniuk. An information-theoretic approach to distributed compressed sensing. In Allerton Conference on Communication, Control, and Computing, [6] Robert Calderbank, Stephen Howard, and Sina Jafarpour. Construction of a large class of deterministic matrices that satisfy the statistical isometry property. IEEE Transactions on Signal Processing, to appear, [7] Emmanuel Candes. The restricted isometry property and its implications for compressed sensing. Comptes Rendus Mathematique, 346(9-10): , May [8] Emmanuel Candes and Justin Romberg. Sparsity and incoherence in compressive sampling. Inverse Problems, 23(3): , [9] Emmanuel Candes, Justin Romberg, and Terence Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 52(2): , February [10] Emmanuel Candes, Justin Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics, 59(8): , Aug [11] Emmanuel Candes and Terence Tao. Decoding by linear programming. IEEE Trans. on Information Theory, 51(12): , December [12] Emmanuel Candes and Terence Tao. Near optimal signal recovery from random projections: Universal encoding strategies? IEEE Trans. on Information Theory, 52(12): , Dec [13] Emmanuel Candes and Michael Wakin. An introduction to compressive sampling. IEEE Signal Processing Magazine, 25(2):21 30, March [14] Ronald A. DeVore. Deterministic constructions of compressed sensing matrices. J. of Complexity, 23: , August [15] David Donoho. Compressed sensing. IEEE Transactions on Information Theory, 52(4): , April [16] Yonina C. Eldar. Compressed sensing of analog signals. Preprint, [17] Yonina C. Eldar and Moshe Mishali. Robust recovery of signals from a structured union of subspaces. IEEE Trans. on Information Theory, to appear, [18] Alyson Fletcher, Sundeep Rangan, and Vivek K Goyal. Rate-distortion bounds for sparse approximation. In IEEE Statistical Signal Processing Workshop, Aug

12 [19] Shamgar Gurevich, Ronny Hadani, and Nir Sochen. On some deterministic dictionaries supporting sparsity. Journal of Fourier Analysis and Applications, 14: , December [20] Shriram Sarvotham, Dror Baron, and Richard Baraniuk. Measurements vs. bits: Compressed sensing meets information theory. In Allerton Conference on Communication, Control, and Computing, [21] C. E. Shannon. Communication in the presence of noise. Proc. Institute of Radio Engineers, 37(1):10 21, January [22] Joel Tropp. Just relax: Convex programming methods for subset selection and sparse approximation. Technical Report 0404, University of Texas, Austin, TX, February [23] Manqi Zhao, Shuchin Aeron, and Venkatesh Saligrama. Sensing capacity and compressed sensing: Bounds and algorithms. In Allerton Conference on Communication, Control, and Computing,