Compressed Sensing, Sparse Inversion, and Model Mismatch

Transcription

1 Compressed Sensing, Sparse Inversion, and Model Mismatch Ali Pezeshki, Yuejie Chi, Louis L. Scharf, and Edwin K. P. Chong Abstract The advent of compressed sensing theory has revolutionized our view of imaging, as it demonstrates that subsampling has manageable consequences for image inversion, provided that the image is sparse in an apriori known dictionary. For imaging problems in spectrum analysis (estimating complex exponential modes), and passive and active radar/sonar (estimating Doppler and angle of arrival), this dictionary is usually taken to be a DFT basis (or frame) constructed for resolution of 2p/n, with n a window length, array length, or pulse-to-pulse processing length. However, in reality no physical field is sparse in a DFT frame or in any apriori Ali Pezeshki Department of Electrical and Computer Engineering, and Department of Mathematics, Colorado State University, Fort Collins, CO 8523, USA ali.pezeshki@colostate.edu Yuejie Chi Department of Electrical and Computer Engineering, and Department of Biomedical Informatics, The Ohio State University, Columbus, OH, 432, USA chi.97@osu.edu Louis L. Scharf Department of Mathematics, Colorado State University, Fort Collins, CO 8523, USA louis.scharf@colostate.edu Edwin K. P. Chong Department of Electrical and Computer Engineering, and Department of Mathematics, Colorado State University, Fort Collins, CO 8523, USA edwin.chong@colostate.edu The authors were supported in part by the NSF by Grants CCF-8472, CCF-743, CCF- 9634, CCF-95299, and CCF c 2 IEEE. Reprinted, with permission, from Y. Chi, L. L. Scharf, A. Pezeshki, A. R. Calderbank, Sensitivity to basis mismatch in compressed sensing, IEEE Transactions on Signal Processing, vol. 59, no. 5, pp , May 2. c 2 IEEE. Reprinted, with permission, from L. L. Scharf, E. K. P. Chong, A. Pezeshki, and J. R. Luo, Sensitivity considerations in compressed sensing, Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), Pacific Grove, CA, 6-9 Nov. 2, pp

2 2 Ali Pezeshki, Yuejie Chi, Louis L. Scharf, and Edwin K. P. Chong known frame. No matter how finely we grid the parameter space (e.g., frequency, delay, Doppler, and/or wavenumber) the sources may not lie in the center of the grid cells and consequently there is always mismatch between the assumed and the actual frames for sparsity. But what is the sensitivity of compressed sensing to mismatch between the physical model that generated the data and the mathematical model that is assumed in the sparse inversion algorithm? In this chapter, we study this question. The focus is on the canonical problem of DFT inversion for modal analysis. Introduction In a great number of fields of engineering and applied science the problem confronting the designer is to invert an image, acquired from a sensor suite, for the underlying field that produced the image. And typically the desired resolution for the underlying field exceeds the temporal or spatial resolution of the image itself. Certainly this describes the problem of identifying field elements from electromagnetic and acoustic images, multipath components in wireless communication, sources of information in signal intelligence, cyber attackers in data networks, and radiating sources in radar and sonar. Here we give image its most general meaning to encompass a time series, a space series, a space-time series, a 2-D image, and so on. Similarly, we give field its most general meaning to encompass complex-exponential modes, radiating modes, packetized voice or data, coded modulations, multipath components, and the like. Broadly speaking there are two classical principles for inverting the kinds of images that are measured in optics, electromagnetics, and acoustics. The first principle is one of matched filtering, wherein a sequence of rank-one subspaces, or one-dimensional test images, is matched to the measured image by filtering or correlating or phasing. The sequence of test images is generated by scanning a prototype image (e.g., a waveform or a steering vector) through frequency, wavenumber, doppler, and/or delay. In time series analysis, this amounts to classical spectrum analysis to identify the frequency modes, and the corresponding mode amplitudes, of the signal (see, e.g., [23], [28]). In phased-array processing, it amounts to spectrum analysis in frequency and wavenumber to identify the frequency-wavenumber coordinates of sources impinging on a sensor array (see, e.g., [36] and [2]). In space-time adaptive processing (STAP) radar and sonar, it amounts to spectrum analysis in delay, frequency, and wavenumber to reconstruct the radar/sonar field (see, e.g., [37] and [2]). This matched filtering principle actually extends to subspace matching for those cases in which the model for the image is comprised of several dominant modes [28], [3]. When there is a known or estimatable noise covariance, then this matched filter concept may be further extended to a whitened matched filter or to a minimum variance unbiased (MVUB) filter (see, e.g., [28]). The MVUB estimator has the interpretation of a generalized sidelobe canceler that scans two subspaces, one matched to the signal subspace as in matched filtering, and the other to its orthogonal complement [27].

3 Compressed Sensing, Sparse Inversion, and Model Mismatch 3 The second principle is one of parameter estimation in a separable nonlinear model, wherein a sparse modal representation for the field is posited and estimates of linear parameters (complex amplitudes of modes) and nonlinear mode parameters (frequency, wavenumber, delay, and/or doppler) are extracted, usually based on maximum likelihood, or some variation on linear prediction (see, [28], [35], and [34]). There is a comprehensive literature in electrical engineering, mathematics, physics, and chemistry on parameter estimation and performance bounding in such models (see, e.g, [4], [28], [32], [22], [35], and [34]). One important limitation of the classical principles is that any subsampling of the measured image has consequences for resolution (or bias) and for variability (or variance). stands in contrast to these principles. It says that complex baseband data may be compressed before processing, when it is known apriori that the field to be imaged is sparse in a known basis or dictionary. For imaging problems in electromagnetics, acoustics, nuclear medicine, and active/passive sensing, this dictionary is usually taken to be a DFT basis or frame that is constructed for resolution of 2p/n, with n a window length, array length, or pulse-to-pulse processing length. A great number of articles (see, e.g., [], [7], [5], [6], and [3]) have considered the use of compressed sensing theory for active/passive imaging, when the sources of interest are taken to be on a regular grid in delay, Doppler, and wavenumber, and point to the potential of this theory as a new high resolution imaging principle. But no matter how large n is, the actual field will not place its sources on the grid points {2p`/n} n `= of the image model. This means the image is not actually sparse in the DFT frame, as any mode of the field that does not align with the DFT frame will produce spectral leakage into all DFT modes through the Dirichlet kernel, which decays only as / f in frequency f. This observation raises the following questions. What is the sensitivity of compressed sensing for image inversion to mismatch between the assumed frame for sparsity and the actual basis or frame in which the image is sparse? How does the performance of compressed sensing in the presence of model mismatch compare with the performance of classical matched filtering for imaging in frequency, wavenumber, delay, and/or doppler, or with that of linear prediction or other approximations to maximum likelihood? These are the general questions that we discuss in this chapter. Our discussion follows our results and analysis in [] and [29], where we have studied these questions in great detail. The reader is referred to [8], [6], and [2] for other studies on the treatment of model mismatch on compressed sensing. The chapter is organized as follows. In Section 2, we discuss the main cause of model mismatch in compressed sensing, by contrasting overdetermined separable nonlinear models with underdetermined sparse linear models as their surrogates. In Section 3, we discuss the particularly problematic nature of model mismatch in compressed sensing for DFT inversion. In Section 4, we present numerical examples that show the degradation of compressed sensing inversions in the presence of DFT grid mismatch, even when the DFT grid is made very fine. In Section 5, we discuss general analytical results that show how guarantee bounds for sparse inversion

4 4 Ali Pezeshki, Yuejie Chi, Louis L. Scharf, and Edwin K. P. Chong degenerate in presence of basis mismatch. Finally, in Section 6, we briefly review a few promising new results that may provide an alternative to gridding. 2 Model Mismatch in Compressed Sensing When viewed from the point of view of separable nonlinear models (see, e.g., [4] and [28]), most inverse imaging problems are decidedly not under-determined. However, when the problem is approximated with a high resolution linear model, where the unknown modal basis is replaced with a highly-resolved apriori dictionary, then this approximating linear model is underdetermined. But this model is always mismatched to the actual model determined by the physics of the problem. More specifically, in the separable nonlinear model, parameters such as complex scattering coefficients compose a small number of modes that are nonlinearly modulated by mode parameters such as frequency, wavenumber, and delay, and the problem is to estimate both the mode parameters and the scattering coefficients. In the approximating linear model, scattering coefficients compose a large number of modes whose frequencies, wavenumbers, and delays are assumed to be on a prespecified grid, and the problem is to determine which modes on this prespecified grid are active and to estimate the complex scattering coefficients of those modes. So the question of model mismatch in compressed sensing is really a question of quantifying the consequences of replacing an overdetermined separable nonlinear model, in which the basis or frame for sparsity is to be determined, with an underdetermined linear model in which a basis or frame is assumed to render a sparse model. In order to frame our question more precisely, let us begin with two models for a measured image x 2 C n. The first is the mathematical model that is assumed in the compressed sensing procedure, and the other is the physical model that has actually produced the image x from the field. In the mathematical model, the image is composed as x = Yc, () where the basis Y 2 C n n is known (apriori selected), and is typically a gridded imaging matrix (e.g., an n-point DFT matrix or an n-point discretized ambiguity matrix) and c 2 C n is assumed to be k-sparse. But, as a matter of fact, the image x is composed by the physics as x = Ỹ c, (2) where the basis Ỹ 2 C n k is determined by a point spread function, a Green s function, or an impulse response, and c is a k-dimensional complex vector. Typically Ỹ is determined by frequency, wavenumber, delay, and/or doppler parameters that are unknown apriori. These are the true but unknown modal parameters that nonlinearly parameterize the image x. More importantly, these parameters do not lie exactly on

5 Compressed Sensing, Sparse Inversion, and Model Mismatch 5 the gridding points corresponding to the columns of Y (e.g., DFT vectors). In other words, the columns of Ỹ almost never coincide with a subset of columns of Y. We call this basis mismatch, and note that it is present in all imaging problems, no matter how large n is, or equivalently no matter how fine-grained the gridding procedure is. If m (with k m n) linear measurements y = Ax are taken with a compressive measurement matrix A 2 C m n, then we have two different inversion problems depending on which model we consider for x. With the physical model, we have an overdetermined separable nonlinear problem, y = AỸ c, (3) where we wish to invert the m-dimensional measurement vector y for the 2k < m unknown parameters in Ỹ and c. We refer to this case as a separable nonlinear case, where k linear parameters determine the elements of c and k nonlinear parameters determine the columns of Ỹ. With the mathematical model, we have an underdetermined linear problem, y = AYc, (4) which we typically regularize by assuming sparsity for c. In this latter case, the problem of estimating the nonlinear parameters in Ỹ is assumed away by replacing the unknown physical basis Ỹ by the preselected mathematical basis Y. The goal is then to extract the linear parameters c that are active (nonzero) in the assumed mathematical model from the relatively small m < n number of linear measurements. This replacement of an overdetermined separable nonlinear model with an underdetermined linear one would be plausible if the parameter vector c would infact be k-sparse and the k modes of the physical model are well-approximated by the k modes of the mathematical basis Y. Typically, the sparse recovery of c from y is solved as a basis pursuit (BP) problem (see Chapter, Section.3.): ĉ = argmin z kzk s.t. y = Fz (5) where F = AY. If F satisfies the so-called Restricted Isometry Property (RIP) with restricted-isometry constant (RIC) d 2k (F) < p 2, then the estimate ĉ for the presumably sparse c is exact (see Chapter, Theorem.6, and also [5] and [3]). With noisy measurements, typically a basis pursuit denoising (BPDN) [4], [3] is considered: ĉ = argminkzk s.t. ky Fzk 2 apple e, (6) z where e upper-bounds the 2-norm of the noise. If F satisfies the RIP, then the 2- norm of the error in estimating c will be bounded above by a term that scales linearly with e. The reader is refrerred to [4], [3] for details. Many other approaches such as orthogonal matching pursuit (OMP) (see Chapter, Section.4.4.2, and [33]), ROMP [26] and CoSaMP [24] for finding a sparse solution to (4) also exist.

6 6 Ali Pezeshki, Yuejie Chi, Louis L. Scharf, and Edwin K. P. Chong But is it plausible to assume that c in () (and subsequently (4)) is k-sparse? This is the crux of the issue when comparing compressed sensing to the classical estimation methods such linear prediction, exact and approximate maximum likelihood, subspace decompositions, and so on. To answer this question, let us consider the coordinate transformation between the physical representation vector c and the mathematical representation vector c: c = Y Ỹ c. (7) We see that unless Y Ỹ is an n k slice of the n n identity matrix the parameter vector c is not sparse in the standard basis. But as we discussed, the columns of Ỹ almost never coincide with the columns of Y, which are posited on a preselected grid (e.g., an n-point DFT at resolution 2p/n). As a result c is almost never sparse, and in an array of problems including line spectrum estimation, Doppler estimation, and direction of arrival estimation, where Y is an n-point DFT matrix, c is not even compressible. So the question is, what is the consequence of assuming that c is sparse in the identity basis, when in fact it is only sparse in an unknown basis Y Ỹ that is determined by the mismatch between Y and Ỹ? In the rest of this chapter, we study this question in detail. Our study follows that of [], where this question was originally posed and answered. Our emphasis is on the canonical problem of sparse DFT inversion for modal analysis. 3 A Canonical Problem A mismatch case of particular interest arises in Fourier imaging when a sparse signal with arbitrary frequency components is taken to be sparse in a DFT basis. Our objective in this section is to highlight the particularly problematic nature of basis mismatch in this application. Suppose the sparsity basis Y in the mathematical model (), assumed by the compressed sensing procedure, is the unitary n-point DFT basis. Then, the `th column of Y is a Vandermonde vector of the form and the basis Y is y` = e e j 2p` n. j 2p`(n ) n C A (8)

7 Compressed Sensing, Sparse Inversion, and Model Mismatch Y = p e j 2p 2p(n ) n e j n n (9) 2p(n ) 2p(n )2 e j n e j n Without loss of generality, suppose that the `th column ỹ ` of the physical basis Ỹ is closest, in its (normalized) frequency, to y` of Y, but it is mismatched to y` by Dq`, ` in (normalized) frequency, where apple Dq`, ` < 2p n. In other words, ỹ ` = e e j(+dq`, `) j( 2p` n +Dq`, `) e j( 2p` n. +Dq`, `)(n ) C A. () Then, it is easy to see that the (r,`)th element of Y Ỹ 2 C n k, in the coordinate transformation (7), is obtained by sampling the Dirichlet kernel D n (q)= n n  n = e jnq = q(n ) j sin(q n/2) e 2 n sin(q/2) () 2p n (r at q = Dq`, ` `). The Dirichlet kernel D n (q), shown in Fig. (ignoring the unimodular phasing term) for n = 64, decays slowly as D n (q) apple(nq/2p) for q applep, with D()=. This decay behavior follows from the fact that sin(q/2) 2 q/2p for q apple p, where the equality holds when q = p. This means that (nq/2p) is in fact the envelope of D n (q). Therefore, every mismatch between a physical frequency and the corresponding (closest in normalized frequency) DFT frequency produces a column in Y Ỹ for which the entries vanish slowly as each column is traversed. The consequence of this is that the parameter vector c in the mathematical model (), for which the compressed sensing procedure is seeking a sparse solution, is in fact not sparse, because the entries of c leak into all entries of c = Y Ỹ c. In addition, to frequency mismatch, ỹ ` may also be mismatched to y` by damping factors l`, `. In such a case, the (r,`)th element of Y Ỹ is (Y Ỹ) r,` = n n  e nh n = l`, `+ j(dq`, ` i 2p(r `) n ). (2) In general, the basis mismatch problem exists in almost all applications and is not limited to Fourier imaging. However, we have emphasized Fourier imaging in this section as a canonical problem, where basis mismatch seems to have a particularly destructive effect. Other problems such as multipath resolution, where discretely-

8 8 Ali Pezeshki, Yuejie Chi, Louis L. Scharf, and Edwin K. P. Chong Fig. The Dirichlet kernel sin(nq/2) n sin(q/2) vs. q/(2p/n). delayed Nyquist pulses are correlated with Nyquist pulses whose delays do not lie on these discrete delays, appear not as troublesome as the case of Fourier imaging. 4 Effect of Mismatch on Modal Analysis We now present several numerical examples to study the effect of DFT grid mismatch on modal analysis based on compressed sensing measurements and compare these results with those obtained using classical image inversion principles, namely standard DFT imaging (matched filtering) and linear prediction (LP). The reader is referred to [28], [35], and [34] for a description of linear prediction. 4. Inversion in a DFT Basis In all the numerical examples in this section, the dimension of the image x is n = 64. The number m of measurements y used for inversion is the same for all methods and we report results for m = n/4 = 6 to n = n/2 = 32 to m = n/ = 64. For each inversion method (DFT, LP, and CS), we choose a compression matrix A that is typically used in that type of inversion. For DFT and LP inversions, we choose A as the first m rows of the n n identity matrix. For CS inversion (in a DFT basis), we choose A by drawing m rows from the n n identity matrix uniformly at random. Example : no mismatch and noise free. We first consider the case where the field we wish to invert for contains only modes that are aligned with the DFT frequencies. This is to demonstrate that compressed sensing (from here on CS) and LP both

9 Compressed Sensing, Sparse Inversion, and Model Mismatch 9 provide perfect field recovery when there is no mismatch. BP is used as the inversion algorithm for CS. No noise is considered for now. The inversion results are shown in Fig. 2(a)-(c). In each subfigure (a) through (c) there are four panels. In the topleft panel the true underlying modes are illustrated with stems whose locations on the unit disc indicate the frequencies of the modes, and whose heights illustrate the mode amplitudes. The phases of the modes are randomly chosen, and not indicated on the figures. The frequencies at which modes are placed, and their amplitudes, are (9 2p n,), ( 2p n,), (2 2p n,.5), and (45 2p n,.2). These frequencies are perfectly aligned with the DFT frequencies. But what is the connection between the circular plots in Fig. 2 and the models Y c and Ỹ c? The top-left panel (actual modes) in each subplot is an illustration of (Ỹ, c), with the locations of the bars on the unit disc corresponding to modes in Ỹ and the heights of the bars corresponding to the values of entries in c. The top-right panel (conventional DFT) illustrates (Y,ĉ), where ĉ is the estimate of c obtained by DFT processing the measurement vector y. The bottom-left (CS) illustrates (Y,ĉ), where ĉ is the solution of the basis pursuit (BP) problem (5). The bottom-right panel (LP) illustrates (Ŷ, ĉ), where Ŷ and ĉ are, respectively, estimates of Y and c obtained by LP (order 8). We observe that both CS and LP provide perfect recovery, when there is no mismatch and noise. The DFT processing however has leakage according to the Dirichlet kernel unless the measurement dimension is increased to the full dimension n = 64. This was of course expected. Example 2: mismatched but noise free. We now introduce basis mismatch either by moving some of the modes off the DFT grid or by damping them. For frequency mismatch, the first two modes are moved to (9.25 2p n,) and (9.75 2p n,). For damping mismatch the mode at (9 2p n,) is drawn off the unit circle to radius.95, so that the mode is damped as (.95) n at the n sampling instance. The rest of the modes are the same as in the mismatch free case. Figs. 3 and 4 show the inversion results for DFT, CS, and LP (order 8) for m = n/4 = 6, m = n/2 = 32, and m = n/ = 64, in the presence of frequency mismatch and damping mismatch. In all cases, DFT and CS result in erroneous inversions. The inaccuracy in inversion persists even when the number of measurements is increased to the full dimension. However, we observe that LP is always exact. These are all noise free cases. Example 3: noisy with and without mismatch. We now consider noisy observations for both mismatched and mismatch-free cases. In the mismatch-free case, the frequencies at which the modes are placed, and their amplitudes, are (9 2p n,), ( 2p n,), (2 2p n,.5), and (45 2p n,.2). For frequency mismatch, the first two modes are moved to (9.25 2p n,) and (.75 2p n,). For damping mismatch the mode at (9 2p n,) is drawn off the unit circle to radius.95. The number of measurements is m = n/2 = 32. The noise is additive and complex proper Gaussian with variance s 2. The signal-to-noise-ratio (SNR) is log (/s 2 )=7dB, relative to the unit amplitude modes. The LP order is changed to 6, but rank reduction (see [35] and [34]) is applied to reduce the order back to 8 as is typical in noisy cases. The inversion results are shown in Figs. 5 and 6 for the mismatch-free and mismatched cases, respectively. BPDN is used as the inversion method for CS. In the mismatch-free

10 Ali Pezeshki, Yuejie Chi, Louis L. Scharf, and Edwin K. P. Chong.5.5 Linear Prediction.5.5 (a) m = n/4 = Linear Prediction.5.5 (b) m = n/2 = Linear Prediction.5.5 (c) m = n/ = 64 Fig. 2 Comparison of DFT, CS, and LP inversions in the absence of basis mismatch (a) m = n/4 = 6, (b) m = n/2 = 32, and (c) m = n/ = 64.

11 Compressed Sensing, Sparse Inversion, and Model Mismatch.5.5 Linear Prediction.5.5 (a) m = n/4 = Linear Prediction.5.5 (b) m = n/2 = Linear Prediction.5.5 (c) m = n/4 = 64 Fig. 3 Comparison of DFT, CS, and LP inversions in the presence of basis mismatch (frequency mismatch), for (a) m = n/4 = 6, (b) m = n/2 = 32, and (c) m = n/ = 64.

12 2 Ali Pezeshki, Yuejie Chi, Louis L. Scharf, and Edwin K. P. Chong.5.5 Linear Prediction.5.5 (a) m = n/4 = Linear Prediction.5.5 (b) m = n/2 = Linear Prediction.5.5 (c) m = n/4 = 64 Fig. 4 Comparison of DFT, CS, and LP inversions in the presence of basis mismatch (damping mismatch), for (a) m = n/4 = 6, (b) m = n/2 = 32, and (c) m = n/ = 64.

13 Compressed Sensing, Sparse Inversion, and Model Mismatch 3 case in Fig. 5, CS provides relatively accurate estimates of the modes. However, in the mismatched case in Figs. 6(a),(b), CS breaks down and LP provides more reasonable estimates of the modes..5.5 Linear Prediction with Rank Reduction.5.5 Fig. 5 Comparison of DFT, CS, and LP inversions with m = n/2 = 32 noisy observations but no mismatch. 4.2 Inversion in an Overresolved DFT Dictionary A natural question to ask is can model mismatch sensitivities of CS be mitigated by overresolving the mathematical model (relative to Rayleigh limit) to ensure that mathematical modes are close to physical modes? One might expect that overresolution of a mathematical basis would produce a performance that only bottoms out at the quantization variance of the presumed frame for sparsity. But in fact, aggressive over resolution in a frame can actually produce worse performance at high SNR than a frame with lower resolution as we now show through an example. Our study in this section follows [29], where a detailed study of effects of model mismatch in an overresolved DFT dictionary has been presented. For our example, we consider the problem of estimating the frequency f =.5 Hz of a unit-amplitude complex exponential from m = 25 samples, taken in presence of a unit-amplitude interfering complex exponential at frequency f 2 =.52 Hz and complex proper Gaussian noise of variance s 2. The.2 Hz separation between the two tones is a separation of half the Rayleigh limit of /25 Hz. Both complex exponentials have zero phase. We consider both BPDN and OMP (see Chapter,

14 4 Ali Pezeshki, Yuejie Chi, Louis L. Scharf, and Edwin K. P. Chong.5.5 Linear Prediction with Rank Reduction.5.5 (a) Frequency mismatch.5.5 Linear Prediction with Rank Reduction.5.5 (b) Damping mismatch Fig. 6 Comparison of DFT, CS, and LP inversions with m = n/2 = 32 noisy observations: (b) frequency mismatch and (b) damping mismatch. Section.4.4.2) for estimating f through sparse inversion of the measurements in an over-resolved DFT dictionary. The measurement model to be inverted is y = AYc, where here A is the m m identity matrix and Y is the m ml DFT frame at resolution 2p/mL:

15 Compressed Sensing, Sparse Inversion, and Model Mismatch 5 2 l SOCP MSE (db) L = 2 L = 8 L = 4 L = CRB SNR (db) Fig. 7 Experimental mean-squared errors vs. SNR for estimating f using BPDN inversion in a DFT frame with resolution 2p/mL for m = 25 and different values of L. The asterisks indicate the width of the half-cell (/2mL Hz) for the various L values (L = 2,4,6,8). 2 Y = p e j ml 2p e m p(m ) e j ml e j 2p(mL ) ml j 2p(m )(ml ) ml (3) The over-resolution factor L is used to grid the frequency space L times finer than the Rayleigh limit 2p/m. The number of modes in the frame Y is n = ml, corresponding to complex exponentials spaced at /ml Hz. This may be viewed as the resolution of the DTFT frame, and BPDN and OMP can only return frequency estimates at multiples of /ml. Figs. 7 and 8 show experimental mean-squared errors (MSEs) in db for estimating f using BPDN and OMP, respectively, for different values of L. The values of L are selected such that the two frequencies f and f 2 always lie half way between two grid points of the DFT frame. The half-cell widths /2mL Hz (half of distance between grid points) are shown in the plots with asterisks on the right side of each plot. These asterisks show the amounts of bias-squared (/2mL) 2 in estimating f, for various values of L, if the sparse inversion algorithms (BPDN or OMP) happen to choose the closest point on the DFT grid to f. The linear dotted line is the Cramer-Rao Bound (see, e.g., [28]). The SNR in db is defined to be log (/s 2 ). Fig. 7 demonstrates that at L = 2 the BPDN inversions are noise-defeated below 5 db and resolution-limited above 5 db, meaning they are limited by the resolution of the frame. That is, below 5 db, mean-squared error is bias-squared plus variance, while above 5 db, mean-squared error is bias-squared due to the resolution of the

16 6 Ali Pezeshki, Yuejie Chi, Louis L. Scharf, and Edwin K. P. Chong 2 OMP MSE (db) L = 2 L = 4 L = 6 L = CRB SNR (db) (a) OMP inversions for L = 2, 4, 6, 8 2 OMP MSE (db) L = 4 L = 8 L = CRB SNR (db) (b) OMP inversions for L = 8,2,4 Fig. 8 Experimental mean-squared errors vs. SNR for estimating f using OMP inversion in a DFT frame with resolution 2p/mL for m = 25 and different values of L. The asterisks indicate the width of the half-cell (/2mL Hz) for the various L values. (a) L = 2,4,6,8 and (b) L = 8,2,4. frame. At L = 4, the inversions are noise-defeated below 5 db, noise-limited from 5 to db, and resolution-limited above db. As the expansion factor is increased, corresponding to finer- and finer-grained resolution in frequency, the frame loses its incoherence, meaning the dimension of the null space increases so much that there are many sparse inversions that meet a fitting constraint. As a consequence, we see that at L = 6 and L = 8 the mean-squared errors never reach their resolution limits, as variance overtakes bias-squared to produce mean-squared errors that are larger than those of inversions that used smaller expansion factors. This suggests that there is a

17 Compressed Sensing, Sparse Inversion, and Model Mismatch 7 clear limit to how much bias-squared can be reduced with frame expansion, before variance overtakes bias-squared to produce degraded BPDN inversions. Figs. 8(a) and (b) make these points for OMP inversions. The OMP inversions extend the threshold behavior of the inversions, they track the CRB more closely in the noise-limited region, and they reach their resolution limit for larger values of L before reaching their null-space limit at high SNRs. For example, as L = 8 the null-space limit has not yet been reached at SNR= 3 db, whereas for L = 4, the null-space limit is reached before the resolution limit can be reached. We note that the results in Figs. 7 and 8 are actually too optimistic, as the two mode amplitudes are equal. For a weak mode in the presence of a strong interfering mode, the results are much worse. Also, in all of these experiments, the fitting error is matched to the noise variance. With mismatch between fitting error and noise variance the results are more pessimistic. Nonetheless, they show that the consequence of over-resolution is that performance follows the Cramer-Rao bound more closely at low SNR, but at high SNR it departs more dramatically from the Cramer- Rao bound. This matches intuition that has been gained from more conventional spectrum analysis where there is a qualitatively similar trade-off between bias and variance. That is, bias may be reduced with frame expansion (over-resolution), but there is a penalty to be paid in variance. Figs. 9(a) (d) are scatter plots of BPDN and OMP inversions at 7 db SNR, for L = 2,5,9. Fig. 9(a) scatters estimator errors for ( f, f 2 ) for ` inversions and Fig. 9(b) is the same data, plotted as normalized errors in estimating (( f + f 2 ),( f f 2 )), the sum and difference frequencies. At L = 2 mean-squared error is essentially bias-squared, whereas for L = 9 it is essentially variance. The normalized errors in Fig. 9(b) demonstrate that the average frequency of the two tones is easy to estimate and the difference frequency is hard to estimate. (The vertical scale is nearly times the horizontal scale.) This scatter plot demonstrates that BPDN inversions favor large negative differences over large positive differences, suggesting that the algorithm more accurately estimates the mode at frequency f than it estimates the mode at frequency f 2. The geometry of the scatter plots indicates that a concentration ellipse drawn from the inverse of the Fisher information matrix would be a poor descriptor of errors. Figs. 9(c) and (d) make these points for OMP inversions as Figs. 9(a) and (b) made for BPDN inversions. However now the preference for large negative errors in estimating the difference frequency disappears. This suggests that the first dominant mode that is removed is equally likely to be near one or the other of the two modes in the data. The correlation between sum and difference errors reflects the fact that a large error in extracting the first mode will produce a large error in extracting the second. For OMP, it is possible that a concentration ellipse will accurately trap the solutions.

18 8 Ali Pezeshki, Yuejie Chi, Louis L. Scharf, and Edwin K. P. Chong.3 l with SNR = 7 db 2 l with SNR = 7 db.2.5 f 2 estimation error...2 L = 2 L = 5 L = f estimation error.3 (a) OMP with SNR = 7 db Normalized difference of estimates.5.5 L = 2 L = 5 L = Normalized sum of estimates 2 (b) OMP with SNR = 7 db f 2 estimation error L = 2 L = 9 L = f estimation error (c) Normalized difference of estimates L = 2 L = 5 L = Normalized sum of estimates (d) Fig. 9 Scatter plots of BP and OMP inversions at 7 db SNR, for L = 2,5,9: (a) Scatter plot of estimator errors for ( f, f 2 ) for BP inversions. (b) Scatter plot of normalized errors in estimating (( f + f 2 ),( f f 2 )) using BP inversion. (c) Scatter plot of estimator errors of ( f, f 2 ) for OMP inversions. (d) Scatter plot of normalized errors in estimating (( f + f 2 ),( f f 2 )) using OMP. 5 Performance Bounds for Compressed Sensing with Basis Mismatch When the RIC of F = AY satisfies d 2k (F) < p 2 for 2k-sparse signals, the solution ĉ to the basis pursuit problem (5) satisfies (see Chapter, Theorem.6, and also [5], [4], and [3]) kĉ ck apple C kc s k (c) k (4) and kĉ ck 2 apple C k /2 kc s k (c) k, (5) where s k (c) is the best k-term approximation to c in ` norm and C is a constant. These inequalities are at the core of compressed sensing theory, as they demonstrate that for sparse signals, where the best k-term approximation error c s k (c) vanishes, the signal recovery using basis pursuit is exact. The analysis of [], however, indicates that under basis mismatch the k-term approximation degenerates considerably and it fails to provide any guarantee for the

19 Compressed Sensing, Sparse Inversion, and Model Mismatch 9 solution of the mismatched basis pursuit problem. This is captured in the following theorem from []. Theorem. (best k-term approximation error). Consider the coordinate transformation c = Y Ỹ c. Let b = max hy i,ỹ j i be the worst-case coherence between i, j the columns of Y =[y,...,y n ] and the columns of Ỹ =[ỹ,...,ỹ k ]. Then, kc s k (c) k apple (n k)bk ck. (6) In [], it is shown that under certain mismatch conditions the upper bound in (6) is achieved, resulting in the worst-case error. The bound in (6) can be combined with (4) and (5) to produce upper bounds on the inversion error for basis pursuit in the presence of basis mismatch. The error bounds scale linearly with n so that the bound increases with the size of the original image, independently of the compressed dimension, provided the RIP condition is maintained. The bound in (6) is relevant to any sparse inversion algorithm that relies on best k-term approximation error bounds for its inversion guarantees. This class of algorithms includes regularized orthogonal matching pursuit (ROMP) [25] and CoSaMP [24] among others. However, we note that above theorem does not imply that the compressed sesing inversions will necessarily be bad. Rather it says that the inversions cannot be guaranteed to be good based on best k-term approximation reasoning. 6 Compressed Sensing Off The Grid Recently, several approaches based on atomic norm minimization [7] have been proposed to eliminate the need for an apriori selected basis (or grid) in compressed sensing. However, the guaranteed theoretical resolution of these methods is a few Rayleigh limits, and sub-rayleigh resolution of modes is not guaranteed. In particular, in [2] the authors show that a line spectrum with minimum frequency separation D f > 4/k can be recovered from the first 2k Fourier coefficients via atomic norm minimization. In [3], the authors improve the resolution result by showing that a line spectrum with minimum frequency separation D f > 4/n can be recovered from most subsets of the first n Fourier coefficients of size at least m = O(k log(k)log(n)). This framework has been extended in [] for 2-D line spectrum estimation. Another approach is proposed in [8] and [9], where the problem is reformulated as a structured matrix completion inspired by a classical work on matrix pencil [9]. These are all promising new directions that may provide pathways to high-resolution modal analysis in measurement deprived scenarios without the need to use prespecified grids for inversion. We refer the interested reader to [7], [2], [3], [], [8], and [9] for details.

20 2 Ali Pezeshki, Yuejie Chi, Louis L. Scharf, and Edwin K. P. Chong References. Baraniuk, R., Steeghs, P.: Compressive radar imaging. In: Proc. 27 IEEE Radar Conf., pp Waltham, Massachusetts (27) 2. Candes, E., Fernandez-Granda, C.: Towards a mathematical theory of super-resolution. preprint (Mar. 22, arxiv: ) 3. Candés, E.J.: The restricted isometry property and its implications for compressed sensing. Académie des Sciences (346), (28) 4. Candés, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inform. Theory 52(2), (26) 5. Candés, E.J., Tao, T.: Decoding by linear programming. IEEE Trans. Inform. Theory 5, (25) 6. Cevher, V., Gurbuz, A., McClellan, J., Chellappa, R.: Compressive wireless arrays for bearing estimation of sparse sources in angle domain. In: Proc. IEEE Int. Conf. on Acoustics, Speech, and Signal Processing (ICASSP), pp Las Vegas, Nevada (28) 7. Chandrasekaran, V., Recht, B., Parrilo, P.A., Willsky, A.S.: The convex geometry of linear inverse problems. Foundations of Computational Mathematics 2(6), (22) 8. Chen, Y., Chi, Y.: Spectral compressed sensing via structured matrix completion. In: Proc. International Conference on Machine Learning (ICML) (23) 9. Chen, Y., Chi, Y.: Robust spectral compressed sensing via structured ma- trix completion. preprint (23, arxiv: ). Chi, Y., Chen, Y.: Compressive recovery of 2-D off-grid frequencies. In: Conf. Rec. Asilomar Conf. Signals, Systems, and Computers (23). Chi, Y., Scharf, L.L., Pezeshki, A., Calderbank, R.: Sensitivity to basis mismatch in compressed sensing. IEEE Trans. Signal Processing 59(5), (2) 2. Duarte, M.F., Baraniuk, R.G.: Spectral compressive sensing. Applied and Computational Harmonic Analysis 35(), 29 (23) 3. Fannjiang, A., Yan, P., Strohmer, T.: Compressed remote sensing of sparse objects. SIAM J. Imaging Sciences 3(3), (2) 4. Golub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn. John Hopkins Univ. Press, Baltimore, MD (996) 5. Gurbuz, A., McClellan, J., Cevher, V.: A compressive beamforming method. In: Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), pp Las Vegas, NV (28) 6. Herman, M.A., Needell, D.: Mixed operators in compressed sensing. In: Proc. 44th Annual Conference on Information Sciences and Systems (CISS). Princeton, NJ (2) 7. Herman, M.A., Strohmer, T.: High-resolution radar via compressed sensing. IEEE Trans. Signal Processing 57(6), (29) 8. Herman, M.A., Strohmer, T.: General deviants: An analysis of perturbations in compressed sensing. IEEE J. Selected Topics in Signal Processing: Special Issue on Compressive Sensing 4(2), (2) 9. Hua, Y.: Estimating two-dimensional frequencies by matrix enhancement and matrix pencil. IEEE Transactions on Signal Processing 4(9), (992) 2. Karim, H., Viberg, M.: Two decades of array signal processing research: The parametric approach. IEEE Signal Process. Mag. 3(4), (996) 2. Klemm, R.: Space-Time Adaptive Processing. IEEE Press (998) 22. McWhorter, L.T., Scharf, L.L.: Cramer-Rao bounds for deterministic modal analysis. IEEE Trans. Signal Process. 4(5), (993) 23. Mullis, C.T., Scharf, L.L.: Quadratic estimators of the power spectrum. In: S. Haykin (ed.) Advances in Spectrum Estimation, vol., chap., pp. 57. Prentice Hall, Englewood Cliffs, NJ (99) 24. Needell, D., Tropp, J.: CoSaMP: Iterative signal recovery from incomplete and inaccurate samples. Applied and Computational Harmonic Analysis 26, 3 32 (28)

21 Compressed Sensing, Sparse Inversion, and Model Mismatch Needell, D., Vershynin, R.: Uniform uncertainty principle and signal recovery via regularized orthogonal matching pursuit. Foundations of Computational Mathematics 9, (29) 26. Needell, D., Vershynin, R.: Signal recovery from inaccurate and incomplete measurements via regularized orthogonal matching pursuit. IEEE Journal of Selected Topics in Signal Processing 4(2), 3 36 (2) 27. Pezeshki, A., Veen, B.D.V., Scharf, L.L., Cox, H., Lundberg, M.: Eigenvalue beamforming using a generalized sidelobe canceller and subset selection. IEEE Trans. Signal Proc. 56(5), (28) 28. Scharf, L.L.: Statistical Signal Processing. Addison-Wesley, MA (99) 29. Scharf, L.L., Chong, E.K.P., Pezeshki, A., Luo, J.: Sensitivity considerations in compressed sensing. In: Conf. Rec. Forty-fifth Asilomar Conf. Signals, Syst. Pacific Grove, CA (2) 3. Scharf, L.L., Friedlander, B.: Matched subspace detectors. IEEE Trans. Signal Process. 42(8), (994) 3. Tang, G., Bhaskar, B.N., Shah, P., Recht, B.: off the grid. preprint (Jul. 22, arxiv: ) 32. Trees, H.L.V., Bell, K.L.: Bayesian Bounds for Parameter Estimation and Nonlinear Filtering/Tracking. IEEE Press (27) 33. Tropp, J.A., Gilbert, A.C.: Signal recovery from random measurements via orthogonal matching pursuit. IEEE Transactions on Information Theory 53(2) (992) 34. Tufts, D.W., Kumaresan, R.: Estimation of frequencies of multiple sinusoids: making linear prediction perform like maximum likelihood. Proc. IEEE. 7, (982) 35. Tufts, D.W., Kumaresan, R.: Singular value decomposition and improved frequency estimation using linear prediction. IEEE Trans. Acoust., Speech, Signal Processing 3(4), (982) 36. Van Trees, H.L.: Optimum Array Processing. Wiley Interscience (22) 37. Ward, J.: Maximum likelihood angle and velocity estimation with space-time adaptive processing radar. In: Conf. Rec. 996 Asilomar Conf. Signals, Systs., Comput., pp Pacific Grove, CA (996)