Compressed Sensing with Very Sparse Gaussian Random Projections

Transcription

1 Compressed Sensing with Very Sparse Gaussian Random Projections arxiv: v stat.me] Aug 04 Ping Li Department of Statistics and Biostatistics Department of Computer Science Rutgers University Piscataway, NJ 08854, USA Abstract Cun-Hui Zhang Department of Statistics and Biostatistics Rutgers University Piscataway, NJ 08854, USA We study the use of very sparse random projections for compressed sensing sparse signal recovery) when the signal entries can be either positive or negative. In our setting, the entries of a Gaussian design matrix are randomly sparsified so that only a very small fraction of the entries are nonzero. Our proposed decoding algorithm is simple and efficient in that the major cost is one linear scan of the coordinates. We have developed two estimators: i) the tie estimator, and ii) the absolute minimum estimator. Using only the tie estimator, we are able to recover a -sparse signal of length N using.55elog/δ measurements whereδ is the confidence). Using only the absolute minimum estimator, we can detect the support of the signal using e log N/δ measurements. For a particular coordinate, the absolute minimum estimator requires fewer measurements i.e., with a constant e instead of.55e). Thus, the two estimators can be combined to form an even more practical decoding framework. Prior studies have shown that existing one-scan or roughly one-scan) recovery algorithms using sparse matrices would require substantially more e.g., one order of magnitude) measurements than L decoding by linear programming, when the nonzero entries of signals can be either negative or positive. In this paper, following a known experimental setup ], we show that, at the same number of measurements, the recovery accuracies of our proposed method are at least) similar to the standard L decoding.

2 Introduction Compressed Sensing CS) 9, 4] has become an important and popular topic in several fields, including Computer Science, Engineering, Applied Mathematics, and Statistics. The goal of compressed sensing is to recover a sparse signalx R N from a small number of non-adaptive linear measurementsy = xs, where S R N M is the design matrix or sensing matrix). Typically, the signal x is assumed to be -sparse i.e., nonzero entries) and neither the magnitudes nor locations of the nonzero coordinates are known. Many streaming/database applications can be naturally formulated as compressed sensing problems 5, 7, 4] even before the name compressed sensing was proposed). The idea of compressed sensing may be traced back to many prior papers, for example 0, 8]. In the literature of compressed sensing, entries of the design matrix S are often sampled i.i.d. from a Gaussian distribution or Gaussian-like distribution, e.g., a distribution with a finite second moment). Well-known recovery algorithms are often based on linear programming LP) e.g., basis pursuit 6] or L decoding) or greedy methods such as orthogonal matching pursuit OMP) 6, 3, 8, 7]. In general, L decoding is computationally expensive. OMP is often more efficient than L decoding but it can still be expensive especially when is large.. Compressed Sensing with Very Sparse Random Projections The process of collecting measurements, i.e., y = xs, is often called random projections. ] studied the idea of very sparse random projections by randomly sparsifying the sensing matrix S so that only a very small fraction of the entries can be nonzero. In this paper, we will continue to investigate on the idea of very sparse random projections in the context of compressed sensing. Our work is related to sparse recovery with sparse matrices 3,,, 5], for example, the SMP Sparse Matching Pursuit) algorithm 3]. There is a nice well-known wiki page ], which summarizes the comparisons of L decoding with count-min sketch 7] and SMP. Their results have shown that, in order to achieve similar recovery accuracies, count-min sketch needs about 0 to 5 times more measurements than L decoding and SMP needs about half of the measurements of count-min sketch. In comparison, our experimental section e.g., Figure ) demonstrates that the proposed method can be as accurate as or even more accurate than) L decoding, at the same number of measurements. The major cost of our method is one linear scan of the coordinates, like count-min sketch.. Linear Measurements from Sparse Projections In this paper, our procedure for compressed sensing first collects M non-adaptive linear measurements y j = x i s r ], j =,,...,M ) i= Here, s is the i,j)-th entry of the design matrix with s N0,) i.i.d. Instead of using a dense design matrix, we randomly sparsify γ)-fraction of the entries of the design matrix to be zero, i.e., r = { with prob. γ 0 with prob. γ i.i.d. ) And any s and r are also independent. Our proposed decoding scheme utilizes two simple estimators: i) the tie estimator and ii) the absolute minimum estimator. For convenience, we will theoretically analyze them separately. In practice, these two estimators should be combined to form a powerful decoding framework.

3 .3 The Tie Estimator y j s r. The tie estimator is developed based on the following interesting observation on the ratio statistics Conditional on r =, we can write y j N t= s r = x N ts tj r tj t i = x i + x ts tj r tj = x i +η ) / S 3) r = s s S where S,S N0,), i.i.d., and η = x t r tj = t i x t r tj 4) t i y Note that η has certain probability of being zero. If η = 0, then j s r r = x i. Thus, given M = measurements, if η = 0 happens at least) twice i.e., a tie occurs), we can exactly identify the value x i. This is the key observation which motivates our proposal of the tie estimator. Another key observation is that, if x i = 0, then we will not see a nonzero tie i.e., the probability of nonzero tie is 0). This is due to the fact that we use a Gaussian design matrix, which excludes unwanted ties. It is also clear that the Gaussian assumption is not needed, as long as s follows from a continuous distribution. In this paper we focus on Gaussian design because it makes some detailed analysis easier..4 The Absolute Minimum Estimator It turns out that, if we just need to detect whether x i = 0, the task is easier than estimating the value of x i, for a particular coordinate i. Given M measurements, if η = 0 happens at least) once, we will be able to determine whether x i = 0. Note that unlike the tie estimator, this estimator will generate false positives. In other words, if we cannot be certain that x i = 0, then it is still possible that x i = 0 indeed. From the practical perspective, at a particular coordinate i, it is preferable to first detect whether x i = 0 because that would require fewer measurements than using the tie estimator. Later in the paper, we can see that the performance can be potentially further improved by a more general estimator, i.e., the so-called absolute minimum estimator: ˆx i,min,γ = z i,t, where t = argmin z i,j, z = y j 5) j M s r We will also introduce a threshold and provide a theoretical analysis of the event ˆx i,min,γ. When = 0, it becomes the zero-detection algorithm. Our analysis will show that by using > 0 we can better exploit the prior knowledge we have about the signal and hence improve the accuracy..5 The Practical Procedure We will separately analyze the tie estimator and the absolute minimum estimator, for the convenience of theoretical analysis. However, we recommend a mixed procedure. That is, we first run the absolute minimum estimator in one scan of the coordinates, i = ton. Then we run the tie estimator only on those coordinates which are possibly not zero. Recall that the absolute minimum estimator may generate false positives. As an option, we can iterate this process for several rounds. After one iteration i.e., the absolute minimum estimator followed by the tie estimator), there might be a set of coordinates for which we cannot decide their values. We can compute the residuals and use them as the measurements for the next iteration. Typically, a few e.g., 3 or 4) iterations are sufficient and the major computational cost is computing the absolute minimum estimator in the very first iteration. 3

4 Analysis of the Absolute Minimum Estimator The important task is to analyze the false positive probability: Pr ˆx i,min,γ >,x i = 0) for some chosen threshold > 0. Later we will see that is irrelevant if we only care about the worst case. y Recall that, conditional on r =, we can express j s r = x i +η ) / S S, where S,S N0,) i.i.d. and η is defined in 4). It is known that S /S follows the standard Cauchy distribution. Therefore, ) S Pr t = π tan t), t > 0 6) S We are ready to present the Lemma about the false positive probability, including a practically useful data-dependent bound, as well as a data-independent bound which is convenient for worst-case analysis).. The False Positive Probability Lemma Data-dependent bound: { Pr ˆx i,min,γ >,x i = 0) = γe π tan γ π tan Data-independent worst case) bound: γ t x t )}] M 7) M 8) Pr ˆx i,min,γ >,x i = 0) γ γ) ] M 9) Remark: The data-dependent bound 7) and 8) can be numerically evaluated if we have information about the data. The bound will help us understand why empirically the performance of our proposed algorithm is substantially better than the worst-case bound. On the other hand, the worst case bound 9) is convenient for theoretical analysis. In fact, it directly leads to the e log N complexity bound. Proof of Lemma : For convenience, we define the set T i = {j, j M, r = }. )) y j Pr ˆx i,min,γ >,x i = 0) = E Pr s >, x i = 0, j T i T i =E )] S Pr >,x i = 0 = E )] π tan j T i =E E S { π tan )}] Ti { { = γ +γ E π tan { )}] M = γe π tan j T i )}}] M 4

5 By noticing that fx) = tan a x, where a > 0), is a convex function of x > 0, we can obtain an upper bound by using Jensen s inequality. Pr ˆx i,min,γ >, x i = 0) { )}] M = γe π tan { )}] M γ π tan Eη ) / Jensen s Inequality) M = γ π tan γ ) / t i x t M = γ π tan γ t x t We can further obtain a worst case bound as follows. Note that η has some mass at 0.. The False Negative Probability Pr ˆx i,min,γ >, x i = 0) { )}] M = γe π tan { γ π tan ) } M Prη = 0)] 0 = γ γ) ] M It is also necessary to control the false negative probability: Pr ˆx i,min,γ, x i 0). Lemma Pr ˆx i,min,γ, x i 0) { γe π tan = +x i ) π tan )}] M x i 0) M ] π γtan ) Remark: Again, if we know information about the data, we might be able to numerically evaluate the exact false negative probability 0). The loose) upper bound ) is also insightful because it means this probability 0 if 0. Note that in Lemma, the worst case bound is actually independent of. This implies that, if we only care about the worst case performance, we do not have to worry about the false positive probability since we can always choose 0. 5

6 Proof of Lemma : Pr ˆx i,min,γ,x i 0) = Pr ˆx i,min,γ >,x i 0) )) y j = E Pr s >, x i 0, j T i T i = E )] Pr x i + S >,x i 0 j T i S = E )] π x i ) π +x i tan tan j T i = E E { π tan x i )}] )+ π Ti +x i tan { { )}}] M x i = γ +γ E )+ π +x i π tan tan { )}] M +x i = γe ) π x i π tan tan Note that tan z +) tan z ) tan, for 0. Therefore, Pr ˆx i,min,γ,x i 0) { +x i = γe π tan ] M π γtan ) π tan )}] M x i which approaches zero as 0..3 The Worst Case Complexity Bound From the worst-case false positive probability bound: Pr ˆx i,min,γ >,x i = 0) γ γ) ] M, by choosing γ = / and 0), we can easily obtain the following Theorem regarding the sample complexity of only using the absolute minimum estimator. Theorem Using the absolute minimum estimator and γ = /, for perfect support recovery with probability > δ), it suffices to use measurements. log N/δ M ) log ) e log N/δ 3) Remark: The term log ) approaches e = very quickly. For example, the difference is only 0. when = 0. 6

7 3 Analysis of the Absolute Minimum Estimator on Ternary Signals Although the complexity result in Theorem can be theoretically exciting, we would like to better understand why empirically we only need substantially fewer measurements. In this section, for convenience, we consider the special case of ternary signals, i.e., x i {,0,}. The exact expectation 7), i.e., Pr ˆx i,min,γ >, x i = 0) = which, in the case of ternary data, becomes γe { π tan )}] M η = x t r tj Binomial,γ) 4) i= For convenience, we write Pr ˆx i,min,γ >, x i = 0) = { γ)e π tan )}] M = H,,γ) ] M 5) where { )} H,,γ) = γ)e π tan, Z Binomial,γ) 6) Z which can be easily computed numerically for givenγ,, andm. In order forpr ˆx i,min,γ >, x i = 0) δ for all i, we should have M logn/δ 7) H,,γ) It would be much more convenient if we do not have to worry about all combinations of γ and. In fact, we can resort to the well-studied poisson approximation by considering λ = γ and defining { )} h,λ) =λe π tan, Z Poissonλ) 8) Z { )} e λ λ k =λ π tan k=0 k k! { )} λ =λe λ +λe λ k π tan 9) k k! k= Figure plots H,,γ) and h,λ) to confirm that the Poisson approximation is very accurate as one would expect). At γ = / i.e., λ = ), the two terms H,,γ) and h,λ) are upper bounded by e. However, when is not too small, the constant e can be conservative. Basically, the choice of reflects the level of prior information about the signal. If the signals are significantly away from 0, then we can choose a larger and hence the algorithm would require less measurements. For example, if we know the signals are ternary, we can perhaps choose = or larger. Also, we can notice that γ = / is not necessarily the optimum choice for a given. In general, the performance is not too sensitive to the choice γ = λ/ as long as is not too small and the λ is reasonably large. This might be good news for practitioners. 7

8 /H, /h ε = 0.0 ε = 0. ε = 0. ε = 0..5 = 00 ε = Exact Poisson ε = γ /h ε=0.0 ε=0. ε=0. ε=0.3 ε=0.4 ε= ε= λ Figure : Left Panel: H,,γ) solid) and h,λ) dashed), for = 00 and {0.0,0.,0.,,.0}. This plot confirms that the Poisson approximation is indeed very accurate as expected). Right Panel: Poisson approximation h,λ) for {0.0,0.,0.,0.3,0.4,,0.6,0.7,0.8,0.9,.0}. In both panels, we use the horizontal line to indicate e = Whenγ = /, i.e.,λ =, both H,,γ) and h,λ) are upper bounded bye. 4 Analysis of the Absolute Minimum Estimator with Measurement Noise We can also analyze the absolute minimum estimator when measurement noise is present, i.e., ỹ j = y j +n j = x i s r ]+n j, where n j N0,σ ), j =,,...,M 0) i= Again, we compute the ratio statistic y j +n j s r = r = where S,S N0,), i.i.d., and N t= x ts tj r tj +n j s = x i + N t i x ts tj r tj +n j s = x i + η ) / S S ) η = x t r tj +σ = t i x t r tj +σ ) t i Lemma 3 Data-dependent bound: Data-independent bound: Pr ˆx i,min,γ >,x i = 0) = γe γ Pr ˆx i,min,γ >, x i = 0) { π tan { π tan { )}] M 3) σ +γ t x t) / }}] M 4) { γ π tan ) } ] M γ) 5) σ 8

9 Data-independent complexity bound: Withγ = /, in order to achievepr ˆx i,min,γ >, x i = 0) δ for all i, it suffices to use { M e π tan ) } logn/δ 6) σ measurements. Proof of Lemma 3: { )}] M Pr ˆx i,min,γ >,x i = 0) = γe π tan { )}] M γ π tan E η ) / Jensen s Inequality) { { }}] M = γ π tan σ +γ ) / t x t which is still expressed in terms of the summary of the signal. To obtain a data-independent bound, we have { )}] M { Pr ˆx i,min,γ >, x i = 0) = γe π tan γ π tan ) } ] M γ) σ 5 Analysis of the Tie Estimator To construct the tie estimator, we first compute z = y j s r which is anyway needed for the absolute minimum estimator. At each i of interest, we sort those M z values and examine the order statistics, z i,) z i,)... z i,m), and their consecutive differences, z i,j+) z i,j) forj =,,...,M. Then ˆx i,tie,γ = z i,ji ), if z i,ji +) z i,ji ) = 0, and z i,ji ) The analysis of the tie estimator is actually not difficult. Recall y j N t= s r = x N ts tj r tj t i = x i + x ts tj r tj = x i +η ) / S r = s s S where S,S N0,), i.i.d., and η = N t i x t r tj, which has a certain probability of being zero. If y η = 0, then j s r r = x i. To reliably estimate the magnitude of x i, we need η = 0 to happen more = than once, i.e., there should be a tie. Note that { γ γ) if x Prη = 0,r = ) = i = 0 γ γ) 7) if x i 0 For a given nonzero coordinate i, we would like to have η = 0 more than once among M measurements. This is a binomial problem, and the error probability is simply γ γ) ] M +M γ γ) ) γ γ) ] M 8) 9

10 Suppose we use γ = /. To ensure this error is smaller than δ for all nonzero coordinates, it suffices to choose M so that { γ γ) ] M +M γ γ) ) γ γ) ] } M δ 9) It is easy to see that this choice ofm suffices for recovering the entire signal, not just the nonzero entries. This is due to the nice property of the tie estimator, which has no false positives. That is, if there is a tie, we know for sure that it reveals the true value of the coordinate. For any zero coordinate, either there is no tie or is the tie zero. Therefore, it suffices to choose M to ensure all the nonzero coordinates are recovered. Theorem Using the tie estimator and γ =, for perfect signal recovery with probability > δ), it suffices to choose the number of measurements to be M.55e log /δ, δ 30) Proof of Theorem : The recovery task is trivial when =. Consider andp =, ) i.e., p /4. We need to choose M such that p) M +Mp p) M ) δ. Let M be such that p) M = δ, i.e.,m = logδ/. Suppose we choose M = +α)m. Then. log p) = log/δ log p p) +α)m ++α)m p p) +α)m ) = δ Therefore, we need to find theαso that δ/) α ++α) log/δ log p ) δ/) α p p Tδ,,α) = δ/) α + +α)log/δ)δ/)α log p) /p) Since p /4, we have p log p) /p) = log p)+p)/p < 0. Because p is decreasing in, we know that log p) /p) is decreasing in. Also, note that log/δ)δ/)α ] = δ/) α / αlog/δ) δ log/δ)δ/)α ] = δ/) α /δ +αlog/δ) As we consider and δ, we know that, as long as α /log = /log40, the term log/δδ/) α is increasing in δ and decreasing in. Combining the calculations, we know that Tδ,, α) is decreasing in and increasing in δ, for α > / log 40. It is thus suffices to consider δ = and =. Because T,,α) is decreasing in α, we only need to numerically find the α so that T,, α) =, which happens to be log /δ Therefore, it suffices to choose M =.55M =.55 measurements. It remains to log ) show that x, 0 < x <, we have e. Due to log log ) x log ) ) = ) = + ) e 0

11 6 An Experimental Study Compressed sensing is an important problem of broad interest, and it is crucial to experimentally verify that the proposed method performs well as predicted by our theoretical analysis. In this study, we closely follow the experimental setting as in the well-known wiki page see ]), which compared count-min sketch, SMP, and L decoding, on ternary i.e., {, 0, }) signals. In particular, the results for N = 0000 are available for all three algorithms. Their results have shown that, in order to achieve similar recovery accuracies, count-min sketch needs about 0 to 5 times more measurements than L decoding and SMP only needs about half of the measurements of count-min sketch. As shown in the success probability contour plot in Figure for γ = /), the accuracy of our proposed method is at least) similar to the accuracy of L decoding based on ]). This should be exciting because, at the same number of measurements, the decoding cost of our proposed algorithm is roughly the same as count-min sketch. M M N = N = M N = N = Figure : Contour plot of the empirical success probabilities of our proposed method, for N = 000, 0000, 00000, and For each combinationn, M, ), we repeated the simulation 00 times. For N = 0000, we can see from the wiki page ] that our prosed method provides accurate recovery results compared to L decoding. M

12 7 Conclusion Compressed sensing has become a popular and important research topic. Using a sparse design matrix has a significant advantage over dense design. For example, in sensing networks, we can replace a dense constellation of sensors by a randomly sparsified one, which may result in substantially saving of sensing hardware and labor costs. In this paper, we show another advantage from the computational perspective of the decoding step. It turns out that using a very sparse design matrix can lead to a computationally very efficient recovery algorithm without losing accuracies compared to L decoding). References ] Sparse recovery experiments with sparse matrices. ] R. Berinde, A.C. Gilbert, P. Indyk, H. arloff, and M.J. Strauss. Combining geometry and combinatorics: A unified approach to sparse signal recovery. In Communication, Control, and Computing, th Annual Allerton Conference on, pages , Sept ] R. Berinde, P. Indyk, and M. Ruzic. Practical near-optimal sparse recovery in the L norm. In Communication, Control, and Computing, th Annual Allerton Conference on, pages 98 05, Sept ] Emmanuel Candès, Justin Romberg, and Terence Tao. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 5): , Feb ] Moses Charikar, evin Chen, and Martin Farach-Colton. Finding frequent items in data streams. Theor. Comput. Sci., 3):3 5, ] Scott Shaobing Chen, David L. Donoho, Michael, and A. Saunders. Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 0:33 6, ] Graham Cormode and S. Muthukrishnan. An improved data stream summary: the count-min sketch and its applications. Journal of Algorithm, 55):58 75, ] Daivd L. Donoho and Xiaoming Huo. Uncertainty principles and ideal atomic decomposition. IEEE Transactions on Information Theory, 407):845 86, nov ] David L. Donoho. Compressed sensing. IEEE Transactions on Information Theory, 54):89 306, April ] David L. Donoho and Philip B. Stark. Uncertainty principles and signal recovery. SIAM Journal of Applied Mathematics, 493):906 93, 989. ] Anna Gilbert and Piotr. Indyk. Sparse recovery using sparse matrices. Proceedings of the IEEE, 986): , june 00. ] Ping Li, Trevor J. Hastie, and enneth W. Church. Very sparse random projections. In DD, pages 87 96, Philadelphia, PA, ] S.G. Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. Signal Processing, IEEE Transactions on, 4): , 993.

13 4] S. Muthukrishnan. Data streams: Algorithms and applications. Foundations and Trends in Theoretical Computer Science, :7 36, ] Dapo Omidiran and Martin J. Wainwright. High-dimensional variable selection with sparse random projections: Measurement sparsity and statistical efficiency. Journal of Machine Learning Research, :36 386, 00. 6] Y.C. Pati, R. Rezaiifar, and P. S. rishnaprasad. Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. In Signals, Systems and Computers, Conference Record of The Twenty-Seventh Asilomar Conference on, pages vol., Nov ] J.A. Tropp. Greed is good: algorithmic results for sparse approximation. IEEE Transactions on Information Theory, 500):3 4, oct ] Tong Zhang. Sparse recovery with orthogonal matching pursuit under RIP. IEEE Transactions on Information Theory, 579):65 6, sept. 0. 3