arxiv: v1 [cs.it] 21 Apr 2013

Transcription

1 Nonlinear Basis Pursuit Henrik Ohlsson, Allen Y. Yang, Roy Dong and S. Shankar Sastry arxiv: v [cs.it] Apr 03 Abstract In compressive sensing, the basis pursuit algorithm aims to find the sparsest solution to an underdetered linear euation system. In this paper, we generalize basis pursuit to finding the sparsest solution to higher order nonlinear systems of euations, called nonlinear basis pursuit. In contrast to the eisting nonlinear compressive sensing methods, the new algorithm that solves the nonlinear basis pursuit problem is conve and not greedy. The novel algorithm enables the compressive sensing approach to be used for a broader range of applications where there are nonlinear relationships between the measurements and the unknowns. I. INTRODUCTION Consider the problem of finding the sparsest solution to a linear system of euations: y i = b T i R, i =,..., N, R n, () where y i and b i are presumed given. This problem has received considerable attention recently in the area of compressive sensing. The eact solution to () is known as l 0 -imization (l 0 -): 0 (a) subj. to y i = b T i, i =,..., N. (b) It is well known that solving l 0 - is an NP-hard problem. Hence, it is not practical to directly solve the problem ecept when the system dimension n is very small. In the literature, algorithms for finding an approimate solution to () can be divided into two categories. The first category includes greedy algorithms [5], [8], [4]. Because these greedy algorithms iteratively update their estimate of based on local information, their computational compleity in the processing of computing the estimation updates is significantly lower than that of l 0 -. However, their main disadvantage is a weaker guarantee of convergence, especially comparing to the second category below. The second category of algorithms solves a conve relaed problem known as l -imization or basis pursuit [3]: (3a) subj. to y i = b T i, i =,..., N. (3b) Ohlsson gratefully acknowledge support by the Swedish Research Council in the Linnaeus center CADICS, the European Research Council under the advanced grant LEARN, contract 6738, by a postdoctoral grant from the Sweden-America Foundation, donated by ASEA s Fellowship Fund, and by a postdoctoral grant from the Swedish Research Council. Yang is supported in part by ARO 6309-MA-II. Yang and Sastry are also supported in part by DARPA FA and ONR N Ohlsson, Yang, Dong, and Sastry are with the Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, USA. Ohlsson is also with the Division of Automatic Control, Department of Electrical Engineering, Linköping University, Sweden. ohlsson@eecs.berkeley.edu. Compared to greedy algorithms, basis pursuit provably recovers the eact solution as l 0 - under some mild conditions as described in compressive sensing theory [6], [8], [7]. As the nonconve l 0 -norm function is replaced by a conve l -norm function, basis pursuit can be solved by conve optimization. In sparse optimization literature, there have been etensive discussions about accelerating the implementation of basis pursuit. The interested reader is referred to [4], [39]. For further readings on greedy algorithms, basis pursuit, and compressive sensing, see [7]. More recently, compressive sensing theory has been etended to solving nonlinear problem, which is called nonlinear compressive sensing (NLCS) [4], []: 0 subj. to y i = f i () R, i =,..., N, where f i () in general can be considered a smooth function. We are particularly interested in solving (4), as it can etend the compressive sensing approach to a broad range of applications where the relationship between the measurements and the unknowns is nonlinear. For eample, the problem of uadratic basis pursuit (QBP) [3], [3], [30] is a special case of the NLCS formulation, which is a fundamental solution to the compressive phase retrieval problem in the applications of diffraction imaging [7] and sub-wavelength imaging [36]. Similar to solving l 0 -, directly solving (4) in its original form is very epensive and intractable in practice. Therefore, there is a need to seek more efficient numerical algorithms to estimate sparse solutions in (4) while their convergence can be also guaranteed. As the topic of nonlinear compressive sensing is rather new, there are just a few algorithms that have been proposed in the literature. For instance, one of the first papers discussing this problem is [5], which proposed a greedy gradient-based algorithm. Another greedy approach was also proposed in [3]. In [], the authors proposed several iterative hard-thresholding and sparse simple pursuit algorithms. As these algorithms are nonconve greedy solutions, the analysis of their convergence typically only concerns about their local behavior. In [4], the author also considered a generalization of the restricted isometry property (RIP) to support the use of similar iterative hard-thresholding algorithms for solving general NLCS problems. In this paper, we present a novel solution to NLCS, called nonlinear basis pursuit (NLBP). The work etends our previous publications in uadratic compressive sensing and compressive phase retrieval [3], [3], and proposes a conve algorithm to solve NLBP via a high-order Taylor epansion and a lifting techniue. The work was inspired by several recent works on CS applied to the phase retrieval problem [7], [6], [], [36], [], [9], [33], [38], [], [34], [35]. (4)

2 The paper is organized as follows. We will first discuss some basic notation for this paper in Section II. Then we will present the NLBP problem and its analysis of theoretical convergence in Section III. In Section V, we will present a numerical efficient conve solver to estimate the sparse solution of NLBP. Finally, in Section VI, we will present numerical evaluation to validate the performance of our algorithm and compare with other eisting solutions. II. NOTATION AND ASSUMPTIONS In this paper, we use bold face to denote vectors and matrices, and normal font for scalars. We denote the transpose of a real vector by T and the conjugate transpose of a comple vector by H. X(i, j) denotes the (i, j)th element. Similarly, we let (i) be the ith element of the vector. Given two matrices X and Y, we use the following fact that their product in the trace function commutes, namely, Tr(XY ) = Tr(Y X), under the assumption that the dimensions match. 0 counts the number of nonzero elements in a vector or matri; similarly, denotes the element-wise l -norm of a vector or matri, i.e.,, the sum of the magnitudes of the elements; whereas represents the l -norm for vectors and the spectral norm for matrices. We assume that the functions f i ( ), i =,..., N are analytic functions. III. NONLINEAR BASIS PURSUIT Since f i ( ), i =,..., N, are analytic functions, we can epress them using their Taylor epansions. Using multi-inde notation, we can write the Taylor epansion of f i around 0 as f i ( 0 ) = ( 0 ) α α f i ( 0 ). (5) α! α 0 If is an even integer, we can now rewrite a order Taylor epansion as f i ( 0 ) = ( 0 ) α α f i ( 0 ) (6) α! 0 α = T Q i (7) where Q i is a ( n+ ( n+ -symmetric matri, and contains all the monomials of the elements of with degree /. Eample : Let = [ ] T and f() = We consider a 4th order Taylor epansion around 0 = 0. Then the set of -tuple α s is A = {(0,0), (,0), (0,), (,), (,0), (0,), (,), (,), (3,0), (0,3), (,), (3,), (,3), (4,0), (0,4)}. (8) We can easily verify that A = ( ) ( n+ = +4 ) 4 = 5. Hence, we can rewrite f() as f() = / / / / T Again, we verify that the dimension of is eual to ( ) + = 6. From the eample it is easy to see that elements of are generally dependent. The dependencies between elements of needs to be made eplicit. Eample : Let, as in the previous eample = [ ] T. (9) It is clear that e.g., the second element times the third element gives the third element of. This can be epressed as T / / = 0 / / Constructively, the dependences between elements can be generated as follows. Let Q N+ be a ( n+ ( n+ - symmetric matri with Q N+ (, ) = and set all other elements to zero. Set y N+ =, i =, k = l = and m = N +. ) If k l and (i) = (k) (l), let Q m be a ( n+ ( n+ -symmetric matri with Qm (k, l) = Q m (l, k) = /, Q m (, i) = Q m (i, ) = /, set all other elements to zero, y m = 0 and set m = m +. ) If k = l and (i) = (k) (l), let Q m be a ( n+ ) ( n+ -symmetric matri with Qm (l, l) =, Q m (, i) = Q m (i, ) = /, set all other elements to zero, y m = 0 and set m = m +. 3) If k < ( n+, set k = k + and return to step, otherwise continue to the net step. 4) If l < ( n+, set l = l +, k = and return to step, otherwise continue to the net step. 5) If i < ( n+, set i = i +, k = l = and return to step. Otherwise set M = m, return {(y i, Q i )} M i= and abort. The dependencies can now be epressed as y i = Q i, i = N +,..., M. (0) Note that these constraints are all uadratic. Assug that the th order Taylor epansion is a good approimation for {f i ()} N i=, then the problem of finding the sparsest solution to (4) can be written as: 0 subj. to y i = T Q i, i =,..., M. () Net, we employ a lifting techniue used etensively in uadratic programg [37], [5], [9], [9] and define a positive semi-definite matri T = X, which satisfies, for i =,..., M, the following eualities y i = T Q i = Tr( T Q i ) = Tr(Q i T ) = Tr(Q i X). ()

3 The nonconve problem in () can now be shown euivalent to X X 0 subj. to y i = Tr(Q i X), i =,..., M, (3) rank(x) =, X 0. Due to the l 0 -norm function and the rank condition in (3), the problem is still combinatorial. To rela the l 0 -norm function in (3), we can replace the l 0 -norm with the l - norm. To rela the rank condition, we can remove the rank constraint and instead imize rank(x) in the objective function. Furthermore, since the rank of a matri is still not a conve function, we choose the replace the rank with the nuclear norm of X, which is known to be a conve heuristic of the rank function [8], [0]. For a semidefinite matri X, it is also eual to the trace of X. Note that there are several other heuristics for the rank. For eample, the log det heuristic described in [8] is an interesting alternative (see also [36]). However, we choose to use the nuclear norm heuristic here even though our theoretical results also holds for e.g., the log det heuristic. We finally obtain the conve program Tr(X) + λ X X subj. to y i = Tr(Q i X), i =,..., M, X 0, (4) where λ 0 is a design variable. We refer to (4) as nonlinear basis pursuit (NLBP). IV. THEORETICAL ANALYSIS NLBP is a conve relaation of the combinatorial problem given in (). It is of interest to know when the relaation is tight. As a special case, when the degree of the analytic functions f i () are no greater than two, the problem is known as uadratic basis pursuit (QBP) [3], [3]. In fact, the underlying optimization problem of NLBP is the same as that of QBP. Therefore, the theoretical analysis about the performance guarantees of QBP also applies to NLBP in this paper. In this section, we only highlight several key results that etend from the proofs given in [3], [3]. First, it is convenient to introduce a linear operator B: B : X R (n+ ) ( n+ {Tr(Qi X)} i M R M. (5) We consider a generalization of the restricted isometry property (RIP) of the linear operator B. Definition (RIP): A linear operator B( ) as defined in (5) is (ɛ, k)-rip if B(X) X < ɛ (6) for all X 0 k and X 0. We can now state the following theorem: Theorem (Guaranteed recovery using RIP): Let be the solution to (). The solution of NLBP X is eual to T if it has rank and B( ) is (ɛ, X 0 )-RIP with ɛ <. The RIP-type argument may be difficult to check for a given matri and are more useful for claig results for classes of matrices/linear operators. For instance, it has been shown that random Gaussian matrices satisfy the RIP with high probability. However, given realization of a random Gaussian matri, it is indeed difficult to check if it actually satisfies the RIP. In the literature, there eist two alternative arguments, namely, the spark condition [3] and the mutual coherence [6]. The spark condition usually gives tighter bounds but is known to be difficult to compute as well. On the other hand, mutual coherence may give less tight bounds, but is more tractable. Net, we focus our discussion on mutual coherence, which is defined as: Definition 3 (Mutual coherence): For a matri A, define the mutual coherence as µ(a) = ma i,j n,i j A T :,ia :,j A :,i A :,j. (7) Let B be the matri satisfying y = BX s = B(X) with X s being the vectorized version of X. We are now ready to state the following theorem: Theorem 4 (Recovery using mutual coherence): Let be the solution to (). The solution of NLBP, X, is eual to T if it has rank and X 0 < 0.5( + /µ(b)). V. NUMERICAL SOLVERS Naturally, efficient numerical solvers that implement nonlinear basis pursuit are desirable. There are many classes of methods, commonly used in compressed sensing, which can be used to solve non-smooth SDPs. Among these include interior point methods, which is used in the popular software package CVX [0], gradient projection methods [], and augmented Lagrangian methods (ALM) [], outer approimation methods [], and the alternating directions method of multipliers (ADMM), see for instance [3], [6]. Interior point methods are known generally to scale poorly to moderate- or large-scale problems. Gradient projection methods reuire a projection onto the feasible set; for our problem, this feasible set is the intersection of a subspace with the positive semidefinite cone. The compleity of this projection operator limits the benefits of using a gradient projection method. For ALM, the augmented primal and dual objection functions are still SDPs, which are eually difficult to solve in each iteration as the original problem. Also, we found that outer approimation methods converge very slowly. On the other hand, (4) decomposes nicely into the ADMM framework. That is, we can define the following functions: { Tr(X) if y i = Tr(Q h (X) = i X), i =,..., M otherwise h (X) = g(z) = λ Z Then, (4) becomes: { 0 if X 0 otherwise X,X,Z subj. to h (X ) + h (X ) + g(z) X = X = Z (8a) (8b) (8c) (9)

4 which has the form that ADMM applies to. ADMM has strong convergence results, converges uickly in practice, and scales well to large data sets [6]. For more details of this implementation, we refer the reader to [3], [3]. A. A Simple Eample VI. NUMERICAL EVALUATION Let n = 5, N = 50 and consider y i = α α, i =,..., N, R n, (0) 4 α 0 { α } 4 α 0 was generate by sampling from a Gaussian distribution and was zero ecept for two elements that both were one. A mote Carlo simulation consisting of 00 trials was performed. We compared NLBP (fourth order Taylor epansion), QBP (second order Taylor epansion), and LASSO (first order Taylor epansion). 0 = 0 was used. The results are given in Table I. TABLE I SUCCESS RATES OF NLBP, QBP AND LASSO FOR N = 50 AND n = 5. Method NLBP QBP LASSO Success rates 00% 74% 0% B. Polynomial Euation System In this eample we consider the problem of finding the dense that solves a system of 4th order polynomials y i = α α, i =,..., N, R n, () 4 α 0 given the coefficients { α } 4 α 0. We will let N = 60, n = 5 and generate by sampling from a Gaussian distribution with a standard deviation of 0. Figure shows the bo plots of the suared residuals from a monte Carlo simulation consisting of 00 trials. New polynomial coefficients were generated at random for each trial and a new. NLBP found the true solution (within machine precision) in 99 out of the 00 trials. QBP never found the correct solution. Both used λ = 0. LASSO with λ = 0 (the least suares estimate) did not give a satisfactory estimate. VII. CONCLUSION AND FUTURE WORK The main contribution of this paper is a nonlinear compressive sensing algorithms based on conve relaations. The algorithm, referred to as nonlinear basis pursuit, is rather general in that it applies to any analytic nonlinearity by approimate it by a Taylor epansion of desired order. Nonlinear basis pursuit inherits theoretical guarantees, such as guaranteed recovery etc from its linear relative (basis pursuit) and therefore comes with theoretical guarantees that greedy algorithms often lack. Nonlinear basis pursuit takes the form of a conve non-smooth SDP which can be solved using conventional software for problems of interest. It should be noticed that solving a nonlinear euation system is by itself a difficult problem. It is therefore uiet remarkable NLBP Fig.. Bo plot comparing NLBP and QBP for finding the dense solution of polynomial euation systems. The bo plot summarizes the result from 00 trials. that we with rather high succes rate manage to find the sparsest solution to the nonlinear euation system. In addition, solving an overdetered nonlinear euation system is also difficult. As shown in the eample section, nonlinear basis pursuit not only find sparse solutions but also can also provide dense solutions to nonlinear euation system when no sparse solutions is available. Conveifying nonlinear contraints using SDPs via its Taylor epansion is up to our knowledge novel and should find applications in many areas. This is seen as future research. Last, we have not discussed noise in this paper. However, this etension is trivial and a nonlinear etension of basis pursuit denoising follows. QBP VIII. ACKNOWLEDGMENTS The authors would like to acknowledge useful discussions and inputs from Yonina C. Eldar, Laura Waller, Filipe Maia, Stefano Marchesini and Michael Lustig. REFERENCES [] A. Beck and Y. C. Eldar. Sparsity constrained nonlinear optimization: Optimality conditions and algorithms. Technical Report arxiv: , Technion, 0. [] D. P. Bertsekas. Nonlinear Programg. Athena Scientific, 999. [3] D. P. Bertsekas and J. N. Tsitsiklis. Parallel and Distributed Computation: Numerical Methods. Athena Scientific, 997. [4] T. Blumensath. Compressed sensing with nonlinear observations and related nonlinear optimization problems. Technical Report arxiv:05.650, University of Oford, 0. [5] T. Blumensath and M. Davies. Gradient pursuit for non-linear sparse signal modelling. In European Signal Processing Conference, 008. [6] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 0. [7] A. Bruckstein, D. Donoho, and M. Elad. From sparse solutions of systems of euations to sparse modeling of signals and images. SIAM Review, 5():34 8, 009. [8] E. Candès. Compressive sampling. In Proceedings of the International Congress of Mathematicians, volume 3, pages , Madrid, Spain, 006. [9] E. Candès, Y. Eldar, T. Strohmer, and V. Voroninski. Phase retrieval via matri completion. Technical Report arxiv: , Stanford University, September 0. [0] E. Candès, X. Li, Y. Ma, and J. Wright. Robust Principal Component Analysis? Journal of the ACM, 58(3), 0.

5 [] E. Candès, T. Strohmer, and V. Voroninski. PhaseLift: Eact and stable signal recovery from magnitude measurements via conve programg. Technical Report arxiv: , Stanford University, September 0. [] A. Chai, M. Moscoso, and G. Papanicolaou. Array imaging using intensity-only measurements. Technical report, Stanford University, 00. [3] S. Chen, D. Donoho, and M. Saunders. Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 0():33 6, 998. [4] W. Dai and O. Milenkovic. Subspace pursuit for compressive sensing signal reconstruction. IEEE Transactions on Information Theory, 55:30 49, 009. [5] G. Davis, S. Mallat, and M. Avellaneda. Adaptive greedy approimations. Journal of Constructive Approimations, 3:57 98, 997. [6] D. Donoho and M. Elad. Optimally sparse representation in general (nonorthogonal) dictionaries via l -imization. PNAS, 00(5):97 0, March 003. [7] Y. C. Eldar and G. Kutyniok. Compresed Sensing: Theory and Applications. Cambridge University Press, 0. [8] M. Fazel, H. Hindi, and S. P. Boyd. A rank imization heuristic with application to imum order system approimation. In Proceedings of the 00 American Control Conference, pages , 00. [9] M. X. Goemans and D. P. Williamson. Improved approimation algorithms for maimum cut and satisfiability problems using semidefinite programg. J. ACM, 4(6):5 45, November 995. [0] M. Grant and S. Boyd. CVX: Matlab software for disciplined conve programg, version.. August 00. [] K. Jaganathan, S. Oymak, and B. Hassibi. Recovery of Sparse -D Signals from the Magnitudes of their Fourier Transform. ArXiv e-prints, June 0. [] H. Konno, J. Gotoh, T. Uno, and A. Yuki. A cutting plane algorithm for semi-definite programg problems with applications to failure discriant analysis. Journal of Computational and Applied Mathematics, 46():4 54, 00. [3] L. Li and B. Jafarpour. An iteratively reweighted algorithm for sparse reconstruction of subsurface flow properties from nonlinear dynamic data. Technical Report arxiv:09.70, University of Southern California, 009. [4] I. Loris. On the performance of algorithms for the imization of l -penalized functionals. Inverse Problems, 5: 6, 009. [5] L. Lovász and A. Schrijver. Cones of matrices and set-functions and 0- optimization. SIAM Journal on Optimization, :66 90, 99. [6] S. Marchesini. Ab Initio Undersampled Phase Retrieval. Microscopy and Microanalysis, 5, July 009. [7] M. Moravec, J. Romberg, and R. Baraniuk. Compressive phase retrieval. In SPIE International Symposium on Optical Science and Technology, 007. [8] D. Needell and J. Tropp. CoSaMP: Iterative signal recovery from incomplete and inaccurate samples. Appl. Comp. Harmonic Anal., 6:30 3, 008. [9] Y. Nesterov. Semidefinite relaation and nonconve uadratic optimization. Optimization Methods & Software, 9:4 60, 998. [30] H. Ohlsson, A. Yang, R. Dong, and S. S. Sastry. CPRL an etension of compressive sensing to the phase retrieval problem. In P. Bartlett, F.C.N. Pereira, C.J.C. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 5, pages [3] H. Ohlsson, A. Y. Yang, R. Dong, and S. Sastry. Compressive Phase Retrieval From Suared Output Measurements Via Semidefinite Programg. Technical Report arxiv:.633, University of California, Berkeley, November 0. [3] H. Ohlsson, A. Y. Yang, R. Dong, M. Verhaegen, and S. Sastry. Quadratic basis pursuit. Technical Report arxiv:30.700, University of California, Berkeley, 03. [33] E. Osherovich, Y. Shechtman, A. Szameit, P. Sidorenko, E. Bullkich, S. Gazit, S. Shoham, E. B. Kley, M. Zibulevsky, I. Yavneh, Y. C. Eldar, O. Cohen, and M. Segev. Sparsity-based single-shot subwavelength coherent diffractive imaging. CLEO: Science and Innovations, 0. CF3C.7. [34] P. Schniter and S. Rangan. Compressive phase retrieval via generalized approimate message passing. In Proceedings of Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA, October 0. [35] Y. Shechtman, A. Beck, and Y. C. Eldar. GESPAR: Efficient Phase Retrieval of Sparse Signals. ArXiv e-prints, January 03. [36] Y. Shechtman, Y. C. Eldar, A. Szameit, and M. Segev. Sparsity based sub-wavelength imaging with partially incoherent light via uadratic compressed sensing. Opt. Epress, 9(6): , Aug 0. [37] N.Z. Shor. Quadratic optimization problems. Soviet Journal of Computer and Systems Sciences, 5:, 987. [38] A. Szameit, Y. Shechtman, E. Osherovich, E. Bullkich, P. Sidorenko, H. Dana, S. Steiner, E. B. Kley, S. Gazit, T. Cohen-Hyams, S. Shoham, M. Zibulevsky, I. Yavneh, Y. C. Eldar, O. Cohen, and M. Segev. Sparsitybased single-shot subwavelength coherent diffractive imaging. Nature Materials, (5): , May 0. [39] A. Yang, A. Ganesh, Z. Zhou, S. Sastry, and Y. Ma. Fast l - imization algorithms for robust face recognition. Technical Report arxiv: , University of California, Berkeley, 0.