Weighted l 1 Minimization for Sparse Recovery with Prior Information

Transcription

1 Weighted l Minimization for Sparse Recovery with Prior Information M. Amin Khajehnejad amin@caltech.edu Weiyu Xu weiyu@altech.edu A. Salman Avestimehr Caltech CMI avestim@caltech.edu Babak Hassibi hassibi@caltech.edu Abstract In this paper we study the compressed sensing problem of recovering a sparse signal from a system of underdetermined linear equations when we have prior information about the probability of each entry of the unknown signal being nonzero. In particular, we focus on a model where the entries of the unknown vector fall into two sets, each with a different probability of being nonzero. We propose a weighted l minimization recovery algorithm and analyze its performance using a Grassman angle approach. We compute explicitly the relationship between the system parameters (the weights, the number of measurements, the size of the two sets, the probabilities of being non-zero) so that an iid random Gaussian measurement matrix along with weighted l minimization recovers almost all such sparse signals with overwhelming probability as the problem dimension increases. This allows us to compute the optimal weights. We also provide simulations to demonstrate the advantages of the method over conventional l optimization. I. INTRODUCTION Compressed sensing is an emerging technique of joint sampling and compression that has been recently proposed as an alternative to Nyquist sampling (followed by compression) for scenarios where measurements can be costly [4]. The whole premise is that sparse signals (signals with many zero or negligible elements in a known basis) can be recovered with far fewer measurements than the ambient dimension of the signal itself. In fact, the major breakthrough in this area has been the demonstration that l minimization can efficiently recover a sufficiently sparse vector from a system of underdetermined linear equations []. The conventional approach to compressed sensing assumes no prior information on the unknown signal other than the fact that it is sufficiently sparse in a particular basis. In many applications, however, additional prior information is available. In fact, in many cases the signal recovery problem (which compressed sensing attempts to address) is a detection or estimation problem in some statistical setting. Some recent work along these lines can be found in [5] (which considers compressed detection and estimation) and [6] (on Bayesian compressed sensing). In other cases, compressed sensing may be the inner loop of a larger estimation problem that feeds prior information on the sparse signal (e.g., its sparsity pattern) to the compressed sensing algorithm. In this paper we will consider a particular model for the sparse signal that assigns a probability of being zero or nonzero to each entry of the unknown vector. The standard compressed sensing model is therefore a special case where these probabilities are all equal (for example, for a k-sparse vector the probabilities will all be k n, where n is the number of entries of the unknown vector). As mentioned above, there are many situations where such prior information may be available, such as in natural images, medical imaging, or in DNA microarrays where the signal is often block sparse, i.e., the signal is more likely to be nonzero in certain blocks rather than in others [7]. While it is possible (albeit cumbersome) to study this model in full generality, in this paper we will focus on the case where the entries of the unknown signal fall into two categories: in the first set (with cardinality n ) the probability of being nonzero is, and in the second set (with cardinality n = n n ) this probability is P. (Clearly, in this case the sparsity will with high probability be around n +n P.) This model is rich enough to capture many of the salient features regarding prior information, while being simple enough to allow a very thorough analysis. While it is in principle possible to extend our techniques to models with more than two categories of entries, the analysis becomes increasingly tedious and so is beyond the scope of this short paper. The contributions of the paper are the following. We propose a weighted l minimization approach for sparse recovery where the l norms of each set are given different weights w i (i =, ). Clearly, one would want to give a larger weight to those entries whose probability of being nonzero is less (thus further forcing them to be zero). The second contribution is to compute explicitly the relationship between the p i, the w i, the ni n, i =, and the number of measurements so that the unknown signal can be recovered with overwhelming probability as n (the so-called weak threshold) for measurement matrices drawn from an iid Gaussian ensemble. The analysis uses the high-dimensional geometry techniques first introduced by Donoho and Tanner [], [3] (e.g., Grassman angles) to obtain sharp thresholds for compressed sensing. However, rather than use the neighborliness condition used in [], [3], we find it more convenient to use the null space characterization of Xu and Hassibi [4], [3]. The resulting A somewhat related method that uses weighted l optimization is Candes et al [8]. The main difference is that there is no prior information and at each step the l optimization is re-weighted using the estimates of the signal obtained in the last minimization step.

2 Grassmanian manifold approach is a general framework for incorporating additional factors into compressed sensing: in [4] it was used to incorporate measurement noise; here it is used to incorporate prior information and weighted l optimization. Our analytic results allow us to compute the optimal weights for any p, p, n, n. We also provide simulation results to show the advantages of the weighted method over standard l minimization. II. MODEL The signal is represented by a n vector x = (x,x,..., x n ) T of real valued numbers, and is non-uniformly sparse with sparsity factor over the (index) set K {,,..n} and sparsity factor P over the set K = {,,..., n} \K. By this, we mean that if i K, x i is a nonzero element with probability and zero with probability. However, if i K the probability of x i being nonzero is P. We assume that K = n and K = n = n n. The measurement matrix A is a m n ( m n = δ< ) matrix with i.i.d N (, ) entries. The observation vector is denoted by y and obeys the following: y = Ax () As mentioned in Section I, l -minimization can recover a vector x with k = µn non-zeros, provided µ is less than a known function of δ. l minimization has the following form: min x () Ax=y () is a linear programming and can be solved polynomially fast (O(n 3 )). However, it fails to encapsulate additional prior information of the signal nature, might there be any such information. One might simply think of modifying () to a weighted l minimization as follows: min x w = min Ax=y Ax=y n w i x i (3) The index w is an indication of the n positive weight vector. Now the question is what is the optimal set of weights, and can one improve the recovery threshold using the weighted l minimization of (3) with those weights rather than ()? We have to be more clear with the objective at this point and what we mean by extending the recovery threshold. First of all note that the vectors generated based on the model described above can have any arbitrary number of nonzeros. However, their support size is typically (with probability arbitrary close to one) around n + n P ). Therefore, there is no such notion of strong threshold as in the case of []. We are asking the question of for what and P signals generated based on this model can be recovered with overwhelming probability as n. Moreover we are wondering if by adjusting w i s according to and P can one extend the typical sparsity to dimension ratio ( n +n P n ) for which reconstruction is successful with high probability. This is the topic of next section. III. COMPUTATION OF THE WEAK THRESHOLD Because of the partial symmetry of the sparsity of the signal we know that the optimum weights should take only two positive values W and W. In other words { W if i K i {,,,n} w i = W if i K Let x be a random sparse signal generated based on the non-uniformly sparse model of section II and be supported on the set K. K is called ɛ-typical if K K n ɛn and K K n P ɛn. Let E be the event that x is recovered by (3). : P[E c ] = P [E c K is ɛ-typical]p [K is ɛ-typical] + P [E c K not ɛ-typical]p [K not ɛ-typical] For any fixed ɛ > P [K not ɛ-typical] will exponentially approach zero as n grows according to the law of large numbers. So, to bound the probability of failed recovery we may assume that K is ɛ-typical for any small enough ɛ. Therefore we just consider the case K = k = n + n P. Similar to the null-space condition of [3], we present a necessary and sufficient condition for x to be the solution to (3). It is as follows: Z N (A) i K w i Z i i K w i Z i Where N (A) denotes the right nullspace of A. We can upper bound P(E c ) with P K, which is the probability that a vector x of a specific sign pattern (say non-positive) and supported on the specific set K is not recovered correctly by (3) (A difference between this upper bound and the one in [4] is that here there is no ( n k) k factor, and that is because we have fixed the support set K and the sign pattern of x). Exactly as done in [4], by restricting x to the cross-polytope {x R n x w =} 3, and noting that x is on a (k )- dimensional face F of the skewed cross-polytope SP = {y R n y w }, P K, is essentially the probability that a uniformly chosen (n m)-dimensional subspace Ψ shifted by the point x, namely (Ψ + x), intersects SP nontrivially at some other point besides x. P K, is then interpreted as the complementary Grassmann angle [9] for the face F with respect to the polytope SP under the Grassmann manifold Gr ( n m)(n). Building on the works by L.A.Santalö [] and P.McMullen [] etc. in high dimensional integral geometry and convex polytopes, the complementary Grassmann angle for the (k )-dimensional face F can be explicitly expressed as the sum of products of internal angles and external angles []: s G I m++s (SP) β(f, G)γ(G, SP), (4) Also we may assume WLG that W = 3 This is because the restricted polytope totally surrounds the origin in R n

3 where s is any nonnegative integer, G is any (m + + s)- dimensional face of the skewed crosspolytope (I m++s (SP) is the set of all such faces), β(, ) stands for the internal angle and γ(, ) stands for the external angle. The internal angles and external angles are basically defined as follows [][]: An internal angle β(f,f ) is the fraction of the hypersphere S covered by the cone obtained by observing the face F from the face F. The internal angle β(f,f ) is defined to be zero when F F and is defined to be one if F = F. An external angle γ(f 3,F 4 ) is the fraction of the hypersphere S covered by the cone of outward normals to the hyperplanes supporting the face F 4 at the face F 3. The external angle γ(f 3,F 4 ) is defined to be zero when F 3 F 4 and is defined to be one if F 3 = F 4. Note that F here is a typical face of SP corresponding to a typical set K. β(f, G) depends not only on the dimension of the face G, but also depends on the number of its vertices supported on K and K. In other words if G is supported on a set L, then β(f, G) is only a function of L K and L K. So we write β(f, G) =β(t,t ) and similarly γ(g, SP )= γ(t,t ) where t = L K n and t = L K n P. Combining the notations and counting the number of faces G, (4) leads to: P(E c ) t ( )n t ( P )n t + t >m k + t +t ( ( P )n t )( ( P )n t ) β(t,t )γ(t,t ) + O(e cn ) (5) for some c >. As n each term in (5) behaves like exp{nψ com (t,t ) nψ int (t,t ) nψ ext (t,t )} where ψ com ψ int and ψ ext are the combinatorial exponent, the internal angle exponent and the external angle exponent of the each term respectively. It can be shown that the necessary and sufficient condition for (5) to tend to zero is that ψ(t,t )= ψ com (t,t ) ψ int (t,t ) ψ ext (t,t ) be uniformly negative for all t and t in (5). In the following sub-sections we will try to evaluate the internal and external angles for a typical face F, and a face G containing F and try to give closed form upper bounds for them. We combine the terms together and compute the exponents using Laplace method in section IV and derive thresholds for nonnegativity of the cumulative exponent using. A. Derivation of the Internal Angles Suppose that F is a typical (k )-dimensional face of the skewed cross-polytope n SP = {y R n y w = w i y i } supported on the subset K with K = k n + n P. Let G be a l dimensional face of SP supported on the set L with F G. Also, let L K = t and L K = t. First we can prove the following lemma: Lemma : Let Con F,G be the positive cone of all the vectors x R n that take the form: b i e i + i=k+ b i e i, (6) where b i, i l are nonnegative real numbers and w i b i = i=k+ e x Con F,G w i b i b w = b = = b k w k dx = β(f, G)V l k (S l k ) e r r l k dx = β(f, G) π (l k)/, (7) where V l k (S l k ) is the spherical volume of the (l k )-dimensional sphere S l k. Proof: Omitted for brevity From (7) we can find the expression for the internal angle. Define U R l k+ as the set of all nonnegative vectors (x,x,,x l k+ ) satisfying: x p, p l k + ( k w p)x = l w px p k+ and define f(x,, x l k+ ):U Con F,G to be the linear and bijective map f(x,,x l k+ )= Con F,G e x ( Γ Γ dx = x w p e p + U e f(x) df (x) e f(x) dx dx l k+ x p k+ w p e p e (P k w p )x P l w p x p k+ dx dx l k+ Γ is the region described by wp)x = (8) px p k+,x p p l k+ (9) where J(A) is due to the change of integral variables and is essentially the determinant of the Jacobian of the variable transform given by the l l k matrix A given by: Ω w iwk+j i k, j l k A i,j = w i k + i l, j = i k Otherwise ()

4 w = 6 = 4 = W 7 w = w = Fig. : Recoverable threshold as a function of W. P =., m =.75n where Ω = k w p. Now J(A) = det(a T A). By finding the eigenvalues of A T A we obtain: J(A) = W t W t Ω+ t W + t W () Ω Now we define a random variable Z =( wp)x wpx p k+ where X,X,,X l k+ are independent random variables, with X p HN(, ), p (l k + wp+k ), as half-normal distributed random variables and X N(, P k ) as a normal distributed random variable. w p by inspection, (8) is equal to Cp Z (). where p Z ( ) is the probability density function for the random variable Z and p Z () is the probability density function p Z ( ) evaluated at the point Z =, and v l k+ π ly ux C = t k wp J(A) = l k w q q=k+ π l k+ l k q (n + t )W +(n P + t )W () Combining (7) and (8): B. Derivation of the External Angle β(t,t )=π k l CpZ () (3) Without loss of generality, assume K = {n k+,,n}. Consider the (l )-dimensional face G = conv{ e n l+ w n l+,..., e n k w n k, e n k+ w n k+,..., e n w n } of the skewed cross-polytope SP. The n l outward normal vectors of the supporting hyperplanes of the facets containing G are given by j i w i e i + n l { n p=n l+ w i e i,j i {, }}. Fig. 3: Successful recovery percentage for different weights. P =. and m =.75n the outward normal cone c(g, SP) at the face G is the positive hull of these normal vectors. Thus e x dx = γ(g, SP )V n l (S n l ) e r r n l dx c(g,sp) = γ(g, SP).π (n l+)/, (4) where V n l (S n l ) is the spherical volume of the (n l)- dimensional sphere S n l. Now define U to be the set {x R n l+ x n l+, x i /w i x n l+, i (n l)} and define f(x,, x n l+ ):U c(g, SP) to be the linear and bijective map f(x,, x n l+ ) = Z c(g,sp) n l x i e i + Z e x dx Z Z w x n l+ w x n l+ e f(x) dx U Z wn l x n l+ w n l x n l+ n i=n l+ e x x n l (P n i=n l+ i )x n l+ dx dx n l+ Z e (P n i=n l+ i )x Z W x «( P )n t Z W x e y dy W x = n l Z e x where ξ = ξ(t,t )= Z Wx ξ q Pn e y dy i=n l+ w i w i x n l+ e i. ( P )n t e dy«y dx W x! r Z W x ξ e y dy! r dx, (5),r = ( P)n t r = ( P )n t. J(A) = det(a T A)=ξ is resulting from the change of variable in the integral. IV. EXPONENT CALCULATION Using the Laplace method we compute the angle exponents. They are given in the following theorems, the proofs of which are omitted for brevity. we assume n = γ n, n = γ n and WLG W =, W = W.

5 9 9 = = w * w = = = w = = (a) (b) Fig. : Successful recovery percentage for weighted l minimization with different weights and suboptimal weights in a nonuniform sparse setting. P =.5 and m =.5n Theorem : Let t = t n, t = t n, g(x) = π e x, G(x) = π x e y dy. Also define C = (t + γ )+ W (t + γ P ), D = γ ( ) t and D = γ ( P ) t. Let x be the unique solution to x of the following: C g(x)d xg(x) Wg(Wx)D = xg(wx) ψ ext (t,t )=Cx D log G(x ) D log G(Wx ) (6) Theorem : Let b = t +W t t +t and ϕ(.) and Φ(.) be the standard Gaussian pdf and cdf functions respectively. Also t let Q(s) = ϕ(s) (t + Wt ϕ(ws) +t )Φ(s) (t +t )Φ(Ws). Define the function ˆM(s) = s Q(s) and solve for s in ˆM(s) = m mb+ω. Let the unique solution be s and set y = s (b ). Compute ˆM(s ) t the rate function Λ (y) =sy t t +t Λ (s) t +t Λ (Ws) at the point s = s, where Λ (s) = s + log(ϕ(s)). The internal angle exponent is then given by: ψ int (t,t ) = (Λ (y)+ m Ω y + log )(t + t ) (7) As an illustration of these results, for P =. and δ = m n =.75 using Theorems and and combining the exponents with the combinatorial exponent, we have calculated the threshold for for different values of in the range [, 3], below which the signal can be recovered. The curve is depicted in Figure. As expected, the curve is suggesting that in this setting weighted l minimization boosts the weak threshold in comparison with l minimization. This is verified in the next section by some examples. V. SIMULATION We demonstrate by some examples that appropriate weights can boost the recovery percentage. We fix P and n =m =, and try l and weighted l minimization for various values of. We choose n = n = n Figure a shows one such comparison for P =.5 and different values of. Note that the optimal value of varies as changes. Figure b illustrates how the optimal weighted l minimization surpasses the ordinary l minimization. The optimal curve is basically achieved by selecting the best weight of Figure a for each single value of. Figure 3 shows the result of simulations in another setting where P =. and m =.75n (similar to the setting of the previous section). It is clear from the figure that the recovery success threshold for has been shifted higher when using weighted l minimization rather than standard l minimization. Note that this result very well matches the theoretical result of Figure. REFERENCES [] David Donoho, High-Dimensional Centrally Symmetric Polytopes with Neighborliness Proportional to Dimension, Discrete and Computational Geometry, (7), pp , 6, Springer. [] Emmanuel Candes and Terence Tao, Decoding by linear programming, IEEE Trans. on Information Theory, 5(), pp , December 5. [3] David Donoho and Jared Tanner, Neighborliness of randomly-projected simplices in high dimensions, Proc. National Academy of Sciences, (7), pp , 5. [4] W. Xu and Babak Hassibi, Compressed Sensing Over the Grassmann Manifold: A Unified Analytical Framework, Forty-Sixth Annual Allerton Conference, September 3-6. [5] Mark Davenport, Michael Wakin, and Richard Baraniuk, Detection and estimation with compressive measurements, Rice ECE Department Technical Report TREE 6, November 6 [6] Shihao Ji, Ya Xue, and Lawrence Carin, Bayesian compressive sensing, IEEE Trans. on Signal Processing, 56(6) pp , June 8 [7] M. Stojnic, F. Parvaresh and B. Hassibi, On the reconstruction of blocksparse signals with an optimal number of measurements, Preprint 8. [8] E. J. Cands, M. Wakin and S. Boyd. Enhancing sparsity by reweighted l minimization. J. Fourier Anal. Appl., [9] Branko Grünbaum, Grassmann angles of convex polytopes. Acta Math., :pp.93-3, 968. [] Branko Grünbaum, Convex polytopes, volume of Graduate Texts in Mathematics. Springer-Verlag, New York, second edition, 3. Prepared and with a preface by Volker Kaibel, Victor Klee and GnterüM. Ziegler. [] L.A.Santaló, Geometría integral en espacios de curvatura constante, Rep.Argetina Publ.Com.Nac.Energí Atómica, Ser.Mat,No.(95) [] Peter McMullen. Non-linear angle-sum relations for polyhedral cones and polytopes. Math. Proc. Cambridge Philos. Soc., 78():pp.47-6, 975. [3] M. Stojnic, W. Xu, and B. Hassibi, Compressed sensing - probabilistic analysis of a null-space characterization, IEEE International Conference on Acoustics, Speech and Signal Processing, Pages: , March 3 8-April 4 8. [4]