arxiv: v5 [cs.it] 16 Mar 2012

Transcription

1 ONE-BIT COMPRESSED SENSING BY LINEAR PROGRAMMING YANIV PLAN AND ROMAN VERSHYNIN arxiv:09.499v5 [cs.it] 6 Mar 0 Abstract. We give the first coputationally tractable and alost optial solution to the proble of one-bit copressed sensing, showing how to accurately recover an s-sparse vector x R n fro the signs of O(s log (n/s)) rando linear easureents of x. The recovery is achieved by a siple linear progra. This result extends to approxiately sparse vectors x. Our result is universal in the sense that with high probability, one easureent schee will successfully recover all sparse vectors siultaneously. The arguent is based on solving an equivalent geoetric proble on rando hyperplane tessellations.. Introduction Copressed sensing is a odern paradig of data acquisition, which is having an ipact on several disciplines, see []. The scientist has access to a easureent vector v R obtained as (.) v = Ax, where A is a given n easureent atrix and x R n is an unknown signal that one needs to recover fro v. One would like to take n, rendering A non-invertible; the key ingredient to successful recovery of x is take into account its assued structure sparsity. Thus one assues that x has at ost s nonzero entries, although the support pattern is unknown. The strongest known results are for rando easureent atrices A. In particular, if A has Gaussian i.i.d. entries, then we ay take = O(s log(n/s)) and still recover x exactly with high probability [0, 7]; see [6] for an overview. Furtherore, this recovery ay be achieved in polynoial tie by solving the convex iniization progra (.) in x subject to Ax = v. Stability results are also available when noise is added to the proble [9, 8, 3, 7]. However, while the focus of copressed sensing is signal recovery with inial inforation, the classical set-up (.), (.) assues infinite bit precision of the easureents. This disaccord raises an iportant question: how any bits per easureent (i.e. per coordinate of v) are sufficient for tractable and accurate sparse recovery? This paper shows that one bit per easureent is enough. There are any applications where such severe quantization ay be inherent or preferred analog-to-digital conversion [0, 8], binoial regression in statistical odeling and threshold group testing [], to nae a few... Main results. This paper deonstrates that a siple odification of the convex progra (.) is able to accurately estiate x fro extreely quantized easureent vector y = sign(ax). Date: Septeber 9, Matheatics Subject Classification. 94A; 60D05; 90C5. Y.P. is supported by an NSF Postdoctoral Research Fellowship under award No R.V. is supported by NSF grants DMS and 0089.

2 Here y is the vector of signs of the coordinates of Ax. Note that y contains no inforation about the agnitude of x, and thus we can only hope to recover the noralized vector x/ x. This proble was introduced and first studied by Boufounos and Baraniuk [6] under the nae of one-bit copressed sensing; soe related work is suarized in Section.. We shall show that the signal can be accurately recovered by solving the following convex iniization progra (.3) in x subject to sign(ax ) y and Ax =. The first constraint, sign(ax ) y, keeps the solution consistent with the easureents. It is defined by the relation a i, x y i 0 for i =,,...,, where a i is the i-th row of A. The second constraint, Ax =, serves to prevent the progra fro returning a zero solution. Moreover, this constraint is linear as it can be represented as one linear equation y i a i, x = where y i denote the coordinates of y. Therefore (.3) is indeed a convex iniization progra; furtherore one can easily represent it as a linear progra, see (5.3) below. Note also that the nuber in (.3) is chosen for convenience of the analysis; it can be replaced by any other fixed positive nuber. Theore. (Recovery fro one-bit easureents). Let n,, s > 0, and let A be an n rando atrix with independent standard noral entries. Set (.4) δ = C ( s log(n/s) log(n/ + /n) ) /5. Then, with probability at least C exp( cδ), the following holds uniforly for all signals x R n satisfying x / x s. Let y = sign(ax). Then the solution ˆx of the convex iniization progra (.3) satisfies ˆx x ˆx x δ. Here and thereafter C and c denote positive absolute constants; other standard notation is explained in Section.3. Reark (Effective sparsity). The Cauchy-Schwarz inequality iplies that x / x x 0 where x 0 = supp(x) is the nuber of nonzero eleents of x. Therefore one can view the paraeter ( x / x ) as a easure of effective sparsity of the signal x. The effective sparsity is thus a real valued and robust extension of the sparsity paraeter x 0, which allows one to handle approxiately sparse vectors. Let us then state the partial case of Theore. for sparse signals: Corollary. (Sparse recovery fro one-bit easureents). Let n,, s > 0, and set δ as in (.4). Then, with probability at least C exp( cδ), the following holds uniforly for all signals x R n satisfying x 0 s. Let y = sign(ax). Then the solution ˆx of the convex iniization progra (.3) satisfies ˆx x ˆx x δ. To be precise, for a scalar z 0 we define sign(z) = z/ z, and sign(0) = 0. We allow the sign function to act on a vector by acting individually on each eleent.

3 Reark (Nuber of easureents). The conclusion of Corollary. can be stated in the following useful way. With high probability, an arbitrarily accurate estiation of every s-sparse vector x can be achieved fro = O(s log (n/s)) one-bit rando easureents. The iplicit factor in the O( ) notation depends only on the desired accuracy level δ; ore precisely δ 5 s log (n/s) up to an absolute constant factor. The sae holds if x is only effectively s-sparse as in Theore.. The central point here is that the nuber of easureents is alost linear in the sparsity s, which can be uch saller than the abient diension n. Reark 3 (Non-gaussian easureents). Most results in copressed sensing, and in rando atrix theory in general, are valid not only for Gaussian rando atrices but also for general rando atrix ensebles. In one-bit copressed sensing, since the easureents sign(ax) do not depend on the scaling of the rows of A, it is clear that our results will not change if the rows of A are sapled independently fro any rotationally invariant distribution in R n (for exaple, the unifor distribution on the unit Euclidean sphere S n ). However, in contrast to the widespread universality phenoenon, one-bit copressed sensing cannot be generalized to soe of the siplest discrete distributions, such as Bernoulli. Indeed, suppose the entries of A are independent ± valued syetric rando variables. Then for the vectors x = (, 0, 0,..., 0) and x = (,, 0,..., 0) one can easily check that sign(ax) = sign(ax ) for any nuber of easureents. So one-bit easureents can not distinguish between two fixed distinct signals x and x no atter how any easureents are taken. Reark 4 (Optiality). For a fixed level of accuracy, our estiate on the nuber of easureents = O(s log (n/s)) atches the best known nuber of easureents in the classical (not quantized) copressed sensing proble up to the exponent of the logarith, and up to an absolute constant factor. However, we believe that the exponent can be reduced to. We also believe that the error δ in Theore. ay decrease ore quickly as s/ 0. In particular, Jacques et al. [8] deonstrate that x is exactly sparse and is estiated using an l 0 -iniization-based approach, the error is upper bounded as δ = O((s/) o() log n); they also deonstrate a lower error bound δ = Ω(s/) regardless of what algorith is used. In fact, such a result is not possible when x is only known to be effectively sparse (i.e., x / x s). Instead, the best possible bound is of the for δ = O( (s/) log(n/s)) (this can be checked via entropy arguents). We believe this is achievable (and is optial) for the convex progra (.3)... Prior work. While there have been several nuerical results for quantized copressed sensing [6, 4, 5, 0, 8], as well as guarantees on the convergence of any of the algoriths used for these nuerical results, theoretical accuracy guarantees have been uch less developed. One ay endeavor to circuvent this proble by considering quantization errors as a source of noise, thereby reducing the quantized copressed sensing proble to the noisy classical copressed sensing proble. Further, in soe cases the theory and algoriths of noisy copressed sensing ay be adapted to this proble as in [8,, 7, 5]; the ethod of quantization ay be specialized in order to iniize the recovery error. As noted in [9] if the range of the signal is unspecified, then such a noise source is unbounded, and so the classical theory does not apply. However, in the setup of our paper we ay assue without loss of generality that x =, and thus it is possible that the ethods of Candes and Tao [8] can be adapted to derive a version of Corollary. for a fixed sparse signal x. Nevertheless, we do not see any way to deduce by these ethods a unifor result over all sparse signals x. In a copleentary line of research Ardestanizadeh et al. [] consider copressed sensing with a finite nuber of bits per easureent. However, the nuber of bits per easureent there is not 3

4 one (or constant); this nuber depends on the sparsity level s and the dynaic range of the signal x. Siilarly, in the work of Gunturk et al. [4, 5] on siga-delta quantization, the nuber of bits per easureent depends on the dynaic range of x. On the other hand, by considering sigadelta quantization and ultiple bits, the Gunturk et al. are able to provide excellent guarantees on the speed of decay of the error δ as s/ decreases. The fraework of one-bit copressed sensing was introduced by Boufounos and Baraniuk in [6]. Jacques et al. [8] show that O(s log n) one-bit easureents are sufficient to recover an s-sparse vector with arbitrary precision; their results are also robust to bit flips. In particular, their results require the estiate ˆx to be as sparse as x, have unit nor, and be consistent with the data. The difficulty is that the first two of these constraints are non-convex, and thus the only known progra which is known to return such an estiate is l 0 iniization with the unit nor constraint this is generally considered to be intractable. Gupta et al. [6] deonstrate that one ay tractably recover the support of x fro O(s log n) easureents. They give two easureent schees. One is non-adaptive, but the nuber of easureents has a quadratic dependence on the dynaic range of the signal. The other has no such dependence but is adaptive. Our results settle several of these issues: (a) we ake no assuption about the dynaic range of the signal, (b) the one-bit easureents are non-adaptive, and (c) the signal is recovered by a tractable algorith (linear prograing)..3. Notation and organization of the paper. Throughout the paper, C, c, C, etc. denote absolute constants whose values ay change fro line to line. For integer n, we denote [n] = {,..., n}. Vectors are written in bold italics, e.g., x, and their coordinates are written in plain text so that the i-th coponent of x is x i. For a subset T [n], x T is the vector x restricted to the eleents indexed by T. The l and l nors of a vector x R n are defined as x = n x i and x = ( n x i )/ respectively. The nuber of non-zero coordinates of x is denoted by x 0 = supp(x). The unit balls with respect to l and l nors are denoted by B n = {x Rn : x } and B n = {x Rn : x } respectively. The unit Euclidean sphere is denoted S n = {x R n : x = }. The rest of the paper is devoted to proving Theore.. In Section we reduce this task to the following two ingredients: (a) Theore.3 which states states that a solution to (.3) is effectively sparse, and (b) Theore. which analyzes a sipler but non-convex version of (.3) where the constraint Ax = is replaced by x =. The latter result can be interpreted in a geoetric way in ters of rando hyperplane tessellations of a subset K of the Euclidean sphere, specifically for the set of effectively sparse signals K = S n sb n. In Section 3 we estiate the etric entropy of K, and we use this in Section 4 to prove our ain geoetric result of independent interest: = O(s log(n/s)) rando hyperplanes are enough to cut K into sall pieces, yielding that all cells of the resulting tessellation have arbitrarily sall diaeter. This will coplete part (b) above. For part (a), we prove Theore.3 on the effective sparsity of solutions in Section 5. The proof is based on counting all possible solutions of (.3), which are the vertices of the feasible polytope. This will allow us to use standard concentration inequalities fro the Appendix and to conclude the arguent by a union bound. Acknowledgeent. The authors are grateful to Sinan Güntürk for pointing out an inaccuracy in the stateent of Lea 3.4 in an earlier version of this paper.. Strategy of the proof Our proof of Theore. has two ain ingredients which we explain in this section. Throughout the paper, a i will denote the rows of A, which are i.i.d. standard noral vectors in R n. 4

5 Let us revisit the second constraint Ax = in the convex iniization progra (.3). Consider a fixed signal x for the oent. Taking the expectation with respect to the rando atrix A, we see that E Ax = E a i, x = c x where c = /π. Here we used that the first absolute oent of the standard noral rando variable equals c. So in expectation, the constraint Ax = is equivalent to x = up to constant factor c. This observation suggests that we ay first try to analyze the sipler iniization progra (.) in x subject to sign(ax ) = y and x =. This optiization progra was first proposed in [6]. Unfortunately, it is non-convex due to the constraint x =, and therefore sees to be coputationally intractable. On the other hand, we find that the non-convex progra (.) is ore aenable to theoretical analysis than the convex progra (.3). The first ingredient of our theory will be to deonstrate that the non-convex optiization progra (.) leads to accurate recovery of an effectively sparse signal x. One can reforulate this as a geoetric proble about rando hyperplane tessellations. We will discuss tessellations in Section 4; the ain result of that section is Theore 4. which iediately iplies the following result: Theore.. Let n,, s > 0, and set (.) δ = C ( s log(n/s) ) /5. Then, with probability at least C exp( cδ), the following holds uniforly for all x, ˆx R n that satisfy x = ˆx =, x s, ˆx s: sign(aˆx) = sign(ax) iplies ˆx x δ. Theore. yields a version of our ain Theore. for the non-convex progra (.): Theore. (Non-convex recovery). Let n,, s > 0, and set δ as in (.). Then, with probability at least C exp( cδ), the following holds uniforly for all signals x R n satisfying x / x s. Let y = sign(ax). Then the solution ˆx of the non-convex iniization progra (.) satisfies ˆx x x δ. Proof. We can assue without loss of generality that x = and thus x s. Since x is feasible for the progra (.), we also have ˆx x s, and thus ˆx S n. Therefore Theore. applies to x, ˆx, and it yields that ˆx x δ as required. Reark 5 (Prior work). A version of Theore. was recently proved in [8] for exactly sparse signals x, ˆx, i.e. such that x = ˆx =, x 0 s, ˆx 0 s. This latter result holds with δ = C(s/) o() log(n). However, fro the proof of Theore. given above one sees that the result of [8] would not be sufficient to deduce our ain results, even Corollary. for exactly sparse vectors. The reason is that our goal is to solve a tractable progra that involves the l nor, and thus we cannot directly assue that our estiate will be in the low-diensional set of exactly sparse vectors. Our proof of Theore. has to overcoe soe additional difficulties copared to [8] caused by the absence of any control of the supports of the signals x, ˆx. In particular, the 5

6 etric entropy of the set of unit-nored, sparse vectors only grows logarithically with the inverse of the covering accuracy. This allows the consideration of a very fine cover in the proofs in [8]. In contrast, the etric entropy of the set of vectors satisfying x and x s is uch larger at fine scales, thus necessitating a different strategy of proof. Theore. would follow if we could deonstrate that the convex progra (.3) and the nonconvex progra (.) were equivalent. Rather than doing this explicitly, we shall prove that the solution ˆx of the convex progra (.3) essentially preserves the effective sparsity of a signal x, and we finish off by applying Theore.. Theore.3 (Preserving effective sparsity). Let n, s > 0 and suppose that Cs log(n/s). Then, with probability at least C exp( c), the following holds uniforly for all signals x satisfying x / x s. Let y = sign(ax). Then the solution ˆx of the convex iniization progra (.3) satisfies ˆx x C log(n/ + /n). ˆx x This result is the second ain ingredient of our arguent, and it will be proved in Section 5. Now we are ready to deduce Theore.. Proof of Theore.. Consider a signal x as in Theore., so x / x s. In view of the application of Theore.3, we ay assue without loss of generality that Cs log(n/s). Indeed, otherwise we have δ and the conclusion of Theore. is trivial. So Theore.3 applies and gives ˆx C s log(n/ + /n) =: s 0. ˆx Also, as we noted above, x / x s s 0. So Theore. applies for the noralized vectors x/ x, ˆx/ ˆx and for s 0. Note that sign(aˆx) = sign(ax) = y because ˆx is a feasible vector for the progra (.3). Therefore Theore. yields ˆx x ˆx x δ where ( s0 ) ( /5 δ = C log(n/s) = C s ) /5 log(n/s) log(n/ + /n). This copletes the proof. For the rest of the paper, our task will be to prove the two ingredients above Theore., which we shall relate to a ore general hyperplane tessellation proble, and Theore.3 on the effective sparsity of the solution. 3. Geoetry of signal sets Our arguents are based on the geoetry of the set of effectively s-sparse signals and the set of s-sparse signals K n,s := { x R n : x, x s } = B n sb n S n,s := { x R n : x, x 0 s }. While the set S n,s is not convex, K n,s is, and oreover it is a convexification of S n,s in the following sense. Below, for a set K, we define conv(k) to be its convex hull. Lea 3. (Convexification). One has conv(s n,s ) K n,s conv(s n,s ). 6

7 Proof. The first containent follows by Cauchy-Schwartz inequality, which iplies for each x S n,s that x s. The second containent is proved using a coon technique in the copressed sensing literature. Let x K n,s. Partition the support of x into disjoint subsets T, T,... so that T indexes the largest s eleents of x (in agnitude), T indexes the next s largest eleents, and so on. Since all x Ti S n,s, in order to coplete the proof it suffices to show that x Ti. i To prove this, first note that x T x. Second, note that for i, each eleent of x Ti is bounded in agnitude by x Ti /s, and thus x Ti x Ti / s. Cobining these two facts we obtain (3.) x Ti + x Ti + x Ti / s + x / s, i i i where in the last inequality we used that x s for x K n,s. The proof is coplete. Our arguents will rely on entropy bounds for the set K n,s. Consider a ore general situation, where K is a bounded subset of R n and ε > 0 is a fixed nuber. A subset N K is called an ε-net of K if for every x K one can find y N so that x y ε. The inial cardinality of an ε-net of K is called the covering nuber and denoted N(K, ε). The nuber log N(K, ε) is called the etric entropy of K. The covering nubers are (alost) increasing by inclusion: (3.) K K iplies N(K, ε) N(K, ε). Specializing to our sets of signals K n,s and S n,s, we coe across a useful exaple of an ε-net: Lea 3. (Sparse net). Let s t. Then S n,t K n,s is an s/t-net of K n,s. Proof. Let x K n,s, and let T [n] denote the set of the indices of the t largest coefficients of x (in agnitude). Using the decoposition x = x T + x T c and noting that x T S n,t K n,s, we see that it suffices to check that x T c s/t. This will follow fro the sae steps as in (3.). In particular, we have x T c x / T s/t as required. Next we pass to quantitative entropy estiates. The entropy of the Euclidean ball can be estiated using a standard volue coparison arguent, as follows (see [4, Lea 4.6]): (3.3) N(B n, ε) (3/ε) n, ε (0, ). Fro this we deduce a known bound on the entropy of S n,s : Lea 3.3 (Entropy of S n,s ). For ε (0, ) and s n, we have ( 9n ) log N(S n,s, ε) s log. εs Proof. We represent S n,s as the union of the unit Euclidean balls B n RI in all s-diensional coordinate subspaces, I [n], I = s. Each ball B n RI has an ε-net for of cardinality at ost (3/ε) s, according to (3.3). The union of these nets fors an ε-net of S n,s, and since the nuber of possible I is ( ) ( n s, the resulting net has cardinality at ost n s ) (3/ε) s (3en/εs) s. Taking the logarith copletes the proof. As a consequence, we obtain an entropy bound for K n,s : 7

8 Lea 3.4 (Entropy of K n,s ). For ε (0, ), we have { n log ( ) 6 log N(K n,s, ε) ε log ( ) 9εn 4s ε Cs ε s log ( n s ). if 0 < ε < s n if s n ε Proof. First note that K n,s B n. Then the onotonicity property (3.) followed by the voluetric estiate (3.3) yield the first desired bound N(K n,s, ε) N(B n, ε/) n log(6/ε) for all ε (0, ). Next, suppose that s n < ε <. Then set t := 4s/ε n. Lea 3. states that S n,t K n,s is an (ε/)-net of K n,s. Furtherore, to find an (ε/)-net of S n,t, we use Lea 3.3 for ε/4 and for t. Taking into account the onotonicity property (3.), we see that there exists an (ε/)-net N of S n,t K n,s and such that ( 36n ) log N t log = 4s ( 9εn ) εt ε log. s It follows that N is an ε-net of K n,s, and its cardinality is as required. 4. Rando hyperplane tessellations In this section we prove a generalization of Theore.. We consider a set K R n and a collection of rando hyperplanes in R n, chosen independently and uniforly fro the Haar easure. The resulting partition of K by this collection of hyperplanes is called a rando tessellation of K. The cells of the tessellation are fored by intersection of K and the rando half-spaces with particular orientations. The ain interest in the theory of rando tessellations is the typical shape of the cells. Figure. Hyperplane tessellation of a subset K of a sphere We shall study the situation where K is a subset of the sphere S n, see Figure. The particular exaple of K = S n is a natural odel of rando hyperplane tessellation in the sperical space S n. The ore classical and well studied odel of rando hyperplane tessellation is in Euclidean space R n, where the hyperlanes are allowed to be affine, see [3] for the history of this field. The rando hyperplane tessellations of the sphere is studied in particular in []. Here we focus on the following question. How any rando hyperplanes ensure that all the cells of the tessellation of K have sall diaeter (such as /)? For the purposes of this paper, we shall address this proble for a specific set, naely for K = S n sb n = S n K n,s. We shall prove that = O(s log(n/s)) hyperplanes suffice with high probability. Our arguent can be extended to ore general sets K, but we defer generalizations to a later paper. 8

9 Theore 4. (Rando hyperplane tessellations). Let s n and be positive integers. Consider the tessellation of the set K = S n sb n by rando hyperplanes in Rn chosen independently and uniforly fro the Haar easure. Let δ (0, ), and assue that Cδ 5 s log(n/s). Then, with probability at least exp( δ), all cells of the tessellation of K have diaeter at ost δ. It is convenient to represent the rando hyperplanes in Theore 4. as (a i ), i =,...,, where a i are i.i.d. standard noral vectors in R n. The clai that all cells of the tessellation of K have diaeter at ost δ can be restated in the following way. Every pair of points x, y K satisfying x y > δ is separated by at least one of the hyperplanes, so there exists i [] such that a i, x > 0, a i, y < 0. Theore 4. is then a direct consequence of the following slightly stronger result. Theore 4. (Separation by a set of hyperplanes). Let s n and be positive integers. Consider the set K = S n sb n and independent rando vectors a,..., a N (0, Id) in R n. Let δ (0, ), and assue that Cδ 5 s log(n/s). Then, with probability at least exp( δ), the following holds. For every pair of points x, y K satisfying x y > δ, there is a set of at least cδ of the indices i [] that satisfy a i, x > cδ, a i, y < cδ. We will prove Theore 4. by the following covering arguent, which will allow us to uniforly handle all pairs x, y K satisfying x y > δ. We choose an ε-net N ε of K as in Lea 3.4. We decopose the vector x = x 0 + x where x 0 N ε is a center and x εb n K is a tail, and we do siilarly for y. An eleentary probabilistic arguent and a union bound will allow us to nicely separate each pair of centers x 0, y 0 N ε satisfying x 0 y 0 > δ by Ω() hyperplanes. (Specifically, it will follow that a i, x 0 > cδ, a i, y 0 < cδ for at least cδ of the indices i [].) Furtherore, the tails x, y εb n sb n can be uniforly controlled using Lea 5.4, which iplies that all tails are in a good position with respect to o() hyperplanes. (Specifically, for sall ε one can deduce that a i, x < cδ/, a i, y < cδ/ for at least cδ/ of the indices i [].) Putting the centers and the tails together, we shall conclude that x and y are separated at least Ω() + o() > Ω() hyperplanes, as required. Now we present the full proof of Theore Step : Decoposition into centers and tails. Let ε (0, ) be a nuber to be deterined later. Let N ε be an ε-net of K. Since K K n,s, Lea 3.4 along with onotonicity property of entropy (3.) guarantee that N ε can be chosen so that (4.) log N ε Cs ε log ( n s Lea 4.3 (Decoposition into centers and tails). Let t = 4s/ε. Then every vector x K can be represented as (4.) x = x 0 + εx where x 0 N ε, x K n,t. 9 ).

10 Proof. Since N ε is an ε-net of K, representation (4.) holds for soe x B n. Since K n,t = B n tb n, it reains to check that x tb n. Note that x K sb n and x 0 N ε K sb n. By the triangle inequality this iplies that εx = x x 0 sb n. Thus x ( s/ε)b n = tb n, as claied. 4.. Step : Separation of the centers. Our next task is to separate the centers x 0, y 0 of each pair of points x, y K that are far apart by Ω() hyperplanes. For a fixed pair of points and for one hyperplane, it is easy to estiate the probability of a nice separation. Lea 4.4 (Separation by one hyperplane). Let x, y S n and assue that x y δ for soe δ > 0. Let a N (0, Id). Then for δ 0 = δ/ we have Proof. Note that P { a, x > δ 0, a, y < δ 0 } δ 0. P { a, x > δ 0, a, y < δ 0 } = P { a, x > 0 and a, y < 0 and a, x / (0, δ 0 ] and a, y / [ δ 0, 0)} P { a, x 0 or a, y 0} P { a, x (0, δ 0 ]} P { a, y [ δ 0, 0)}. The inequality above follows by the union bound. Now, since a, x N (0, ) we have P { a, x (0, δ 0 ]} δ 0 π and P { a, y [ δ 0, 0)} δ 0 π. Also, denoting the geodesic distance in S n by d(, ) it is not hard to show that (see [3, Lea 3.]). Thus as claied. P { a, x 0 or a, y 0} = d(x, y) π x y π P { a, x > δ 0, a, y < δ 0 } δ π δ 0 π δ 0 δ π Now we will pay attention to the nuber of hyperplanes that nicely separate a given pair of points. Definition 4.5 (Separating set). Let δ 0 (0, ). x, y S n is defined as I δ0 (x, y) := {i [] : a i, x > δ 0, a i, y < δ 0 }. The separating index set of a pair of points The cardinality I δ0 (x, y) is a binoial rando variable, which is the su of indicator functions of the independent events { a i, x > δ 0, a i, y < δ 0 }. The probability of each such event can be estiated using Lea 4.4. Indeed, suppose x y δ for soe δ > 0, and let δ 0 = δ/. Then the probability of each of the events above is at least δ 0. Then I δ0 (x, y) Binoial(, p) with p > δ 0. A standard deviation inequality (e.g. [, Theore A..3]) yields (4.3) P { I δ0 (x, y) < δ 0 /} e δ 0/8. Now we take a union bound over pairs of centers in the net N ε that was chosen in the beginning of Section 4.. Lea 4.6 (Separation of the centers). Let ε, δ (0, ), and let N ε be an ε-net of K whose cardinality satisfies (4.). Assue that (4.4) C ( s n ) ε δ log. s 0

11 Then, with probability at least exp( δ/00), the following event holds: (4.5) For every x 0, y 0 N ε such that x 0 y 0 > δ, one has I δ/ (x 0, y 0 ) δ/4. Proof. For a fixed pair x 0, y 0 as above, we can rewrite (4.3) as P { I δ/ (x 0, y 0 ) < δ/4 } e δ/96. A union bound over all pairs x 0, y 0 iplies that the event in (4.5) fails with probability at ost N ε e δ/96. By (4.) and (4.4), this quantity is further bounded by [ Cs ( n ) exp ε log δ ] exp( δ/00) s 96 provided the absolute constant C is chosen sufficiently large. The proof is coplete Step 3: Control of the tails. Now we provide a unifor control of the tails x K n,t that arise fro the decoposition given in Lea 4.3. The next result is a direct consequence of Lea 5.4. Lea 4.7 (Control of the tails). Let t and let a,..., a N (0, Id) be independent rando vectors in R n. Assue that (4.6) Ct log(n/t). Then, with probability at least exp( c), the following event holds: sup ai, x. x K n,t 4.4. Step 4: Putting the centers and tails together. Let ε = c 0 δ for a sufficiently sall absolute constant c 0 > 0. To control the tails, we choose an ε-net N ε of K as in Lea 4.6, and we shall apply this lea with δ/ instead of δ. Note that requireent (4.4) becoes ( n ) C δ 5 s log, s and it is satisfied by the assuption of Theore 4., for a sufficiently large absolute constant C. So Lea 4.6 yields that with probability at least exp( δ/00), the following separation of centers holds: (4.7) For every x 0, y 0 N ε such that x 0 y 0 > δ/, one has I δ/4 (x 0, y 0 ) δ/48. To control the tails, we choose t = 4s/ε s/δ 4 as in Decoposition Lea 4.3, and we shall apply Lea 4.7. Note that requireent (4.6) becoes ( C 3 δ 4 C3 n ) s log, s and it is satisfied by the assuption of Theore 4., for a sufficiently large absolute constant C. So Lea 4.7 yields that with probability at least exp( c), the following control of tails holds: (4.8) For every x K n,t, one has a i, x.

12 Now we cobine the centers and the tails. With probability at least exp( cδ), both events (4.7) and (4.8) hold. Suppose both these events indeed hold, and consider a pair of vectors x, y K as in the assuption, so x y > δ. We decopose these vectors according to Lea 4.3: (4.9) x = x 0 + εx, y = y 0 + εy where x 0, y 0 N ε and x, y K n,t. By the triangle inequality and the choice of ε, the centers are far apart: x 0 y 0 x y ε > δ ε = δ c 0 δ δ/. Then event (4.7) iplies that the separating set (4.0) I 0 := I δ/4 (x, y) satisfies I 0 δ/48. Furtherore, using (4.8) for the tails x and y we see that a i, x + a i, y. By Markov s inequality, the set { I := i [] : ai, x + a i, y 9 } δ satisfies (I ) c δ 96. We clai that I := I 0 I is a set of indices i that satisfies the conclusion of Theore 4.. Indeed, the nuber of indices in I is as required since I I 0 (I ) c δ 48 δ 96 = δ 96. Further, let us fix i I. Using decoposition (4.9) we can write a i, x = a i, x 0 + ε a i, x. Since i I I 0 = I δ/4 (x, y), we have a i, x 0 > δ/4, while fro i I I we obtain a i, x 9/δ. Thus a i, x > δ 4 9ε δ δ 30, where the last estiate follows by the choice of ε = c 0 δ for a sufficiently sall absolute constant c 0 > 0. In a siilar way one can show that a i, y < δ 4 + 9ε δ δ 30. This copletes the proof of Theore Effective sparsity of solutions In this section we prove Theore.3 about the effective sparsity of the solution of the convex optiization proble (.3). Our proof consists of two steps a lower bound for ˆx proved in Lea 5. below, and an upper bound on x which we can deduce fro Lea 5.4 in the Appendix. Lea 5. (Euclidean nor of solutions). Let n, > 0. Then, with probability at least C exp( c log(n/ + /n)), the following holds uniforly for all signals x R n. Let y = sign(ax). Then the solution ˆx of the convex iniization progra (.3) satisfies ˆx c/ log(n/ + /n).

13 Reark 6. Note that the sparsity of the signal x plays no role in Lea 5.; the result holds uniforly for all signals x. Let us assue that Lea 5. is true for a oent, and show how together with Lea 5.4 it iplies Theore.3. Proof of Theore.3. With probability at least C exp( c), the conclusions of both Lea 5. and Lea 5.4 with t = /4 hold. Assue this event occurs. Consider a signal x as in Theore.3 and the corresponding solution ˆx of (.3). By Lea 5., the latter satisfies (5.) ˆx c/ log(n/ + /n). Next, consider λ = a i, x = Ax. Since by the assuption on x we have x/ x K n,s S n, Lea 5.4 with t = /4 iplies that (5.) λ x. By definition of λ, the vector λ x is feasible for the progra (.3), so the solution ˆx of this progra satisfies ˆx λ x = λ x. Putting this together with (5.) and (5.), we conclude that ˆx ˆx x λ ˆx x x ˆx x x C log(n/ + /n). This copletes the proof of Theore.3. In the rest of this section we prove Lea 5.. The arguent based on the observation that the set of possible solutions ˆx of the convex progra (.3) for all x and corresponding y is finite, and its cardinality can be bounded by exp(c log(n/ + /n)). For each fixed solution ˆx, a lower bound on ˆx will be deduced fro Gaussian concentration inequalities, and the arguent will be finished by taking a union bound over ˆx. It ay be convenient to recast the convex iniization progra (.3) as a linear progra by introducing the duy variables u = (u, u,..., u n ): (5.3) in n u i such that: u i x i u i, i =,,..., n; y i a i, x 0, i =,,..., ; y i a i, x. The feasible set of the linear progra (5.3) is a polytope in R n, and the linear progra attains a solution on a vertex of this polytope. Further, since the rando Gaussian vectors a i are in general position, one can check that the solution of the linear progra is unique with probability. Thus, by characterizing these vertices and pointing out the relationship between u i and ˆx i, we ay reduce the space of possible solutions ˆx. This is the content of our next lea. Given subsets T {,,..., n}, Ω {,,..., }, we define A Ω T to be the subatrix of A with coluns indexed by T and rows indexed by Ω. Lea 5. (Vertices of the feasible polytope). With probability, the linear progra (5.3) attains a solution (ˆx, u) at a point which satisfies the following for soe T {,,..., n} and Ω {,,..., }: 3

14 () u i = ˆx i ; () supp(ˆx) = T ; (3) T = Ω + ; (4) A Ω T ˆx T = 0; (5) a i, ˆx =. Proof. Part () follows since we are iniizing u i. Part (5) follows since y i a i, ˆx = a i, ˆx cobined with the fact that we are iplicitly iniizing x. Parts () (4) will follow fro the fact that (5.3) achieves its iniu at a vertex. The vertices are precisely the feasible points for which soe d of the inequality constraints achieve equality, provided ˆx is the unique solution to those d equalities. Since (ˆx, u) R n at least n of the constraints ust be equalities. We now count equalities based on T and Ω. We first consider the constraints u i x i u i, i =,,..., n. If ˆx i = 0 we have two equalities, u i = ˆx i and u i = ˆx i, otherwise, we have one. This gives n + T c equalities. Part (5) gives one ore equality. This leaves us with at least n n T c = T equalities that ust be satisfied out of the equations y i a i, ˆx 0. Thus, we ay take Ω = T. Proof of Lea 5.. We ay disregard the duy variables (u i ) and consider that the solution ˆx = x ust satisfy conditions () (5) above for soe T and Ω. We will show that with high probability, any such vector x R n is lower bounded in the Euclidean nor. Let us first fix sets T and Ω, and consider a vector x satisfying () (5). We represent it as x = µ x for soe µ > 0 and x =. Our goal is to lower bound µ. By condition (4) above, we have A Ω T x T = 0 which, with probability, copletely deterines the vector x up to ultiplication by ± (since T = Ω + and x T c = 0). Moreover, since supp( x) = supp(x ) = T, we have 0 = A Ω T x T = A Ω x, so a i, x = 0 for i Ω. Using this with together with condition (5), we obtain and thus (5.4) = µ a i, x = µ a i, x x = µ = i/ Ω ( ). a i, x We proceed to upper bound i/ Ω a i, x. Since the rando vector x depends entirely on A Ω T, it is independent of a i for i / Ω. Thus, by the rotational invariance of the Gaussian distribution, for any fixed vector z with unit nor, we have the following distributional estiates: i/ Ω a i, x dist = i/ Ω i/ Ω a i, z dist a i, z. For rando variables X, Y, the distributional inequality X dist Y eans that P{X > t} P{Y > t} for all t R. 4

15 The last ter is a su of independent sub-gaussian rando variables, and it can be bounded using standard concentration inequalities. Specifically, applying Lea 5.3 fro the Appendix, we obtain { } P a i, z > t C exp( ct ) for t. Using (5.4), this is equivalent to P { x < /t } C exp( ct ) for t. It is left to upper bound the nuber of vectors satisfying conditions () (5) and to use the union bound. Since T = Ω +, the total nuber of possible choices for T and Ω (and hence of x ) is in(,n ) ( )( ) n exp(c log(n/ + /n)). i + i i=0 Thus, by picking t = C 0 log(n/ + /n) with a sufficiently large absolute constant C0, we find that all x uniforly satisfy the required estiate x c/ log(n/ + /n) with probability at least exp(c log(n/ + /n)) C exp( ct ) = C exp( c log(n/ + /n)). Lea 5. is proved. Appendix. Unifor concentration inequality. In this section we prove concentration inequalities for Ax = a i, x. In the situation where the vector x is fixed, we have a su of independent rando variables, which can be controlled by standard concentration inequalities: Lea 5.3 (Concentration). Let n, N and x R n. Then, for every t > 0 one has { } P a i, x π x > t x C exp( ct ). Proof. Without loss of generality we can assue that x =. Then a i, x are independent standard noral rando variables, so E a i, x = /π. Therefore X i := a i, x /π are independent and identically distributed centered rando variables. Moreover, X i are sub-gaussian rando variable with X i ψ C, see [6, Reark 8]. An application of Hoeffding-type inequality (see [6, Proposition 0]) yields { } P X i > t C exp( ct ). This copletes the proof. We will now prove a stronger version of Lea 5.3 that is unifor over all effectively sparse signals x. Lea 5.4 (Unifor concentration). Let n N, t [0, /π], and suppose that Ct 4 s log(n/s). Then { } P sup a i, x π > t C exp( ct ). x K n,s S n 5

16 Proof. This is a standard covering arguent, although the approxiation step requires a little extra care. Let M be a t/4-net of K n,s S n. Since K n,s S n K n,s, we can arrange by Lea 3.4 that M exp(ct s log(n/s)). By definition, for any x K n,s S n one can find x M such that x x t/4. So the triangle inequality yields a i, x π a i, x π + a i, x x. Note that x x x + x s. Together with x x t/4 this eans that x x t 4 K n,64s/t. Consequently, (5.5) sup x K n,s S n a i, x π sup x M a i, x π + t 4 sup w K n,64s/t a i, w =: R + t 4 R. We bound the ters R and R separately. For siplicity of notation, we assue that 64s/t is an integer, as the non-integer case will have no significant effect on the result. A bound on R follows fro the concentration estiate in Lea 5.3 and a union bound: (5.6) P {R > t/4} M C exp( ct ) C exp(ct s log(n/s) ct ) C exp( ct ) provided that Ct 4 s log(n/s). Next, due to Lea 3. and Jensen s inequality, we have R sup w S n,64s/t ( a i, w sup w S n,64s/t a i, w ) / =: R. The quantity R has been well studied in copressed sensing; it is bounded by the restricted isoetry constant of the atrix A at sparsity level 64s/t. Probabilistic bounds for the restricted isoetry constants of Gaussian atrices are well known, and have been derived in the earliest copressed sensing works [0]. We use the bound in [6, Theore 65] that gives (5.7) P { R >.5 } exp( c) provided that Ct s log(n/s). Thus P {R > 3} exp( c). Cobining this and (5.6) we conclude that { P R + t } 4 R > t C exp( ct ) where we used the assuption that t /π. This and (5.5) coplete the proof. 6

17 References [] Alon, N., and Spencer, J. H. The probabilistic ethod, second ed. Wiley-Interscience (John Wiley and Sons), New York, 000. [] Ardestanizadeh, E., Cheraghchi, M., and Shokrollahi, A. Bit precision analysis for copressed sensing. In International Syposiu on Inforation Theory (ISIT) (009), IEEE. [3] Bickel, P. J., Ritov, Y., and Tsybakov, A. B. Siultaneous analysis of Lasso and Dantzig selector. The Annals of Statistics 37, 4 (Aug. 009), [4] Boufounos, P. T. Greedy sparse signal reconstruction fro sign easureents. In Asiloar Conference on Signals, Systes and Coputers (Noveber 009). [5] Boufounos, P. T. Reconstruction of sparse signals fro distorted randoized easureents. In International Conference on Acoustics, Speech and Signal Processing (00). [6] Boufounos, P. T., and Baraniuk, R. G. -Bit copressive sensing. In 4nd Annual Conference on Inforation Sciences and Systes (CISS) (Mar. 008). [7] Candes, E., Rudelson, M., Vershynin, R., and Tao, T. Error correction via linear prograing. 46th Annual Syposiu on Foundations of Coputer Science (005), [8] Candes, E., and Tao, T. The Dantzig selector: Statistical estiation when p is uch larger than n. The Annals of Statistics 35, 6 (Dec. 007), [9] Candès, E. J., Roberg, J. K., and Tao, T. Stable signal recovery fro incoplete and inaccurate easureents. Counications on Pure and Applied Matheatics 59, 8 (Aug. 006), [0] Candes, E. J., and Tao, T. Near-optial signal recovery fro rando projections: Universal encoding strategies? IEEE Transactions on Inforation Theory 5, (Dec. 006), [] Dai, W., Pha, H. V., and Milenkovic, O. A coparative study of quantized copressive sensing schees. In International Syposiu on Inforation Theory (June 009). [] Daaschke, P. Threshold group testing. Electronic Notes in Discrete Matheatics (005), [3] Goeans, M. X., and Williason, D. P. Iproved approxiation algoriths for axiu cut and satisfiability probles using seidefinite prograing. Journal of the ACM 4, 6 (Nov. 995), [4] Gunturk, C., Laers, M., Powell, A., Saab, R., and Ylaz, O. Siga delta quantization for copressed sensing. In 44th Annual Conference on Inforation Sciences and Systes (CISS) (00), IEEE. [5] Gunturk, C., Laers, M., Powell, A., Saab, R., and Ylaz, O. Sobolev duals for rando fraes and siga-delta quantization of copressed sensing easureents. Preprint. Available at [6] Gupta, A., Nowak, R., and Recht, B. Saple coplexity for -bit copressed sensing and sparse classification. In International Syposiu on Inforation Theory (ISIT) (00), IEEE. [7] Jacques, L., Haond, D., and Fadili, J. Dequantizing copressed sensing: When oversapling and nongaussian constraints cobine. IEEE Transactions on Inforation Theory 57, (0), [8] Jacques, L., Laska, J. N., Boufounos, P. T., and Baraniuk, R. G. Robust -bit copressive sensing via binary stable ebeddings of sparse vectors. Preprint. Available at [9] Laska, J. N., Boufounos, P. T., Davenport, M. A., and Baraniuk, R. G. Deocracy in action: Quantization, saturation, and copressive sensing. Applied and Coputational Haronic Analysis (Feb. 0). [0] Laska, J. N., Wen, Z., Yin, W., and Baraniuk, R. G. Trust, but verify: Fast and accurate signal recovery fro -bit copressive easureents. Preprint. Available at files/publications/journal-article/00/trustbutverify.pdf. [] Mackenzie, D. Copressed sensing akes every pixel count. What s Happening in the Matheatical Sciences, 7 (009), 4 7. [] Miles, R. Rando points, sets and tessellations on the surface of a sphere. Sankhyā: The Indian Journal of Statistics, Series A 33 (97), [3] Møller, J., and Stoyan, D. Stochastic geoetry and rando tessellations. In Tessellations in the sciences: virtues, techniques and applications of geoetric tilings, R. van de Weijgaert, G. Vegter, V. Icke, and J. Ritzerveld, Eds. Springer-Verlag. To appear. [4] Pisier, G. The volue of convex bodies and Banach space geoetry. Cabridge University Press, 989. [5] Sun, J., and Goyal, V. Optial quantization of rando easureents in copressed sensing. In IEEE International Syposiu on Inforation Theory (ISIT) (009), IEEE. [6] Vershynin, R. Introduction to the non-asyptotic analysis of rando atrices. In Copressed Sensing: Theory and Applications, Y. Eldar and G. Kutyniok, Eds. Cabridge University Press. To Appear (00). [7] Wojtaszczyk, P. Stability and instance optiality for Gaussian easureents in copressed sensing. Foundations of Coputational Matheatics 0, (Apr. 009), 3. 7

18 [8] Zynis, A., Boyd, S., and Candes, E. Copressed sensing with quantized easureents. IEEE Signal Processing Letters 7, (Feb. 00), Departent of Matheatics, University of Michigan, 530 Church St., Ann Arbor, MI 4809, U.S.A. E-ail address: 8