Beyond Worst-Case Reconstruction in Deterministic Compressed Sensing

Transcription

1 Beyond Worst-Case Reconstruction in Deterministic Comressed Sensing Sina Jafarour, ember, IEEE, arco F Duarte, ember, IEEE, and Robert Calderbank, Fellow, IEEE Abstract The role of random measurement in comressive sensing is analogous to the role of random codes in coding theory In coding theory, decoders that can correct beyond the minimum distance of a code allow random codes to achieve the Shannon limit In comressed sensing, the counterart of minimum distance is the sark of the measurement matrix, ie, the size of the smallest set of linearly deendent columns This aer constructs a family of measurement matrices where the columns are formed by exonentiating codewords from a classical binary error-correcting code of block length The columns can be artitioned into mutually unbiased bases, and the sark of the corresonding measurement matrix is shown to be O( ) by identifying a configuration of columns that lays a role similar to that of the Dirac comb in classical Fourier analysis Further, an exlicit basis for the null sace of these measurement matrices is given in terms of indicator functions of binary self-dual codes Reliable reconstruction of k-sarse inuts is shown for k of order /log() which is best ossible and far beyond the worst case lower bound rovided by the sark I INTRODUCTION The central goal of comressed sensing (CS) is to cature a signal using very few measurements In most work to date, this broader objective is exemlified by the imortant secial case in which the measurement data constitute a vector f = +e where is an N matrix called the sensing matrix, R N is a k-sarse signal, and e R is the additive noise There are two distinct CS frameworks with different objectives Worst-case CS [], []: In the worst-case CS framework, the goal is to recover every k-sarse vector from the corresonding measurement vector f It is known that certain robabilistic rocesses generate sensing matrices that suort worst-case CS [3] However, the random sensing framework suffers from storage and comutation limitations As a result, there has been a significant amount of research on designing alternative deterministic matrices for worst-case CS framework over the last few years [4] ost such constructions rely on the coherence between the columns of the matrix When the coherence follows the Welch Bound µ = O, the Gerschgorin Circle Theorem guarantees reconstruction of any k-sarse signal with k = O( ) Average-case CS [4] [6]: In many ractical alications, including wireless communications and radar, it is not nec- SJ is with the ultimedia Research Grou, Yahoo! Research, sinaj@yahoo-inccom D is with the Deartment of Electrical and Comuter Engineering, University of assachusetts-amherst, mduarte@ecsumassedu RC is with the Deartment of Comuter Science, Duke University, robertcalderbank@dukeedu The work is suorted in art by ONR under grant N and by AFOSR under grant FA essary to reconstruct every sarse vector [7] The goal of average-case CS is to recover most (in contrast to all) k-sarse vectors Here, the sarse vector is often modeled to have a uniformly random suort and random sign for the k nonzero entries The average-case CS framework relies on the coherence and the sectral norm of the deterministic sensing matrix The ideal case is when coherence follows the Welch bound [8], and different measurements are orthogonal Then, as long as k = O (/log N), with high robability a k- sarse vector has a unique sarse reresentation, and can be efficiently recovered from the comressive measurements [6] In this aer, we construct an exlicit basis for the null sace of a large family of deterministic sensing matrices designed for the average-case CS framework (see [5] and the references therein) using the indicator vectors of binary selfdual codes Characterizing the null sace of these matrices makes it ossible to investigate and analyze the geometric roerties of these matrices more recisely ore secifically, we introduce a family of deterministic sensing matrices called the extended Delsarte-Goethals frames (EDGFs) that hold the following three roerties simultaneously: (i) as long as k ale c it is ossible to recover every k-sarse vector from the measurement vector ; (ii) there exists a k-sarse vector with k = c such that no reconstruction algorithm can uniquely recover from ; and (iii) it is ossible to recover most k-sarse vectors from the measurement vector as long as k ale c 3 log N (which can be much larger than c ), where c, c, and c 3 are fixed constants The EDGFs meet the coherence-based lower bound on worst-case reconstruction and the order-otimal uer bound on average-case reconstruction The rest of the aer is organized as follows Section II reviews Delsarte-Goethals frames (DGFs) Section III exlains the roerties of the self-dual codes used in this aer Sections IV and V characterize the null sace of the DGFs and the EDGFs The exeriments are rovided in Section VI Section VII concludes the aer II BACKGROUND AND NOTATION A Worst-case vs Average-case Comressed Sensing A vector is k-sarse if it has at most k non-zero entries The suort of a k-sarse vector indicates the ositions of its non-zero entries Let be an N matrix Then is a tightframe with redundancy if = I N N, where denotes the conjugate transose of The sark of the measurement matrix, denoted as sark( ), is the size of the smallest set

2 of linearly deendent columns of Let ' i denote the i th column of The coherence between the columns of is defined as the maximum inner roduct between two distinct columns of : µ = max i6=j ' i ' j k' i k k' j k In this aer, we simlify the analysis by focusing on the noiseless CS roblem, and note that it is straightforward to generalize the analysis to include noise The following two theorems relate the maximum sarsity level k to the arameters of the sensing matrix, so that it is ossible to efficiently recover all (resectively most) k-sarse vectors from Theorem (Worst-case CS [9]): Let be a an N sensing matrix with worst-case coherence µ Then as long as k = O(µ ), it is ossible to efficiently recover every k-sarse vector from the measurement vector In contrast, when sark( ) k, there exist multile k-sarse vectors that are maed to the same measurement vector, rendering recovery imossible Theorem (Average-case CS [6], [0]): Let be a an N tight-frame with redundancy = n N/ and with µ o worst-case coherence µ If k = O min log N, log N, then with robability /N, it is ossible to efficiently recover a k-sarse vector with uniformly random suort and uniformly random sign from the measurement vector Throughout this aer, m denotes an integer and = m Given a vector v with binary entries, we let v(x) denote the entry of v indexed by x The inner roduct of two binary vectors u, v is denoted by u > v B Delsarte-Goethals Frames The Delsarte-Goethals frames (DGFs) are a class of CS matrices that have been recently introduced by Calderbank, Howard, and Jafarour [5] Secifically, let m be an odd ositive integer, and r to be an integer smaller than m Next, let DG(m, r) denote the Delsarte-Goethals set of binary symmetric matrices, as described in [] Then, given = m and N = r+, the N DGF is constructed from DG(m, r) in the following way Index the rows of by binary vectors x F m and index the columns of by airs (P, b), where P ranges over all (r+)m binary symmetric matrices DG(m, r) and b ranges over all members of F m The entries of are given by ' (P,b) (x) = i x> Px+b >x, where i = It is easy to see from this descrition that (i) DGFs are unions of orthonormal bases and (ii) each DGF can be reresented as = [D R H,, D H, H], () where R = N = r+, H is the Hadamard matrix, and each D i is a diagonal matrix whose (x, x) entry is i x> P ix where P i is a binary symmetric matrix from the Delsarte-Goethals set DG(m, r) A DGF is a tight-frame with redundancy r+ and worst-case coherence µ = r [5] As a result, it follows from Theorems and that as long as k = O( / r ), it is ossible to recover every k- sarse vector from using the `-minimization method; moreover, even if k = O r log N it is still ossible to recover most k-sarse vectors from using the same techniques III BINARY SELF-DUAL CODES We start by defining a binary self-dual code and exlaining some of its roerties Definition 3 (Binary Self-Dual Code): A binary code C is self-dual if C? = u : u > w =08 w C = C, () Let C be a self-dual code of length m, and let b be a binary m-tule vector in the finite field F m Throughout the aer, by b C we mean {b c : c C} Definition 3: Let C be a binary self-dual code of length m The binary vector v of length = m with entries v(x) = if x C and v(x) =0if x/ C is called the indicator of C A direct calculation, catured in the following lemma, shows that the indicator of a self-dual code can be viewed as the binary counterart of the Dirac comb in Fourier analysis Lemma 3: Let C be a binary self-dual linear code of length m, and let v {0, } be the indicator of C Let H be the Hadamard matrix Then Hv = C v Next, we use the vector v to construct h ai sarse vector in the null sace of the matrix 0 = I, H h i Theorem 3: Let 0 = I, H be an matrix generated from concatenating the identity matrix and the normalized Hadamard matrix Let C be a binary linear selfdual code with indicator v Define v =[ v, v] > Then I 0 is a tight frame with redundancy II v is a -sarse vector in the null sace of 0 Therefore sark( 0 ) ale Proof: We rove each art searately: I 0 is a union of two orthonormal bases, therefore = I II Every self-dual code of length m has dimension m, and hence exactly different codewords Therefore, v is -sarse, and by construction, v has exactly non-zero entries oreover, it follows from Lemma 3 that Hv = h i Iv or equivalently I, H [ v, v] > = 0 Corollary 3: Let D be an diagonal matrix, and let = [DH, H] be an matrix generated from concatenating the modulated Hadamard h (DH) matrix and the i Hadamard (H) matrix Define v N = H D v ; v Then v N is in the null sace of

3 Proof: Theorem 3 guarantees that H v I v =0 On the other hand, since the Hadamard matrix has orthogonal binary-valued rows, we have H = H, and therefore H v DH HD v =Hv DH (DH) v =0 Finally, we consider the sarsity degree of v N Since v is -sarse, only of the second entries of vn are nonzero Now we analyze the first entries of v N Let denote the diagonal of D, let =D v, and denote =H Here we focus on a secial but imortant case where there exists an m m binary symmetric matrix P such that (x) =i x> Px for every x F m This is the case for a large class of CS matrices, including the DGFs For every binary m-tule x we have (x) if x C (x) = 0 otherwise, and (x) = ( ) x>y (y) = ( ) x> y yc yf m (y) = yc( ) x>y i y> Py Note that the calculation y > Py+x > y is now over Z 4 and not Z Let (x) denote the comlex conjugate of (x) Observe that (x) is zero if and only if (x) is zero Therefore, it is sufficient to analyze (x) Theorem 3: Let P be an m m binary symmetric matrix, and let E be the null sace of P Define (x) = yc i y> Py+x >y, where x F m Let d P denote the diagonal of P and assume that d P C This assumtion is easily satisfied if we only consider zero-diagonal matrices in the construction of the DGFs Define C 0 = {z C :Pz C} (3) Then (x) 6= 0for at most t values of x, where t = m dim(c 0 ) Proof: We have (x) = i y> Py+y 0> Py 0 +x > (y+y 0 ) y,y 0 C = y,y 0 C i (y+y0 ) > P(y+y 0 )+y > Py 0 +x > (y+y 0) Changing variables to z = y + y 0 and y gives (x) = zc = zc i z> Pz+x > z yc ( ) (z+y)> Py i z> Pz+x > z yc( ) (dp+pz)>y The inner sum is zero if (d P +Pz) / C Otherwise the inner sum is C, and we have (x) = C zc 0 i z> Pz+x >z (4) Now observe that for any z,z C 0, (z + z ) > P(z + z )+x > (z + z ) = z > Pz > +x > z + z > Pz +x > z +z > Pz (5) = z > Pz > +x > z + z > Pz +x > z (mod 4), where the second equality follows from the fact that both Pz and z are codewords of C If this linear ma is the zero ma on x, then (x) = C C 0, and otherwise (x) =0 As a result, (x) vanishes for all but t values of x, where t = m dim(c 0 ) IV THE NULL SPACE OF DELSARTE-GOETHALS FRAES In this section, we construct a basis for the null sace of matrices of the form of Equation () To do this, we first rovide h an orthogonal i basis for the null sace of the matrix 0 = I, H, Let a be a binary m-dimensional vector The time-shift matrix A a is a circulant matrix so that every row x of A a has only one at the corresonding column indexed at x a and zeros elsewhere Similarly, the frequency-shift matrix B a is a diagonal matrix with diagonal entries ( ) xa>, where the m-dimensional binary vector x ranges over all = m rows A direct calculation reveals that HA a = B a H, and HB a = A a H Lemma 4: Let 0 and v be as above, and let a and b be any two binary m-dimensional offsets Then the vector w = [A a B b v; B a A b v] is in the null sace of 0 Proof: We have 0w = A a B b v HB a A b v = A a B b v A a B b Hv (6) = A a B b alei, H [v; v] =0 Lemma 4: Let W = {w : w =[A a B b v; B a A b v],a,b F m } Then W is an orthogonal basis forming the null sace of 0 Proof: Since v is the indicator of a self dual code, it is -sarse oreover, there are exactly offsets a that roduce distinct shifts of v The reason is that since C is linear, if c is any codeword of C, then both A a v and A a c v rovide the same vector whose non-zero entries corresond to indices of the form z a where z ranges over all codewords That is A a v = A a c v Similarly, since C is self-dual, for every air of codewords z and c we have ( ) b>z =( ) (b c)>z This imlies that there are distinct choices b for the frequency shift of the vector v Now we show that these vectors are orthogonal To see this, let (a, b) and (a 0,b 0 ) be two distinct airs of time and frequency shift offsets Let = B a A b v and = BaA 0 0 b v Then, it is sufficient to show that if and are distinct, then > =0 Since v is the indicator vector of a linear self-dual code, then if Su( ) \ Su( ) contains some element y, then

4 the set y C is also in the intersection of the two suorts This set has elements, and since both suorts also have elements, we must have Su( )=Su( ) As a result, if A b v 6= A 0 b v then Su( ) \ Su( )=Ø, and therefore B a A b v and BaA 0 0 bv are orthogonal On the other hand, if A b v = A 0 b v, then and are two distinct Walsh tones restricted to the set b C Now let c be an element in C such that c > (a a 0 )= Then ( ) c> (a a 0 ) = bc bc ( ) x> (a a 0 ) ( ) (x c)> (a a 0 )!! = bc ( ) x> (a a 0) Therefore, we must have P bc ( (a a 0 ) )x> = 0, which roves that and are orthogonal A similar argument can be used to show that if = A a B b v and = A 0 abb 0v, then, if and are distinct, then > =0 As a result, we have distinct linearly indeendent vectors of the form w =[A a B b v; B a A b v] which are all in the null sace of 0 On the other hand, since 0 is, its null sace has dimension, and therefore the null sace vectors above form a basis for the null sace of 0 Next we show that the same argument can be used to form an orthogonal basis for the null sace of matrices of form = [DH, H], where D is a diagonal matrix with h diagonal entries i xpx>i, x F m Theorem 4: Let = [DH, H], where D=[i xpx> ], and where P is a zero-diagonal binary symmetric matrix Let C be a self-dual code such that for any codeword c, Pc is also a codeword in C Let v denote the indicator vector of the codewords of C Let W = w : w = (7) ale HD A a B b v; B a A b v,a,b F m Then W forms an orthonormal basis for the null sace of the matrix ; moreover, every element of W is -sarse Proof: Let w be any vector in W We have w = ale HD [DH, H] A a B b v; B a A b v (8) =[A a B b v HB a A b v]=0 Now we show that any vector w W is -sarse First, note that since v is the indicator of a self-dual code, and the oerators A b and B a do not change the sarsity level, B a A b v is -sarse Hence, we need to show that! = HD A a B b v is also -sarse We have The zero-diagonal assumtion is only for simlifying the calculations, and is easy satisfied by using the Gray ma []!(x) = ( ) b>y i (y a)> P (y a) ( ) (y a)> x yc = i a> Pa ( ) a> x yc i (b+x+pa)> y+y > Py Since C is self-dual and y and Py are both codewords in C, the maing y! (b + x + ap ) > y + y > Py is a linear ma from C to Z 4 Theorem 3 now guarantees that w(x) is non-zero only for choices of x Remark 4: So far we have shown how to generate a basis for the null sace of a matrix of the form [DH, H] This construction can be generalized to matrices of form [D R H,, D H, H] in a straightforward manner by first zeroadding each null sace vector, so that it becomes (R + )dimensional, and then collecting all these R vectors For instance, a basis for the null sace of a 3 matrix [D H, D H, H] has the form 4 0 V 0 V 0 0 V V 3 (9) 5, (0) where [V; 0 V ] is a basis for the null sace of [D H, H], and [V; 0 V ] is a basis for the null sace of [D H, H] V ETENDED DELSARTE-GOETHALS FRAES In this section, we use the results of the revious sections and design an N sensing matrix with N for which there exists constants c, c, and c 3, such that (i) it is ossible to uniquely recover every k-sarse vectors as long as k ale c ; (ii) there exists a k-sarse vector with k = c such that no reconstruction algorithm is able to uniquely recover it from the measurement vector f = ; and (iii) it is ossible to uniquely recover most k-sarse vectors as long as k ale c 3 /log N The extended Delsarte-Goethals frame (EDGF) is constructed by concatenating the identity matrix to a DGF Theorem 5: Let r be a constant integer, and let m>r+ be an integer Then: I The EDGF has = m rows, and N = r+ + columns II The EDGF is a tight frame with redundancy r+ +, and coherence r Proof: The roof of Part I follows from the construction of the EDGF To rove Part II observe that the EDGF is a union of orthonormal bases, and therefore it is a tight-frame with redundancy N The coherence bound follows from the fact that the inner roduct between two distinct columns of a DGF is bounded by a column of a DGF, and a column of the identity matrix is bounded by r, and that the inner roduct between Since the matrix 0 = [I, H] is a submatrix of any EDGF, it follows from Theorem 3 that the sark of any

5 =56, N=5, self dual submatrix dimension (56*80) with rank 76 # <=$"(>6?="#$!>6,857!@A56,A4B/096@0B8C,03C6D$"(E%$F6G0H6/CI6%# 6 09!&+ 08!&* robability of exact recovery / :6/8:3;8/!&)!&(!&"!&'!&% 0!&$ 0 Self dual Uniform sarsity!&# J857!@A5 KC073/B! 6! " #! #" $! $" %! %",-/,0 Fig : Probability of exact recovery of the Basis Pursuit algorithm in recovering sarse vectors using a 56 5 submatrix of the DG(8, 0) frame The matrix has a rank-deficient submatrix with rank 76 Fig : Probability of exact recovery of the Basis Pursuit algorithm in recovering sarse vectors using a submatrix of the extended DG(8, 0) frame The matrix has a 56 3 rank-deficient submatrix with rank 3 EDGF is at most As a result, there can be two distinct -sarse vectors maed to the same low-dimemsional measurement vector On the other hand, Theorem states that it is ossible touniquely recover every k-sarse vector with sarsity k ale c In contrast, it follows from Theorem that if is a k-sarse vector with uniformly random suort and random sign, then as long as k ale c 3 /log N (which can be much larger than c ), is uniquely recoverable from f = with overwhelming robability VI EPERIENTAL RESULTS The exeriments of this section comare the erformance of the Basis Pursuit algorithm [6] in recovering sarse vectors with uniformly random suorts versus recovering sarse vectors suorted on a rank-deficient submatrix obtained using a self-dual code We rovide the comarisons for the DGFs (Fig ), as well as for the EDGFs (Fig ) In Fig we used a 56 5 matrix of the form = [H, D H], where D is a diagonal matrix with d D (x) = i x> Px, and P is the 8 8 zero-diagonal binary symmetric matrix obtained from alying the Gray ma [] to a 7 7 binary DG(7, 0) matrix We also used the self-dual Hamming code C of length 8 in order to find a rank-deficient submatrix of A simle calculation reveals that only 4 codewords are in C 0 (defined by eq (3)) Therefore, dim(c 0 )=, and Theorem 3 redicts that HD v must have 56/4 = 64 non-zero entries A direct calculation reveals that HD v indeed has 64 nonzero entries Therefore, the null sace of [H, D H] has a 80-sarse element That is, there exists, a submatrix of that is rank-deficient As illustrated in Fig, the Basis Pursuit Algorithm fails to recover some k-sarse vectors with sarsity level k > 40 which are suorted on this rankdeficient submatrix However, it is still ossible to recover most k-sarse vectors with uniformly random suort over the 5 columns, even for sarsity level k = 80 Fig shows the result of the same exeriment using an EDGF VII CONCLUSION We have determined a natural basis for the null sace of an extended Delsarte-Goethals frame and shown that this null sace contains a vector that is -sarse We have demonstrated that this family of measurement matrices meets the lower bound of k = O( ) on worst-case CS as well as the order otimal uer bound of k = O(/log(N)) REFERENCES [] E Candès, J Romberg, and T Tao Stable signal recovery from incomlete and inaccurate measurements Communications on Pure and Alied athematics, Vol 59 (8), 07-3, 006 [] D Donoho Comressed Sensing IEEE Transactions on Information Theory, Vol 5 (4), , Aril 006 [3] R Baraniuk, Davenort, R DeVore, and Wakin A simle roof of the restricted isometry roerty for random matrices Constructive Aroximation, Vol 8 (3), 53-63, December 008 [4] W Bajwa, R Calderbank, and S Jafarour odel Selection: Two fundamental measures of coherence and their algorithmic significance Proceedings of IEEE Symosium on Information Theory (ISIT), 00 [5] R Calderbank, S Howard, and S Jafarour Construction of a large class of matrices satisfying a statistical isometry roerty IEEE Journal of Selected Toics in Signal Processing, Secial Issue on Comressive Sensing, Vol 4(), , 00 [6] E Candès and J Plan Near-ideal model selection by ` minimization Annals of Statistics, Vol 37, 45-77, 009 [7] L Alebaum, S Howard, S Searle, and R Calderbank Chir sensing codes: Deterministic comressed sensing measurements for fast recovery Alied and Comutational Harmonic Analysis, Vol 6 (), 83-90, arch 009 [8] W Bajwa, J Haut, G Raz, S Wright, and R Nowak Toelitzstructured comressed sensing matrices Statistical Signal Processing IEEE/SP 4th Worksho on, ages 94 98, August 007 [9] D L Donoho and Elad Otimally sarse reresentation in general (nonorthogonal) dictionaries via ` minimization Proc Nat Acad Sci, 00(5):97 0, 003 [0] J Tro On The Conditioning of Random Subdictionaries Alied and Comutational Harmonic Analysis, Vol 5, 4, 008 [] R Calderbank and S Jafarour Reed-uller sensing matrices and the LASSO International Conference on Sequences and their Alications (SETA), , 00 [] AR Calderbank, PJ Cameron, W Kantor, and JJ Seidel Z 4 Kerdock Codes, orthogonal sreads and extremal euclidean line sets Proceedings of London ath Society, vol 75, , 997