Randomness-in-Structured Ensembles for Compressed Sensing of Images

Transcription

1 Randomness-in-Structured Ensembles for Compressed Sensing of Images Abdolreza Abdolhosseini Moghadam Dep. of Electrical and Computer Engineering Michigan State University Hayder Radha Dep. of Electrical and Computer Engineering Michigan State University Abstract Leading compressed sensing (CS) methods require m = O (k log(n)) compressive samples to perfectly reconstruct a k-sparse signal x of size n using random projection matrices (e.g., Gaussian or random Fourier matrices). For a given m, perfect reconstruction usually requires high complexity methods, such as Basis Pursuit (BP), which has complexity O(n 3 ). Meanwhile, low-complexity greedy algorithms do not achieve the same level of performance (as BP) in terms of the quality of the reconstructed signal for the same m. In this paper, we introduce a new CS framework, which we refer to as Randomness-in-Structured Ensemble (RISE) projection. RISE projection matrices enable compressive sampling of image coefficients from random locations within the k-sparse image vector while imposing small structured overlaps. We prove that RISE-based compressed sensing requires only m = ck samples (where c is not a function of n) to perfectly recover a k-sparse image signal. For the case of n O(k 2 ), the complexity of our solver is O(nk) which is less than the complexity of the popular greedy algorithm Orthogonal Matching Pursuit (OMP). Moreover, in practice we only need m = 2k samples to reconstruct the signal. We present simulation results that demonstrate the RISE framework s ability to recover the original image with higher than 50 db PSNR, whereas other leading approaches (such as BP) can achieve PSNR values around 30 db only. I. INTRODUCTION Traditional compressed sensing considers a length n signal S (e.g. an n pixel image or an n pixel image region) which has a sparse decomposition k in a known basis Ψ : S = Ψx. By k sparse signal, it is usually meant that x is non-zero only in k coordinates k = x 0 n. In some applications such as CT scanners and MRIs, it is either expensive or impossible to sense all n samples from the original signal [1]. However Tao, Candes and Donoho have shown that S can be recovered by much fewer linear incoherent measurements [1][2]. More specifically if we project S into an incoherent frame φ (with respect to Ψ) then given compressive samples y m 1 = φs = φψx and the projection matrix P m n = φψ we can recover x (or equivalently S) by solving: min x 0 subject to P m n x n 1 = y m 1 (1) Under state of the art compressed sensing methods, we need m = O (k log(n)) compressive samples to perfectly reconstruct x using random projection matrices [3]-[5]. In this paper we introduce a novel Randomness-in-Structured Ensembles (RISE) of projection matrices. More importantly, we prove that RISE-based compressed sensing requires only m = ck samples (where c is not a function of n) to recover the signal/image without introducing any error. For the case of n O ( k 2), the complexity of our solver is O(nk) which is less than the complexity of the greedy algorithm OMP [6]. Moreover, in practice we only need m = 2k samples to reconstruct the signal. The organization of this paper is as follow: In section II we show that if we divide the image transform coefficients x (or coordinates) into roughly 8k random disjoint subsets, then x in all subsets will be non-zero in less than three coordinates with high probability. Then we introduce the RISE framework, which can solve an underdetermined system of three linear equations with w unknowns if these unknowns are non-zero in less than three coordinates. Thus if we sense each of the 8k subsets using RISE then we can solve all sub-problems and thus reconstruct the original image. Moreover we show that if these subsets of indices have small overlap with each other (for instance similar to the incidence matrix of a Balanced Incomplete Block Design [7]) then we will need only m = 2k samples in practice for exact recovery. The simulation results are presented in section III. II. RANDOMNESS-IN-STRUCTURED ENSEMBLES Following traditional compressed sensing literature, for simplicity and without loss of generality we refer to the sparse representation x as being the image. Consider an n pixel, real valued image x which is non-zero only in k coordinates: k = x 0 n. Hence the image is represented here by a k-sparse vector: x R n. We begin by dividing the image into some random (equal-size) blocks/windows of size w. In other words, we select w random pixels from {x 1, x 2,..., x n } to form a block/window of size w. Next we take a few samples m W < w by processing each of these blocks. More precisely, this can be stated as follows: Choose B = n/w equal-size partitions of the original set {1, 2,..., n} labeled W i such that: B=n/w i=1 W i = 1, 2,..., n, i : W i = w (2) The subset of x with indices in W i (x Wi ) is the i th block/window of the signal. We then sense m W samples from

2 each block of the signal x Wi, leading to the following equation: y i = f i (x Wl ), l [1, B], i [(l 1) m W + 1, lm W ]. This results in a compression ratio of m W /w. We show that the image can now be recovered using the compressive samples y provided each block spans a random collection of coordinates/indices. Further, if we allow these blocks to have a small amount of overlaps (similar to incidence matrix of a Balanced Incomplete Block Design [7] where the rows correspond to the subsets W i ) we can achieve higher compression ratios. In order to prove this claim we need to employ some results from the classical problem of balls in bins [8] [9]. A. Design of Blocks It will be useful to model our proposed sampling procedure in order to simplify the analysis. Dividing the length n imagevector x which is non-zero in k indices into B blocks is equivalent to randomly throwing k balls into B bins. In [8] it has been proven that on average, the fraction of bins (blocks) containing no more than r balls (non-zeros) is: Q( r, B, k) = r Q(j, B, k) = j=0 r j=0 ( ) j 1 k e B k (3) j! B So for instance setting r to two and B (number of blocks) to eight times the number of non-zeros k, then Q( r, B, k) 1 or equivalently with high probability all of blocks contains less than three non-zeros. Assume now the existence of a template projection matrix T of dimension, say m W w (where m W = 3) which could solve an underdetermined system of equations of y i = T x Wi exactly, if x Wi has at most two non-zero entries. Now, we can sense three samples from each block x Wi of the image x using our template T. Thus we can solve all of those sub-problems y i = T i x Wi, l [1, n/w], i [3l w, 3l] without introducing any error (with very high probability). Consequently, we can recover the original image exactly. We have now reduced the problem to that of finding the appropriate projection template matrix T. B. Canonical RISE As stated before, we begin by looking for a 3 w template matrix T such that if we obtain three sensed samples from an image-vector x of length n = w, according to the template: y = T x, we would be guaranteed the recovery of x exactly if x has less than three non-zero entries. (To simplify notations in this section, we are replacing x Wi by x, replacing y i by y; and hence we are using n = w). Let us consider the matrix T (Fig. 1) with the following properties: The first row consists of w random complex numbers i.e. i [1, w] : T 1,i = r 1,i exp jφ i. In the first half of the second row there is one random complex number and in the next half there is another random complex number i.e., the phase is: T 2,i = φ 1, T 2,l = φ 2 and the magnitude T 2,i = r 2,1, T 2,l = r 2,2 for i [1, w/2] and l [1 + w/2, w]. In the last row, there are four unique random complex numbers. For each quarter partition the magnitude is constant and in each half partition the phase is constant i.e., the phase is T 2,i = φ 1, T 2,l = φ 2 for i [1, w/2] and l [1 + w/2, w] and the magnitude is T 3,p = r 3,1, T 3,q = r 3,2, T 3,r = r 3,3, T 3,s = r 3,4 for p [1, w/4], q [1+w/4, w/2], r [1+w/2, 3w/4] and s [1 + 3w/4, w]. It is well-known that one efficient solution to finding a specific value in a sorted list of w numbers is a Binary Search Tree. Here we utilize a similar method: We begin by evaluating the sum of the signal in each quarter (using the second and the third rows of the template) corresponding to a BST of depth two. Following this step we use the first row to locate the non-zero elements in each sub-blocks. Additionally, the first sample provides us a sanity check (i.e. the template T can detect if there are more than two non-zeros in the signal) as demonstrated in Lemma 1. It is easy to show that we can remove one of the parameters (r i,j ) and still have the same properties. But for now we focus on this general template. Lemma 1. Assume we sense three samples y i : i = {1, 2, 3} according to the canonical RISE template from the length n = w image x which is non-zero in k indices: y = T x, k = x 0. We can recover x exactly if k 2 with the complexity of O(w 2 ) in the worst case. Proof: We will prove the lemma for each value of k independently: 1) k = 0 or the image is zero over all indices: i : x i = 0. This is a trivial case because all three compressive samples would be zero. 2) 2. k = 1 or x is non-zero only in one index. We know that the phase of the sum of two vectors is different from each of the individual phases (except for the case when the two vectors lie along the same line): i.e., (a+b) / { a, b}. Using the fact that x is real-valued and entries of the first row of the template are random, if we could find the phase of the first sample among the phases of the first row entries, we can conclude that the signal is non-zero only in one index: y 1 = φ i x j = { y 1 r 1,j exp(jφ 2) if j = i 0 otherwise 3) k = 2: Let us define four partial-sums S i such that: S i = iw/4 j=1+(i 1)w/4 (4) x j, i = {1, 2, 3, 4} (5) We can now express the second and the third compressive samples by: y 2 = (S 1 + S 2 )r 2,1 e jφ 1 + (S3 + S 4 )r 2,2 e jφ 2 (6) y 3 = (r 3,1 S 1 + r 3,2 S 2 )e jφ 1 + (r3,3 S 3 + r 3,4 S 4 )e jφ 2 (7) Since each complex number can be uniquely written as the linear combination of two given complex numbers and all parameters of the template are random hence

3 T = r 1,1 e jφ 1 r 1,2 e jφ r 1,w e jφ w r 2,1 e jφ 1 r 2,1 e jφ 1... r 2,1 e jφ 1 r 2,2 e jφ 2 r 2,2 e jφ 2... r 2,2 e jφ 2 r 3,1 e jφ 1... r 3,2 e jφ 1... r 3,3 e jφ 2... r 3,4 e jφ 2... Fig. 1: The canonical RISE projection matrix we can re-write the second compressive sample into the full-search is O(w 2 ). If there were more than one solution to following full-rank system of equations: the system of equations, it implies that the system of equations is underdetermined or x is non-zero in more than two indices. R{y 2 } = (S 1 + S 2 )r 2,1 R{e jφ 1 } + (S3 + S 4 )r 2,2 R{e jφ 2 } (8) Thus we can detect this case as well. In summary, if the image I{y 2 } = (S 1 + S 2 )r 2,1 I{e jφ 1 } + (S3 + S 4 )r 2,2 I{e jφ 2 } is (9) non-zero (at most) in two indices, we are able to recover By solving this system, we can obtain S 1 +S 2 and S 3 + S 4. Using the same arguments, we can re-state the third compressive sample as: R{y 3 } = (r 3,1 S 1 + r 3,2 S 2 )R{e jφ 1 } + (r 3,3 S 3 + r 3,4 S 4 )R{e jφ 2 } (10) equations: r 1,b e jφ b r1,aejφa [ ˆxa ˆx b ] = y 1 S t S k (16) The only (ˆx a, ˆx b ) that makes these equations consistent is the pair of (x c, x d ). Thus we are performing a full-search to find a subspace of rank two in three samples. We can find the subspace with required rank if the rank of samples is really two. Moreover this subspace is unique. If t = l then it implies that there are at least two non-zeros in the t th sub-block. Again in our full-search if we could find a unique solution to the system of equations 16, then it implies that there are exactly two non-zeros in that sub-block. Clearly the complexity of this it exactly just by three samples. Moreover we can detect that image has more than two non-zero. C. The RISE Projection Matrix So far we have proved that when dividing a sparse image I{y 3 } = (r 3,1 S 1 + r 3,2 S 2 )I{e jφ 1 } + (r 3,3 S 3 + r 3,4 S 4 )I{e jφ 2 vector } (11) into blocks (randomly), if the number of blocks is This system of equation will give us r 3,1 S 1 +r 3,2 S 2 and around eight times the number of non-zeros, then all of blocks r 3,3 S 3 + r 3,4 S 4. Since we already have and then we can would contain less than three non-zeros and the RISE template obtain the individual partial-sums S i s. Clearly if each can solve such problems. According to (3) if the number of S i turns out to be zero, we can conclude that the image blocks equals to k then on average less than 8% of blocks is zero over all associated indices with high probability: would contain more than two non-zeros and RISE template (i 1)w if S i = 0 x j = 0, j [1 +, iw 4 4 ] (12) cannot identify the signal values in those blocks. However if we design the blocks W i such that they have small overlaps Assume S t and S l are non-zero, and t l. Then we conclude that there is at least one non-zero each, in the following ranges: with each others, then we do not need to ensure that all blocks be solvable; because if we are not able to solve a specific block, then there are other blocks which jointly cover those (t 1)w [1 +, tw (l 1)w ] and [1 +, lw indices and we know that most of the blocks are solvable ] (13) Moreover, by adding overlaps between blocks, the rank of the submatrices of the projection matrix will increase which means For now suppose that there are only two non-zeros in the x at the indices c and d and they are in t th and l th that even when other blocks could not help us in determining sub-blocks the signal values at those unsolvable blocks, then we might respectively. Now we perform an exhaustive search to find directly find those values after some simple operations. The indices where the signal was non-zero. More specifically we next question is: How should the blocks overlap? We propose evaluate all ˆx a and ˆx b such that: using objects from combinatorial designs called the Incidence ˆx a r 1,a e jφa + ˆx b r 1,b e jφ b = y 1 (14) Matrix of a Balanced Incomplete Block Designs [7]. (t 1)w a [1 +, tw (l 1)w ], b [1 +, lw ] (15) The incidence matrix E of a BIBD with integer parameters (v, K, λ) ; v > K 2, λ > 0 has the following properties: every column of E contains exactly K 1 s and every row Although for each possible pair of (a, b) there is a unique pair of (ˆx a, ˆx b ) such that the above equation holds true. However, of E contains exactly r 1 s; such that r = λ(v 1)/(k 1). just one of these pairs is consistent with the first sample. Moreover two distinct rows of E both contain 1 s in exactly For proof consider the following overdetermined system of λ columns. Assume M is the matrix formed by randomly permuting the columns of E and Mi is the set of columns whose i th row of M is one. Then the i th block (W i ) will be Mi. Therefore each of image (transform) coefficient will appear in K blocks, and every two blocks will overlap in exactly λ coefficients. In summary the proposed projection matrix would be in the form of: P i,j = { T j M [ i 3] 0 otherwise D. The RISE Solver, T is the RISE template. (17) Suppose we divide the image into overlapping blocks as described in the previous sub-section. Thus it is the case that

4 most of the time there are less than three non-zeros in a block and just a small fraction of blocks contains more than two nonzeros. For the case that n O(k 2 ), we have m, w = O(k). Thus the complexity of the solver will be mo(w 2 ) = O(nk) which is less than the complexity of OMP (O(nk 2 )). Let us denote the indices which we were successful to evaluate their pixel values by C (Confident Set) and let NC (Not-Confident Set) be the indices that we could not determine the pixel values at. Clearly C NC =. We can express our compressive samples by: y = P x = P C x C + P NC x NC. After the first pass of our algorithm, we reduce the dimension of the system of equations by substituting the compressive samples from the Confident Set: y NC = y P C x C = P NC x NC (18) We expect that the size of NC will be less than the rank of our projection matrix. If this was the case, we can find the values of the remaining unknowns by: x NC = (P NC ) 1 y NC. If the cardinality of N C was larger than the rank of (P NC ) 1, we can send the new underdetermined problem y NC = P x NC to an available solver such as Basis Pursuit or StOM [5]. Since with a high probability we were able to determine the signal at least in some coordinates, then this new problem is sparser than the original problem y = P x. Hence BP or OMP can solve this problem much better than the main problem. III. SIMULATION RESULTS We tested the proposed RISE-based scheme with overlapping blocks on a large number of standard compressed sensing signals and images. Here, we present the results for the cameraman image and compare our results with two dominant solvers/projection matrices. We performed compressive sampling on each n = 8 8 block of the image. More specifically the target image is formed by keeping the largest k = 16 DCT coefficients in all 8 8 blocks and set the rest of coefficients to zero. For RISE we divide each block of the image to 10 random windows/blocks with BIBID parameters of λ = 1, K = 2 and sense three samples from each window according to the RISE template. Moreover we sensed two random compressive samples in order to increase the rank of the resulting submatrices to m = 32 total samples for each block. Furthermore we sensed 32 samples from each block using a Gaussian random matrix and applied these samples to the OMP and BP solvers. We maintain the largest 16 DCT coefficients of each 8 8 image block. This represented our target image to be recovered Fig. (2a) by the three schemes that are being compared. As it is clearly shown in Fig. 2, RISE was able to reconstruct the image (virtually) perfectly from its compressive samples while BP and OMP failed to achieve perfect reconstruction. Here we should note that in the cameraman image, the DCT coefficients (excluding the DC value) are very small in most of the blocks therefore if an algorithm just recovers the largest coefficient (e.g. the DC coefficient), the errors in those blocks will be unnoticeable. On the other hand, in blocks which are in the center of the image, all of the 16 non-zero DCT coefficients are significant and consequently introducing any error in the recovery of these blocks are obtrusive. As it is clearly illustrated in Fig.2, BP and OMP failed in recovering these regions of the image, however RISE reconstructed these areas perfectly. To demonstrate the effect of the number of compressive samples m on the quality of recovery, we have measured the PSNR of reconstructed images for a range of m, Fig. 3. Each data point shows the average of 10 iterations. In our simulations, the average performance was very close to the typical performance of a run. Although by increasing the number of compressive samples, OMP and BP improve in terms of the overall image quality, but these methods cannot achieve perfect reconstructions even in the case of m 3k samples. Meanwhile, when using RISE, we have achieved (virtually) the perfect reconstruction for all of these scenarios except in the case of m = 38 samples. Although even in this case the quality of the reconstructed images were significantly higher than the quality of the images recovered by BP and OMP, due to the random nature of defining windows W i, there were a few blocks where the cardinality of the NC was higher than the rank of the RISE projection matrix. Finally Fig. 4 illustrates the required time for recovering the image from its compressive samples (average of 10 iterations for each data point). In this paper, we claimed that the complexity of RISE solver is less than the complexity of OMP. This figure verifies this assertion. PSNR (db) RISE OMP BP Number of Samples Fig. 3: PSNR of recovered image as a function of the number of compressive samples m (average of 10 iterations for each data point) IV. CONCLUSION In this paper, we presented a new approach for solving the compressed sensing problem which does not involve any optimization algorithm or greedy decisions. In our approach, we divide the image transform coefficients into random disjoint (or with very small overlap) subsets such that with high probability in all subsets, the number of non-zeros is less than a predefined threshold. Then we introduce a template matrix which can solve an underdetermined system of equations when

5 Target Image, n=64,k=18 RISE, n=64, m=36, k=18, PSNR= Guassian Mat. BP, n=64, m=36, k=18, PSNR= Guassian Mat. OMP, N=64, m=36, k=18, PSNR= (a) Target image (b) RISE recovered (c) BP recovered Fig. 2: Comparison of RISE with Basis Pursuit and Orthogonal Matching Pursuit (d) OMP recovered Time (seconds) RISE OMP BP [4] Scott Chen, David Donoho and Michael A. Saunders, Atomic Decomposition by Basis Pursuit, Siam J. Sci, Compu, Vol. 20, issue 1, pp , 1998 [5] David L. Donoho, Yaakov Tsaig, Iddo Drori, and Jean-Luc Starck, Sparse solution of underdetermined linear equations by stagewise orthogonal matching pursuit. Preprint, 2007 [6] J. Tropp, A. C. Gilbert, Signal Recovery from Partial Information via Orthogonal Matching Pursuit, Submitted for publication, April [7] Combinatorial Designs: Construction and Analysis by Douglas R. Stinson, Springer-Verlag, New York inc, 2004 [8] [9] Raab, Steger, Balls into Bins - A Simple and Tight Analysis, Lecture notes in Computer Science, Springer, Number of Samples Fig. 4: Required time for the recovery of image from its compressive samples for RISE, BP and OMP (average of 10 iterations for each data point) number of non-zero unknowns is less than that threshold. Thus if we sense each of the subsets of the image by the template then all sub-problems are solvable. In practice the complexity of the solver can be lower than complexity of the OMP and the quality of the recovery is better than BP. ACKNOWLEDGMENT This work was supported in part by NSF Award CNS , NSF Award CCF , NSF Award CCF , and unrestricted gift from Microsoft Research. REFERENCES [1] Emmanuel Cands, Justin Romberg, and Terence Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. on Information Theory, 52(2) pp , February 2006 [2] David Donoho, Compressed sensin. IEEE Trans. on Information Theory, 52(4), pp , April 2006 [3] Emmanuel Cands and Terence Tao, Near optimal signal recovery from random projections: Universal encoding strategies?, IEEE Trans. on Information Theory, 52(12), pp , December 2006