Compressive Sensing of Sparse Tensor

Transcription

1 Shmuel Friedland Univ. Illinois at Chicago Matheon Workshop CSA2013 December 12, 2013 joint work with Q. Li and D. Schonfeld, UIC

2 Abstract Conventional Compressive sensing (CS) theory relies on data representation in the form of vectors. Many data types in various applications such as color imaging, video sequences, and multi-sensor networks, are intrinsically represented by higher-order tensors. We propose Generalized Tensor Compressive Sensing (GTCS) a unified framework for compressive sensing of higher-order spare tensors. Similar to Caiafa-Cichocki GTCS offers an efficient means for representation of multidimensional data by providing simultaneous acquisition and compression from all tensor modes. We compare the performance of the proposed method with Kronecker compressive sensing (KCS, Duarte-Baraniuk), and multi-way compressive sensing (MWCS, Sidiropoulus-Kyrillidis). We demonstrate experimentally that GTCS outperforms KCS and MWCS in terms of both accuracy and speed.

3 Overview CS of vectors CS of matrices Simulations of CS for matrices CS of tensors Simulations of CS for tensors Conclusions

4 Compressive sensing of vectors: Noiseless Σ s,n is the set of all x R N with at most s nonzero coordinates Sparse version of CS: Given x Σ s,n compress it to a short vector y = (y 1,..., y M ), M << N and send it to receiver receiver gets y, possible with noise, decodes to x Compressible version: coordinates of x have fast power law decay Solution: y = Ax, A R M N a specially chosen matrix, e.g. s-n. p. Sparse noiseless recovery: x = arg min{ z 1, Az = y} A has s-null property if for each Aw = 0, w 0, w 1 > 2 w S 1 S [N] := {1,..., N}, S = s, w S has zero coordinates outside S and coincides with w on S Recovery condition M cs log(n/s), noiseless reconstruction O(N 3 )

5 Compressive sensing of vectors with noise A R M N satisfies restricted isometry property (RIP s ): (1 δ s ) x 2 Ax 2 (1 + δ s ) x 2 for all x Σ s,n and for some δ s (0, 1) Recover with noise: ˆx = arg min{ z 1, Az y} 2 ε} reconstruction O(N 3 ) THM: Assume that A satisfies RIP 2s property with δ 2s (0, 2 1). Let x Σ s,n, y = Ax + e, e 2 ε. Then ˆx x 2 C 2 ε, where C 2 = 4 1+δ 2s 1 (1+ 2)δ 2s

6 Compressive sensing of matrices I - noiseless X = [x ij ] = [x 1... x N1 ] R N 1 N 2 is s-sparse. Y = U 1 XU 2 = [y 1,..., y M2 ] R M 1 M 2, U 1 R M 1 N 1, U 2 = R M 2 N 2 M i cs log(n i /s) and U i has s-null property for i = 1, 2 Thm M: X is determined from noiseless Y. Algo 1: Z = [z 1... z M2 ] = XU 2 RN 1 M 2 each z i a linear combination of columns of X hence s-sparse Y = U 1 Z = [U 1 z 1,..., U 1 z M2 ] so y i = U 1 z i for i [M 2 ] Recover each z i to obtain Z Cost: M 2 O(N 3 1 ) = O((log N 2)N 3 1 ) Z = U 2 X = [U 2 x 1... U 2 x N1 ] Recover each x i from i th column of Z Cost: N 1 O(N 3 2 ) = O(N 1N 3 2 ), Total cost: O(N 1N (log N 2)N 3 1 )

7 Compressive sensing of matrices II - noiseless Algo 2: Decompose Y = r i=1 u iv i, u 1,..., u r, v 1,..., v r span column and row spaces of Y respectively for example a rank decomposition of Y : r = rank Y Claim u i = U 1 a i, v j = U 2 b j, a i, b j are s-sparse, i, j [r]. Find a i, b j. Then X = r i=1 a ib i Explanation: Each vector in column and row spaces of X is s-sparse: Range(Y ) = U 1 Range(X), Range(Y ) = U 2 Range(X ) Cost: Rank decomposition: O(rM 1 M 2 ) using Gauss elimination or SVD Note: rank Y rank X s Reconstructions of a i, b j : O(r(N N3 2 )) Reconstruction of X: O(rs 2 ) Maximal cost: O(s max(n 1, N 2 ) 3 )

8 Why algorithm 2 works Claim 1: Every vector in Range X and Range X is s-sparse. Claim 2: Let X 1 = r i=1 a ib i. Then X = X 1. Prf: Assume 0 X X 1 = k j=1 c jd j, c 1,..., c k & d 1,..., d k lin. ind. as Range X 1 Range X, Range X 1 Range X c 1,..., c k Range X, d 1,..., d k Range X Claim: U 1 c 1,... U 1 c k lin.ind.. Suppose 0 = k j=1 t iu 1 c j = U 1 k j=1 t jc j. As c := k j=1 t jc j Range X, c is s-sparse. As U 1 has null s-property c = 0 t 1 =... = t k = 0. 0 = Y Y = U 1 (X X 1 )U 2 = k j=1 (U 1c j )(d j U 2 ) U 2 d 1 =... = U 2 d k = 0 d 1 =... d k = 0 as each d i is s-sparse So X X 1 = 0 contradition

9 Sum.-Noiseless CS of matrices & vectors as matrices 1. Both algorithms are highly parallelizable 2. Algorithm 2 is faster by factor s min(n 1, N 2 ) at least 3. In many instances but not all algorithm 1 performs better. 4. Caveat: the compression is : M 1 M 2 C 2 (log N 1 )(log N 2 ). 5. Converting vector of length N to a matrix Assuming N 1 = N α, N 2 = N 1 α the cost of vector compressing is O(N 3 ) the cost of algorithm 1 is O((log N)N 9 5 ), α = 3 5 the cost of algorithm 2 is O(sN 3 2 ), α = 1 2, s = O(log N)(?) Remark 1: The cost of computing Y from s-sparse X: 2sM 1 M 2 (Decompose X as sum of s standard rank one matrices)

10 Compressive sensing of matrices with noise - I Y = U 1 XU 2 + E, E F ε Algo 1: Recover each z i by ẑ i = arg min{ w i 1, U 1 w i c i (Y )} 2 ε} Form Ẑ = [z 1... z M2 ] R N 1 m 2 c i (Ẑ ) c i(xu 2 2 C 2 ε (optimistically C 2ε M ) Ẑ XU 2 F MC 2 ε (optimistically C 2 ε) Obtain ˆX by recovering each row of X: ˆb i = arg min{ w i 1, U 2 w i c i (Ẑ )} 2 MC 2 ε} ˆb i c i (X ) 2 Mε (optimistically ˆb i c i (X ) 2 ε Variation: Estimate s and take best rank s-approximation of Y : Y s Similarly, after computing Ẑ from Y s replace Ẑ by Z s. Costlier and no estimates

11 Numerical simulations We experimentally demonstrate the performance of GTCS methods on sparse and compressible images and video sequences. Our benchmark algorithm is Duarte-Baraniuk 2010 named Kronecker compressive sensing (KCS) Another method is multi-way compressed sensing of Sidoropoulus-Kyrillidis (MWCS) 2012 Our experiments use the l 1 -minimization solvers of Candes-Romberg. We set the same threshold to determine the termination of l 1 -minimization in all subsequent experiments. All simulations are executed on a desktop with 2.4 GHz Intel Core i5 CPU and 8GB RAM. We set M i = K

12 UIC logo (a) The original sparse image (b) GTCS-S recovered image recov- (d) KCS image (c) GTCS-P ered image recovered

13 PSNR and reconstruction times for UIC logo PSNR (db) GTCS S GTCS P KCS Reconstruction time (sec) GTCS S GTCS P KCS Normalized number of samples (K*K/N) (a) PSNR comparison Normalized number of samples (K*K/N) (b) Recovery time comparison Figure : PSNR and reconstruction time comparison on sparse image.

14 Explanation of UIC logo representation I The original UIC black and white image is of size (N = 4096 pixels). Its columns are 14-sparse and rows are 18-sparse. The image itself is 178-sparse. For each mode, the randomly constructed Gaussian matrix U is of size K 64. So KCS measurement matrix U U is of size K The total number of samples is K 2. The normalized number of samples is K 2 N. In the matrix case, GTCS-P coincides with MWCS and we simply conduct SVD on the compressed image in the decomposition stage of GTCS-P. We comprehensively examine the performance of all the above methods by varying K from 1 to 45.

15 Explanation of UIC logo representation II Figure?? and?? compare the peak signal to noise ratio (PSNR) and the recovery time respectively. Both KCS and GTCS methods achieve PSNR over 30dB when K = 39. As K increases, GTCS-S tends to outperform KCS in terms of both accuracy and efficiency. Although PSNR of GTCS-P is the lowest among the three methods, it is most time efficient. Moreover, with parallelization of GTCS-P, the recovery procedure can be further accelerated considerably. The reconstructed images when K = 38, that is, using 0.35 normalized number of samples, are shown in Figure??????. Though GTCS-P usually recovers much noisier image, it is good at recovering the non-zero structure of the original image.

16 Cameraman simulations I 8000 Magnitude Row Index Column index (a) Cameraman in space domain (b) Cameraman in DCT domain Figure : The original cameraman image (resized to pixels) in space domain and DCT domain.

17 Cameraman simulations II PSNR (db) GTCS S GTCS P KCS Reconstruction time (sec) GTCS S GTCS P KCS Normalized number of samples (K*K/N) (a) PSNR comparison Normalized number of samples (K*K/N) (b) Recovery time comparison Figure : PSNR and reconstruction time comparison on compressible image.

18 Cameraman simulations III (a) GTCS-S, K = 46, PSNR = db (b) GTCS-P/MWCS, K = 46, PSNR = db (c) KCS, K = 46, PSNR = db (d) GTCS-S,K = 63, PSNR = db (e) GTCS-P/MWCS, K = 63, PSNR = db (f) KCS, K = 63, PSNR = db

19 Cameraman explanations As shown in Figure??, the cameraman image is resized to (N = 4096 pixels). The image itself is non-sparse. However, in some transformed domain, such as discrete cosine transformation (DCT) domain in this case, the magnitudes of the coefficients decay by power law in both directions (see Figure??), thus are compressible. We let the number of measurements evenly split among the two modes. Again, in matrix data case, MWCS concurs with GTCS-P. We exhaustively vary K from 1 to 64. Figure?? and?? compare the PSNR and the recovery time respectively. Unlike the sparse image case, GTCS-P shows outstanding performance in comparison with all other methods, in terms of both accuracy and speed, followed by KCS and then GTCS-S. The reconstructed images when K = 46, using 0.51 normalized number of samples and when K = 63, using 0.96 normalized number of samples are shown in Figure??.

20 Compressive sensing of tensors M = (M 1,..., M d ), N = (N 1,..., N d ) N d, J = {j 1,..., j k } [d] Tensors: d i=1 RN i = R N 1... N d = R N Contraction of A = [a ij1,...,i jk ] jp JR N jp with T = [t i1,...,i d ] R N : A T = i j p [N jp ],jp J a i j1,...,i jk t i1,...,i d l [d]\j R N l X = [x i1,...,i d ] R N, U = U 1 U 2... U d R (M 1,N 1,M 2,N 2,...,M d,n d ) U p = [u (p) i pj p ] R Mp Np, p [d], U Kronecker product of U 1,..., U d. Y = [y i1,...,i d ] = X U := X 1 U 1 2 U 2... d U d R M y i1,...,i p = j q [N q],q [d] x j 1,...,j d q [d] u i q,j q Thm X is s-sparse, each U i has s-null property then X uniquely recovered from Y. Algo 1: GTCS-S Algo 2: GTCS-P

21 Algo 1- GTCS-S Unfold Y in mode 1: Y (1) = U 1 W 1 R M 1 (M 2... M d ), W 1 := X (1) [ 2 k=d U k] R N 1 (M 2... M d ) As for matrices recover the M 2 := M 2 M d columns of W 1 using U 1 Complexity: O( M 2 N1 3). Now we need to recover Y 1 := X 1 I 1 2 U 2... d U d RN 1 M 2... M d Equivalently, recover N 1, d 1 mode tensors in R N 2... N d R M 2... M d using d 1 matrices U 2,..., U d. Complexity d i=1 Ñi 1 M i+1 N 3 i from Ñ 0 = M d+1 = 1, Ñ i = N 1... N i, Mi = M i... M d d = 3: M 2 M 3 N N 1M 3 N N 1N 2 N 3 3

22 Algo 2- GTCS-P Unfold X in mode k: X (k) R N k N N k, N = d i=1 N i. As X is s-sparse rank k X := rank X (k) s. Y (k) = U k X (k) [ i k U i ] Range Y (k) U k Range X (k), rank Y (k) s. X (1) = R 1 j=1 u iv i, u 1,..., u R1 spans range of X (1) so R 1 s Each v i corresponds to U i R N 2...N d So (1) X = R j=1 u 1,j... u d,j, R s d 1 u k,1,..., u k,r R N k which is s-sparse span Range X (k) and each is s-sparse Compute decomposition Y = R j=1 w 1,j... w d,j, R s d 1, w k,1,..., w k,r R M k span Range Y (k), Compl: O(s d 1 d i=1 M i) Find u k,j from w k,j = U k u k,j and reconstruct X from (1) Complexity O(ds d 1 max(n 1,..., N d ) 3 ), s = O(log(max(N 1,..., N d )))

23 Summary of complexity converting linear data N i = N α i, M i = O(log N), α i > 0, d i=1 α i = 1, s = log N d = 3 GTCS-S: O((log N) 2 N ) GTCS-P: O((log N) 2 N) GTCS-P: O((log N) d 1 N 3 d ) for any d. Warning: the roundoff error in computing parfac decomposition of Y and then of X increases significantly with d.

24 Sparse video representation We compare the performance of GTCS and KCS on video data. Each frame of the video sequence is preprocessed to have size and we choose the first 24 frames. The video data together is represented by a tensor and has N = voxels in total. To obtain a sparse tensor, we manually keep only nonzero entries in the center of the video tensor data and the rest are set to zero. The video tensor is 216-sparse and its mode-i fibers are all 6-sparse i = 1, 2, 3. The randomly constructed Gaussian measurement matrix for each mode is now of size K 24 and the total number of samples is K 3. The normalized number of samples is K 3 N. We vary K from 1 to 13.

25 PSNR and reconstruction time of sparse video PSNR (db) GTCS S GTCS P KCS Reconstruction time (sec) GTCS S GTCS P KCS Normalized number of samples (K*K*K/N) (a) PSNR comparison Normalized number of samples (K*K*K/N) (b) Recovery time comparison Figure : PSNR and reconstruction time comparison on sparse video.

26 Reconstruction errors of sparse video Reconstruction error x Row index 20 5 x Column index Reconstruction error Row index Column index Reconstruction error Row index Column index x (a) Reconstruction error of GTCS-S (b) Reconstruction error of GTCS-P (c) Reconstruction error of KCS Figure : Visualization of the reconstruction error in the recovered video frame 9 by GTCS-S (PSNR = db), GTCS-P (PSNR = db) and KCS (PSNR = db) when K = 12, using normalized number of samples.

27 Conclusion Real-world signals as color imaging, video sequences and multi-sensor networks, are generated by the interaction of multiple factors or multimedia and can be represented by higher-order tensors. We propose Generalized Tensor Compressive Sensing (GTCS)-a unified framework for compressive sensing of sparse higher-order tensors. We give two reconstruction procedures, a serial method (GTCS-S) and a parallelizable method (GTCS-P). We compare the performance of GTCS with KCS and MWCS experimentally on various types of data including sparse image, compressible image, sparse video and compressible video. Experimental results show that GTCS outperforms KCS and MWCS in terms of both accuracy and efficiency. Compared to KCS, our recovery problems are in terms of each tensor mode, which is much smaller comparing with the vectorization of all tensor modes. Unlike MWCS, GTCS manages to get rid of tensor rank estimation, which considerably reduces the computational complexity and at the same time improves the reconstruction accuracy.

28 References 1 C. Caiafa and A. Cichocki, Multidimensional compressed sensing and their applications, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 3(6), , (2013). E. Candes, J. Romberg and T. Tao, Robust uncertainty principles: exact signal reconstruction from highly incomplete information, Information Theory, IEEE Transactions on 52 (2006), E. Candes and T. Tao, Near optimal signal recovery from random projections: Universal encoding strategies, Information Theory, IEEE Transactions on 52 (2006), D. Donoho, Compressed sensing, Information Theory, IEEE Transactions on 52 (2006), M. Duarte and R. Baraniuk, Kronecker compressive sensing, Image Processing, IEEE Transactions on, 2 (2012),

29 References 2 Q. Li, D. Schonfeld and S. Friedland, Generalized tensor compressive sensing, Multimedia and Expo (ICME), 2013 IEEE International Conference on, July 2013, ISSN: , 6 pages. S. Friedland, Q. Li and D. Schonfeld, Compressive Sensing of Sparse Tensors, arxiv: N. Sidiropoulus and A. Kyrillidis, Multi-way compressive sensing for sparse low-rank tensors, Signal Processing Letters, IEEE, 19 (2012),