Compressive Sensing and Beyond Sohail Bahmani Gerorgia Tech.
Signal Processing Compressed Sensing
Signal Models
Classics: bandlimited The Sampling Theorem Any signal with bandwidth B can be recovered from its samples collected uniformly at a rate no less that f s = 2B. Claude Shannon Harry Nyquist 1 x(t) = n Z x n sinc f s t n 0 x n = x t, sinc f s t n -3-2 -1 0 1 2 3 1
Classics: time-frequency X(τ, ω) = x t, w t τ e jωt x t = 1 2π R R X τ, ω e jωt dωdτ ψ m,n (t) = a m 2 ψ a m t nb x t = x t, ψ m,n (t) ψ m,n (t) m Z n Z D Gabor A. Haar I. Daubechies R. Coifman 2
Sparsity x F x = x Ω F Ω x T. Tao E. Candès J. Romberg = D. Donoho 3
Structured sparsity Tree Structured Models Graph Structured Models Discrete Wavelet Transform Protein-Protein Interactions Gene Regulatory Network Couch Lab, George Mason University, http://osf1.gmu.edu/~rcouch/protprot.jpg 4
Low rank Model for Images B. Recht Other applications minimum order linear system realization low-rank matrix completion low-dimensional Euclidean embedding M.Fazel P. Parrilo 5
Compressive Sensing
Linear inverse problems y A x Ax = y m = n x x = A y m < n No unique inverse, but 6
Sparse solutions argmin x 0 x subject to Ax = y x 1 x 0 = supp x supp x = i x i 0 x 2 Ax 1 Generally NP-hard! Tractable for many interesting A Ax 2 7
Specialize A Many sufficient conditions proposed Nullspace property, Incoherence, Restricted Eigenvalue, Restricted Isometry Property 1 δ k x 2 2 Ax 2 2 1 + δ k x 2 2, x: x 0 k x 1 d Ax 1 1 ± δd x 2 Ax 2 8
Randomness comes to rescue Random matrices can exhibit RIP: Random matrices with iid entries Gaussian, Rademacher (symmetric Bernoulli), Uniform Structured random matrices Random partial Fourier matrices Random circulant matrices RIP holds with high probability if m k log γ n 9
Basis Pursuit (l 1 -minimization) argmin x 1 x subject to Ax = y x 1 = i x i Theorem [Candès] If A satisfies δ 2k -RIP with δ 2k < 2 1, then BP recovers any k-sparse target exactly. 10
Value Basis Pursuit (l 1 -minimization) 1.5 Spar se Signal ` 1 Reconst r uct ion 1 CVX 0.5 0-0.5-1 -1.5-2 0 100 200 300 400 500 600 700 800 900 1000 I ndex 11
Greedy algorithms OMP, StOMP, IHT, CoSAMP, SP, Iteratively estimate the support and the values on the support Many of them have RIP-based convergence guarantees CoSaMP (Needell and Tropp) Proxy vector z = A T (Ax y) supp x Z = supp z 2k 2 b = argmin x Ax y 2 s. t. x T c= 0 Converges with δ 4k 0.1 x z b 1 4 5 3 2 Z T b k Update 12
Rice single-pixel camera Wakin et al oroginal object dim. reduction @ 20% dim. reduction @ 40% 13
Compressive MRI Traditional MRI Lustig et al CS MRI Acquisition Speed 14
Image super-resolution Yang et al D h α Sparse representation α D l α CS-based reconstruction Original 15
Low-Rank Matrix Recovery
Linear systems w/ low-rank solutions r y i = X = n 2 n U V 1 r, A i Simple least squares requires n 1 n 2 measurements Degrees of freedom is r n 1 + n 2 1 Can we close the gap? 16
Rank minimization argmin rank X X subject to A X = y If identifiable, it recovers the low-rank solution exactly Just like l 0 -minimization, it is generally NP-hard Special measurement operators A admit efficient solvers 17
Random measurements A i with iid entries exhibit low-rank RIP Gaussian Rademacher Universal Uniform Some structured matrices Rank-one A i = a i b i A i with dependent entries Instance optimal Standard RIP usually doesn t hold, but other approaches exist 18
Nuclear-norm minimization argmin X X subject to A X = y A X = y X = i σ i X Theorem [Recht, Fazel, Parrilo] If A obeys δ 5r -RIP with δ 5r < 0.1 then nuclear-norm minimization recovers any rank-r target exactly. 19
Matrix completion Alice 4 5? 3?? 1 Bob? 2? 4 5??... Yvon 5? 4 3 2 2? Zelda 3 2?? 5 2 2 Theorem [Candès and Recht] If M is a rank-r matrix from the random orthogonal model, then w.h.p. the nuclear-norm minimization recovers M exactly from O(rn 5 4 log n) uniformly observed entries, where n = n 1 n 2. 20
Robust PCA Ordinary PCA is sensitive to noise and outliers low-rank component Background Modeling M = L + S argmin L,S L + λ S 1 subject to P Ωobs L + S = Y sparse component Candès et al 21
Compressive multiplexing Ahmed and Romberg B x m t = α m ω e j2πωt ω= B Ω~R M + W log 3 MW W = 2B + 1 22
Nonlinear CS
Sparsity-constrained minimization Squared error isn t always appropriate example : non-gaussian noise models argmin x f x subject to x 0 k It s challenging in its general form Objectives with certain properties allow approximations convex relaxation: l 1 -regularization greedy methods 23
Gene classification a i : microarray sample y i : binary sample label (healthy/diseased) model: p y a; x DNA Microarray Sparse Logistic Regression m x : sparse weights for genes argmin log 1 + e a Tx i y i a T i x x i=1 subject to x 0 k, x 2 1 24
Imaging under photon noise Photon noise follows a Poisson distribution y Ax i p y A; x = i y i! i Physical constraints A 0, Ax 0, x 0 SPIRAL-TAP argmin x 0 e Ax i log p y A; x + τr(x) Harmany et al x 1 W T x 1 x TV 25
Coherent diffractive imaging Szameit et al * I : diffracted field intensity Cx : real image * C : Shifts of generating function L : low-pass filter F : Fourier transform f x = LFCx 2 I 2 2 * Lima et al., ESRF (www.esrf.eu) 26
Gradient Support Pursuit w/ Raj and Boufounos Generalizes CoSaMP x 1 z 4 b 5 b k Proxy vector z = f x Z = supp z 2k b = argmin x f x s. t. x T c= 0 supp x 2 3 Z T Converges under SRH the Hessian 2 f x is well-conditioned when restricted to sparse subspaces a generalization of the RIP Update 27
There s much more 28
Resources a collection of CS papers, tutorials, and softwares http://dsp.rice.edu/cs sparse and low-rank algorithms wiki http://ugcs.caltech.edu/~srbecker/wiki/main_page a weblog focusing on CS and broader computational areas http://nuit-blanche.blogspot.com/ 29