Compressive sensing of low-complexity signals: theory, algorithms and extensions

Transcription

1 Compressive sensing of low-complexity signals: theory, algorithms and extensions Laurent Jacques March 7, 9, 1, 14, 16 and 18, 216 9h3-12h3 (incl. 3 ) Graduate School in Systems, Optimization, Control and Networks (SOCN)

2 General content of this doctoral course Sparse signal models: from Fourier to *-lets transforms (wavelets, curvelets, ) Beyond sparsity: grouped sparsity and low-rank priors General overview of compressed sensing theory (aka incoherent sensing) Random isometries and Johnson-Lindenstrauss lemma Signal recovery in compressed sensing: optimization and greedy methods Quantizing compressed sensing and quasi-isometric embeddings Compressed sensing applications 2

3 Main course material S. Mallat, A Wavelet Tour of Signal Processing, 3 rd ed., Academic Press, dec. 28. ISBN S. Foucart, H. Rauhut, A Mathematical Introduction to Compressive Sensing, ISBN plus internet resources: The Numerical Tours of Signal Processing Rice University's Compressive Sensing Resources: The Nuit Blanche blog : (on CS, Machine/Deep Learning, low-rank minimization, ) 3

4 Caveats The course will focus on a discrete formalism (signals, images, videos, are vectors!) Emphasize both on intuition & proofs (when these are short enough ;-) ) Reference books/resources available for further reading 4

5 Part Course overview

6 Generally, sampling is... + Shannon/Nyquist Human Readable Signal 6

7 Generally, sampling is... + Shannon/Nyquist Human Readable Signal Example: Daguerreotype Camera obscura + photochemical recording Boulevard du Temple, Paris, 1839 (wikipedia) 7

8 But, new ways to sample signals!!! Paradigm shift: Computer readable sensing + prior information 8

9 But, new ways to sample signals!!! Paradigm shift: Computer readable sensing + prior information World Sensing Device Signal Sensing Signal Human Optimized blocs! Sampling rate information! 9

10 Prior information? Informative signals are composed of structures Transformée continue en ondelettes sur la sphère 37 avec ˆψ a (l, m) = Yl m ψ a la transformée en harmonique sphérique 12 de ψ a = D a ψ. Une condition plus simple à manipuler et presque équivalente à (2.65) est d imposer que [Van98] ψ(θ, ϕ) dµ(θ, ϕ) =, (2.66) S 1+cosθ 2 condition homologue à l annulation de la moyenne des ondelettes planes. En remarquant que dµ(θ, ϕ) D aψ(θ, ϕ) S 1+cosθ 2 = a dµ(θ, ϕ) S 2 ψ(θ, ϕ) 1+cosθ, (2.67) 3-D data la condition (2.66) permet de créer toute une classe d ondelettes admissibles de la forme pour une certaine fonction φ L 2 (S 2 ). Speech signal ψ(θ, ϕ) = φ(θ, ϕ) 1 α D αφ(θ, ϕ), (2.68) Data on Graph Fig. 2.3 L ondelette DOG pour α =1.25 dilatée de a =.1. Biology En particulier, pour φ =exp ( tan 2 ( 1 2 θ)), c.-à-d. la projection stéréographique inverse de la gaussienne sur la sphère, nous obtenons l ondelette sphérique DOG 13 Spherical data ψ(θ, ϕ) = exp ( tan 2 ( 1 2 θ)) 1 α λ(α, θ) 1 2 exp ( 1 α 2 tan 2 ( 1 2 θ)), (2.69) Astronomy dont une représentation dilatée d un facteur a =.1 est donnée sur la Figure Nommée également transformée de Fourier sur S

11 Origin: sparse models Hypothesis: any informative signal x can be decomposed in a sparsity basis with few non-zero elements : x ' X i i i = = sparse vector can be an ONB (e.g. Fourier, wavelets) or a dictionary (atoms) f atom Non-linear approximation # atoms improved quality 11

12 What are low-complexity models (LC)? Include the case of sparse models + mixed-norm sparsities & model-based + low-rank data models + union of low-dimension subspaces + parametric models + manifolds +. Intuition: model = small domain, low-effective dimension allows, e.g., inverse problem regularization 12

13 Common applications for LC models 1. Data Compression/Transmission: (by definition) 2. Data restoration: e.g.,... Inpainting Deconvolution Wavelets, Curvelets, *-lets, Dictionaries,... (inverse problem solving) Matrix completion 3. Simplified model and interpretation (e.g. in ML) (not covered here) 13

14 4. and Compressed Sensing! 14

15 Compressed Sensing in a nutshell: Generalize Dirac/Nyquist sampling: 1 ) ask few (linear) questions 2.4. Transformée continue en ondelettes sur la sphère 37 avec ˆψ about your informative a (l, m) = Yl m ψ a la transformée en harmonique sphérique 12 de ψ signal a = D a ψ. Une condition plus simple à manipuler et presque équivalente à (2.65) est d imposer que [Van98] ψ(θ, ϕ) dµ(θ, ϕ) S 1+cosθ 2 =, (2.66) 2 ) and recover it condition differently homologue à l annulation de la moyenne des ondelettes (non-linearly) planes. En remarquant que dµ(θ, ϕ) D aψ(θ, ϕ) S 1+cosθ 2 = a dµ(θ, ϕ) S 2 ψ(θ, ϕ) 1+cosθ, (2.67) la condition (2.66) permet de créer toute une classe d ondelettes admissibles de la forme pour une certaine fonction φ L 2 (S 2 ). ψ(θ, ϕ) = φ(θ, ϕ) 1 α D αφ(θ, ϕ), (2.68) Fig. 2.3 L ondelette DOG pour α =1.25 dilatée de a =.1. En particulier, pour φ =exp ( tan 2 ( 1θ)), c.-à-d. la projection stéréographique inverse 2 e.g., sparse, structured, de la gaussienne sur la sphère, nous obtenons l ondelette low-rank, sphérique DOG ψ(θ, ϕ) = exp ( tan 2 ( 1θ)) 1 λ(α, θ) 1 2 exp ( 1 tan 2 ( 1θ)), (2.69) 2 α α 2 2 dont une représentation dilatée d un facteur a =.1 est donnée sur la Figure Nommée également transformée de Fourier sur S Pour Difference of Gaussians. 15

16 Compressed Sensing... M questions y Sensing method Signal x OBSERVATIONS M ' noise SENSOR M N Sparsity Prior ( = Id) N A signal in this discrete world 16

17 Compressed Sensing... M questions y Sensing method Signal x y i ' i ' noise Sparsity Prior ( = Id) M M N Generalized Linear Sensing! y i 'h' i, xi = ' T i x 1 apple i apple M e.g., to be realized optically/analogically N A signal in this discrete world 17

18 Compressed Sensing... M questions Sensing method Signal y x y i ' i ' noise Sparsity Prior ( = Id) M M N But why does it work? Identifiability of x from x? (sparse) N A signal in this discrete world 18

19 Compressed Sensing... Two K-sparse signals x, x 2 K := {u : kuk := supp u 6 K} For many random constructions of (e.g., Gaussian, Bernoulli, structured) and M & K log(n/k), with high probability, Geometry of ( K ) Geometry of K 19

20 Compressed Sensing... Two K-sparse signals x, x 2 K := {u : kuk := supp u 6 K} For many random constructions of (e.g., Gaussian, Bernoulli, structured) and M & K log(n/k), with high probability, Geometry of ( K ) Geometry of K x x, x x K ( K ) R M R N 2

21 Compressed Sensing... Two K-sparse signals x, x 2 K := {u : kuk := supp u 6 K} For many random constructions of and M & K log(n/k), with high probability, Mathematically, Geometry of ( K ) Geometry of K (e.g., Gaussian, Bernoulli, structured) respects the Restricted Isometry Property RIP(K, ) (1 )kuk 2 apple 1 M k uk2 apple (1 + )kuk 2 for all u 2 K and < < 1. embeds the low-dimensional domain K in R M! 21

22 Compressed Sensing... Possible reconstruction: (others exist, e.g., greedy) If 1 p M Basis Pursuit DeNoise [Chen, Donoho, Saunders, 1998] ˆx 2 arg min kuk 1 s.t. ky uk apple u 2 R N Sparsity promotion kuk 1 = P j u j Level of noise y = x + n, knk 6 respects the Restricted Isometry Property (RIP) Then, if < p 2 1 [Candès, 9], (with f. g 9c>:f 6 cg) Robustness: vs sparse deviation + noise. kx ˆxk. 1 p K kx x K k 1 + p M hidden constant e (K): error of the model noise 22

23 The Power of Random Projections At the heart of CS: random projections! as realized by random sensing matrices e.g., Gaussian: 2 R M N,with ij iid N (, 1) But also: random sub-gaussian ensembles (e.g., Bernoulli, bounded); or structured sensing matrices: random Fourier/Hadamard ensembles (e.g., for CT, MRI); random convolutions, spread-spectrum (e.g., for imaging) (see, e.g., [Foucart, Rauhut, 213]) 23

24 Related Concepts? For many random constructions of (e.g., Gaussian, Bernoulli, structured) and M & intrinsic dimension of M, with high probability, Geometry of (M) Geometry of M ULS, manifolds Hilbert spaces x x, x x M Of specific interest beyond CS! (M) R M R N 24

25 Related Concepts? For many random constructions of and M & intrinsic dimension of M, with high probability, (e.g., Gaussian, Bernoulli, structured) Geometry of f( (M)) Geometry of M x x, x x f( x) f( x ), x x Of specific interest... f( ) ULS, manifolds Hilbert spaces M beyond CS! (M) R M R N with f non-linear (e.g., quantification, sign operator) 25

26 Related Concepts? Connection with Quantized Compressed Sensing: Recover/estimate x from sign ( x) 2 { 1, +1}M Q( x) 2 C RM finite codebook CS QCS

27 CS Applications? MANY! 27

28 CS Applications? Proof of concept, 27 Satellite imaging Magnetic Resonance Imaging MANY! Hyperspectral imaging Internet of Thing Radio-interferometry 28

29 General outline of this course: Part I: Low-complexity signal models Part II: Compressed Sensing (CS) Part III: Quantized aspects of CS Part IV: CS applications 29

30 So, let s start! 3