Compressed Sensing and Related Learning Problems

Transcription

1 Compressed Sensing and Related Learning Problems Yingzhen Li Dept. of Mathematics, Sun Yat-sen University Advisor: Prof. Haizhang Zhang Advisor: Prof. Haizhang Zhang 1 /

2 Overview Overview Background Compressed Sensing Norms Indicating Sparsity Restricted Isometry Property (RIP) Recovery Algorithms Applications Learning Methods Probabilistic Methods Dictionary Learning Conclusion Please refer to my thesis for details. Advisor: Prof. Haizhang Zhang 2 /

3 Background Background Signal processing everywhere. Entertainment: music, videos, images... Engineering: telecommunication, medical use... Recognition: speech, face, moving object... Limitation of the Nyquist-Shannon Sampling Theorem Era of the big data. Daily life accompanied with data. Knowledge discovery from data. Any fast algorithms? Advisor: Prof. Haizhang Zhang 3 /

4 Why use l 1 -Norm? Compressed Sensing Norms Indicating Sparsity (a) p = 2 (b) p = 1 (c) p = 0 Recovery of the signal f R n from y R m : f = arg min f f p s.t. y = Φf. (1) (P 0 ) f = arg min f 0 s.t. y = Φf (Matching Pursuit) f (P 1 ) f = arg min f 1 s.t. y = Φf (Basis Pursuit) f Advisor: Prof. Haizhang Zhang 4 /

5 Compressed Sensing Restricted Isometry Property Restricted Isometry Property (RIP) Definition (Restricted Isometry Property [CT05]) The sensing matrix Φ is said to obey the restricted isometry property of order S if δ S s.t. k-sparse f such that k S, and f s support T {1, 2,..., n}( T = k), (1 δ S ) f 2 2 Φ T f 2 2 (1 + δ S ) f 2 2. (2) Advisor: Prof. Haizhang Zhang 5 /

6 Compressed Sensing Restricted Isometry Property (RIP) Sampling Constraints Theorem (Equivalence of problem (P 0 ) and (P 1 )) Suppose S 1 and δ 2S < 1, the solutions of (P 0 ) and (P 1 ) coincides if that solution f has its support T satisfying T S. Theorem (Noiseless incoherent sampling [CT06]) If f is k-sparse, then for any β > 0, with probability at least 1 5/n e β the signal can be perfectly recovered if m O(k log n) Random sensing matrix works better. Advisor: Prof. Haizhang Zhang 6 /

7 Compressed Sensing Recovery Algorithms Simulations: solving BP by LP (d) recovery (p = 1) (e) recovery (p = 2) (f) error (p = 1) (g) error (p = 2) Figure: Recovery compared to the raw signal. Advisor: Prof. Haizhang Zhang 7 /

8 Compressed Sensing Recovery Algorithms Noisy Recovery (a) noiseless (b) noisy In practise y = Φf + σz. LASSO: l 2 -minimization with l 1 -penalty. f 1 = arg min f 2 Φf y λσ m f 1 (3) Similar (weak) RIP guarantees the recovery. Advisor: Prof. Haizhang Zhang 8 /

9 Single-Pixel Camera Compressed Sensing Applications Figure: Compressed sensing v.s. wavelet decomposition [TLW + 06]. Advisor: Prof. Haizhang Zhang 9 /

10 CS-MRI Compressed Sensing Applications Figure: Applying CS techniques to MRI [LDSP08]. Advisor: Prof. Haizhang Zhang 10 /

11 MAP Learning Methods Probabilistic Methods f = arg max(log P(y f ; Φ) + log P(f )). (4) f y Φf = z N (0, σ 2 ) Prior P(f ) = 1 p e λp f Z f = arg min y Φf λ f p f f = arg min f p s.t. y Φf 2 ɛ f (5) p = 1: standard LASSO ɛ 0: perfect recovery. Advisor: Prof. Haizhang Zhang 11 /

12 Learning Methods Dictionary Learning Dictionary Learning Finding the dictionary minimizing the error of representation: X is supposed to be sparse. K-SVD [AEB06]. Expectation-Maximization. Ψ = arg min Ψ Y ΦΨX 2 F (6) Advisor: Prof. Haizhang Zhang 12 /

13 Maximum Likelihood Learning Methods Dictionary Learning Signal can be represented by some fixed codes from an over-complete dictionary: f = Ψx + v, v N (0, σ 2 ) (7) Best dictionary gives sparse representations x 1 ɛ: Ψ = arg max e 1 2σ 2 f Ψx 2 2 λ 1 x 1 (8) Ψ Trick: fast online methods. f x 1 ɛ Advisor: Prof. Haizhang Zhang 13 /

14 Conclusion Conclusion Revolutionary sensing theories without Nyquist rate constraints. Benefited from data sparsity. The RIP helps yield perfect recovery with high probability. Random sensing matrix works better. Learning algorithms benefits CS methods. Advisor: Prof. Haizhang Zhang 14 /

15 Conclusion Future Research Can the algorithms keep high performance when n grows? Can P(Φ) be learned? Can we loose the restriction of (weak) RIP? Any other learning approaches? Advisor: Prof. Haizhang Zhang 15 /

16 Acknowledgements Acknowledgements Thesis supervisor Prof. Haizhang Zhang. Prof. Guocan Feng, Dr. Lei Zhang and Mr. Zhihong Huang. Professors who have taught me or given me advices. My friends, my family, and myself. Advisor: Prof. Haizhang Zhang /

17 [AEB06] [Can08] References Michal Aharon, Michael Elad, and Alfred Bruckstein. K -svd : An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing, 54(11): , E. J. Candès. The restricted isometry property and its implications for compressed sensing. Comptes Rendus Mathematique, pages , [CDMS98] Scott Shaobing Chen, David L. Donoho, Michael, and A. Saunders. Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 20:33 61, [CT05] E.J. Candès and T. Tao. Decoding by linear programming. Advisor: Prof. Haizhang Zhang /

18 [CT06] [DH99] References IEEE Transactions on Information Theory, pages , E.J. Candès and T. Tao. Near-optimal signal recovery from random projections: Universal encoding strategies? IEEE Transactions on Information Theory, 52(12): , David L. Donoho and Xiaoming Huo. Uncertainty principles and ideal atomic decomposition, [LDSP08] M. Lustig, D.L. Donoho, J.M. Santos, and J.M. Pauly. Compressed sensing mri. Signal Processing Magazine, IEEE, 25(2):72 82, Advisor: Prof. Haizhang Zhang /

19 References [TLW + 06] Dharmpal Takhar, Jason N. Laska, Michael B. Wakin, Marco F. Duarte, Dror Baron, Shriram Sarvotham, Kevin F. Kelly, and Richard G. Baraniuk. A new compressive imaging camera architecture using optical-domain compression. In Proceedings of Computational Imaging IV at SPIE Electronic Imaging, pages 43 52, [Wei] Fred Weinhaus. Fourier transforms. Advisor: Prof. Haizhang Zhang /