Learning-based compression

Transcription

1 Nature Learning-based compression Value / Knowledge Volkan Cevher Laboratory for Information and Inference Systems

2 A paradigm shift in data generation hours

3 A paradigm shift in data generation hours

4 Key tool: Compression

5 A familiar example 12MPix & 24bits/pixel = 36MB Power: OK Storage: NO

6 A familiar example 12MPix & 24bits/pixel = 36MB Power: OK Storage: NO 1000 images + no apps

7 A familiar example 12MPix & 24bits/pixel = 36MB Power: OK Storage: OK Compression actual: 1.4MB images + no apps (vs 1000 images)

8 A familiar example Bandwidth: OK 12MPix & 24bits/pixel = 36MB Power: OK Storage: OK Compression actual: 1.4MB images + no apps (vs 1000 images)

9 Compression helps! Bandwidth: OK 12MPix & 24bits/pixel = 36MB Power: OK Storage: OK Compression actual: 1.4MB images + no apps (vs 1000 images)

10 Compression: The basics p x \ y \ y \ = x \

11 Compression: The basics = n n p p x \ y \ y \ = x \ JPEG2000: Wavelets sparsity

12 Compression: The basics = n n p p x \ y \ Add decades & math + eng JPEG2000: Wavelets y \ = x \ sparsity

13 Compression: The basics = n n p p x \ y \ Add decades & math + eng JPEG2000: Wavelets y \ = x \ Strategy: Encode b = P x \ sparsity Decode ˆx = P : Subset selector P b

14 Compression: The basics = n n p p x \ y \ Add decades & math + eng JPEG: DCT y \ = x \ Strategy: Encode b = P x \ sparsity Decode ˆx = P : Subset selector P b

15 The core challenge: Can we automatically teach any sensor how to compress its own data well?

16 Compression helps! Bandwidth: OK 12MPix & 24bits/pixel = 36MB Power: OK Storage: OK Compression actual: 1.4MB Caveats for generalization: Collected the full data & Performed a full transformation!

17 The core challenge: Can we automatically teach any sensor how to compress its own data well? Our twist: Compress without transforming or sampling the whole data!

18

19 (Old) Compressive sensing (CS) Goal: Directly obtain the compressed version Off-load the difficulty to computation - encoding model: b = P Fx \ & x \ is s sparse in - decoding algorithm: convex optimization ˆx = arg min x {k xk 1 : b = P Fx} - Theorem: If s (log p) & is su ciently random then ˆx = x \ with hp [Candes et al 2006; ]

20 Challenges to the old CS High computational cost & latency: O(n 2 p 1.5 ) Oversampling: p vs s vs s(log p) Dictionary : hidden need for training data

21 ``When solving a given problem, try to avoid a more general problem as an intermediate step.'' Vladimir Vapnik [main developer of statistical learning theory (along with Alexey Chervonenkis)] Given training data, we will bypass dictionary learning & design the whole compressive sampling system directly

22 Statistical Learning Theory meets Compressive Sensing Learning data triage (simplified)

23 A statistical learning framework for CS with sample signals I Probabilistic model: y = P Fx I x follows some unknown probability distribution P. I Sample signals: {xi } iæm, i.i.d. random vectors following P I Fix an estimator: ˆx = F H P T y =(P F) y I Loss function: L(x ; )= Έx x Î 2 2 Îx Î 2 2 I Goal: Fix = n. Findasub-samplingpattern, given {x i } iæm, such that the risk E L(x ; ) is minimized.

24 A statistical learning framework for CS with sample signals I Probabilistic model: y = P Fx I x follows some unknown probability distribution P. I Sample signals: {xi } iæm, i.i.d. random vectors following P I Fix an estimator: ˆx = F H P T y =(P F) y simplification is here I Loss function: L(x ; )= Έx x Î 2 2 Îx Î 2 2 I Goal: Fix = n. Findasub-samplingpattern, given {x i } iæm, such that the risk E L(x ; ) is minimized. simplification is here

25 Empirical risk minimization-i If P were known, the optimal optimization problem: is given by solving the discrete opt œ arg min : Æn E L(x ; ) Proposition We have L(x ; )=1 ÎP Fx Î 2 2 Îx Î 2 2 =: 1 (x ; ). Therefore, we can write opt œ arg max : Æn E (x ; ), and we have E L(x ; opt) = min : Æn E L(x ; )=1 E ÎP opt Fx Î 2 2 Îx Î 2 2 =: 1 Á P.

26 Empirical risk minimization-ii While P is unknown, we have i.i.d. samples {x i } iæn from P. Hence we may consider the empirical risk minimizer given by: ˆ œ arg max : Æn 1 m ÿ iæm (x i; ). Since in general ˆ, opt, we can only expect that E L(x ; ˆ) Æ E L(x ; opt)+á m =1 Á P + Á m.

27 Statistical analysis Recall that E L(x ; ˆ) Æ E L(x ; opt)+á m =1 Á P + Á m. Theorem For any œ (0, 1), wehave Û 2 Á m Æ m 5 log 3 4 p n + log , with probability at least 1. Corollary Number of sample signals required is of O(n log p).

28 Solving the discrete optimization problem Define x i = x i /Îx i Î 2.Recallthat ˆ œ= arg max : Æn ÿ iæm ÎP Fx i Î 2 2 Îx i Î 2 2 = arg max : Æn ÿ iæm ÎP F x iî 2 2. Proposition (Existence of a simple greedy algorithm) Let i be the i-th row of F. We can compute ˆ exactly by the following greedy algorithm. 1. For all i Æ p, compute v i = q jæm È i, x j Í Let be the set of indices of the n largest v i s. I If F is the Fourier transform, then the computational complexity is O(mp log p), nearly linear time.

29 Applications

30 Images - I Old CS vs Learning based CS vs JPEG

31 Images - II 1Gpix at 1MPix rate! Opens up the possibility of streaming video at 30FPS

32 Wireless neural implants - I µ-electrode input AFE ADC DSP TX f s 5KHz ~.3μW/accumulator 1nJ/bit > 30dB quality Stream out Full comp. LBCS AFE + ADC 10μW 10μW 10μW DSP 0 80μW ~2.5μW TX 50μW ~2.5μW ~3μW

33 Wireless neural implants - I µ-electrode input AFE ADC DSP TX f s 5KHz ~.3μW/accumulator 1nJ/bit > 30dB quality Stream out Full comp. LBCS AFE + ADC 10μW 10μW 10μW DSP 0 80μW ~2.5μW TX 50μW ~2.5μW ~3μW

34 Wireless neural implants - II µ-electrode input AFE ADC DSP TX f s I Dataset: billion samples length from ieeg.org LBCS: Indices are learned for Hadamard basis on the training data SNR comparison in [db], for N=256 and B i = 10 Old CS Method Compression rate LBCS SHS BERN MCS n.a. n.a. SHS: Structured Hadamard Sampling [Baldassarre, 15] BERN: Random Bernoulli [Chen, JSSC 12] MCS: Multi-Channel Sampling [Shoaran, TBioCAS 15]

35 Wireless neural implants - II µ-electrode input AFE ADC DSP TX f s

36 other trade-offs are possible! Wireless neural implants - II µ-electrode input AFE ADC DSP TX f s 5KHz ~.3μW/accumulator 1nJ/bit > 30dB quality Stream out Full comp. LBCS AFE + ADC 10μW 10μW 10μW DSP 0 80μW ~2.5μW TX 50μW ~2.5μW ~3μW

37 Wireless neural implants - III µ-electrode input AFE ADC DSP TX f s Actual circuit:

38 Magnetic Resonance Imaging MRI of the Brain minute scan time. MRI of the Orbits minute scan time. MRI of the TMJ minute scan time. MRI of the Soft Tissue Neck minute scan time. MRI of the Cervical Spine minute scan time. MRI of the Upper Extremity minute scan time. MRI of the Thoracic Spine minute scan time. MRI of the Chest minute scan time. MRI of the Abdomen minute scan time.

39 MRI - multi coil (4xaccel)

40 VD MRI - multi coil (4xaccel)

41 MRI - multi coil (4xaccel) VD LBCS Decoder: BP with shearlets

42 MRI - multi coil (4xaccel) 6dB improvement on the average

43 MRI - multi coil (4xaccel) LBCS Patient #22 VD 40.80dB 35.08dB

44 MRI - multi coil (4xaccel) LBCS Patient #29 VD 41.39dB 34.80dB

45 Learning-based CS Middle-out compression unleashed with machine learning