An Adaptive Sublinear Time Block Sparse Fourier Transform

Transcription

1 An Adaptive Sublinear Time Block Sparse Fourier Transform Volkan Cevher Michael Kapralov Jonathan Scarlett Amir Zandieh EPFL February 8th 217

2 Given x C N, compute the Discrete Fourier Transform (DFT) of x: x i = 1 x N j ω ij, j [N] where ω = e 2πi/N is the N-th root of unity.

3 Given x C N, compute the Discrete Fourier Transform (DFT) of x: x i = 1 x N j ω ij, j [N] where ω = e 2πi/N is the N-th root of unity. Assume that N is a power of 2.

4 Given x C N, compute the Discrete Fourier Transform (DFT) of x: x i = 1 x N j ω ij, j [N] where ω = e 2πi/N is the N-th root of unity. Assume that N is a power of 2. compression schemes (JPEG, MPEG) signal processing data analysis imaging (MRI, NMR)

5 Fast Fourier Transform (FFT) Computes Discrete Fourier Transform (DFT) of a length N signal in O(N logn) time

6 Fast Fourier Transform (FFT) Computes Discrete Fourier Transform (DFT) of a length N signal in O(N logn) time Cooley and Tukey, 1964

7 Fast Fourier Transform (FFT) Computes Discrete Fourier Transform (DFT) of a length N signal in O(N logn) time Cooley and Tukey, 1964 Gauss, 185

8 Sparse FFT Say that x is k-sparse if x has k nonzero entries time frequency

9 Sparse FFT Say that x is k-sparse if x has k nonzero entries Say that x is approximately k-sparse if x is close to k-sparse in some norm time frequency

10 Sparse approximations JPEG = Image and video compression schemes (e.g. JPEG, MPEG) Compute top k coefficients of x faster than FFT?

11 Sample complexity Sample complexity=number of samples accessed in time domain.

12 Sample complexity Sample complexity=number of samples accessed in time domain. In medical imaging (MRI, NMR), one measures Fourier coefficients x of imaged object x (which is often sparse)

13 Given access to signal x in time domain, (approximately) compute top k coefficients of x Minimize runtime and sample complexity

14 Given access to signal x in time domain, (approximately) compute top k coefficients of x Minimize runtime and sample complexity Formally, want to find ŷ such that x ŷ 2 (1 + ε) min k sparse ẑ x ẑ 2 (l 2 /l 2 sparse recovery guarantees)

15 Uniform bounds (for all): Candes-Tao 6 Rudelson-Vershynin 8 Cheraghchi-Guruswami-Velingker 12 Bourgain 14 Haviv-Regev 15 Deterministic, Ω(N) runtime O(k log 2 k logn) Non-uniform bounds (for each): Goldreich-Levin 89 Kushilevitz-Mansour 91, Mansour 92 Gilbert-Guha-Indyk-Muthukrishnan-Strauss 2 Gilbert-Muthukrishnan-Strauss 5 Hassanieh-Indyk-Katabi-Price 12a Hassanieh-Indyk-Katabi-Price 12b Indyk-K.-Price 14(k logn(loglogn) C, but only in 1d) Indyk-K. 14 K 16 K?? Randomized, O(k poly(logn)) runtime O(k logn) Lower bound: k log(n/k) for non-adaptive algorithms Do-Ba-Indyk-Price-Woodruff 1 (Also Boufounos-Cevher-Gilbert-Li-Strauss 12, Prince-Song 14 on continuous Sparse FFT)

16 Uniform bounds (for all): Candes-Tao 6 Rudelson-Vershynin 8 Cheraghchi-Guruswami-Velingker 12 Bourgain 14 Haviv-Regev 15 Deterministic, Ω(N) runtime O(k log 2 k logn) Non-uniform bounds (for each): Goldreich-Levin 89 Kushilevitz-Mansour 91, Mansour 92 Gilbert-Guha-Indyk-Muthukrishnan-Strauss 2 Gilbert-Muthukrishnan-Strauss 5 Hassanieh-Indyk-Katabi-Price 12a Hassanieh-Indyk-Katabi-Price 12b Indyk-K.-Price 14(k logn(loglogn) C, but only in 1d) Indyk-K. 14 K 16 K?? Randomized, O(k poly(logn)) runtime O(k logn) Lower bound: k log(n/k) for non-adaptive algorithms Do-Ba-Indyk-Price-Woodruff 1 (Also Boufounos-Cevher-Gilbert-Li-Strauss 12, Prince-Song 14 on continuous Sparse FFT) Nearly optimal results under sparsity assumption alone can we exploit structure beyond sparsity?

17 Sparse FFT beyond the sparsity assumption Signals arising in applications often have structure beyond sparsity Can we exploit further structure to reduce sample complexity and runtime?

18 Block sparsity in Fourier domain 1 k blocks of width k 1.8 magnitude }{{} width k frequency This work: suppose that dominant frequencies are contained in k clusters (intervals) of length k 1 each? O(k logn +k k 1 ) sample complexity in sublinear time?

19 Model based compressed sensing Framework for exploiting structured sparsity to reduce sample complexity, introduced by Baraniuk, Cevher, Duarte and Hegde 1 Baraniuk et al 1: optimal O(k logn +k k 1 ) sample complexity using Gaussian measurements in Õ(k k 1 n) time 1 k blocks of width k 1.8 magnitude }{{} width k frequency

20 Adaptivity in compressed sensing An algorithm is adaptive its signal access pattern is guided by samples taken All Sparse FFT and compressed sensing results mentioned so far are non-adaptive

21 Main result Theorem There exists an adaptive (k,k 1 )-block sparse Fourier transform with sample complexity O (k log(1 +k )logn +k k 1 ) logsnr samples and O (k k 1 log 3 n) logsnr runtime. Sample complexity optimal up to lower order terms for constant SNR

22 head tail SNR=ratio of total signal energy to total noise energy

23 Main result Theorem There exists an adaptive (k,k 1 )-block sparse Fourier transform with sample complexity O (k log(1 +k )logn +k k 1 ) logsnr samples and O (k k 1 log 3 n) logsnr runtime. Sample complexity optimal up to lower order terms for constant SNR

24 Main result Theorem There exists an adaptive (k,k 1 )-block sparse Fourier transform with sample complexity O (k log(1 +k )logn +k k 1 ) logsnr samples and O (k k 1 log 3 n) logsnr runtime. Sample complexity optimal up to lower order terms for constant SNR Crucially use adaptivity!

25 Main result Theorem There exists an adaptive (k,k 1 )-block sparse Fourier transform with sample complexity O (k log(1 +k )logn +k k 1 ) logsnr samples and O (k k 1 log 3 n) logsnr runtime. Sample complexity optimal up to lower order terms for constant SNR Crucially use adaptivity! Theorem Any non-adaptive Sparse FFT algorithm must take Ω(k k 1 log n k k 1 ) samples.

26 Main result Theorem There exists an adaptive (k,k 1 )-block sparse Fourier transform with sample complexity O (k log(1 +k )logn +k k 1 ) logsnr samples and O (k k 1 log 3 n) logsnr runtime. Sample complexity optimal up to lower order terms for constant SNR Crucially use adaptivity! Theorem Any non-adaptive Sparse FFT algorithm must take Ω(k k 1 log n k k 1 ) samples. No improvement upon vanilla sparsity possible with non-adaptive Fourier measurements

27 First result along the following directions: First sublinear time model based compressed sensing primitive Separation between adaptive vs non-adaptive Sparse FFT Structured (Fourier) vs unstructured (Gaussian) measurements in model based compressed sensing

28 1. Sparse recovery from Fourier measurements 2. Main result

29 1. Sparse recovery from Fourier measurements 2. Main result

30 Structure of Sparse FFT algorithms Input: access to signal x C N in time domain Output: top k coefficients of x, approximately

31 Structure of Sparse FFT algorithms Input: access to signal x C N in time domain Output: top k coefficients of x, approximately Iterative process: recover dominant coefficients of input signal

32 Structure of Sparse FFT algorithms Input: access to signal x C N in time domain Output: top k coefficients of x, approximately Iterative process: recover dominant coefficients of input signal subtract recovered coefficients (e.g. in time domain)

33 Structure of Sparse FFT algorithms Input: access to signal x C N in time domain Output: top k coefficients of x, approximately Iterative process: recover dominant coefficients of input signal subtract recovered coefficients (e.g. in time domain) repeat

34 Structure of Sparse FFT algorithms Input: access to signal x C N in time domain Output: top k coefficients of x, approximately Iterative process: recover dominant coefficients of input signal subtract recovered coefficients (e.g. in time domain) repeat Summary of techniques from Gilbert-Guha-Indyk-Muthukrishnan-Strauss 2, Akavia-Goldwasser-Safra 3, Gilbert-Muthukrishnan-Strauss 5, Iwen 1, Akavia 1, Hassanieh-Indyk-Katabi-Price 12a, Hassanieh-Indyk-Katabi-Price 12b

35 1-sparse recovery from Fourier measurements magnitude.5 magnitude time frequency x a = ω a f +noise 2πf /n O(logn) measurements for random a

36 Reducing k-sparse recovery to 1-sparse recovery Permute with a random linear transformation and phase shift magnitude.5.5 magnitude time frequency

37 Reducing k-sparse recovery to 1-sparse recovery Permute with a random linear transformation and phase shift magnitude.5.5 magnitude time frequency

38 Reducing k-sparse recovery to 1-sparse recovery Permute with a random linear transformation and phase shift magnitude.5.5 magnitude time frequency

39 Reducing k-sparse recovery to 1-sparse recovery Permute with a random linear transformation and phase shift magnitude.5.5 magnitude time frequency Choose a filter G,Ĝ such that Ĝ rectangle G has support k Compute x Ĝ = (x G) Sample complexity=supp G!

40 Reducing k-sparse recovery to 1-sparse recovery Permute with a random linear transformation and phase shift magnitude.5.5 magnitude time frequency Choose a filter G,Ĝ such that Ĝ rectangle G has support k Compute x Ĝ = (x G) Sample complexity=supp G!

41 1. Sparse recovery from Fourier measurements 2. Main result

42 Block sparsity in Fourier domain 1 k blocks of width k 1.8 magnitude }{{} width k frequency This work: suppose that dominant frequencies are contained in k clusters (intervals) of length k 1 each?

43 Block sparsity in Fourier domain 1 k blocks of width k 1.8 magnitude }{{} width k frequency This work: suppose that dominant frequencies are contained in k clusters (intervals) of length k 1 each? Standard techniques destroy structure (by a random permutation), and need Ω(k k 1 logn) samples

44 Block sparsity in Fourier domain 1 k blocks of width k 1.8 magnitude }{{} width k frequency This work: suppose that dominant frequencies are contained in k clusters (intervals) of length k 1 each? Standard techniques destroy structure (by a random permutation), and need Ω(k k 1 logn) samples O(k logn +k k 1 ) sample complexity in sublinear time?

45 Main result Theorem There exists an adaptive (k,k 1 )-block sparse Fourier transform with sample complexity O (k log(1 +k )logn +k k 1 ) logsnr samples and O (k k 1 log 3 n) logsnr runtime. Sample complexity optimal up to lower order terms for constant SNR

46 Main result Theorem There exists an adaptive (k,k 1 )-block sparse Fourier transform with sample complexity O (k log(1 +k )logn +k k 1 ) logsnr samples and O (k k 1 log 3 n) logsnr runtime. Sample complexity optimal up to lower order terms for constant SNR Crucially use adaptivity!

47 Spectrum is well approximated by k blocks of length k 1 : 1 k blocks of width k 1.8 magnitude k = 3 in the example }{{} width k frequency Natural idea: can we treat blocks as individual frequencies, and thus reduce to standard k -sparse recovery? YES, to some extent...

48 Spectrum is well approximated by k blocks of length k 1 : 1.8 magnitude k = 3 in the example frequency Natural idea: can we treat blocks as individual frequencies, and thus reduce to standard k -sparse recovery? YES, to some extent...

49 Spectrum is well approximated by k blocks of length k 1 : 1.8 magnitude k = 3 in the example frequency Natural idea: can we treat blocks as individual frequencies, and thus reduce to standard k -sparse recovery? YES, to some extent...

50 Spectrum is well approximated by k blocks of length k 1 : 1 k -sparse reduced signal.8 magnitude k = 3 in the example frequency Natural idea: can we treat blocks as individual frequencies, and thus reduce to standard k -sparse recovery? YES, to some extent...

51 Reduced signals 1.8 magnitude frequency For each j = 1,...,n/k 1 define reduced signal Z by Ẑ j := ( X Ĝ) j k 1 (G has small support, and Ĝ indicator of a block)

52 Reduced signals 1.8 magnitude frequency For each j = 1,...,n/k 1 define reduced signal Z by Ẑ j := ( X Ĝ) j k 1 (G has small support, and Ĝ indicator of a block)

53 Reduced signals filter Ĝ 1.8 magnitude frequency For each j = 1,...,n/k 1 define reduced signal Z by Ẑ j := ( X Ĝ) j k 1 (G has small support, and Ĝ indicator of a block)

54 Reduced signals filter Ĝ 1.8 magnitude frequency For each j = 1,...,n/k 1 define reduced signal Z by Ẑ j := ( X Ĝ) j k 1 (G has small support, and Ĝ indicator of a block)

55 Reduced signals filter Ĝ 1.8 magnitude frequency For each j = 1,...,n/k 1 define reduced signal Z by Ẑ j := ( X Ĝ) j k 1 Good news: if X is approximately (k,k 1 )-block sparse, then Z is approximately O(k )-sparse.

56 Reduced signals filter Ĝ 1.8 magnitude frequency For each j = 1,...,n/k 1 define reduced signal Z by Ẑ j := ( X Ĝ) j k 1 Major problem: some blocks of X might have essentially zero contribution to Z due to cancellations!

57 Energy concentration of 1-block sparse signals In standard Sparse FFT k 1 = 1, so energy of a block is uniformly spread over time.

58 Energy concentration of 1-block sparse signals In standard Sparse FFT k 1 = 1, so energy of a block is uniformly spread over time time frequency

59 Energy concentration of 1-block sparse signals In standard Sparse FFT k 1 = 1, so energy of a block is uniformly spread over time time frequency

60 Energy concentration of 1-block sparse signals In standard Sparse FFT k 1 = 1, so energy of a block is uniformly spread over time. If X is 1-block sparse with block size k 1, its energy could be entirely concentrated on only a 1/k 1 fraction of the time domain! time frequency

61 Energy concentration of 1-block sparse signals In standard Sparse FFT k 1 = 1, so energy of a block is uniformly spread over time. If X is 1-block sparse with block size k 1, its energy could be entirely concentrated on only a 1/k 1 fraction of the time domain! time frequency

62 Energy concentration of 1-block sparse signals In standard Sparse FFT k 1 = 1, so energy of a block is uniformly spread over time. If X is 1-block sparse with block size k 1, its energy could be entirely concentrated on only a 1/k 1 fraction of the time domain! time frequency

63 Energy concentration of 1-block sparse signals In standard Sparse FFT k 1 = 1, so energy of a block is uniformly spread over time. If X is 1-block sparse with block size k 1, its energy could be entirely concentrated on only a 1/k 1 fraction of the time domain! time frequency

64 Energy concentration of 1-block sparse signals In standard Sparse FFT k 1 = 1, so energy of a block is uniformly spread over time. If X is 1-block sparse with block size k 1, its energy could be entirely concentrated on only a 1/k 1 fraction of the time domain! time frequency

65 Energy concentration of 1-block sparse signals In standard Sparse FFT k 1 = 1, so energy of a block is uniformly spread over time. If X is 1-block sparse with block size k 1, its energy could be entirely concentrated on only a 1/k 1 fraction of the time domain! time frequency

66 Recovery of block sparse signals For each r = 1,...,2k 1 define reduced signal Z r by ( ( Ẑj r ) ) n := X r Ĝ 2k 1 (Shift X first, use 2k 1 regular shifts) j k 1

67 Recovery of block sparse signals For each r = 1,...,2k 1 define reduced signal Z r by ( ( Ẑj r ) ) n := X r Ĝ 2k 1 (Shift X first, use 2k 1 regular shifts) If each reduction requires k logn samples, we are back to the k k 1 logn bound? j k 1

68 Recovery of block sparse signals For each r = 1,...,2k 1 define reduced signal Z r by ( ( Ẑj r ) ) n := X r Ĝ 2k 1 (Shift X first, use 2k 1 regular shifts) If each reduction requires k logn samples, we are back to the k k 1 logn bound? j k 1 An adaptive solution can first probe which signals carry most energy, then allocate hashing budgets carefully

69 Importance sampling ALLOCATEBUDGETS(X,k,k 1 ) For t = 1,...,O(k ) Sample r Z r 2 2, q Geom Set s t (r,2 q ) End For O(k k 1 ) samples

70 Importance sampling ALLOCATEBUDGETS(X,k,k 1 ) For t = 1,...,O(k ) Sample r Z r 2 2, q Geom Set s t (r,2 q ) End For O(k k 1 ) samples BUDGETEDRECOVERY(X,{s t }) For t = 1,...,O(k ) Hash Z r into s t buckets, Recover top s t frequencies End For O(k log(1 +k )logn) samples

71 Importance sampling ALLOCATEBUDGETS(X,k,k 1 ) For t = 1,...,O(k ) Sample r Z r 2 2, q Geom Set s t (r,2 q ) End For O(k k 1 ) samples 8 BUDGETEDRECOVERY(X,{s t }) For t = 1,...,O(k ) Hash Z r into s t buckets, Recover top s t frequencies End For O(k log(1 +k )logn) samples time time

72 Importance sampling ALLOCATEBUDGETS(X,k,k 1 ) For t = 1,...,O(k ) Sample r Z r 2 2, q Geom Set s t (r,2 q ) End For O(k k 1 ) samples 12 BUDGETEDRECOVERY(X,{s t }) For t = 1,...,O(k ) Hash Z r into s t buckets, Recover top s t frequencies End For O(k log(1 +k )logn) samples time time

73 Adaptivity is necessary Algorithm crucially uses adaptivity: sample a few points first to find a non-uniform sampling distribution for second stage Theorem Any non-adaptive Sparse FFT algorithm must take Ω(k k 1 log n k k 1 ) samples.

74 Our results: first Sparse FFT algorithm that exploits structure beyond sparsity adaptivity is crucial

75 Our results: first Sparse FFT algorithm that exploits structure beyond sparsity adaptivity is crucial Other forms of structured sparsity (e.g. tree sparsity, graph sparsity?)

76 Our results: first Sparse FFT algorithm that exploits structure beyond sparsity adaptivity is crucial Other forms of structured sparsity (e.g. tree sparsity, graph sparsity?) A new class of Sparse FFT techniques that adapt to structure of the spectrum?

77 Our results: first Sparse FFT algorithm that exploits structure beyond sparsity adaptivity is crucial Other forms of structured sparsity (e.g. tree sparsity, graph sparsity?) A new class of Sparse FFT techniques that adapt to structure of the spectrum? Experimental evaluation?

78 Our results: first Sparse FFT algorithm that exploits structure beyond sparsity adaptivity is crucial Other forms of structured sparsity (e.g. tree sparsity, graph sparsity?) A new class of Sparse FFT techniques that adapt to structure of the spectrum? Experimental evaluation? Thank you!