Sample Optimal Fourier Sampling in Any Constant Dimension

Transcription

1 1 / 26 Sample Optimal Fourier Sampling in Any Constant Dimension Piotr Indyk Michael Kapralov MIT IBM Watson October 21, 2014

2 Fourier Transform and Sparsity Discrete Fourier Transform Given x C n, compute x i = x j ω ij, j [n] where ω is the n-th root of unity. Assume n is a power of 2. Fundamental tool: Compression (image, audio, video) Signal processing Data analysis Medical imaging (MRI, NMR) 2 / 26

3 3 / 26 Fourier Transform and Sparsity Sparse Fourier Transform The fast algorithm for DFT is FFT, runs in O(nlogn) time improving on FFT runtime in full generality a major open problem

4 3 / 26 Fourier Transform and Sparsity Sparse Fourier Transform The fast algorithm for DFT is FFT, runs in O(nlogn) time improving on FFT runtime in full generality a major open problem Most interesting signals are sparse (have few nonzero entries) or approximately sparse in the Fourier domain. k-sparse=at most k non-zeros Hassanieh-Indyk-Katabi-Price 12 compute approximate sparse FFT in O(k lognlog(n/k)) time

5 4 / 26 Sample complexity Fourier Transform and Sparsity Sample complexity=number of samples accessed in time domain. In some applications at least as important as runtime Shi-Andronesi-Hassanieh-Ghazi- Katabi-Adalsteinsson ISMRM 13

6 4 / 26 Sample complexity Fourier Transform and Sparsity Sample complexity=number of samples accessed in time domain. In some applications at least as important as runtime Shi-Andronesi-Hassanieh-Ghazi- Katabi-Adalsteinsson ISMRM 13 Given access to x C n, find ŷ such that x ŷ 2 C min k sparse ẑ x ẑ 2 Use smallest possible number of samples?

7 5 / 26 Uniform bounds (for all): Fourier Transform and Sparsity Candes-Tao 06 Rudelson-Vershynin 08 Candes-Plan 10 Cheraghchi-Guruswami-Velingker 12 Non-uniform bounds (for each): Goldreich-Levin 89 Kushilevitz-Mansour 91, Mansour 92 Gilbert-Guha-Indyk-Muthukrishnan-Strauss 02 Gilbert-Muthukrishnan-Strauss 05 Hassanieh-Indyk-Katabi-Price 12a Hassanieh-Indyk-Katabi-Price 12b Indyk-K.-Price 14 Deterministic, Ω(n) runtime O(k log 3 k logn) Randomized, O(k poly(logn)) runtime O(k logn (loglogn) C ) Lower bound: Ω(k log(n/k)) for non-adaptive algorithms Do-Ba-Indyk-Price-Woodruff 10

8 Fourier Transform and Sparsity Uniform bounds (for all): Candes-Tao 06 Rudelson-Vershynin 08 Candes-Plan 10 Cheraghchi-Guruswami-Velingker 12 Non-uniform bounds (for each): Goldreich-Levin 89 Kushilevitz-Mansour 91, Mansour 92 Gilbert-Guha-Indyk-Muthukrishnan-Strauss 02 Gilbert-Muthukrishnan-Strauss 05 Hassanieh-Indyk-Katabi-Price 12a Hassanieh-Indyk-Katabi-Price 12b Indyk-K.-Price 14 Deterministic, Ω(n) runtime O(k log 3 k logn) Randomized, O(k poly(logn)) runtime O(k logn (loglogn) C ) Lower bound: Ω(k log(n/k)) for non-adaptive algorithms Do-Ba-Indyk-Price-Woodruff 10 Theorem There exists an algorithm for l 2 /l 2 sparse recovery from Fourier measurements using O(k logn) samples and O(nlog 3 n) runtime. Optimal up to constant factors for k n 1 δ. First sample-optimal algorithm even if exponential runtime is allowed. 5 / 26

9 6 / 26 Fourier Transform and Sparsity Higher dimensional Fourier transform is needed in some applications Given x C [n]d, N = n d, compute x j = 1 ω it j x i and x j = 1 N i [n] d N i [n] d ω it j x i where ω is the n-th root of unity, and n is a power of 2.

10 7 / 26 Fourier Transform and Sparsity Previous sample complexity bounds: O(klog d N) in sublinear time algorithms runtime k log O(d) N, for each O(klog 4 N) for any d Ω(N) time, for all

11 Fourier Transform and Sparsity Previous sample complexity bounds: O(klog d N) in sublinear time algorithms runtime k log O(d) N, for each O(klog 4 N) for any d Ω(N) time, for all Our result: Theorem There exists an algorithm for l 2 /l 2 sparse recovery from Fourier measurements using O d (k logn) samples and O(N log 3 N) runtime. Sample-optimal up to constant factors for any constant d, first such algorithm. 7 / 26

12 Fourier Transform and Sparsity Previous sample complexity bounds: O(klog d N) in sublinear time algorithms runtime k log O(d) N, for each O(klog 4 N) for any d Ω(N) time, for all Our result: Theorem There exists an algorithm for l 2 /l 2 sparse recovery from Fourier measurements using O d (k logn) samples and O(N log 3 N) runtime. Sample-optimal up to constant factors for any constant d, first such algorithm. Also: sublinear time recovery, but with loglog 2 n loss in sample complexity (extending Indyk-K.-Price 14 to higher dimensions) 7 / 26

13 8 / 26 Fourier Transform and Sparsity Outline of talk: 1 l 2 /l 2 sparse recovery guarantee 2 Summary of techniques for recovery from Fourier measurements 3 Sample-optimal algorithm in O(N log 3 N) time for d = 1

14 9 / 26 l 2 /l 2 sparse recovery l 2 /l 2 sparse recovery guarantees: x ŷ 2 C min k sparse ẑ x ẑ 2 head tail µ tail noise/ k

15 10 / 26 l 2 /l 2 sparse recovery l 2 /l 2 sparse recovery guarantees: x ŷ 2 C min k sparse ẑ x ẑ 2 x 1... x k x k+1 x k+2... Err 2 k ( x) = n j=k+1 x j 2 Residual error bounded by noise energy Err 2 k ( x) head tail µ tail noise/ k

16 11 / 26 l 2 /l 2 sparse recovery l 2 /l 2 sparse recovery guarantees: x ŷ 2 C Err 2 k ( x) x 1... x k x k+1 x k+2... Err 2 k ( x) = n j=k+1 x j 2 Residual error bounded by noise energy Err 2 k ( x) head tail µ tail noise/ k

17 12 / 26 l 2 /l 2 sparse recovery l 2 /l 2 sparse recovery guarantees: x ŷ 2 C Err 2 k ( x) x 1... x k x k+1 x k+2... Err 2 k ( x) = n j=k+1 x j 2 Residual error bounded by noise energy Err 2 k ( x) head tail µ tail noise/ k

18 13 / 26 Recovery from Fourier measurements Iterative refinement Many algorithms use the iterative refinement scheme: Input: x C n ŷ 0 0 For t = 1 to L ẑ REFINEMENT(x y t 1 ) Update ŷ t ŷ t 1 +ẑ Takes random samples of x y REFINEMENT(x) return dominant Fourier coefficients ẑ of x (approximately) Gilbert-Guha-Indyk-Muthukrishnan-Strauss 02, Akavia-Goldwasser-Safra 03, Gilbert-Muthukrishnan-Strauss 05, Iwen 10, Akavia 10, Hassanieh-Indyk-Katabi-Price 12a, Hassanieh-Indyk-Katabi-Price 12b

19 14 / 26 Recovery from Fourier measurements magnitude magnitude time frequency

20 15 / 26 Recovery from Fourier measurements Task:approximate top k coeffs of x using few samples Natural idea: look at the value of the signal on the first O(k) points magnitude magnitude time frequency

21 15 / 26 Recovery from Fourier measurements Task:approximate top k coeffs of x using few samples Natural idea: look at the value of the signal on the first O(k) points magnitude magnitude time frequency

22 16 / 26 Recovery from Fourier measurements Task:approximate top k coeffs of x using few samples Natural idea: look at the value of the signal on the first O(k) points This convolves spectrum with sinc: (x G) = x Ĝ magnitude magnitude time frequency

23 16 / 26 Recovery from Fourier measurements Task:approximate top k coeffs of x using few samples Natural idea: look at the value of the signal on the first O(k) points This convolves spectrum with sinc: (x G) = x Ĝ magnitude 0 magnitude time frequency (G x) f = x f Ĝf f f [n]

24 16 / 26 Recovery from Fourier measurements Task:approximate top k coeffs of x using few samples Natural idea: look at the value of the signal on the first O(k) points This convolves spectrum with sinc: (x G) = x Ĝ magnitude 0 magnitude time frequency (G x) f = x f + x f Ĝf f f [n],f f

25 Recovery from Fourier measurements REFINEMENT(x) return dominant Fourier coefficients ẑ of x (approximately) Take M = C logn independent measurements: y j (P σj,a j,q j x) G Estimate each f [n] as { ŵ f median ỹ 1 f,...,ỹm f }. Sample complexity= filter support logn 17 / 26

26 18 / 26 Recovery from Fourier measurements magnitude 0 magnitude time frequency Like hashing heavy hitters into buckets (COUNTSKETCH, COUNTMIN), but buckets leak

27 19 / 26 Iterative refinement Sample-optimal algorithm Many algorithms use the iterative refinement scheme: Input: x C n ŷ 0 0 For t = 1 to L ẑ REFINEMENT(x y t 1 ) Update ŷ t ŷ t 1 +ẑ Takes random samples of x y

28 19 / 26 Iterative refinement Sample-optimal algorithm Many algorithms use the iterative refinement scheme: Input: x C n ŷ 0 0 For t = 1 to L ẑ REFINEMENT(x y t 1 ) Update ŷ t ŷ t 1 +ẑ In most prior works sampling complexity is Takes random samples of x y samples per REFINEMENT step number of iterations

29 20 / 26 Iterative refinement Sample-optimal algorithm Many algorithms use the iterative refinement scheme: Input: x C n ŷ 0 0 For t = 1 to L ẑ REFINEMENT(x y t 1 ) Update ŷ t ŷ t 1 +ẑ In most prior works sampling complexity is Takes random samples of x y filter support logn number of iterations

30 20 / 26 Iterative refinement Sample-optimal algorithm Many algorithms use the iterative refinement scheme: Input: x C n ŷ 0 0 For t = 1 to L ẑ REFINEMENT(x y t 1 ) Update ŷ t ŷ t 1 +ẑ In most prior works sampling complexity is Takes random samples of x y filter support logn number of iterations Lots of work on carefully choosing filters, reducing number of iterations: Hassanieh-Indyk-Katabi-Price 12, Ghazi-Hassanieh-Indyk-Katabi-Price-Shi 13, Indyk-K.-Price 14 still lose Ω(loglogn) in sample complexity (number of iterations) lose Ω((logn) d 1 loglogn) in higher dimensions

31 Iterative refinement Sample-optimal algorithm Many algorithms use the iterative refinement scheme: Input: x C n ŷ 0 0 For t = 1 to L ẑ REFINEMENT(x y t 1 ) Update ŷ t ŷ t 1 +ẑ Takes random samples of x y Our sampling complexity is filter support logn number of iterations 21 / 26

32 Iterative refinement Sample-optimal algorithm Many algorithms use the iterative refinement scheme: Input: x C n ŷ 0 0 For t = 1 to L ẑ REFINEMENT(x y t 1 ) Update ŷ t ŷ t 1 +ẑ Takes random samples of x y Our sampling complexity is filter support logn number of iterations 21 / 26

33 Iterative refinement Sample-optimal algorithm Many algorithms use the iterative refinement scheme: Input: x C n ŷ 0 0 For t = 1 to L ẑ REFINEMENT(x y t 1 ) Update ŷ t ŷ t 1 +ẑ Takes random samples of x y Our sampling complexity is samples per REFINEMENT step number of iterations 21 / 26

34 Iterative refinement Sample-optimal algorithm Many algorithms use the iterative refinement scheme: Input: x C n ŷ 0 0 For t = 1 to L ẑ REFINEMENT(x y t 1 ) Update ŷ t ŷ t 1 +ẑ Takes random samples of x y Our sampling complexity is samples per REFINEMENT step number of iterations Do not use fresh randomness in each iteration! In general challenging: only one paper Bayati-Montanari 11 gives provable guarantees, with Gaussians 21 / 26

35 Iterative refinement Sample-optimal algorithm Many algorithms use the iterative refinement scheme: Input: x C n ŷ 0 0 For t = 1 to L ẑ REFINEMENT(x y t 1 ) Update ŷ t ŷ t 1 +ẑ Takes random samples of x y Our sampling complexity is samples per REFINEMENT step number of iterations Do not use fresh randomness in each iteration! In general challenging: only one paper Bayati-Montanari 11 gives provable guarantees, with Gaussians Can use very simple filters! (Essentially) boxcar filter 21 / 26

36 22 / 26 G B B B Let y m (P m x) G m = 0,...,M = C logn ẑ 0 0 For t = 1,...,T = O(logn): Sample-optimal algorithm For f [n]: { ŵ f median ỹ 1 f,...,ỹm f If ŵ f < 2 T t µ/3 then ŵ f 0 End ẑ t+1 = ẑ t +ŵ y m y m (P m w) G for m = 1,...,M End } Take samples of x Loop over thresholds Estimate, prune small elements Update samples

37 22 / 26 G B B B Let y m (P m x) G m = 0,...,M = C logn ẑ 0 0 For t = 1,...,T = O(logn): Sample-optimal algorithm For f [n]: { ŵ f median ỹ 1 f,...,ỹm f If ŵ f < 2 T t µ/3 then ŵ f 0 End ẑ t+1 = ẑ t +ŵ y m y m (P m w) G for m = 1,...,M End } µ

38 22 / 26 G B B B Let y m (P m x) G m = 0,...,M = C logn ẑ 0 0 For t = 1,...,T = O(logn): Sample-optimal algorithm For f [n]: { ŵ f median ỹ 1 f,...,ỹm f If ŵ f < 2 T t µ/3 then ŵ f 0 End ẑ t+1 = ẑ t +ŵ y m y m (P m w) G for m = 1,...,M End } µ

39 22 / 26 G B B B Let y m (P m x) G m = 0,...,M = C logn ẑ 0 0 For t = 1,...,T = O(logn): Sample-optimal algorithm For f [n]: { ŵ f median ỹ 1 f,...,ỹm f If ŵ f < 2 T t µ/3 then ŵ f 0 End ẑ t+1 = ẑ t +ŵ y m y m (P m w) G for m = 1,...,M End } µ

40 22 / 26 G B B B Let y m (P m x) G m = 0,...,M = C logn ẑ 0 0 For t = 1,...,T = O(logn): Sample-optimal algorithm For f [n]: { ŵ f median ỹ 1 f,...,ỹm f If ŵ f < 2 T t µ/3 then ŵ f 0 End ẑ t+1 = ẑ t +ŵ y m y m (P m w) G for m = 1,...,M End } µ

41 22 / 26 G B B B Let y m (P m x) G m = 0,...,M = C logn ẑ 0 0 For t = 1,...,T = O(logn): Sample-optimal algorithm For f [n]: { ŵ f median ỹ 1 f,...,ỹm f If ŵ f < 2 T t µ/3 then ŵ f 0 End ẑ t+1 = ẑ t +ŵ y m y m (P m w) G for m = 1,...,M End } µ

42 22 / 26 G B B B Let y m (P m x) G m = 0,...,M = C logn ẑ 0 0 For t = 1,...,T = O(logn): Sample-optimal algorithm For f [n]: { ŵ f median ỹ 1 f,...,ỹm f If ŵ f < 2 T t µ/3 then ŵ f 0 End ẑ t+1 = ẑ t +ŵ y m y m (P m w) G for m = 1,...,M End } µ Main challenge: lack of fresh randomness. Why does median work?

43 23 / 26 Sample-optimal algorithm Let S denote the set of heavy hitters: S = { i [n] : x i > µ }. There cannot be too many of them: S = O(k) head tail

44 23 / 26 Sample-optimal algorithm Let S denote the set of heavy hitters: S = { i [n] : x i > µ }. There cannot be too many of them: S = O(k) head tail Main invariant: never modify x outside of S Prove: samples taken have error-correcting properties wrt set S of head elements Set of head elements does not change, only their values

45 24 / 26 Experimental evaluation Experimental evaluation Problem: recover support of a random k-sparse signal from Fourier measurements. Parameters: n = 2 15, k = 10,20,...,100 Filter: boxcar filter with support k + 1

46 Experimental evaluation Comparison to l 1 -minimization (SPGL1) O(k log 3 k logn) sample complexity, requires LP solve 2500 n=32768, L1 minimization n=32768, B=k, random phase, non monotone number of measurements number of measurements sparsity sparsity Within a factor of 2 of l 1 minimization 25 / 26

47 26 / 26 Experimental evaluation Open questions O(k logn) samples in O(k log O(1) n) time? Thank you!