On the coherence barrier and analogue problems in compressed sensing

Transcription

1 On the coherence barrier and analogue problems in compressed sensing Clarice Poon University of Cambridge June 1, 2017 Joint work with: Ben Adcock Anders Hansen Bogdan Roman (Simon Fraser) (Cambridge) (Cambridge) 1 / 40

2 Compressed sensing Given an isometry U C N N, can we recover x C N from P Ω Ux, where P Ω is the restriction of a vector to an index set Ω and Ω = m << N? Theorem (Candès & Plan, 2011) Define the coherence of U be µ(u) := max u k,j 2. j,k=1,...,n If x C N is s-sparse, then by choosing Ω is uniformly at random such that Ω = O ( µ(u) N s log(n) log(ɛ 1 + 1) ) x is recovered exactly from its samples P Ω Ux with probability exceeding 1 ɛ by solving min β C N β 1 subject to P Ω Uβ = P Ω Ux Notation: if y = P Ω x, then y j = x j for all j Ω and y j = 0 otherwise. 2 / 40

3 Compressed sensing Given an isometry U C N N, can we recover x C N from P Ω Ux, where P Ω is the restriction of a vector to an index set Ω and Ω = m << N? Theorem (Candès & Plan, 2011) Let U be incoherent µ(u) := max u k,j 2 = O ( N 1) j,k=1,...,n If x C N is s-sparse, then by choosing Ω uniformly at random such that Ω = O ( µ(u) N s log(n) log(ɛ 1 + 1) ) x is recovered exactly from its samples P Ω Ux, with probability exceeding 1 ɛ by solving min β C N β 1 subject to P Ω Uβ = P Ω Ux. 3 / 40

4 Compressed sensing in action Applications: Magnetic Resonance Imaging (MRI), X-ray Computed Tomography, Electron Microscopy, Seismology, Radio interferometry,... Mathematically: recover f L 2 (R 2 ) from Ff (ω), ω Ω where Ff denotes the continuous Fourier transform of f and Ω is a finite set of frequencies. Standard approach: Solve min z C N2 U z 1 subject to P Ω U df z P Ω ˆf δ 2 where U df is the discrete Fourier transform, and U is some sparsifying transform (e.g. wavelets). 4 / 40

5 Compressed sensing in action If U is a discrete wavelet transform, then µ(udf U 1 ) = 1, so µ(udf U 1 ) N s log(n) log( 1 + 1) > N! Also, uniform random sampling does not work. Original Ω Rec. Wavelet Rec. TV Test phantom constructed by Guerquin-Kern, Lejeune, Pruessmann, Unser, / 40

6 Compressed sensing in action If U is a discrete wavelet transform, then µ(u df U 1 ) = 1, so µ(u df U 1 ) N s log(n) log(ɛ 1 + 1) > N! Also, uniform random sampling does not work. Original Ω Rec. Wavelet Rec. TV Lustig, Donoho & Pauli 07, Lustig et al. 08: Sample more densely at low Fourier frequencies and less at higher Fourier frequencies. Why? Test phantom constructed by Guerquin-Kern, Lejeune, Pruessmann, Unser, / 40

7 This talk The assumptions of incoherence and sparsity are not appropriate. Instead, we have asymptotic incoherence and asymptotic sparsity. CS is primarily finite dimensional, our theory will cover an infinite dimensional setting. Present a theoretical result which shows how the presence of asymptotic incoherence and asymptotic sparsity will allow for significant subsampling when solving min z 1 subject to P Ω Uz P Ω Ux 2 δ. 6 / 40

8 Outline Asymptotic incoherence Sparsity structure Compressed sensing in infinite dimensions The recovery statement 7 / 40

9 Outline Asymptotic incoherence Sparsity structure Compressed sensing in infinite dimensions The recovery statement 8 / 40

10 The coherence barrier For essentially any wavelet basis {ϕ j } j N, if U dw,n is its discrete wavelet transform and U df,n is the discrete Fourier transform, then U df,n U 1 WOT dw,n U, N where {ψ j } j N = {e 2πik } k Z, ϕ 1, ψ 1 ϕ 2, ψ 1 U = ϕ 1, ψ 2 ϕ 2, ψ 2, µ(u) c Any systems arising from the discretization of continuous problems will always run into the coherence barrier. 9 / 40

11 Asymptotic incoherence If U is the Fourier-wavelets matrix, then µ(p N U), µ(up N ) = O ( N 1). Fourier to DB4 Fourier to Legendre polynomials Notation: P N x = (x 1,, x N, 0, 0, ) and P N x = (0,, 0, x N+1, x N+2, ) 10 / 40

12 Local coherence Instead of coherence, divide U into rectangular blocks using N = (N j ) r j=1, M = (M j) r j=1, and consider local coherence µ N,M (k, l) = µ(p N k N k 1 UP M l M l 1 ). where P m n α = (..., 0, α n+1, α n+2,..., α m, 0...). Implication of asymptotic incoherence: sample more at low Fourier frequencies where the local coherence is high and less at higher Fourier frequencies. 11 / 40

13 Local coherence Instead of coherence, divide U into rectangular blocks using N = (N j ) r j=1, M = (M j) r j=1, and consider local coherence µ N,M (k, l) = µ(p N k N k 1 UP M l M l 1 ). where P m n α = (..., 0, α n+1, α n+2,..., α m, 0...). Implication of asymptotic incoherence: sample more at low Fourier frequencies where the local coherence is high and less at higher Fourier frequencies. Krahmer & Ward (2014): O ( s log 3 (s) log 2 (N) ) for Haar wavelet sparsity from Fourier samples and O ( s log 5 (s) log 3 (N) ) for TV sparsity with Fourier sampling / 40

14 Outline Asymptotic incoherence Sparsity structure Compressed sensing in infinite dimensions The recovery statement 12 / 40

15 Sparsity and the flip test In standard CS, the only signal structure considered is sparsity and RIP based results consider the recovery of all s-sparse signals using one Ω. In contrast, the flip test will demonstrate that we must look beyond sparsity. Consider the reconstruction of x from P Ω U df x by solving min z 1 subject to P Ω U df U 1 dw z = P ΩU df x. Ω x Reconstruction 13 / 40

16 The flip test Let α be the wavelet coefficients of x. 2 α 2 α flip 1.5 Truncated (max = ) 1.5 Truncated (max = ) x x10 5 Let α flip = (α N,..., α 1 ) and x flip = U 1 dw αflip. If it is enough to consider sparsity when choosing Ω, then for the same Ω, would yield α f arg min z z 1 subject to P Ω U df U 1 dw z = P ΩU df x flip, α f α flip = α flip f α ˆx = U 1 dw αflip f x = U 1 dw α 14 / 40

17 The flip test Ω Standard Reconstruction from reconstruction flipped coefficients We can repeat this test for different images, different sampling patterns, or even the sparsifying transform to that of *-lets or total variation, min z C N2 z TV subject to P ΩU df z = P Ω U df x... and observe that the optimal choice of Ω cannot depend on sparsity alone. 15 / 40

18 The flip test Ω Standard Reconstruction from reconstruction flipped coefficients We can repeat this test for different images, different sampling patterns, or even the sparsifying transform to that of *-lets or total variation, min z C N2 z TV subject to P ΩU df z = P Ω U df x... and observe that the optimal choice of Ω cannot depend on sparsity alone. 16 / 40

19 The flip test Ω Image 1 Image We can repeat this test for different images, different sampling patterns, or even different sparsifying transforms such as *-lets or total variation, min z C N2 z TV subject to P ΩU df z = P Ω U df x... and observe that the optimal choice of Ω cannot depend on sparsity alone. 17 / 40

20 The flip test Ω Image 1 Image 2 reconstruction reconstruction We can repeat this test for different images, different sampling patterns, or even the sparsifying transform to that of *-lets or total variation, min z C N2 z TV subject to P ΩU df z = P Ω U df x The optimal choice of Ω cannot depend on sparsity alone. 18 / 40

21 Remark on the RIP The flip test demonstrated that Ω cannot depend on sparsity alone, and in fact, the RIP is absent. This is in contrast to random Gaussian measurements are insensitive to sparsity structure: Standard reconstruction Reconstruction from flipped coefficients 19 / 40

22 Asymptotic sparsity Natural images are not just sparse, but asymptotically sparse. Given ɛ (0, 1) and f = j N α j ϕ j. we define the number of significant wavelet coefficients of f in the k th scale as s k (ɛ) = min n : n M k α π(j) ϕ π(j) ɛ α j ϕ j j=1 j=1+m 2 k 1 where the {M k 1 + 1,..., M k } be indices corresponding to the k th scale and π is a permutation of the indices in {M k 1 + 1,..., M k } such that α π(1) α π(2) α π(3) / 40

23 Asymptotic sparsity Sparsity, sk(ǫ)/(mk Mk 1) Level 1 Level 2 Level 3 Level 4 Level 5 Level 6 Level 7 Level 8 Worst sparsity Best sparsity Sparsity, sk(ǫ)/(mk Mk 1) Level 1 Level 2 Level 3 Level 4 Level 5 Level 6 Level 7 Level 8 Worst sparsity Best sparsity Relative threshold, ǫ Relative threshold, ǫ s k (ɛ) M k M k 1 0, k Variable density sampling patterns work because they exploit this additional structure. 21 / 40

24 Sparsity in levels For M = (M j ) r j=1 Nr, s = (s j ) r j=1 Nr with 0 = M 0 < M 1 <... < M r, α l 2 (N) is (s, M)-sparse if {j : α j 0} {M k 1 + 1,..., M k } = s k. 22 / 40

25 Outline Asymptotic incoherence Sparsity structure Compressed sensing in infinite dimensions The recovery statement 23 / 40

26 A continuous framework Typical model for sampling operator: U df U 1. But, our samples are of the continuous Fourier transform and not of the discrete one. In general, modelling infinite dimensional problems using finite dimensional techniques will lead to samples mismatch. 24 / 40

27 A continuous framework Typical model for sampling operator: U df U 1. But, our samples are of the continuous Fourier transform and not of the discrete one. In general, modelling infinite dimensional problems using finite dimensional techniques will lead to samples mismatch. Consider ˆf = {ˆf (j) : j N}, U df : C 2N+1 C 2N+1, min U dw g 1 subject to P Ω U df g = ˆf If Ω = {1,..., 2N + 1}, then the solution is the truncated Fourier representation f (x) = j N ˆf (j)e πijx/n. But the wavelet representation of f need not be sparse! 24 / 40

28 A continuous framework Consider two orthonormal bases of a Hilbert space H, e.g. L 2 (0, 1) d : Sampling basis: S = {ψ j } j N, e.g. Fourier basis. Sparsity basis: R = {ϕ j } j N. e.g. wavelet basis. Let f H be the object of interest. Write x j = f, ϕ j for the coefficients of f, i.e. f = j N x jϕ j, y j = f, ψ j for the measurements of f. Define the infinite matrix U = ( ϕ j, ψ i ) i,j N B(l 2 (N)) and note that Ux = y. 25 / 40

29 A continuous framework Consider two orthonormal bases of a Hilbert space H, e.g. L 2 (0, 1) d : Sampling basis: S = {ψ j } j N, e.g. Fourier basis. Sparsity basis: R = {ϕ j } j N. e.g. wavelet basis. Let f H be the object of interest. Write x j = f, ϕ j for the coefficients of f, i.e. f = j N x jϕ j, y j = f, ψ j for the measurements of f. Define the infinite matrix U = ( ϕ j, ψ i ) i,j N B(l 2 (N)) and note that P Ω Ux = P Ω y. 25 / 40

30 Infinite dimensional CS Solve min z l 1 (N) z 1 subject to PΩ Uz P Ω ˆf δ. (P δ (ˆf )) 2 Adcock & Hansen. Generalized sampling and infinite-dimensional compressed sensing (2011). Implementation: Guerquin-Kern, Haberlin, Pruessmann & Unser. A Fast Wavelet-Based Reconstruction Method for Magnetic Resonance Imaging (2011). 26 / 40

31 Infinite dimensional CS Solve min z l 1 (N) z 1 subject to PΩ UP K z P Ω ˆf δ. (P δ (ˆf )) 2 Adcock & Hansen. Generalized sampling and infinite-dimensional compressed sensing (2011). Implementation: Guerquin-Kern, Haberlin, Pruessmann & Unser. A Fast Wavelet-Based Reconstruction Method for Magnetic Resonance Imaging (2011). In the Fourier sampling/wavelet reconstruction case, if Ω {1,..., N}, then choosing K = O (N) yields the same error bounds as the infinite dimensional problem. 26 / 40

32 Infinite dimensional CS Considerations: What is the bandwidth N that we should draw from? i.e. Ω {1,..., N}. Given a finite sampling bandwidth, we cannot expect to recover any s-sparse vector x over N stably. We can only expect the recovery of sparse vectors such that Supp(x) {1,..., M}. 27 / 40

33 Infinite dimensional CS Considerations: What is the bandwidth N that we should draw from? i.e. Ω {1,..., N}. Given a finite sampling bandwidth, we cannot expect to recover any s-sparse vector x over N stably. We can only expect the recovery of sparse vectors such that Supp(x) {1,..., M}. Adcock & Hansen: N, K = N Ω should satisfy the balancing property relative to M and s: ( P M U P N UP M P M 8 log 1/2 ( ) ) skm, Typically, N > M. P MU P N UP M / 40

34 Infinite dimensional CS Considerations: What is the bandwidth N that we should draw from? i.e. Ω {1,..., N}. Given a finite sampling bandwidth, we cannot expect to recover any s-sparse vector x over N stably. We can only expect the recovery of sparse vectors such that Supp(x) {1,..., M}. Adcock & Hansen: N, K = N Ω should satisfy the balancing property relative to M and s: ( P M U P N UP M P M 8 log 1/2 ( ) ) skm, Typically, N > M. P MU P N UP M 1 8. Under the balancing property, one can guarantee ( the recovery of s-sparse ) vectors with support on {1,..., M} from O s N µ(u) log(ñ) samples from uniformly at random from {1,..., N}. 27 / 40

35 Remarks for the Fourier wavelets case Let S be a Fourier basis for L 2 [0, 1] and let R be any compactly supported orthonormal wavelet basis. min z z l 1 1 subject to PΩ Uz P Ω ˆf δ. (N) 2 Fast transforms (Gataric & P. 16): Given x C M and y C N, P N UP M x and P M U P N y can be computed in O (N log(m)) basic operations. Linear scaling (P. 14): Suppose the scaling function Φ and wavelet Ψ satisfy for some α 2 ˆΦ (k) (ξ) (1 + ξ ) α, ˆΨ (k) (ξ) (1 + ξ ) α, k = 0, 1, 2. If x are the wavelet coefficients of f, then by choosing Ω = {1,..., N} with N = O(M) and δ = 0, any minimizer ˆx satisfies x ˆx l 1 P M x / 40

36 Example from electron microscopy f (x, y ) = cos2 (7πx/2) cos2 (7πy /2) exp( x y ). Original Infinite-dimensional CS Finite-dimensional CS Err = 4.1% Err = 16.0% 29 / 40

37 Outline Asymptotic incoherence Sparsity structure Compressed sensing in infinite dimensions The recovery statement 30 / 40

38 Multi-level sampling scheme Let r N, N = {N k } r k=1 Nr, m = {m k } r k=1 Nr be such that 0 = N 0 < N 1 < < N r = N, m k N k N k 1. Ω = Ω 1 Ω r is an (N, m)-sampling scheme if Ω k consists of m k indices drawn uniformly at random from {N k 1 + 1,..., N k }. 31 / 40

39 Multi-level sampling scheme Let r N, N = {N k } r k=1 Nr, m = {m k } r k=1 Nr be such that 0 = N 0 < N 1 < < N r = N, m k N k N k 1. Ω = Ω 1 Ω r is an (N, m)-sampling scheme if Ω k consists of m k indices drawn uniformly at random from {N k 1 + 1,..., N k }. Q: Let U B(l 2 (N)) be an isometry. If x is approximately (s, M)-sparse σ s,m (α) = inf z α 1 << 1. z is (s,m)-sparse then how should N and m be chosen so as to guarantee robust and stable recovery of x by solving inf z z l 1 1 subject to P ΩUx P Ω Uz 2 δ? (N) 31 / 40

40 Recovery result (Adcock, Hansen, P. & Roman) Let ɛ (0, e 1 ] and x be approximately (s, M)-sparse. Suppose that Ω = Ω N,m satisfies the following. (i) N r, K = max r N k N k 1 k=1 m k, satisfy the balancing property: ( P Mr U P Nr UP Mr P Mr 8 log 1/2 ( ) ) skmr, PM r U P Nr UP Mr 1 8. (ii) m k (N k N k 1 ) ( r l=1 µ ) ( N,M(k, l) s l log(ɛ 1 ) log K M ) s, ( (iii) m k ˆm k log(ɛ 1 ) log K M ) s, where ˆm k satisfies 1 r k=1 ( ) Nk N k 1 1 µ N,M (k, l) s k. ˆm k Then, with probability exceeding 1 sɛ, any minimizer ξ l 1 (N) satisfies ξ x 2 δ K ( ) log2 (6ɛ ) log 2 (4KM s + σ s,m (x). s) 32 / 40

41 Recovery of wavelet coefficients from partial Fourier data {N k } r k=1 and {M k} r k=1 correspond to wavelet scales. The mother wavelet Ψ has v vanishing moments. There exists α 1, C > 0 such that ˆΨ(ξ) C (1+ ξ ) for all ξ R. α It suffices that (i) (ii) N r M 1+1/(2α 1) r (log 2 (4M r K s)) 1/(2α 1) (or even N r M r (log 2 (4M r K s)) 1/(2α 1) when Ψ is sufficiently smooth). ( m k 1 L ŝ k + N k N k 1 N k 1 k 2 s j 2 α(k l) + l=1 r l=k+2 s l 2 v(l k) ) where ŝ k = max {s k 1, s k, s k+1 } and L = log(ɛ 1 ) log ( KN s ) NB: m m r L (s s r ). 33 / 40

42 Resolution Dependence (5% samples, varying resolution) Asymptotic sparsity and asymptotic incoherence are only witnessed when N is large. Thus, V. D. sampling only reaps their benefits for large values of N and the success of compressed sensing is resolution dependent. 256x256 Error: 19.86% 512x512 Error: 10.69% 34 / 40

43 Resolution Dependence (5% samples, varying resolution) 1024x1024 Error: 7.35% 2048x2048 Error: 4.87% 4096x4096 Error: 3.06% 35 / 40

44 Recovering Fine Details At finer wavelet scales, the presence of sparsity and incoherence with Fourier samples allows us to subsample. Thus, compressed sensing allows one to enhance fine details without increasing the number of samples. In the next example, consider the reconstruction of a test phantom with details added at the finest wavelet scale. 36 / 40

45 Recovering Fine Details Figure: linear reconstruction from the first Fourier samples (6.25%) 37 / 40

46 Recovering Fine Details Figure: reconstruction from a multilevel scheme using Fourier samples (6.25%) 38 / 40

47 Conclusions There are many real world problems where there is no incoherence or RIP. We introduced a CS theory based on local coherence and local sparsities, and demonstrated that the successful recovery of wavelet coefficients from partial Fourier data can be explained by asymptotic incoherence and asymptotic sparsity. Two key consequences of our theory: (1) CS is resolution dependent. (2) Successful recovery is signal dependent, thus, an understanding of the structure imposed by the sparsifying transform can lead to optimal sampling patterns. On a practical note, one should see compressed sensing as a means of enhancing resolution / 40

48 References Breaking the coherence barrier: A new theory for compressed sensing. Adcock, Hansen, Poon & Roman, Forum of Mathematics, Sigma. Vol. 5. Cambridge University Press (2017). Generalized sampling and infinite-dimensional compressed sensing. Adcock & Hansen, Foundations of Computational Mathematics 16.5 (2016): A consistent and stable approach to generalized sampling. Poon. Journal of Fourier Analysis and Applications, 20(5), (2014). A practical guide to the recovery of wavelet coefficients from Fourier measurements. Gataric & Poon. SIAM Journal on Scientific Computing, 38(2), A1075-A1099 (2016). 40 / 40

49 References Breaking the coherence barrier: A new theory for compressed sensing. Adcock, Hansen, Poon & Roman, Forum of Mathematics, Sigma. Vol. 5. Cambridge University Press (2017). Generalized sampling and infinite-dimensional compressed sensing. Adcock & Hansen, Foundations of Computational Mathematics 16.5 (2016): A consistent and stable approach to generalized sampling. Poon. Journal of Fourier Analysis and Applications, 20(5), (2014). A practical guide to the recovery of wavelet coefficients from Fourier measurements. Gataric & Poon. SIAM Journal on Scientific Computing, 38(2), A1075-A1099 (2016). Thank you for you attention. 40 / 40