Greedy Signal Recovery and Uniform Uncertainty Principles

Transcription

1 Greedy Signal Recovery and Uniform Uncertainty Principles SPIE - IE 2008 Deanna Needell Joint work with Roman Vershynin UC Davis, January 2008 Greedy Signal Recovery and Uniform Uncertainty Principles p.1/24

2 Outline Problem Background Setup L1 Minimization Greedy Algorithms ROMP Algorithm Main Theorem Empirical Results Future Work Greedy Signal Recovery and Uniform Uncertainty Principles p.2/24

3 Setup Consider v R d, v 0 := supp v n d. Greedy Signal Recovery and Uniform Uncertainty Principles p.3/24

4 Setup Consider v R d, v 0 := supp v n d. We call such signals n-sparse. Greedy Signal Recovery and Uniform Uncertainty Principles p.3/24

5 Setup Consider v R d, v 0 := supp v n d. We call such signals n-sparse. Given some N d measurement matrix Φ, we collect N d nonadaptive linear measurements of v, in the form x = Φv. Φ v = x Greedy Signal Recovery and Uniform Uncertainty Principles p.3/24

6 Setup ctd. From these measurements we wish to efficiently recover the original signal v. Greedy Signal Recovery and Uniform Uncertainty Principles p.4/24

7 Setup ctd. From these measurements we wish to efficiently recover the original signal v. How many measurements N d are needed? Greedy Signal Recovery and Uniform Uncertainty Principles p.4/24

8 Setup ctd. From these measurements we wish to efficiently recover the original signal v. How many measurements N d are needed? Exact recovery is possible with just N = 2n. However, recovery in this regime is not numerically feasible. Greedy Signal Recovery and Uniform Uncertainty Principles p.4/24

9 Setup ctd. Work in Compressed Sensing has shown that the signal v can be efficiently exactly recovered from x = Φv with just N n polylog d. Greedy Signal Recovery and Uniform Uncertainty Principles p.5/24

10 Setup ctd. Work in Compressed Sensing has shown that the signal v can be efficiently exactly recovered from x = Φv with just N n polylog d. Two major algorithmic approaches: Greedy Signal Recovery and Uniform Uncertainty Principles p.5/24

11 Setup ctd. Work in Compressed Sensing has shown that the signal v can be efficiently exactly recovered from x = Φv with just N n polylog d. Two major algorithmic approaches: L1-Minimization (Donoho et. al.) Greedy Signal Recovery and Uniform Uncertainty Principles p.5/24

12 Setup ctd. Work in Compressed Sensing has shown that the signal v can be efficiently exactly recovered from x = Φv with just N n polylog d. Two major algorithmic approaches: L1-Minimization (Donoho et. al.) Iterative methods such as Orthogonal Matching Pursuit (Tropp-Gilbert) Greedy Signal Recovery and Uniform Uncertainty Principles p.5/24

13 L1-Minimization Methods The sparse recovery problem can be stated as solving the optimization problem: min z 0 subject to Φz = Φv Greedy Signal Recovery and Uniform Uncertainty Principles p.6/24

14 L1-Minimization Methods The sparse recovery problem can be stated as solving the optimization problem: min z 0 subject to Φz = Φv For certain measurement matrices Φ this hard problem is equivalent to: min u 1 subject to Φu = Φv (Donoho, Candès-Tao) Greedy Signal Recovery and Uniform Uncertainty Principles p.6/24

15 Restricted Isometry Condition A measurement matrix Φ satisfies the Restricted Isometry Condition (RIC) with parameters (m, ε) for ε (0, 1) if we have (1 ε) v 2 Φv 2 (1+ε) v 2 m-sparse v. Greedy Signal Recovery and Uniform Uncertainty Principles p.7/24

16 Restricted Isometry Condition A measurement matrix Φ satisfies the Restricted Isometry Condition (RIC) with parameters (m, ε) for ε (0, 1) if we have (1 ε) v 2 Φv 2 (1+ε) v 2 m-sparse v. Every set of n columns of Φ forms approximately an orthonormal system. Greedy Signal Recovery and Uniform Uncertainty Principles p.7/24

17 L1 and the RIC Assume that the measurement matrix Φ satisfies the Restricted Isometry Condition with parameters (2n, 2 1). Greedy Signal Recovery and Uniform Uncertainty Principles p.8/24

18 L1 and the RIC Assume that the measurement matrix Φ satisfies the Restricted Isometry Condition with parameters (2n, 2 1). Then the L 1 method recovers any n-sparse vector. (Candes-Tao) Greedy Signal Recovery and Uniform Uncertainty Principles p.8/24

19 L1 and the RIC Assume that the measurement matrix Φ satisfies the Restricted Isometry Condition with parameters (2n, 2 1). Then the L 1 method recovers any n-sparse vector. (Candes-Tao) What kinds of matrices satisfy the RIC? Greedy Signal Recovery and Uniform Uncertainty Principles p.8/24

20 L1 and the RIC Assume that the measurement matrix Φ satisfies the Restricted Isometry Condition with parameters (2n, 2 1). Then the L 1 method recovers any n-sparse vector. (Candes-Tao) What kinds of matrices satisfy the RIC? Random Gaussian, Bernoulli, and partial Fourier matrices, with N n polylog d. Greedy Signal Recovery and Uniform Uncertainty Principles p.8/24

21 Greedy Algorithms: OMP Orthogonal Matching Pursuit (Tropp-Gilbert) finds the support of the n-sparse signal v progressively. Greedy Signal Recovery and Uniform Uncertainty Principles p.9/24

22 Greedy Algorithms: OMP Orthogonal Matching Pursuit (Tropp-Gilbert) finds the support of the n-sparse signal v progressively. Once S = supp(v) is found correctly, we can recover the signal: x = Φv as v = (Φ S ) 1 x. Greedy Signal Recovery and Uniform Uncertainty Principles p.9/24

23 Greedy Algorithms: OMP ctd. At each iteration, OMP finds the largest component of u = Φ x and subtracts off that component s contribution. Greedy Signal Recovery and Uniform Uncertainty Principles p.10/24

24 Greedy Algorithms: OMP ctd. At each iteration, OMP finds the largest component of u = Φ x and subtracts off that component s contribution. For every fixed n-sparse v R d, and an N d Gaussian measurement matrix Φ, OMP recovers v with high probability, provided N n log d. Greedy Signal Recovery and Uniform Uncertainty Principles p.10/24

25 Comparing the approaches L 1 has uniform guarantees. Greedy Signal Recovery and Uniform Uncertainty Principles p.11/24

26 Comparing the approaches L 1 has uniform guarantees. OMP has no such known uniform guarantees. Greedy Signal Recovery and Uniform Uncertainty Principles p.11/24

27 Comparing the approaches L 1 has uniform guarantees. OMP has no such known uniform guarantees. L 1 is based on linear programming. Greedy Signal Recovery and Uniform Uncertainty Principles p.11/24

28 Comparing the approaches L 1 has uniform guarantees. OMP has no such known uniform guarantees. L 1 is based on linear programming. OMP is quite fast, both theoretically and experimentally. Greedy Signal Recovery and Uniform Uncertainty Principles p.11/24

29 Comparing the approaches This gap between the approaches leads us to our new algorithm, Regularized Orthogonal Matching Pursuit. Greedy Signal Recovery and Uniform Uncertainty Principles p.12/24

30 Comparing the approaches This gap between the approaches leads us to our new algorithm, Regularized Orthogonal Matching Pursuit. ROMP has polynomial running time. Greedy Signal Recovery and Uniform Uncertainty Principles p.12/24

31 Comparing the approaches This gap between the approaches leads us to our new algorithm, Regularized Orthogonal Matching Pursuit. ROMP has polynomial running time. ROMP provides uniform guarantees. Greedy Signal Recovery and Uniform Uncertainty Principles p.12/24

32 ROMP INPUT: Measurement vector x R N and sparsity level n Greedy Signal Recovery and Uniform Uncertainty Principles p.13/24

33 ROMP INPUT: Measurement vector x R N and sparsity level n OUTPUT: Index set I {1,..., d} Greedy Signal Recovery and Uniform Uncertainty Principles p.13/24

34 ROMP INPUT: Measurement vector x R N and sparsity level n OUTPUT: Index set I {1,..., d} Initialize: Set I =, r = x. Repeat until r = 0: Greedy Signal Recovery and Uniform Uncertainty Principles p.13/24

35 ROMP INPUT: Measurement vector x R N and sparsity level n OUTPUT: Index set I {1,..., d} Initialize: Set I =, r = x. Repeat until r = 0: Identify: Choose a set J of the n biggest coordinates in magnitude of u = Φ r. Greedy Signal Recovery and Uniform Uncertainty Principles p.13/24

36 ROMP ctd. Regularize: Among all subsets J 0 J with comparable coordinates: u(i) 2 u(j) for all i, j J 0, choose J 0 with the maximal energy u J0 2. Greedy Signal Recovery and Uniform Uncertainty Principles p.14/24

37 ROMP ctd. Regularize: Among all subsets J 0 J with comparable coordinates: u(i) 2 u(j) for all i, j J 0, choose J 0 with the maximal energy u J0 2. Update: the index set: I I J 0, and the residual: y = argmin x Φz 2 ; r = x Φy. z R I Greedy Signal Recovery and Uniform Uncertainty Principles p.14/24

38 Main Theorem Theorem: Stability under measurement perturbations. Assume a measurement matrix Φ satisfies the Restricted Isometry Condition with parameters (4n,.01/ log n). Greedy Signal Recovery and Uniform Uncertainty Principles p.15/24

39 Main Theorem Theorem: Stability under measurement perturbations. Assume a measurement matrix Φ satisfies the Restricted Isometry Condition with parameters (4n,.01/ log n). Let v be an n-sparse vector in R d. Greedy Signal Recovery and Uniform Uncertainty Principles p.15/24

40 Main Theorem Theorem: Stability under measurement perturbations. Assume a measurement matrix Φ satisfies the Restricted Isometry Condition with parameters (4n,.01/ log n). Let v be an n-sparse vector in R d. Consider corrupted x = Φv + e. Greedy Signal Recovery and Uniform Uncertainty Principles p.15/24

41 Main Theorem Theorem: Stability under measurement perturbations. Assume a measurement matrix Φ satisfies the Restricted Isometry Condition with parameters (4n,.01/ log n). Let v be an n-sparse vector in R d. Consider corrupted x = Φv + e. Then ROMP produces a good approximation to v: v ˆv 2 C log n e 2. Greedy Signal Recovery and Uniform Uncertainty Principles p.15/24

42 Corollary Stability of ROMP under signal perturbations. Assume a measurement matrix Φ satisfies the Restricted Isometry Condition with parameters (8n,.01/ log n). Greedy Signal Recovery and Uniform Uncertainty Principles p.16/24

43 Corollary Stability of ROMP under signal perturbations. Assume a measurement matrix Φ satisfies the Restricted Isometry Condition with parameters (8n,.01/ log n). Let v be an arbitrary vector in R d. Greedy Signal Recovery and Uniform Uncertainty Principles p.16/24

44 Corollary Stability of ROMP under signal perturbations. Assume a measurement matrix Φ satisfies the Restricted Isometry Condition with parameters (8n,.01/ log n). Let v be an arbitrary vector in R d. Consider corrupted x = Φv + e. Greedy Signal Recovery and Uniform Uncertainty Principles p.16/24

45 Corollary Stability of ROMP under signal perturbations. Assume a measurement matrix Φ satisfies the Restricted Isometry Condition with parameters (8n,.01/ log n). Let v be an arbitrary vector in R d. Consider corrupted x = Φv + e. Then ROMP produces a good approximation to v 2n : ˆv v 2n 2 C ( log n e 2 + v v n 1 ). n Greedy Signal Recovery and Uniform Uncertainty Principles p.16/24

46 Remarks In the noiseless case (e = 0), note that the theorem guarantees exact reconstruction. Greedy Signal Recovery and Uniform Uncertainty Principles p.17/24

47 Remarks In the noiseless case (e = 0), note that the theorem guarantees exact reconstruction. The runtime is polynomial: In the case of unstructured matrices, the runtime is O(dNn). Greedy Signal Recovery and Uniform Uncertainty Principles p.17/24

48 Remarks In the noiseless case (e = 0), note that the theorem guarantees exact reconstruction. The runtime is polynomial: In the case of unstructured matrices, the runtime is O(dNn). The theorem gives uniform guarantees of sparse recovery. Greedy Signal Recovery and Uniform Uncertainty Principles p.17/24

49 Remarks In the noiseless case (e = 0), note that the theorem guarantees exact reconstruction. The runtime is polynomial: In the case of unstructured matrices, the runtime is O(dNn). The theorem gives uniform guarantees of sparse recovery. ROMP succeeds with no prior knowledge about the error vector e. Greedy Signal Recovery and Uniform Uncertainty Principles p.17/24

50 Empirical Results 100 Percentage of input signals recovered exactly (d=256) (Gaussian) Percentage recovered n=4 20 n=12 n=20 10 n=28 n= Number of Measurements (N) Figure 1: Sparse flat signals with Gaussian matrix. Greedy Signal Recovery and Uniform Uncertainty Principles p.18/24

51 Empirical Results ctd % Recovery Trend, d= Sparsity Level Measurements N Figure 2: Sparse flat signals, Gaussian. Greedy Signal Recovery and Uniform Uncertainty Principles p.19/24

52 Empirical Results ctd. mber of Iterations needed to reconstruct sparse compressible signals (d=10,000, N=200) (Gau 6 Number of Iterations p= p=0.7 5 p=0.9 Flat Signal Sparsity Level (n) Figure 3: Number of Iterations. Greedy Signal Recovery and Uniform Uncertainty Principles p.20/24

53 Empirical Results ctd. Normalized Recovery Error from ROMP on Perturbed Measurements, d= Error to Noise Ratio n=4 n=12 n=20 n=28 n= Measurements N Figure 4: Error to noise ratio ˆv v 2 e 2. Greedy Signal Recovery and Uniform Uncertainty Principles p.21/24

54 Empirical Results ctd. Error to Noise Ratio Normalized Recovery Error from ROMP on Perturbed Signals, d=256 n=4 n=12 n=20 n=28 n= Measurements N Figure 5: Error to noise ratio ˆv v 2n 2 v v n 1 / n. Greedy Signal Recovery and Uniform Uncertainty Principles p.22/24

55 Future Work N-Tropp-Vershynin developing Compressive Sampling Matching Pursuit (CoSaMP) Greedy Signal Recovery and Uniform Uncertainty Principles p.23/24

56 Future Work N-Tropp-Vershynin developing Compressive Sampling Matching Pursuit (CoSaMP) Selects O(n) coordinates at each iteration but adds a signal estimation step using least squares. Then prunes this estimation to make sparse. Greedy Signal Recovery and Uniform Uncertainty Principles p.23/24

57 Future Work N-Tropp-Vershynin developing Compressive Sampling Matching Pursuit (CoSaMP) Selects O(n) coordinates at each iteration but adds a signal estimation step using least squares. Then prunes this estimation to make sparse. Same uniform guarantees as ROMP, but removes the log n term in the requirement for ε Greedy Signal Recovery and Uniform Uncertainty Principles p.23/24

58 Thank you! Questions? Greedy Signal Recovery and Uniform Uncertainty Principles p.24/24