Sparse analysis Lecture III: Dictionary geometry and greedy algorithms

Transcription

1 Sparse analysis Lecture III: Dictionary geometry and greedy algorithms Anna C. Gilbert Department of Mathematics University of Michigan

2 Intuition from ONB Key step in algorithm: r, ϕ j = x c i ϕ i, ϕ j = x, ϕ j c i ϕ i, ϕ j

3 Coherence Definition The coherence of a dictionary µ = max j l ϕ j, ϕ l

4 Coherence Definition The coherence of a dictionary µ = max j l ϕ j, ϕ l φ1 φ1 φ2 φ3 φ2 φ3 Small coherence (good) Large coherence (bad)

5 Coherence: lower bound Theorem For a d N dictionary, N d µ d(n 1) 1. d [Welch 73] Theorem For most pairs of orthonormal bases in R d, the coherence between the two is ( log d ) µ = O. d [Donoho, Huo 99]

6 Large, incoherent dictionaries Fourier Dirac, N = 2d, µ = 1 d wavelet packets, N = d log d, µ = 1 2 There are large dictionaries with coherence close to the lower (Welch) bound; e.g., Kerdock codes, N = d 2, µ = 1/ d

7 Approximation algorithms (error) Sparse. Given k 1, solve arg min x Φc 2 s.t. c 0 k c i.e., find the best approximation of x using k atoms. c opt = optimal solution E opt = Φc opt x 2 = optimal error

8 Approximation algorithms (error) Sparse. Given k 1, solve arg min x Φc 2 s.t. c 0 k c i.e., find the best approximation of x using k atoms. c opt = optimal solution E opt = Φc opt x 2 = optimal error Algorithm returns ĉ with (1) ĉ 0 = k (2) E = Φĉ x 2 C 1 E opt

9 Approximation algorithms (error) Sparse. Given k 1, solve arg min x Φc 2 s.t. c 0 k c i.e., find the best approximation of x using k atoms. c opt = optimal solution E opt = Φc opt x 2 = optimal error Algorithm returns ĉ with (1) ĉ 0 = k (2) E = Φĉ x 2 C 1 E opt (Error) approximation ratio: E E opt = C 1E opt E opt = C 1

10 Approximation algorithms (terms) Algorithm returns ĉ with (1) ĉ 0 = C 2 k (2) E = Φĉ x 2 = E opt (Terms) approximation ratio: ĉ 0 c opt 0 = C 2k k = C 2

11 Bi-criteria approximation algorithms Algorithm returns ĉ with (1) ĉ 0 = C 2 k (2) E = Φĉ x 2 = C 1 E opt (Terms, Error) approximation ratio: (C 2, C 1 )

12 Greedy algorithms Build approximation one step at a time...

13 Greedy algorithms Build approximation one step at a time......choose the best atom at each step

14 Orthogonal Matching Pursuit OMP [Mallat 92], [Davis 97] Input. Dictionary Φ, signal x, steps k Output. Coefficient vector c with k nonzeros, Φc x

15 Orthogonal Matching Pursuit OMP [Mallat 92], [Davis 97] Input. Dictionary Φ, signal x, steps k Output. Coefficient vector c with k nonzeros, Φc x Initialize. counter t = 1, residual r 0 = x, c = 0

16 Orthogonal Matching Pursuit OMP [Mallat 92], [Davis 97] Input. Dictionary Φ, signal x, steps k Output. Coefficient vector c with k nonzeros, Φc x Initialize. counter t = 1, residual r 0 = x, c = 0 1. Greedy selection. Find atom ϕ jt s.t. j t = argmax l r t 1, ϕ l

17 Orthogonal Matching Pursuit OMP [Mallat 92], [Davis 97] Input. Dictionary Φ, signal x, steps k Output. Coefficient vector c with k nonzeros, Φc x Initialize. counter t = 1, residual r 0 = x, c = 0 1. Greedy selection. Find atom ϕ jt s.t. j t = argmax l r t 1, ϕ l 2. Update. Find c l1,..., c lt to solve min x c ls ϕ ls s 2

18 Orthogonal Matching Pursuit OMP [Mallat 92], [Davis 97] Input. Dictionary Φ, signal x, steps k Output. Coefficient vector c with k nonzeros, Φc x Initialize. counter t = 1, residual r 0 = x, c = 0 1. Greedy selection. Find atom ϕ jt s.t. j t = argmax l r t 1, ϕ l 2. Update. Find c l1,..., c lt to solve min x c ls ϕ ls s 2 new residual r t x Φc

19 Orthogonal Matching Pursuit OMP [Mallat 92], [Davis 97] Input. Dictionary Φ, signal x, steps k Output. Coefficient vector c with k nonzeros, Φc x Initialize. counter t = 1, residual r 0 = x, c = 0 1. Greedy selection. Find atom ϕ jt s.t. j t = argmax l r t 1, ϕ l 2. Update. Find c l1,..., c lt to solve min x c ls ϕ ls s 2 new residual r t x Φc 3. Iterate. t t + 1, stop when t > k.

20 Many greedy algorithms with similar outline Matching Pursuit: replace step 2. by c lt c lt + r t 1, ϕ kt Thresholding Choose m atoms where x, ϕ l are among m largest Alternate stopping rules: r t 2 ɛ max l r t, ϕ l ɛ Many other variations

21 Convergence of OMP Theorem Suppose Φ is a complete dictionary for R d. For any vector x, the residual after t steps of OMP satisfies r t 2 c 1 t. [Devore-Temlyakov]

22 Convergence of OMP Theorem Suppose Φ is a complete dictionary for R d. For any vector x, the residual after t steps of OMP satisfies r t 2 c 1 t. [Devore-Temlyakov] Even if x can be expressed sparsely, OMP may take d steps before the residual is zero.

23 Convergence of OMP Theorem Suppose Φ is a complete dictionary for R d. For any vector x, the residual after t steps of OMP satisfies r t 2 c 1 t. [Devore-Temlyakov] Even if x can be expressed sparsely, OMP may take d steps before the residual is zero. But, sometimes OMP correctly identifies sparse representations.

24 Sparse representation with OMP Suppose x has k-sparse representation x = l Λ c l ϕ l where Λ = k i.e., c opt is non-zero on Λ.

25 Sparse representation with OMP Suppose x has k-sparse representation x = l Λ c l ϕ l where Λ = k i.e., c opt is non-zero on Λ. Sufficient to find Λ When can OMP do so?

26 Sparse representation with OMP Suppose x has k-sparse representation x = l Λ c l ϕ l where Λ = k i.e., c opt is non-zero on Λ. Sufficient to find Λ When can OMP do so? Define Φ Λ = [ ] ϕ l1 ϕ l2 ϕ lk and l s Λ Ψ Λ = [ ] ϕ l1 ϕ l2 ϕ ln k l s / Λ

27 Sparse representation with OMP Suppose x has k-sparse representation x = l Λ c l ϕ l where Λ = k i.e., c opt is non-zero on Λ. Sufficient to find Λ When can OMP do so? Define Φ Λ = [ ] ϕ l1 ϕ l2 ϕ lk and l s Λ Ψ Λ = [ ] ϕ l1 ϕ l2 ϕ ln k l s / Λ Define greedy selection ratio ρ(r) = max l/ Λ r, ϕ l max l Λ r, ϕ l = Ψ T Λ r Φ T Λ r = max i.p. bad atoms max i.p. good atoms

28 Sparse representation with OMP Suppose x has k-sparse representation x = l Λ c l ϕ l where Λ = k i.e., c opt is non-zero on Λ. Sufficient to find Λ When can OMP do so? Define Φ Λ = [ ] ϕ l1 ϕ l2 ϕ lk and l s Λ Ψ Λ = [ ] ϕ l1 ϕ l2 ϕ ln k l s / Λ Define greedy selection ratio ρ(r) = max l/ Λ r, ϕ l max l Λ r, ϕ l = Ψ T Λ r Φ T Λ r = OMP chooses good atom iff ρ(r) < 1 max i.p. bad atoms max i.p. good atoms

29 Exact Recovery Condition Theorem (ERC) A sufficient condition for OMP to identify Λ after k steps is that Φ + Λ ϕ l 1 < 1 max l/ Λ where A + = (A T A) 1 A T. [Tropp 04]

30 Exact Recovery Condition Theorem (ERC) A sufficient condition for OMP to identify Λ after k steps is that Φ + Λ ϕ l 1 < 1 max l/ Λ where A + = (A T A) 1 A T. [Tropp 04] A + x is a coefficient vector that synthesizes best approximation of x using atoms in A. P = AA + orthogonal projector produces this best approximation

31 Proof. r 0 = x range(φ Λ )

32 Proof. r 0 = x range(φ Λ ) At iteration t + 1 assume l 1, l 2,..., l t Λ, thus a t range(φ Λ ) and r t = x a t range(φ Λ )

33 Proof. r 0 = x range(φ Λ ) At iteration t + 1 assume l 1, l 2,..., l t Λ, thus a t range(φ Λ ) and r t = x a t range(φ Λ ) Express orthogonal projector onto range(φ Λ ) as (Φ + Λ )T Φ T Λ, therefore (Φ + Λ )T Φ T Λ rt = rt.

34 Proof. r 0 = x range(φ Λ ) At iteration t + 1 assume l 1, l 2,..., l t Λ, thus a t range(φ Λ ) and r t = x a t range(φ Λ ) Express orthogonal projector onto range(φ Λ ) as (Φ + Λ )T Φ T Λ, therefore (Φ + Λ )T Φ T Λ rt = rt. Bound Ψ T Λ rt ρ(r t ) = = Φ T Λ rt Ψ T Λ (Φ+ Λ )T = Φ + Λ Ψ Λ 1 = max Φ + Λ ϕ l < 1 l / Λ 1 Ψ T Λ (Φ+ Λ )T Φ T Λ rt Φ T Λ rt

35 Proof. r 0 = x range(φ Λ ) At iteration t + 1 assume l 1, l 2,..., l t Λ, thus a t range(φ Λ ) and r t = x a t range(φ Λ ) Express orthogonal projector onto range(φ Λ ) as (Φ + Λ )T Φ T Λ, therefore (Φ + Λ )T Φ T Λ rt = rt. Bound Ψ T Λ rt ρ(r t ) = = Φ T Λ rt Ψ T Λ (Φ+ Λ )T = Φ + Λ Ψ Λ 1 = max Φ + Λ ϕ l < 1 l / Λ 1 Ψ T Λ (Φ+ Λ )T Φ T Λ rt Φ T Λ rt Then OMP selects an atom from Λ at iteration t and since it chooses a new atom at each iteration,

36 Proof. r 0 = x range(φ Λ ) At iteration t + 1 assume l 1, l 2,..., l t Λ, thus a t range(φ Λ ) and r t = x a t range(φ Λ ) Express orthogonal projector onto range(φ Λ ) as (Φ + Λ )T Φ T Λ, therefore (Φ + Λ )T Φ T Λ rt = rt. Bound Ψ T Λ rt ρ(r t ) = = Φ T Λ rt Ψ T Λ (Φ+ Λ )T = Φ + Λ Ψ Λ 1 = max Φ + Λ ϕ l < 1 l / Λ 1 Ψ T Λ (Φ+ Λ )T Φ T Λ rt Φ T Λ rt Then OMP selects an atom from Λ at iteration t and since it chooses a new atom at each iteration, After k iterations, chosen all atoms from Λ.

37 Coherence Bounds Theorem The ERC holds whenever k < 1 2 (µ 1 + 1). Therefore, OMP can recover any sufficiently sparse signals. [Tropp 04] For most redundant dictionaries, k < 1 2 ( d + 1).

38 Sparse Theorem Assume k 1 3 µ 1. For any vector x, the approximation Φĉ after k steps of OMP satisfies ĉ 0 k and x Φĉ k x Φc opt 2 where c opt is the best k-term approximation to x over Φ. [Tropp 04] Theorem Assume 4 k 1 µ. Two-phase greedy pursuit produces x = Φĉ s.t. x x 2 3 x Φc opt 2. Assume k 1 µ. Two-phase greedy pursuit produces x = Φĉ s.t. x x 2 ( 2µk 2 ) 1 + (1 2µk) 2 x Φc opt 2. [Gilbert, Strauss, Muthukrishnan, Tropp 03]

39 Summary Geometry of dictionary is important but get sufficient conditions on the geometry of the dictionary to solve Sparse problems efficiently. Algorithms are approximation algorithms (wrt error). Algorithmic architecture is greedy pursuit. Next lecture: other algorithmic formulations + solutions