Multilevel Preconditioning and Adaptive Sparse Solution of Inverse Problems

Transcription

1 Multilevel and Adaptive Sparse of Inverse Problems Fachbereich Mathematik und Informatik Philipps Universität Marburg Workshop Sparsity and Computation, Bonn, (joint work with M. Fornasier and T. Raasch)

2 Outline

3 Outline

4 The Problem: treatment of inverse problems y = Ku + e, K : X Y, X, Y Hilbert spaces Minimization of functionals: J(u) := Ku y 2 Y + 2 ( u, ψ λ ) λ I l1,α (I), u lp,α := ( u λ p α λ ) 1/p λ I Ψ := {ψ λ } λ I (wavelet) basis Ψ := { ψ λ } λ I dual basis

5 The Problem: treatment of inverse problems y = Ku + e, K : X Y, X, Y Hilbert spaces Minimization of functionals: J(u) := Ku y 2 Y + 2 ( u, ψ λ ) λ I l1,α (I), u lp,α := ( u λ p α λ ) 1/p λ I Ψ := {ψ λ } λ I (wavelet) basis Ψ := { ψ λ } λ I dual basis

6 The Problem: equivalent formulation: F u := λ I u λ ψ λ, u l 2 (I) J(u) := J α (u) = (K F )u y 2 Y + 2 u l1,α (I), A := K F several iterative methods available: (a) the GPSR-algorithm (b) the l 1 l s algorithm, (c) FISTA (fast iterative soft-thresholding algorithm) (d) LARS...

7 The Problem: equivalent formulation: F u := λ I u λ ψ λ, u l 2 (I) J(u) := J α (u) = (K F )u y 2 Y + 2 u l1,α (I), A := K F several iterative methods available: (a) the GPSR-algorithm (b) the l 1 l s algorithm, (c) FISTA (fast iterative soft-thresholding algorithm) (d) LARS...

8 The Approach: Iterated Soft- Algorithm (ISTA) [Daubechies/DeFrise/DeMol] and others... u (n+1) = S α [u (n) + A y A Au (n)], x τ x > τ S τ (x) = 0 x τ x + τ x < τ ( S α (a) = arg min u a 2 ) + 2 u 1,α u l 2 (I)

9 Problems... slow convergence: The path in the x 1 vs. Kx y 2 plane Acceleration by: decreasing thresholding parameters adaptivity multilevel preconditioning

10 Outline

11 : decreasing iterative soft-thresholding algorithm (D-ISTA): u (n+1) ( = S α (n) u (n) + A (y Au (n) ) ), α (n) λ α λ Restricted Isometry Property (RIP): (1 γ k ) u Λ 2 l 2 A Λ u Λ Y (1 + γ k ) u Λ 2 l 2, for all Λ {1,..., N} with #Λ k Theorem The following conditions are equivalent: (i) A has RIP property (ii) A A Λ Λ is positive definite, eigenvalues in [1 γ k, 1 + γ k ], for all Λ {1,..., N} with #Λ k; (iii) (I A A) Λ Λ γ k, for all Λ {1,..., N} with #Λ k.

12 It works... Theorem Let ū := (I A A)u + A y l w τ (I), 0 < τ < 2. Moreover, let L := 4 u 2 l 2 (I) + 4C ū τ ᾱ 2 l w τ (I)ᾱ τ. RIP of order 2L + #suppu with γ 0 < 1 satisfied. Whenever for γ 0 γ < 1 α λ α (n) λ α λ + (γ γ 0 )L 1/2 ɛ n, for all λ Λ, then #supp u (n) L and u u (n) l2 (I) γ n u l2 (I) =: ɛ n

13 A Comparison: log 10 ( A u n y 2 2 ) Dynamics of the algorithms u n 1 ISTA D ISTA Sparse minimizer u * and approximations due to the algorithms ISTA D ISTA u * Error with respect to the minimizer of J ISTA D ISTA Dynamics of the support sizes of the approximations to u * due to the algorithms 3.5 ISTA 3 D ISTA u * log 10 ( u n u * 2 ) log 10 ( u n 0 ) Number of iterations Number of iterations A : matrix with i.i.d. Gaussian entries, α = 10 3, γ 0 = 0.1 and γ = 0.95.

14 Outline

15 : Typical application: A solution operator of an operator equation Problem: D-ISTA not directly implementable! We need suitable approximations! Use adaptive strategies.

16 : Typical application: A solution operator of an operator equation Problem: D-ISTA not directly implementable! We need suitable approximations! Use adaptive strategies.

17 : Typical application: A solution operator of an operator equation Problem: D-ISTA not directly implementable! We need suitable approximations! Use adaptive strategies.

18 : Typical application: A solution operator of an operator equation Problem: D-ISTA not directly implementable! We need suitable approximations! Use adaptive strategies.

19 Building Blocks: RHS[g, ε] g ε : determines finitely supported g ε s.t. g g ε l2 (I) ε; APPLY[N, v, ε] w ε : determines finitely supported w ε s.t Nv w ε l2 (I) ε; Realization: [Cohen/Dahmen/DeVore] Implementable algorithm A-ISTA: ũ (n+1) = S (ũ(n) α (n) +RHS[A y, δ n ] APPLY[A A, ũ (n), γ n ] )

20 It works... Theorem Technical assumptions (RIP etc.) If δ n = γ n = ɛ n+1 2ρ, ɛ n = γ n u l2 (I), 0 < γ 0 γ < γ < 1, α (n) are chosen according to α λ α (n) λ α λ + (γ γ 0 )L 1/2 ɛ n, then the iterates of A-ISTA fulfill #supp ũ (n) L and u ũ (n) l2 (I) ɛ n.

21 Outline

22 : How can RIP be guaranteed?! X L 2 (Ω) X, Ψ = {ψ λ } λ I Assumptions: wavelet basis K : X L 2 (Ω), K 2 s λ µ 2 σ( λ + µ ) Kψ λ, ψ µ c 1 ( min( λ, µ ) dist(ω µ, Ω λ ) ) r For µ = λ For µ = λ K Kψ λ, ψ λ c 2 2 2σ λ K Kψ λ, ψ µ c 3 2 2σ λ (1 + k k ) r

23 Theorem Let A A = F K KF = ( K Kψ λ, ψ µ ) λ,µ I. Let Dj b = ( K Kψ λ, ψ µ ) λ = µ =j be the diagonal block of A A for refinement level j, D b = (D0 b, Db 1,...). Then (I (D b ) 1/2 A A(D b ) 1/2 ) Λ Λ < C2 s Λ and K((D b ) 1/2 A A (D b ) 1/2 Λ Λ ) 1 + C 2 s Λ 1 C 2 s Λ. increase of s = larger sets Λ (D b ) 1/2 A A (D b ) 1/2 not globally well-conditioned! in practice: diagonal preconditioning works well!

24 Theorem Let A A = F K KF = ( K Kψ λ, ψ µ ) λ,µ I. Let Dj b = ( K Kψ λ, ψ µ ) λ = µ =j be the diagonal block of A A for refinement level j, D b = (D0 b, Db 1,...). Then (I (D b ) 1/2 A A(D b ) 1/2 ) Λ Λ < C2 s Λ and K((D b ) 1/2 A A (D b ) 1/2 Λ Λ ) 1 + C 2 s Λ 1 C 2 s Λ. increase of s = larger sets Λ (D b ) 1/2 A A (D b ) 1/2 not globally well-conditioned! in practice: diagonal preconditioning works well!

25 Theorem Let A A = F K KF = ( K Kψ λ, ψ µ ) λ,µ I. Let Dj b = ( K Kψ λ, ψ µ ) λ = µ =j be the diagonal block of A A for refinement level j, D b = (D0 b, Db 1,...). Then (I (D b ) 1/2 A A(D b ) 1/2 ) Λ Λ < C2 s Λ and K((D b ) 1/2 A A (D b ) 1/2 Λ Λ ) 1 + C 2 s Λ 1 C 2 s Λ. increase of s = larger sets Λ (D b ) 1/2 A A (D b ) 1/2 not globally well-conditioned! in practice: diagonal preconditioning works well!

26 Examples: integral operators with Schwartz kernels Ku(x) = Φ(x, ξ)u(ξ)dξ, x Ω, Ω Ω, Ω R d, dist(ω, Ω) = δ > 0, u X := H t (Ω) Φ : Ω Ω α x β ξ Φ(x, ξ) c α,β x ξ (d+2t+ α + β )

27 Magnetic Tomography: Current Sensor Biot-Savart operator: B(x, j) = µ 0 j(ξ) (x ξ) 4π Ω x ξ 3 dξ = Φ(x, ξ) = µ 0 1 4π x ξ, x ξ j := current density Ω [ x Φ(x, ξ)] j(ξ)dξ

28 Outline

29 : Volterra integral operator K : L 2 (0, 1) L 2 (0, 1), Ku(t) = u(x) = t 0 u(s) ds, K Ku(t) = 1 24x + 1, 0 x < x , 16 x < x , 1 2 x < , otherwise 0 ( 1 max(s, t) ) u(s) ds level j k

30 : no prec. diag. prec. b. diag. prec. cond 2 (A T * AT ) #random columns

31 Linear Convergence: l 2 errors, alpha=1e-05, alphaprec=1e-05, gamma=0.99, eta=0.1 ISTA P-ISTA D-ISTA PD-ISTA l 2 error 1e-06 1e-08 1e-10 1e iteration

32 Linear Convergence: l 2 errors, alpha=1e-06, alphaprec=1e-06, gamma=0.99, eta=0.1 ISTA P-ISTA D-ISTA PD-ISTA l 2 error iteration

33 Support Dynamics: active coefficients, alpha=1e-06, alphaprec=1e-06, gamma=0.99, eta=0.1 ISTA P-ISTA D-ISTA PD-ISTA active coefficients iteration

34 Outline

35 : treatment of inverse problems, minimization of associated functionals thresholding algorithms Adaptive strategies experiments

36 : treatment of inverse problems, minimization of associated functionals thresholding algorithms Adaptive strategies experiments

37 : treatment of inverse problems, minimization of associated functionals thresholding algorithms Adaptive strategies experiments

38 : treatment of inverse problems, minimization of associated functionals thresholding algorithms Adaptive strategies experiments

39 : treatment of inverse problems, minimization of associated functionals thresholding algorithms Adaptive strategies experiments

40 Reformulation after : Observe that: J(u) = Au y 2 Y + 2α u l1 (I) = AD 1/2 D} 1/2 {{ u} y 2 Y + 2α D 1/2 D} 1/2 {{ u} :=z :=z l1 (I) = AD 1/2 z y 2 Y + 2α D 1/2 z l1 (I) := J D (z). Hence, ( ) argmin u l2 (I)J(u) = D 1/2 argmin z D 1/2 l 2 (I) J D (z).

41 Adptivity helps again... to control the supports of the iterands! Theorem We define z := z + (D 1/2 A y D 1/2 A AD 1/2 z ) l 2 (I) and Λ δ ( z) = {λ : δ < z λ + (D 1/2 A y D 1/2 A AD 1/2 z ) λ }, We set Λ = N n=0λ (n) supp z Λ δ ( z), Λ (n) = supp ( z (n) ), where N N is such that z z (N) ɛ N ε, Then for δ(ɛ) sufficiently small Λ (n) Λ, for all n 0.