Iterative Methods for Inverse & Ill-posed Problems

Transcription

1 Iterative Methods for Inverse & Ill-posed Problems Gerd Teschke Konrad Zuse Institute Research Group: Inverse Problems in Science and Technology InverseProblems ESI, Wien, December 2006

2 Outline 1 Scope of the problem 2 Linear Problems & Sparsity 3 Nonlinear Inverse Problems & Sparsity Setting Iteration Minimization Convergence Regularization Examples 4 Adaptivity for linear problems

3 Scope of the problem Scope of the Problem

4 Scope of the problem Computation of an approximation to a solution of T (x) = y where T : X Y and X, Y Hilbert spaces In many relevant cases only noisy data y δ with y δ y δ

5 Scope of the problem There is a very wide range of possible applications Image deblurring + decomposition (Daubechies,T. 05) Audio coding (T. 06) Sparseness (acceleration) of support vector machines (Rätsch,T. 05,06) SPECT (Ramlau,T. 06) Astrophysical data processing (Anthoine 05 + DeMol 04) Geophysics: seismic wave decomposition (Holschneider 06) Meteorological radar data processing (Lehmann,T. 06)...

6 Scope of the problem

7 Scope of the problem

8 Scope of the problem

9 Scope of the problem Mathematical description: div(σ Φ) = divj in Ω σ Φ, n = 0 at Γ = Ω, Inverse problem: R : (σ, j) Φ Ω Variational formulation: J(σ, j) = R(σ, j) Φ δ Ω 2 + αψ(σ, σ, j)

10 Scope of the problem

11 Scope of the problem Linear case: ill posed integral equation of the first kind ( ) Rf (s, ω) = f (sω + tω IL (s, ω) )dt = log I 0 (s, ω) R Nonlinear case: (SPECT) R[f, µ](s, ω) = f (sω + tω )e R t µ(sω+τω )dτ dt R Consider J(f, µ) = y δ R[f, µ] 2 + αψ(f, µ)

12 Scope of the problem

13 Scope of the problem Sparse Approximation of set vectors x i Reduced SVM sparse SVM Sparsifying both simultaneously: Ψ 1 (α, x) Ψ(β, z) 2 + Sparsity(β) + Sparsity(z) where Ψ 1 (α, x) = N x i=1 α iφ(x i )

14 Scope of the problem

15 Linear Inverse Problems & Sparsity Linear Problems & Sparsity

16 Linear Inverse Problems & Sparsity Signal representation v maybe represented by a preassigned basis But sometimes too restrictive - way out: frame But sometimes too restrictive - way out: dictionary of frames,...

17 Linear Inverse Problems & Sparsity Sparsity Constraints Certain physical constraints, e.g. well-known energy norm y AF g 2 + α g 2 promotion of sparsity 0 < p < 2, y AF g 2 + α g p l p more general y Av 2 + α sup v, h h C

18 Linear Inverse Problems & Sparsity Sparsity Constraints

19 Linear Inverse Problems & Sparsity Sparsity Constraints and Iterative Process Consider for instance: y AF g 2 + α g l1 Problem: AF g 2 induce a nonlinear coupling Way out: y AF g 2 + α g l1 + C g a 2 AF (g a) 2 = 2 g, FA y + α g l1 + C g a g, FA AF a y 2 AF a 2

20 Linear Inverse Problems & Sparsity Sparsity Constraints and Iterative Process Define J(g, a) := y AF g 2 +α g l1 +C g a 2 AF (g a) 2 Create an iteration process by setting a = g 0 and g m+1 = arg min g J(g, g m )

21 Linear Inverse Problems & Sparsity Sparsity Constraints and Minimization Reduces to variational equations of the form: a = b α sign(a) Solved by b α a = S α (b) = b + α b = 0 b α b α α < b < α In its full glory g m+1 = S α (FA y + g m FA AF g m )

22 Linear Inverse Problems & Sparsity Provided analysis Daubechies+Defrise+DeMol 2003: minimization by Gaussian surrogate functionals iterative Landweber approach proof of norm convergence and regularization properties general case: y Av 2 + 2α sup v, h h C minimization, norm convergence, (regularization theory) Daubechies + T. + Vese 2006

23 Linear Inverse Problems & Sparsity Well-posed case Theorem (Daubechies/T./Vese 06) Suppose some technical conditions on C, and A A has bounded inverse in its range. If we define T := (A A) 1/2 and, for an arbitrary closed convex set K, S K := Id P K, where P K is the (nonlinear) projection on K, then the minimizing v is given by v = TS αtc {TA y}.

24 Linear Inverse Problems & Sparsity Ill-posed case, convergence Gaussian surrogate approach yields: v n+1 := (Id P αc )(v n + A y A Av n ) by same techniques as in D : v n weak v norm convergence requires special knowledge of C!!!

25 Linear Inverse Problems & Sparsity Ill-posed case, convergence Theorem (Daubechies/T./Vese 06) Suppose v n v weak 0 and P αc (g) P αc (g + v n v) 0. Moreover, assume that v n is orthogonal to g, P C (g). If for some sequence γ n (with γ n ) the convex set C satisfies γ n (v n v) C. then v n v 0

26 Linear Inverse Problems & Sparsity Left: Shepp-Logan Phatom (64x64), right: FBP (0:10:180)

27 Linear Inverse Problems & Sparsity (... here is the movie theater)

28 Linear Inverse Problems & Sparsity

29 Nonlinear Inverse Problems & Sparsity Nonlinear Problems & Sparsity

30 Nonlinear Inverse Problems & Sparsity Setting The setting Nonlinear problem: T : X Y, T (x) = y Variational form (vector valued) J α (g 1,..., g n ) = y δ T (g 1,..., g n ) 2 + 2αΨ(g 1,..., g n )

31 Nonlinear Inverse Problems & Sparsity Setting The setting Requirements on T (essentially): T strongly continuous T L - Lipschitz continuous Further requirements g (l2 ) n cψ(g)... technical conditions

32 Nonlinear Inverse Problems & Sparsity Setting Linear mixing: { r } T (Kg) = T A l,i F g l l=1 i=1,...,n Simple cases: Nonlinear scalar valued: ( n ) T (Kg) = T (K(g 1,..., g n )) = T F g i i=1 Purely linear: T some linear and bounded operator.

33 Nonlinear Inverse Problems & Sparsity Setting Non coupled sparsity (T. 05) Ψ(g) = (Ψ 1 (g 1 ),..., Ψ n (g n )) Joint sparsity (linear case: Fornasier/Rauhut 06) Ψ(u) = λ Λ ω λ u λ q Complementary sparsity,...

34 Nonlinear Inverse Problems & Sparsity Iteration Basic Idea For g (l 2 ) n and some auxiliary a (l 2 ) n, consider J s α(g, a) := J α (g) + C g a 2 (l 2 ) n T (g) T (a) 2 Y Create an iteration process 1 Pick g 0 (l 2 ) n and some proper constant C > 0 2 Derive a sequence {g k } k=0,1,... by the iteration: g k+1 = arg min g k (l 2 ) Js α(g, g k ) k = 0, 1, 2,... n

35 Nonlinear Inverse Problems & Sparsity Iteration Proper Surrogate Functionals Given multi parameter α R + and g 0 (l 2 ) n, define a ball with radius r = J α (g 0 )/(2α) Define C ( C := 2 max K r := {g (l 2 ) n : Ψ(g) r} sup T (g) g K r ) 2, L J(g 0 )

36 Nonlinear Inverse Problems & Sparsity Iteration Proper Surrogate Functionals Properties: C g g 0 2 (l 2 ) n T (g) T (g 0 ) 2 Y 0 All J s α(g, g k ) are bounded from below, g k K r All J α (g k ) and J s α(g k+1, g k ) are non-increasing

37 Nonlinear Inverse Problems & Sparsity Minimization Necessary Condition The necessary condition for a minimum of J s α(g, a) is given by 0 T (g) (y δ T (a)) + Cg Ca + α Ψ(g)

38 Nonlinear Inverse Problems & Sparsity Minimization Recasting the Necessary Condition Let M(g, a) := T (g) (y δ T (a))/c + a then the necessary conditions can be casted as fixed point problem g = α ( ) C C (I P C) M(g, a), α where P C is the orthogonal projection onto the convex set C.

39 Nonlinear Inverse Problems & Sparsity Minimization Fixed Point Iteration with Projection Lemma The fixed point iteration converges towards the minimizer of J s α(g, g k ) Lemma T C 2 : J s α(g, g k ) is strictly convex.

40 Nonlinear Inverse Problems & Sparsity Minimization Joint Sparsity Measure: Ψ(u) = λ Λ ω λ u λ q Fixed point iteration: g l+1 = α ( ) C (I P M(gl, a) C) α C Equivalent description: g l+1 M(g l, a) 2 (l 2 ) n + 2α/C λ Λ (g λ ) l+1 q

41 Nonlinear Inverse Problems & Sparsity Minimization Joint Sparsity Proposition Let 1 q and 1 = 1/q + 1/q. The coefficients of iterates of the fixed point equation are given by (g λ ) l+1 = (g 1 λ,..., g n λ ) l+1 = (I P B q (C 1 αω λ ) )((M(g l, a)) λ ).

42 Nonlinear Inverse Problems & Sparsity Convergence Convergence Theorem Assume that there exists at least one isolated limit g α of a subsequence g k,l of g k. Then g k g α as k. The accumulation point g α is a minimizer for the functional J s α(g, g α) and satisfies the necessary condition for a minimum of J α.

43 Nonlinear Inverse Problems & Sparsity Regularization Regularization Theorem Let α(δ) δ 0 0, δ 2 /α(δ) δ 0 0. Then every sequence {g α(δ) } of minimizers of the functional J α(g) where δ 0 and α = α(δ) has a convergent subsequence. The limit of every convergent subsequence is a solution of T (g) = y with minimal values of Ψ(g).

44 Nonlinear Inverse Problems & Sparsity Examples X y y δ F * 1 g F * 2 g K g discr(red), penalty(green) sparsity 10 0 err J α Add J α (red), J α +Add (blue)

45 Nonlinear Inverse Problems & Sparsity Examples X 1.5 Y+δ 1 G discr(red), penalty(green) x sparsity err J α Add J α (red), J α +Add (blue)

46 Nonlinear Inverse Problems & Sparsity Examples R[f, µ](s, ω) = R f (sω + tω )e R t µ(sω+τω )dτ dt Left: density f, right: attenuation µ

47 Nonlinear Inverse Problems & Sparsity Examples R[f, µ](s, ω) = R f (sω + tω )e R t µ(sω+τω )dτ dt Simulated data R[f, µ]

48 Nonlinear Inverse Problems & Sparsity Examples Reconstruction of density f (3 percent error)

49 Nonlinear Inverse Problems & Sparsity Examples Reduced Support Vector Machines Cascade Classification

50 Iterative Methods for Inverse & Ill-posed Problems Nonlinear Inverse Problems & Sparsity Examples Input image, images showing the amount of rejected pixels at the 1st, 3rd and 50th stages of the cascade

51 Nonlinear Inverse Problems & Sparsity Examples Reduced Support Vector Machines Percentage of rejected non-face patches as a function of the number of operations required

52 Nonlinear Inverse Problems & Sparsity Examples method SVM RVM W-RVM time per patch µs 22.51µs 1.48µs Comparison of speed improvement of the W-RVM to the RVM and SVM

53 Nonlinear Inverse Problems & Sparsity Examples Drawback - High Computational Complexity Adaptivity!

54 Adaptivity for linear problems Frame based concept (Stevenson, Dahlke et.al.) for positive operators Consider the regularized problem Construction of RHS, APPLY, COARSE routines

55 Adaptivity for linear problems Operator s admissibility New definition of s compressibility (NEW: density function) New routine APPLY = if our operator fulfills the new s compressibility, then with the new APPLY routine our operator is s admissible

56 Adaptivity for linear problems Verification for the linear Radon transform 2 λ λ (R R + α)φ λ, φ λ c min( λ, λ ) δ(λ, λ ) remember Lemarié class: 2 σ λ λ (1 + 2 min( λ, λ ) δ(λ, λ )) β with β > n, σ > n/2

57 Adaptivity for linear problems see Matlab images...