Inverse problems Total Variation Regularization Mark van Kraaij Casa seminar 23 May 2007 Technische Universiteit Eindh ove n University of Technology

Transcription

1 Inverse problems Total Variation Regularization Mark van Kraaij Casa seminar 23 May 27

2 Introduction Fredholm first kind integral equation of convolution type in one space dimension: g(x) = 1 k(x x )f(x )dx = (Kf)(x), < x < 1, k(x) = C exp( x 2 /2γ 2 ). (Gaussian kernel)

3 Introduction Fredholm first kind integral equation of convolution type in one space dimension: g(x) = 1 k(x x )f(x )dx = (Kf)(x), < x < 1, k(x) = C exp( x 2 /2γ 2 ). (Gaussian kernel) Direct problem: Given the source f and the kernel k, determine the blurred image g. For a piecewise smooth source and smooth kernel, an accurate approximation of g = Kf is found using standard numerical quadrature.

4 Introduction Fredholm first kind integral equation of convolution type in one space dimension: g(x) = 1 k(x x )f(x )dx = (Kf)(x), < x < 1, k(x) = C exp( x 2 /2γ 2 ). (Gaussian kernel) Inverse problem: Given the blurred image g and the kernel k, determine the source f. Discretize equation using collocation in the independent variable x and quadrature in x to obtain a discrete linear system Kf = d.

5 Introduction Composite midpoint quadrature with h = 1/n: ( f = K 1 ((i ) j)h)2 d, where K ij = h C exp. 2γ 2 To obtain an accurate quadrature approximation, n must be relatively large. Unfortunately, matrix K becomes increasingly ill-conditioned for large n.

6 Introduction Composite midpoint quadrature with h = 1/n: ( f = K 1 ((i ) j)h)2 d, where K ij = h C exp. 2γ 2 To obtain an accurate quadrature approximation, n must be relatively large. Unfortunately, matrix K becomes increasingly ill-conditioned for large n. Errors due to quadrature can be controlled, errors in d may be amplified! Source function f Blurred image g Discrete noisy data d Figure 1: γ =.5, C = 1 γ 2π, n = 8

7 Regularization by filtering Assume a discrete data model for the discrete linear system Kf = d, i.e. d = Kf true + η, with δ := η > the error level. Assuming K is invertible the SVD (K = U diag(s i )V T ) gives n K 1 d = f true + s 1 i (u T i η)v i. i=1

8 Regularization by filtering Assume a discrete data model for the discrete linear system Kf = d, i.e. d = Kf true + η, with δ := η > the error level. Assuming K is invertible the SVD (K = U diag(s i )V T ) gives n K 1 d = f true + s 1 i (u T i η)v i. i=1 Instabilities arise due to division by small singular values. Use regularizing filter function w α (s 2 i ) for which the product w α (s 2 )s 1 as s. Approximate solution n f α = w α (s 2 i )s 1 i (u T i d)v i. i=1

9 Regularization by filtering TSVD Tikhonov w α (s 2 ) = { 1 if s2 > α, if s 2 α. w α (s 2 ) = s2 s 2 + α f α = s 2 i >α s 1 i (u T i d)v i f α = (K T K + αi) 1 K T d 1 TSVD Tikhonov.8 w α (s 2 ) Figure 2: Regularization parameter α = 1 2 s 2

10 Regularization by filtering 5 Norm of TSVD Solution Error α = e α = f α f true α f α (x) α = α = f α (x).4 f α (x) Figure 3: TSVD regularized solutions with 2% error level

11 Regularization by filtering Choose regularization parameter α such that e α = e trunc α + e noise α for δ. For both TSVD and Tikhonov filter e noise α α 1 2 δ, for α = δ p and p < 2, e trunc α, for α thus p >.

12 Regularization by filtering Choose regularization parameter α such that e α = e trunc α + e noise α for δ. For both TSVD and Tikhonov filter e noise α α 1 2 δ, for α = δ p and p < 2, e trunc α, for α thus p >. For TSVD assume δ s min and f true = K T z for z R n then e α α 1 2 z + α 1 2 δ. Minimizing w.r.t. α gives α = δ/ z and e α 2 z 1 2 δ 1 2 = O( δ).

13 Variational regularization methods For very large ill-conditioned systems regularization by filtering is impractical since it requires the SVD of a large matrix. Alternative, Tikhonov variational representation f α = arg min f R n Kf d 2 + α f 2, might be easier to compute incorporate constraints, e.g. nonnegativity of f replace least squares term Kf d 2 by other fit-to-data functionals replace penalty term f 2 by other functionals incorporating a priori information Total Variation Regularization

14 Variational regularization methods For very large ill-conditioned systems regularization by filtering is impractical since it requires the SVD of a large matrix. Alternative, Tikhonov variational representation f α = arg min f R n Kf d 2 + αtv(f), with the discrete one-dimensional total variation function n 1 TV(f) = f i+1 f i = i=1 n 1 f i+1 f i x. x i=1 This penalizes highly oscillatory solutions while allowing jumps in the regularized solution.

15 Variational regularization methods 2 Norm of TV Solution Error 1.2 α = e α = f α f true f α (x) α α = 1e α = f α (x).4 f α (x) Figure 4: TV regularized solutions with 2% error level

16 Definition total variation Definition. The total variation of a function f L 1 (Ω) is defined by TV(f) = sup f div v dx, v V where Ω Ω denotes a simply connected, nonempty, open subset of R d, d = 1, 2,..., with Lipschitz continuous boundary. the space of test functions V = {v C 1 (Ω; R d ) v(x) 1 for all x Ω}. C 1 (Ω; R d ) denotes the space of vector valued functions v = (v 1,..., v d ) whose component functions v i are each continuously differentiable and compactly supported on Ω, i.e., each v i vanishes outside some compact subset of Ω.

17 Definition total variation Example. Let Ω = [, 1] R, and define { f, x < 1, 2 f(x) = f 1 x > 1 2, where f, f 1 are constants. For any v C 1 [, 1], 1 f(x)v (x) dx = 1/2 f(x)v (x) dx + = (f f 1 )v(1/2). 1 1/2 f(x)v (x) dx This quantity is maximized over all v V when v(1/2) = sign(f f 1 ), thus TV(f) = f 1 f.

18 Definition total variation Definition. The total variation of a function f defined on [, 1] is defined by TV(f) = sup i f(x i ) f(x i 1 ), where the supremum is taken over all partitions = x < < x n = 1. Proposition. If f is smooth (f W 1,1 (Ω)) then TV(f) = f dx. Ω

19 Definition total variation Definition. The total variation of a function f defined on [, 1] is defined by TV(f) = sup i f(x i ) f(x i 1 ), where the supremum is taken over all partitions = x < < x n = 1. Proposition. If f is smooth (f W 1,1 (Ω)) then TV(f) = f dx. Ω TV(f) can be interpreted geometrically as the lateral surface area of the graph of f. If f has many large amplitude oscillations f has large lateral surface area TV(f) is large

20 Numerical methods for total variation Find regularized solutions to operator equations Kf = g by minimizing Tikhonov-TV functional T α (f) = 1 2 Kf g 2 + αtv(f), where TV(f) = TV(f) = 1 df 1 1 dx, in 1D, dx f dx dy, in 2D. Standard methods requiring gradient and/or hessian info (e.g. Steepest descent, Newton s method) are not suitable due to the non-differentiability of the Euclidean norm at the origin.

21 Numerical methods for total variation Find regularized solutions to operator equations Kf = g by minimizing approximate Tikhonov-TV functional T α (f) = 1 2 Kf g 2 + αj β (f), where J β (f) = J β (f) = ( df ) 2 + β2 dx, in 1D, dx ( f ) 2 ( f ) β2 dx dy, in 2D. x y Now the standard techniques can be used to minimize the discretized approximate Tikhonov-TV functional T (f) = 1 2 Kf d 2 + αj(f).

22 Numerical methods for total variation A one-dimensional discretization Using a composite midpoint quadrature, central difference approximation for the derivative, the approximation to the one-dimensional TV functional becomes J(f) = 1 n ψ ( (D i f) 2) x, 2 i=1 where f = (f,..., f n ) T, with f i = f(x i ), x i = i x, x = 1/n, D i = [,...,, 1/ x, 1/ x,,..., ] 1 (n+1), ψ(t) = 2 t + β 2.

23 Numerical methods for total variation From the directional derivative the gradient is derived d dτ J(f + τv) τ= n = ψ ( (D i f) 2) (D i f)(d i v) x where i=1 [diag(ψ (f))] i,i = ψ ( (D i f) 2 ) ), = x (Dv) T diag(ψ (f)) (Df) = x D T diag(ψ (f)) Df, v = grad J(f), v =: L(f)f, v, D = [D 1 ;... ; D n ] n (n+1). The matrix L(f) is symmetric and positive semidefinite.

24 Numerical methods for total variation In a similar way the hessian is derived Hess J(f) = L(f) + L (f)f, where L (f)f = x D T diag(2(df) 2 ψ (f))d, [diag(2(df) 2 ψ (f))] i,i = 2(D i f) 2 ψ ((D i f) 2 ).

25 Numerical methods for total variation In a similar way the hessian is derived Hess J(f) = L(f) + L (f)f, where L (f)f = x D T diag(2(df) 2 ψ (f))d, [diag(2(df) 2 ψ (f))] i,i = 2(D i f) 2 ψ ((D i f) 2 ). The gradient and hessian of the discretized approximate Tikhonov-TV functional are now simply given by grad T (f) = K T (Kf d) + αl(f)f, Hess T (f) = K T K + αl(f) + αl (f)f.

26 Numerical methods for total variation 1. Steepest descent with line search for total variation f ν+1 = f ν τ grad T (f ν ) Algorithm: ν := ; f := initial guess; while no convergence g ν := K T (Kf ν d) + αl(f ν )f ν ; % gradient τ ν := arg min τ> T (fν τg ν ); % line search f ν+1 := f ν τ ν g ν ; % update approximate solution ν := ν + 1;

27 Numerical methods for total variation 2. Newton s method with line search for total variation f ν+1 = f ν τ (Hess T (f ν )) 1 grad T (f ν ) Algorithm: ν := ; f := initial guess; while no convergence g ν := K T (Kf ν d) + αl(f ν )f ν ; % gradient H J := L(f ν ) + L (f ν )f ν ; % hessian of penalty functional H := K T K + αh J ; % hessian of cost functional s ν := H 1 g ν ; % newton step τ ν := arg min τ> T (fν + τs ν ); % line search f ν+1 := f ν + τ ν s ν ; % update approximate solution ν := ν + 1;

28 Numerical methods for total variation 3. Lagged diffusivity fixed point iteration f ν+1 = (K T K + αl(f ν )) 1 K T d = f ν (K T K + αl(f ν )) 1 grad T (f ν ). Fixed point form can be derived by setting grad T (f) =. The matrix L(f) can be viewed as a discretization of a steady-state diffusion operator and is evaluated at f ν, hence the name. The equivalent quasi-newton iteration can also be derived by dropping the term αl (f)f from the Hessian. The quasi Newton form tends to be less sensitive to roundoff error.

29 Numerical methods for total variation 3. Lagged diffusivity fixed point iteration Algorithm: ν := ; f := initial guess; while no convergence g ν := K T (Kf ν d) + αl(f ν )f ν ; % gradient H := K T K + αl(f ν ); % approximate hessian of cost functional s ν := H 1 g ν ; % quasi-newton step f ν+1 := f ν + s ν ; % update approximate solution ν := ν + 1; If K T K is positive definite, then this fixed point iteration converges globally and no line-search is needed. Because the Hessian is approximated, only linear convergence is expected.

30 Numerical methods for total variation Results for one-dimensional test problem All three methods give essentially the same reconstruction. Measure for numerical performance is relative iterative solution error e ν α = f α ν f α, f α with f α the minimizer of the approximate Tikhonov-TV functional (use accurate approximation obtained with Primal-Dual Newton method) True solution Approximate TV regularized solution Figure 5: α =.1, β =.1

31 Numerical methods for total variation Results for one-dimensional test problem Steepest Descent: Hessian ill-conditioned slow (linear) convergence Newton: Line search restricts stepsize local quadratic convergence attained late (α small, β small) Cost per iteration is about the same for Newton, Fixed point and Primal- Dual Newton Norm of gradient Steepest Descent Newton Fixed point Primal Dual Newton 1 Norm of solution error Iteration Iteration

32 Summary Variational representation of Tikhonov functional TV is a penalty functional which penalizes oscillatory solutions Approximate TV functional needed for standard optimization tools References [Vog2] Curtis R. Vogel. Computational Methods for Inverse Problems. Society for Industrial and Applied Mathematics, 22.