Regularization methods for large-scale, ill-posed, linear, discrete, inverse problems Silvia Gazzola Dipartimento di Matematica - Università di Padova January 10, 2012 Seminario ex-studenti 2 Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 1 / 21
Outline 1 Introduction Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 2 / 21
Outline 1 Introduction 2 Singular Value Decomposition Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 2 / 21
Outline 1 Introduction 2 Singular Value Decomposition 3 Regularization Truncated Singular Value Decomposition Tikhonov regularization method Iterative regularization methods Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 2 / 21
Outline 1 Introduction 2 Singular Value Decomposition 3 Regularization Truncated Singular Value Decomposition Tikhonov regularization method Iterative regularization methods 4 More recent regularization methods Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 2 / 21
Outline 1 Introduction 2 Singular Value Decomposition 3 Regularization Truncated Singular Value Decomposition Tikhonov regularization method Iterative regularization methods 4 More recent regularization methods 5 Future works Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 2 / 21
Introduction Hadamard 1 : a problem is ill-posed if it has at least one of the following features does not have a solution; does not have an unique solution; the solution does not depend continuously on the given data; However, ill-posed problems arise in many applications! Inverse problems typically are ill-posed problems. We are considering an inverse problem when we seek to determine the cause of an observed effect. Common examples: astronomical, geological, medical imaging. Usually described by Fredholm integral equations of the first kind k(s, t)x(t)dt = g(s), s Ω R q Ω 1 Lectures on Cauchy s problem in linear partial differential equations, Yale University Press (1923). Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 3 / 21
Image Processing I A digital (grayscale) image is mathematically represented by a matrix, whose elements are called pixels. The number of pixels usually varies from 256 2 (small images) to 10 6 (huge images). Recorded images are always damaged by: blur noise Notations: X R m n is the exact image; B ex R m n is the blurred image; B R m n is the blurred and noisy image; E R m n is the noise. Remark: deblurring and denoising an image is indeed an inverse problem. Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 4 / 21
Image Processing II 130 27 23 23 22 50 40 45 52 135 36 34 31 26 53 45 50 43 149 57 63 62 56 84 76 82 51 137 52 65 69 65 95 88 94 76 149 52 59 68 59 86 90 96 85 144 48 44 58 63 85 82 93 65 149 66 42 44 56 71 66 88 76 135 98 80 65 64 61 46 64 32 134 161 181 175 175 173 162 172 158 In this example m = 179, n = 355 and N = 63545. Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 5 / 21
Image Processing III Assumption: the operation of going from the original sharp scene to the recorded blurred image is linear. Rewriting conveniently x = vec(x) R N, b = vec(b) R N, b ex = vec(b ex ), e = vec(e), where N = nm, the blurring process can be represented by the linear system Ax = b. As a consequence, we have a large number of tools from linear algebra and matrix computation at our disposal! Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 6 / 21
Blur Further assumptions on the (linear) blurring functions: spatially invariant local Point Spread Functions (PSF): Exact image Out-of-focus blur Gaussian blur Remark: another ingredient that determines our blurring model is the boundary conditions. Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 7 / 21
Noise Further assumptions on noise: additive; white noise (i.i.d. random variables, drawn from a Gaussian distribution). Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 8 / 21
A first attempt at deblurring Let s try to naively solve our deblurring/denoising problem: x = A 1 b. Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 9 / 21
A first attempt at deblurring Let s try to naively solve our deblurring/denoising problem: This is our result! x = A 1 b. We need regularization! Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 9 / 21
A very important tool Useful (if available) tool for the analysis of the discretized problem: SVD. Theorem If A R m n, then there exist orthogonal matrices U R n n and V R m m (i.e. U T U = I n, V T V = I m ) such that U T AV = diag(σ 1,..., σ p ) =: Σ, p min{m, n} where σ 1 σ 2 σ p > 0. Some useful facts follows: The matrix condition number: cond(a) = σ 1 σ p ; The solution of the linear system Ax = b can be written as x = V Σ 1 U T b = p i=1 u T i b σ i v i. Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 10 / 21
Common features of ill-posed problems The singular values σ i decay towards 0, typically without a particular gap between large and tiny singular values. As a consequence the condition number of the system matrix is huge. In our deblurring problem, this number is 5.9995 10 11. The singular vectors u i and v i often have more sign changes in their elements as the corresponding σ i decreases, i.e. tiny singular values highly oscillating functions large singular values smooth functions the coefficients u T i b ex on the average decay faster to zero then the singular values σ i. (Discrete Picard condition). Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 11 / 21
Example Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 12 / 21
Regularization Since real-world problems are always affected by errors, in practice we solve p ui T b ex p ui T e x = v i + v i. σ i σ i i=1 Definition In order to be able to determine a meaningful approximation of the solution of the exact linear system, we replace the available one by a nearby system that is less sensitive to perturbations of the right-hand side; we then consider the solution of the latter system as an approximation of the exact solution. i=1 We ll explore the class of spectral filtering methods, i.e. p ui T b x reg = φ i v i, φ i [0, 1]. σ i i=1 Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 13 / 21
Truncated Singular Value Decomposition (TSVD) We choose a truncation index k < n. We simply drop all the components above k. k x = V Σ 1 U T ui T b b = v i, k < n. σ i i=1 In this case: φ i = 1 for i k, φ i = 0 for i > k. In the example: m = n = 512 and N = 262144. We respectively choose k = 658, k = 2813, k = 7243. Pay attention to over-regularization and under-regularization. Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 14 / 21
Tikhonov regularization method We want to solve the following penalized minimization problem { x λ = min Ax b 2 2 + λ Lx 2 } x R n 2, where λ > 0, A R m n, L R p n (typically p n m). Equivalent formulations Normal equations (A T A + λl T L)x λ = A T b; Minimization problem ( ) x λ = min λl A x In this case: φ i = σ2 i σ 2 i +λ. x R n ( b 0 Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 15 / 21 ) 2
Choice of the regularization matrix The null space of the regularization matrix is very important! Common choices: L = I L := L 1 = L := L 2 = 1 1 1 1...... 1 1 1 2 1 1 2 1......... 1 2 1 R(n 1) n, R(n 2) n The best situation is when we have an idea of what the solution may be. Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 16 / 21
Choice of the regularization parameter 1 We have an accurate approximation of ε := e Discrepancy principle: we choose λ > 0 so that Ax λ b = ηε, where η 1 2 An estimation of ε is not available L-curve criterion: we plot the curve (log 10 Lx λ, log 10 b Ax λ ), λ > 0 (L curve). We choose the value of λ that corresponds to the vertex of the L-curve. Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 17 / 21
Iterative regularization methods General idea: if the iteration is truncated sufficiently early, the error can be controlled. Therefore the iteration number plays the role of the regularization parameter. In particular: based on Krylov subspaces methods K m (B, c) = {c, Bc, B 2 c,..., B m 1 c}. Example: The Conjugate Gradient applied to the normal equation (CGNR) A T Ax = A T b. We want to find x m K m (A T A, A T b) such that Ax b 2 is minimized. Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 18 / 21
More recent regularization methods Based on the reduction of the dimensions of the involved matrices. Achieved by projection onto Krylov subspaces. An outstanding (and promising!) example: we run the Arnoldi algorithm the Arnoldi-Tikhonov method AU l = U l+1 H l ; we write the reduced minimization problem y l,λ = min{ H l y U y l+1 T 2 + λ y } l once the above problem is solved, the original solution is x l,λ = U l y l,λ Disadvantages: we have to deal with two parameters (l, λ), simple only with L = I. Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 19 / 21
Future works Iterative approach based on the computation of functions of matrices. Quest for new and more suitable regularization operators. Extension on the more general (and realistic) case in which the system matrix is affected by errors, too. New and more efficient methods for the choice of the regularization parameters. Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 20 / 21
Bibliography P. C. Hansen. Rank-Deficient and Discrete Ill-Posed Problems. Numerical Aspects of Linear Inversion. SIAM, 1998. P. C. Hansen, J. G. Nagy, D. P. O Leary. Deblurring Images. Matrices, Spectra and Filtering. SIAM, 2006. M. Hanke, P.C. Hansen. Regularization methods for large-scale problems. Surv. Math. Ind., 3:253-315, 1993. Silvia Gazzola (Dip. Matematica - UniPD) Reg. meth.s for discrete ill-posed pb.s January 10, 2012 21 / 21