PhD Course: Introduction to Inverse Problem. Salvatore Frandina Siena, August 19, 2012

Transcription

1 PhD Course: to Inverse Problem theory Department of Information Engineering, Siena, Italy Siena, August 19, / 68

2 An overview of the theory 2 / 68

3 Direct and Inverse Problems - Direct problem: given a know model x in a model space X and an operator K, evaluate K(x) - Inverse problem: given y and an operator K, solve the equation K(x) = y for determine the model x - Note: * The formulation of an inverse problem requires the definition of the operator K with its domain and range * The formulation as an operator equation allows to distinguish among finite, semi-finite, and infinite-dimensional, linear and nonlinear problems * The evaluation of K(x) requires solving a boundary value problem for a differential equation or evaluating an integral equation theory 3 / 68

4 Well-Posedness I - A mathematical model for a physical problem is well-posed, in the sense of Hadamard, if it satisfies the following properties: 1. there exists a solution of the problem (existence) 2. there is at most one solution of the problem (uniqueness) 3. the solution depends continuously on the data (stability) - The direct problem is well-posed while the inverse problem is ill-posed - Note: * The existence of a solution can be enforced by enlarging the solution space * If a problem has more than one solution, then information about the model is missing * If the solution of a problem does not depend continuously on the data, then in general the computed solution has nothing to do with the true solution theory 4 / 68

5 Well-Posedness II Definition 1 Let X and Y be normed spaces, K : X Y a (linear or nonlinear) mapping. The equation Kx = y is called well-posed if the following holds: 1. Existence: for every y Y there is (at least one) x X such that Kx = y 2. Uniqueness: for every y Y there is at most one x X with Kx = y 3. Stability: the solution x depends continuously on y; that is, for every sequence (x n) X with Kx n Kx (n ), it follows that x n x (n ) - every equations for which (at least) one of these properties does not hold are called ill-posed theory 5 / 68

6 Well-Posedness III - Existence and Uniqueness depend only on the algebraic nature of the spaces and the operator i.e. if the operator is onto (surjective) or one-to-one (injective) - Stability depends also on the topologies of the spaces i.e. if the inverse operator K 1 : Y X is continuous - All these requirements are not independent of each other. A theorem says that the inverse operator K 1 is automatically continuous if K is linear and continuous and X and Y are Banach spaces theory 6 / 68

7 Compact Operator - Integral operators are compact operators in many natural topologies under very weak conditions on the kernels Theorem 1 Let X, Y be normed spaces and K : X Y be a linear compact operator with kernel N (K) := {x X : Kx = 0}. Let the dimension of the factor space X /N (K) be infinite. Then there exists a sequence (x n) X such that Kx n 0 but (x n) does not converge. We can even choose (x n) such that x n. In particular, if K is one-to-one, the inverse K 1 : Y R(K) X is unbounded. Here, R(K) := {Kx Y : x X } denotes the range of K - Linear equations of the form Kx = y with compact operators K are always ill-posed theory 7 / 68

8 Concept of I - Many inverse problems can be formulated as operator equations of the form Kx = y where K is a linear compact operator between Hilbert spaces X and Y over the field K = R or C - A successful reconstruction strategy requires additional a priori information about the solution - The regularization strategies are optimal if they have the same asymptotic order as the worst-case error theory 8 / 68

9 Concept of II - The method and the are two of the most important regularization strategies - The regularization parameter α = α(δ) is chosen a priori i.e. before to start to compute the regularized solution - The optimal regularization parameter α depends on bounds of the exact solution and it is not known in advance theory 9 / 68

10 Assumptions of the problem I - The compact operator K is one-to-one * this is not a restriction since that the domain X can be replaced by orthogonal complement of the kernel of K i.e. N(K) = {x X : x, y = 0, y N(K)} - Exists a solution x X of the unperturbed equation Kx = y i.e. y R(K) Y - The injectivity of K implies that this solution is unique - The right-hand side y Y is never known exactly but only with an error of δ > 0. The assumption is that the δ > 0 and y δ Y are known with y y δ δ theory 10 / 68

11 Assumptions of the problem II - The aim is to solve the perturbed equation Kx δ = y δ - In general, the problem is not solvable because is not known if the measured data y δ are in the range R(K) of K - An approximation x δ X to the exact solution x must have the same asymptotic error of the worst-case error - The approximate solution x δ should depend continuously on the data y δ theory 11 / 68

12 I - The purpose is to construct a suitable bounded approximation R : Y X of the (unbounded) inverse operator K 1 : R(K) X Definition 2 A regularization strategy is a family of linear and bounded operators theory such that for all x X R α : Y X, α > 0, lim RαKx = x α 0 i.e. the operators R αk converge pointwise to the identity 12 / 68

13 II - The following theorem derives the definition 2 and from the compactness of K Theorem 2 Let R α be a regularization strategy for a compact operator K : X Y where dim X =. Then * the operators R α are not uniformly bounded i.e. there exists a sequence (α j ) with R αj for j * the sequence (R αkx) does not converge uniformly on bounded subsets of X i.e. there is no convergence R αk to the identity I in the operator norm - N.B. the definition is based on unperturbed data i.e. the regularizer R αy converges to x for the exact right-hand side y = Kx theory 13 / 68

14 III - An approximation of the solution x of Kx = y is defined as x α,δ = R αy δ where y K(X ) is the exact right-hand side and y δ Y is the measured data with y y δ δ - The error can be split into two parts using triangle inequality theory x α,δ x R αy δ R αy + R αy x R α y δ y + R αkx x and finally x α,δ x condition number {}}{ }{{} δ R α + R αkx x }{{} error in the data approximation error (1) 14 / 68

15 Total Error Trend theory Figure: Trend of the total error - The first term is the error in the data multiplied by the condition number R α of the regularized problem. By theorem 2, this term tends to infinity as α tends to zero - The second term is the approximation error (R α K 1 )y at the exact right-hand side y = Kx. By the definition of a regularization strategy, this term tends to zero with α - The aim is to choose α = α(δ) dependent on δ in order to minimize the total error δ R α + R αkx x 15 / 68

16 Admissible Definition 3 A regularization strategy α = α(δ) is called admissible if, for every x X, α(δ) 0 and sup{ R α(δ) y δ x : Kx y δ δ} 0, δ 0 theory - A method to construct classes of admissible regularization strategies is given by filtering singular systems - Let K : X Y be a linear compact operator, and let (µ j, x j, y j ) be a singular system for K. The solution x of Kx = y is given by Picard s theorem [1] as 1 x = (y, y j )x j µ j j=1 provided the series converges i.e. y R(K) 16 / 68

17 Regularizing Filter I Theorem 3 Let K : X Y be a compact with singular system (µ j, x j, y j ) and q : (0, ) (0, K ] R be a function with the following properties: - (1) for every α > 0 and for all 0 < µ K * q(α, µ) 1 * exists c(α) such that q(α, µ) c(α)µ - (2) for every 0 < µ K, lim α 0 q(α, µ) = 1 Then the operator R α : Y X, α > 0, defined by theory R αy = j=1 q(α, µ j ) µ j (y, y j )x j, y Y, is a regularization strategy with R α c(α). A choice α = α(δ) is admissible if α(δ) 0 and δc(α(δ)) 0 as δ 0. The function q is a regularizing filter for K - the theorem implies that R αy converges to the solution x 17 / 68

18 Regularizing Filter II - A optimal strategy in the sense of worst-case error can be achieved by enforcing the previous theorem 3 Theorem 4 Let assumption (1) of the previous theorem 3 hold. Then (2) can be replaced by the stronger assumption: - (2a) exists c 1 > 0 such that for all α > 0 and 0 < µ K with q(α, µ) 1 c 1 α µ * If, furthermore, x K (Y ), then R αkx x c 1 α z, where x = K z - (2b) exists c 2 > 0 such that for all α > 0 and 0 < µ K with q(α, µ) 1 c 2 α µ 2 * If, furthermore, x K K(X ) R αkx x c 2 α z, where x = K Kz theory 18 / 68

19 Regularizing Filter III Theorem 5 The following three functions q satisfy the assumptions (1), (2), and (2a-b) of the previous theorems 3 and 4: - (a) q(α, µ) = µ 2 /(α + µ 2 ). This satisfies (1) with c(α) = 1/(2 α) while assumptions (2a) and (2b) hold with c 1 = 1/2 and c 2 = 1, respectively - (b) q(α, µ) = 1 (1 aµ 2 ) 1/α for some 0 < a < 1/ K 2. In this case (1) holds with c(α) = a/α while (2a) and (2b) are satisfied with c 1 = 1/ 2a and c 2 = 1/a, respectively - (c) Let q be defined by { 1, µ q(α, µ) = 2 α, 0, µ 2 < α In this case (1) holds with c(α) = 1/ α while (2a) and (2b) are satisfied with c1 = c2 = 1 All of the functions q defined in (a), (b), and (c) are regularizing filters that lead to optimal regularization strategies theory 19 / 68

20 Spectral Cutoff - For the first two choices of q exists a characterization that avoids knowledge of the singular system - The choice (c) of q is called the spectral cutoff. The spectral cutoff solution x α,δ X is therefore defined by x α,δ = µ 2 j α 1 µ j (y δ, y j )x j theory - From the fundamental estimate Definition 1 and with the previous theorem, the following result for the cutoff solution hold 20 / 68

21 Spectral Cutoff I Theorem 6 Let y δ Y be such that y δ y δ, where y = Kx denotes the exact right-hand side - (a) let K : X Y be a compact and injective operator with singular system (µ j, x j, y j ). The operators R αy = µ 2 j α 1 µ j (y, y j )x j, y Y, theory define a regularization strategy with R α 1/ α. This strategy is admissible if α(δ) 0 (δ 0) and δ 2 /α(δ) 0 (δ 0) - (b) let x = K z K (Y ) with z E and c > 0. For the choice α(δ) = cδ/e, we have the estimate ( 1 x α(δ),δ x c + δe c) - (c) let x = K Kz K K(X ) with z E and c > 0. The choice α(δ) = c(δ/e) 2/3 leads to the estimate ( ) 1 x α(δ),δ x c + c δ 2/3 E 1/3 21 / 68

22 Spectral Cutoff II - The spectral cutoff is optimal for the information (K ) 1 x E or (K K) 1 x E, respectively (if K is one-to-one) theory 22 / 68

23 to I - In general for a concrete integral operators it is recommended to avoid the computation of a singular system - Given an overdetermined finite linear system of the form Kx = y, a common method to solve it is to determine the best fit that minimizes the defect Kx y with respect to x X for some norm in Y theory 23 / 68

24 to II - If X is infinite-dimensional and K is compact, this minimization problem is also ill-posed by the following lemma Lemma 1 Let X and Y be Hilbert spaces, K : X Y be linear and bounded, and y Y. There exists ˆx X with K ˆx y Kx y for all x X if and only if ˆx X solves the normal equation K K ˆx = K y. Here, K : Y X denotes the adjoint of K theory 24 / 68

25 Tikhonov functional I - The aim is to: * penalize the defect in term of the optimization theory * replace the equation of the first kind K K ˆx = K y with an equation of the second kind in term of language of integral equation theory - Both viewpoints lead to the following minimization problem: given the linear, bounded operator K : X Y and y Y, determine x α X that minimizes the Tikhonov functional theory J α(x) = Kx y 2 + α x 2, x X 25 / 68

26 Tikhonov functional II Theorem 7 Let K : X Y be a linear and bounded operator between Hilbert spaces and α > 0. Then the Tikhonov functional J α has a unique minimum x α X. This minimum x α is the unique solution of the normal equation theory αx α + K Kx α = K y - The solution x α can be written in the form x α = R αy with R α = (αi + K K) 1 K : Y X (2) 26 / 68

27 Tikhonov functional III - Choosing a singular system (µ j, x j, y j ) for the compact operator K, R αy has the representation R αy = µ j (y, y j )x j = α + µ 2 j n=0 n=0 q(α, µ j ) µ j (y, y j )x j, y Y theory with q(α, µ) = µ 2 /(α + µ 2 ) - Note: this function q is exactly the filter function (a) of the theorem 5 27 / 68

28 Tikhonov regularization method I Theorem 8 Let K : X Y be a linear, compact operator and α > 0. - (a) the operator αi + K K is boundedly invertible. The operators R α : Y X from 2 form a regularization strategy with R α 1/(2 α). It is called the Tikhonov regularization method. R αy δ is determined as the unique solution x α,δ X of the equation of the second kind αx α,δ + K Kx α,δ = K y δ Every choice α(δ) 0 (δ 0) with δ 2 /α(δ) 0 (δ 0) is admissible - (b) let x = K z K (Y ) with z E. Choosing α(δ) = cδ/e for some c > 0. Then the following estimate holds: x α(δ),δ x 1 ( 1 c + δe c) 2 theory - (c) let x = K Kz K K(X ) with z E. The choice α(δ) = c(δ/e) 2/3 for some c > 0 leads to the error estimate ( ) 1 x α(δ),δ x 2 c + c δ 2/3 E 1/3 The regularization method is optimal for the information (K ) 1 x E or (K K) 1 x E, respectively (provided K is one-to-one) 28 / 68

29 Tikhonov regularization method II - The eigenvalues of K tend to zero, and the eigenvalues of αi + K K are bounded away from zero by α > 0 [1] - From the previous theorem 8, the α(δ) has to be chosen in such a way that it converges to zero as δ tends to zero but not as fast as δ 2 - From parts (b) and (c), the smoother solution x is the slower α has to tend to zero - N.B. the convergence can be arbitrarily slow if no a priori assumption about the solution x is available [1] theory 29 / 68

30 Tikhonov regularization method III Theorem 9 Let K : X Y be linear, compact, and one-to-one such that the range R(K) is infinite-dimensional. Furthermore, let x X, and assume that there exists a continuous function α : [0, ) [0, ) with α(0) = 0 such that theory lim x α(δ),δ x δ 2/3 = 0 δ 0 for every y δ Y with y δ Kx δ, where x α(δ),δ X solves αx α,δ + K Kx α,δ = K y δ. Then x = 0 30 / 68

31 Consideration - regularization method is not optimal for stronger smoothness assumptions on the solution x i.e. under the assumption x (K K) r (X ) for some r N, r 2 * this is in contrast to Landweber s method or the conjugate gradient method - The choice of α in theorem 8 is made a priori; that is, before starting the computation of x α by solving the least squares problem - Note: The regularization is stronger related to the differential operators or norms used [1] * one example is the interpretation of regularization by smoothing norms in terms of reproducing kernel Hilbert spaces theory 31 / 68

32 Landweber s Equation - Landweber suggested rewriting the equation Kx = y in the form x = (I ak K)x + ak y for some a > 0 and iterating this equation: * x 0 = 0 * x m = (I ak K)x m 1 + ak y for m = 1, 2,... - This iterative scheme can be viewed as the steepest descent algorithm applied to the quadratic functional x Kx y 2 theory 32 / 68

33 I Lemma 2 Consider the sequence (x m = (I ak K)x m 1 + ak y) and define the functional ψ : X R by ψ(x) = 1 2 Kx y 2, x X. Then ψ is Fréchet differentiable in every z X and theory ψ (z)x = Re(Kz y, Kx) = Re(K (Kz y), x), x X Therefore, x m = x m 1 ak (Kx m 1 y) is the steepest descent step with stepsize a 33 / 68

34 II - x m can be now expressed by the explicit form x m = R my, where the operator R m : Y X is defined by m 1 R m = a (I ak K) k K for m = 1, 2,... k=0 theory - Choosing a singular system (µ j, x j, y j ) for the compact operator K, then R my has the representation R my = j=1 q(m, µ j ) µ j (y, y j )x j, y Y with q(m, µ) = 1 (1 aµ 2 ) m - This filter function q is the same of the theorem 5 where α = 1/m 34 / 68

35 Theorem 10 III - (a) let K : X Y be a compact operator and let 0 < a < 1/ K 2. The linear and bounded operators R m : Y X define a regularization strategy with discrete regularization parameter α = 1/m, m N, and R m am. The sequence x m,δ = R my δ is computed by the iteration * x 0,δ = 0 * x m,δ = (I ak K)x m 1,δ + ak y δ for m = 1, 2,.... Every strategy m(δ) (δ 0) with δ 2 m(δ) 0 (δ 0) is admissible - (b) let x = K z K (Y ) with z E and 0 < c 1 < c 2. For every E choice m(δ) with c 1 m(δ) δ c2 E, the following estimate holds: δ theory x m(δ),δ x c 3 δe for some c 3 depending on c 1, c 2, and a. - (c) let x = K Kz K K(X ) with z E and 0 < c 1 < c 2. For every choice m(δ) with c 1(E/δ) 2/3 m(δ) c 2(E/δ) 2/3, we have x m(δ),δ x c 3E 1/3 δ 2/3 35 / 68

36 Consideration - From the previous slide, the Landweber s iteration is optimal for the information (K ) 1 x E or (K K) 1 x E, respectively - The choice x 0 = 0 simplifies the analysis. In general, the explicit iteration x m is given by m 1 x m = a (I ak K) k K y + (I ak K) m x 0, m = 1, 2,... k=0 theory - R m is affine linear i.e. of the form R my = z m + S my, y Y, for some z m X and some linear operator S m : Y X - Note: * high precision (ignoring the presence of errors) requires a large number m of iterations * stability forces us to keep m small enough * from the theorem, it also holds the following general rule: x (K K) r (X ), r N x m(δ),δ x c E 2r+1 1 δ 2r+1 2r 36 / 68

37 I - Most of the inverse problems in science and engineering areas are ill-posed * computational vision * system identification * nonlinear dynamic reconstruction * density estimation - Given the available input data, the solution to the problem is nonunique (one-to-many) or unstable - The classical regularization techniques, due to Tikhonov, are able to making the solution well-posed * model selection * complexity control theory 37 / 68

38 II - theory was introduced to the machine learning community in the 1990s - A regularization algorithm for learning is equivalent to a multilayer network with a kernel in the form of a radial basis function (RBF), an RBF network [2] - The solution of the classical Tikhonov regularization problem can be derived from the regularized functional defined by a: * linear differential operator in the spatial domain * linear integral operator in Fourier domain - Note: the regularized solution was originally derived by a differential linear operator and its Green s function theory 38 / 68

39 II - One way to solve ill-posed problems is to make the problems well-posed by incorporating prior knowledge into the solutions - The forms of prior knowledge vary and are problem dependent * the most popular and important prior knowledge is the smoothness prior - A possible choice of the smoothness of the functional is to put the functional into the reproducing kernel Hilbert space (RKHS) theory 39 / 68

40 Machine Learning Framework - Given a set of observation data (learning examples) {(x i, y i ) R N R} X Y, the learning machine f has to find the solution to the inverse problem - f has to approximate a real function in the hypothesis satisfying the constraints f (x i ) = y(x i ) y i * where y(x) is supposed to be a deterministic function in the target space - The learning problem can be viewed as a multivariate functional approximation problem - Note: this problem is ill-posed since the approximants satisfying the constraints are not unique theory 40 / 68

41 Loss Function - Statistically, the approximation accuracy is measured by the expectation of the approximation error. The expected risk functional R = L(x, y)p(x, y)dxdy X Y where L(x, y) represents the loss functional - A common loss function is the mean squared error defined by L 2 norm and the expected risk functional is R = [y f (x, y)] 2 p(x, y)dxdy X Y - In practice * the joint probability p(x, y) is unknown * an estimate of R is based on finite l observations and so the expected risk functional becomes an empirical risk functional l R emp = [y i f (x i )] 2 i=1 * which introduces an estimate ŷ(x) (ŷ = y ɛ = f (x)) theory 41 / 68

42 Tikhonov - The expected risk can be decomposed into two part R[f ] = R emp[f ] }{{} the empirical risk functional + R reg[f ] }{{} the regularizer risk functional = 1 l [y i f (x i )] λ Df i=1 theory * is the norm operator * λ is a regularization parameter that controls the trade-off between the good fit of the data and the smoothness of the solution * D is a linear differential operator defined as the Fréchet differential of Tikhonov functional - Note: the smoothness prior implicated in D makes the solution stable and insensitive to noise 42 / 68

43 The Fréchet differential and Riesz Theorem Definition 4 A function f : X Y is Fréchet differentiable at a point x, if for every h X, f (x + ɛh) f (x) lim = df (x, h) ɛ 0 ɛ exists, and defines a linear bounded transformation (in h) mapping X into Y. df (x, h) = F (x)h is the Fréchet differential; F (x) is the Fréchet derivative. Theorem 11 Let H be a Hilbert space, and let H be its dual space, consisting of all continuous linear functionals from H into the field R or C. If x is an element of H, then the function ϕ x defined by theory ϕ x(y) = y, x y H where, denotes the inner product of the Hilbert space, is an element of H. The theorem states that every element of H can be written uniquely in this form 43 / 68

44 The Reproducing Kernel Hilbert Space I Definition 5 Let X be an arbitrary set and H a Hilbert space of complex-valued functions on X. H is a reproducing kernel Hilbert space if every linear functional of the form L x : f f (x) (L x : H C) is continuous for any x X. By the Riesz theorem, this implies that for every x X there exists a unique element K x of H with the property that: f (x) = f, K x f H. The function K x is called the point-evaluation functional at the point x. The function K : X X C defined by K(x, y) = K x(y) is called the reproducing kernel for the Hilbert space H and it is determined entirely by H theory 44 / 68

45 The Reproducing Kernel Hilbert Space II - Note: * H is a space of functions, then the element K x is itself a function * the Riesz theorem guarantees that, for every x in X, the element K x is unique - The reproducing property * K(x, y) = K x (y) = K y, K x * K(x, x) = K x, K x 0, x X * K x = 0 if and only if f (x) = 0 f H theory 45 / 68

46 Fréchet Differential in Spatial Domain - The Fréchet Differential of R is dr(f, h) = d R(f + βh) β=0 where dβ h(x) is a constant function of x R emp(f, h) = d Remp(f + βh) dβ = β=0 l [y i f (x i )]h(x i ) = h, i=1 l [y i f (x i )]δ(x x i ) i=1 theory R reg(f, h) = d Rreg(f + βh) dβ β=0 = D[f + βh]dh dx = β=0 Df Dh dx = Df, Dh * δ( ) is the Dirac delta function *, is the inner product in Hilbert space H 46 / 68

47 Integral Equation I - Given a bounded linear operator A : X Y, its adjoint operator is defined as A : Y X such that Ax, y = x, A y (x X, y Y ) and A = A - An operator is a self-adjoint operator if it is equal to its adjoint operator Ax, x = x, A x (x, x X ) Definition 6 Given a positive-definite kernel function K * an integral operator T is defined by Tf = f (s) = K(s, x)f (x) dx * T is the adjoint operator defined by T f = f (x) = K(x, s)f (s) ds * T is self-adjoint if and only if K(s, x) = K(x, s) for all s and x theory - The integral equation is a Fredholm integral equation of the first kind 47 / 68

48 - Note: Integral Equation II - An integral operator with a symmetric kernel K(s, x) is self-adjoint - K = T Tf = g(s) = K (s, x)f (x)dx with K (s, x) = K(s, t)k(x, t)dt K is a integral self-adjoint operator - L = D D is a differential operator theory 48 / 68

49 Green s Function I - Given a positive integral operator T is possible to find a (pseudo-)differential operator D as its inverse - The operator D corresponds to the inner product of the RKHS with a reproducing kernel K associated to T * the kernel K is called Green s function of the differential operator D Definition 7 Given a linear differential operator L, the function G(x, ξ) is the Green s function for L if it has the following properties: * for a fixed ξ, G(x, ξ) is a function of x and satisfies the given boundary conditions * except at the point x = ξ, the derivatives of G(x, ξ) with respect to x are all continuous; the number of derivatives is determined by the order of operator L * with G(x, ξ) considered as a function of x, it satisfies the partial differential equation LG(x, ξ) = δ(x ξ) theory 49 / 68

50 Green s Function II - The solution of the differential equation LF (x) = ϕ(x) is F (x) = R N G(x, ξ)ϕ(ξ) dξ where ϕ(x) is a continuous function of x R N theory 50 / 68

51 Spectral in Fourier Domain - A spectral operator is an integral operator T, in the case of the Fourier operator K(s, x) = exp( j s, x ) Definition 8 For any functional f (x) H, the Fourier operator T is defined by Tf = F (s) = + f (x)exp( jxs) dx; F (s) H Theorem 12 Given two functionals f, g H and their corresponding Fourier transform F and G, the Parseval theorem says that f (x), g(x) = 1 F (s), G(s). In the 2π operator form it is expressed by f, g = Tf, Tg theory 51 / 68

52 Spectral in Fourier Domain - The differential operator D = + T D = + ( 1) n (js) n = exp( js) n! * T D f = T(Df ) = D(Tf ) ( 1) n n! d n dx n and its spectral operator - The kernel function associated with the operator K is given by K (x, x i ) = exp(jsx)exp( jsx i ) ds = δ(x x i ) - Then Kf = K (x, x i )f (s)ds = f (x x i ) theory 52 / 68

53 I - The regularization strategy for the equation Aϕ = f (A : X Y ) is to find an approximated solution ϕ ɛ related to ϕ, such that * ϕ ɛ ϕ ɛ where ɛ is a small, positive value - The two equivalent conditions are to find: * R λ : Y X (λ > 0) such that lim λ 0 R λ A ϕ = ϕ for all ϕ X (pointwise convergence) * R λ f A 1 f as λ 0 - Note: the operator R λ cannot be uniformly bounded with respect to λ, and the operator R λ A cannot be norm convergent as λ 0 theory 53 / 68

54 II - The approximation error is ϕ ɛ λ ϕ ɛ R λ + R λ Aϕ ϕ }{{}}{{} influence of incorrect data approximation error between R λ and A 1 - The regularization parameter, λ, controls the trade-off between: * stability (the first term increases as λ 0) * accuracy (the second term decreases as λ 0) - Note: the aim of the regularization is to find an appropriate λ to achieve the minimum error of the regularized risk functional theory 54 / 68

55 III Definition 9 The regularization strategy R λ i.e. the choice of regularization parameter λ = λ(ɛ), is called regular if for all f A(X ) and all f ɛ Y with f ɛ f ɛ, there holds R λ(ɛ) f ɛ A 1 f, ɛ 0 theory Theorem 13 For a bounded linear operator A(x) = N(A ) and N(A ) = A(X ) * A(X ) is the orthogonal complement of A(X ) i.e A = {ϕ X : Aϕ, g = 0 g A } * N(A ) is the kernel of A i.e. N(A ) = {A f = 0, f } 55 / 68

56 I Theorem 14 Let X be a Hilbert space, and let A : X X be a self-adjoint compact operator * all eigenvalues of A are real * all eigenspaces N(ξI A) for nonzero eigenvalues ξ have finite dimension * all eigenspaces associated with different eigenvalues are orthogonal Suppose the eigenvalues are ordered such that ξ 1 ξ 2..., and denote by P n : X N(ξ ni A) the orthogonal projection onto the eigenspace for the eigenvalue ξ n; then A = ξ np n n=1 Let Q : X N(A) be the orthogonal projection onto the kernel N(A); then ϕ = P nϕ + Qϕ ϕ X n=1 theory 56 / 68

57 II - In the case of an orthonormal basis, ϕ n, ϕ k = δ n,k, the expansion representation is Aϕ = ϕ = ξ n ϕ, ϕ n ϕ n n=1 ϕ, ϕ n ϕ n + Qϕ n=1 theory 57 / 68

58 Singular Values Decomposition Theorem 15 Let X and Y be Hilbert spaces. Let A : X Y be a linear compact operator and A : Y X be its adjoint. Let σ n denote the singular values of A, which are the square roots of the eigenvalues of the self-adjoint compact operator A A : X X. Let {σ n} be an ordered sequence of nonzero singular values of A according to the dimension of the kernel N(σ 2 ni A A). Then there exist orthonormal sequences {ϕ n} in X and {g n} in Y such that Aϕ n = σ ng n, A g n = σ nϕ n n Z - For each ϕ X, the Singular Values Decomposition (SVD) is ϕ = n=1 ϕ, ϕn ϕn + Qϕ * with the orthogonal projection Q : X N(A) and Aϕ = n=1 σn ϕ, ϕn gn theory 58 / 68

59 Picard s Theorem Theorem 16 Let A : X Y be a linear compact operator with singular system (σ n, ϕ n, g n). The Fredholm integral equation of the first kind Aϕ = f is solvable if and only if f N(A ) and n=1 solution is given by n=1 1 σ 2 n 1 σ n f, g n ϕ n f, g n 2 <. Then a theory - The Picard theorem describes the ill-posed nature of the integral equation Aϕ = f * the perturbation ratio ϕ ɛ / f ɛ = ɛ/σ n determines the degree of ill-posedness * more quickly σ decays and the more severe is the ill-posedness 59 / 68

60 Green s Identity - From the Parseval identity (theorem 12), the R reg(f, h) becomes dr reg(f, h) = Df Dh dx = Dh, Df = T D f T D h ds = T D h, T D f Definition 10 For any pair of functions u(x) and v(x), given a linear differential operator D and its associated Fourier operator T D, with adjoint operators, D and T D, are uniquely determined to satisfy the boundary conditions u(x)dv(x)dx = v(x)d u(x)dx R N R N u(s)t D v(s)ds = v(s)t Du(s)ds Ω Ω theory where Ω represents spectrum support in the frequency domain 60 / 68

61 Euler-Lagrange equation of Tikhonov functional I - The first equation is the Green s identity and it can be used to write dr reg(f, h) = Dh, Df = h, D Df (x) with u(x) = Df (x) and Dv(x) = Dh(x) - From the second equation derives dr reg(f, h) = T D h, T D f = h, T DT D f (s) theory with u(s) = T D f (s) and T D v(s) = T D h(s) - The condition dr(f, h) = dr emp(f, h) + λdr reg(f, h) = 0 becomes dr(f, h) = dr(f, h) = [ h(x), D Df (x) 1 λ [ h(s), T DT D f (s) l ] (y i f (x i ))δ(x x i ) i=1 F }{{} Fourier transform { 1 λ l }] (y i f (x i ))δ(x x i ) i=1 61 / 68

62 Euler-Lagrange equation of Tikhonov functional II - The two equivalent necessary conditions for f (x) is an extremum of R(f ) are * dr(f, h) = 0 for all h H * in the distribution sense D Df λ (x) = 1 l (y i f (x i ))δ(x x i ) and λ i=1 }{{} Euler-Lagrange equation of Tikhonov functional R(f ) T D T Df λ (s) = F{ 1 } l λ i=1 (y i f (x i ))δ(x x i ) = 1 l (y i f (x i ))exp( jx i s) λ i=1 }{{} Fourier transform of Euler-Lagrange equation of Tikhonov functional R(f ) theory 62 / 68

63 Differential and Integral Operators - Consider L = D D and K = T DT D * the G(x, ξ) is the Green s function for the linear differential operator LG(x, ξ) = δ(x ξ) * in frequency domain KG(s, ξ) = exp( jsξ) - The operators L and K are defined as * L = ( 1) n n=0 n!2 n 2n where 2 = l i=1 2 x i 2 is the Laplace operator * K = ( 1) 2n s 2n n=0 n!2 n = exp(s 2 /2) - Since LG(x) = δ(x) and KG(s) = 1 follow that G(s) = exp( s 2 /2) G(x) = exp( x 2 /2) theory 63 / 68

64 Solution of Regularized Problem I - The differential equation Lf (x) = ϕ(x) and integral equation Kf (x) = φ(s) have the same the solution f (x) = R N G(x, ξ)ϕ(ξ) dξ - Note: * Lf (x) = L R N G(x, ξ)ϕ(ξ) dξ = R N δ(x ξ)ϕ(ξ) dξ = ϕ(x) * ϕ(ξ) = 1 l λ i=1 [y i f (x i )]δ(ξ x i ) * f λ (x) = 1 l λ i=1 [y i f (x i )] R N G(x, ξ)δ(ξ x i ) dξ = l i=1 w i G(x, x i ) * equivalently in the frequency domain - The purpose of regularization is to find a subspace, the eigenspace of Lf or Kf, within which the operator becomes well-posed. The solution in the subspace is unique theory 64 / 68

65 Solution of Regularized Problem II - The solution of the regularized problem is f λ (x) = l i=1 w ig(x, x i ) with w i = [y i f (x i )]/λ and G(x, x i ) is a positive-definite Green s function for all i - In general, the solution of the Tikhonov regularization problem is f λ (x) = l i=1 w ig(x, x i ) + β(x) where * β(x) is a term that lies in the kernel of D which satisfies the orthogonal condition l i=1 w i β(x i ) = 0 * the simplest case is β(x) = const * the functional space of the solution f λ is an RKHS of the direct sum of two orthogonal RKHS theory 65 / 68

66 Parzen s Window - Replacing the summation by the integral and G(x, x i ) by K(x, x i ). If K is a reproducing kernel invariant to translation and satisfies K(x, x i ) = K(x x i ). The approximated function can be expressed by the convolution f (x) = 1 (y i f (x i )) K(x x i ) dx i λ }{{}}{{} observation kernel = 1 ( ) y i K(x x i ) dx i f (x i )K(x x i ) dx i λ }{{}}{{} Kernel regression Integral operator theory - The f (x) can be reconstructed by the data sample smoothed by an averaged kernel (Parzen s Window) 66 / 68

67 Regularized Solution in Matrix Form - The regularized solution in matrix form is obtained by taking dr(f ) df setting it to zero and R(f ) = 1 2 y f λ(df )T (Df ) = 1 2 (y f)t (y f) λft }{{} K f D D theory where the solution is f = (I λk) 1 }{{} smoothing matrix y * K is quadratic symmetric penalty matrix associated with L * similarly in the frequency domain where the regularizer is K 67 / 68

68 References Andreas Kirsch. An Introdutction to the Mathematical Theory of the Inverse Problems - Second Edition. Springer, theory S. Haykin Z. Chen. On Different Facets of. Neural Computation, 14: , / 68