Fall 2015: Computational and Variational Methods for Inverse Problems

Transcription

1 Fall 215: Computatonal and Varatonal Methods for Inverse Problems Georg Stadler Courant Insttute of Mathematcal Scences New York Unversty Omar Ghattas Jackson School of Geoscences Department of Mechancal Engneerng Insttute for Computatonal Engneerng & Scences The Unversty of Texas at Austn

2

3 CHAPTER 1 Introducton The goal of these notes s to provde an overvew of basc analyss, regularzaton and soluton methods for nverse problem, wth an emphass on nverse problems that nvolve (partal) dfferental equatons. As general references for nverse problem, and also as sources for these notes we refer to [1 3]. Ths secton ntroduces basc defntons and namng conventons used throughout these notes. Based on an mage deblurrng problem, typcal features of nverse problems are llustrated. Whle the deblurrng problem does not nvolve a dfferental equaton, t shares many features wth problems n whch the parameters and the measurements are lnked through the soluton of a partal dfferental equaton. Ths makes t an llustratve ntroductonary example. Let us ntroduce the notaton used n these notes. We denote vectors wth bold letters, e.g., u R n. Components of these vectors are denoted usng ndces,.e., u = (u 1,..., u n ) and vectors are understood as column vectors. The nner product between two vectors u, v s defned by u T v, and the norm of u by u := u T u. Matrces are denoted usng bold captal letters such as A R n m. Real-valued functons and varables not further specfed are denoted by lower case Latn letters such as g or h, and real scalars are usually denoted by Greek letters such as α or β. 1. Ill-posed problems The basc setup of an nverse problem can be explaned usng the relaton (1) d = F (p) + n, where The varable p, whch we want to reconstruct n the nverse problem s called the parameter (feld), the mage or the model. The set of all parameters s called parameter (or mage) space. In many of the applcatons we are nterested n, p s a functon or, after dscretzaton, a vector n a hgh-dmensonal space. The forward (or drect) model F maps p to the quantty we are able to measure. Ths forward mappng can be lnear or nonlnear n p. Usually, F descrbes a physcal theory, such as the propagaton of waves, the dffuson of a substance, the absorpton of rays when passng through an object, or flud flow. In the cases we are 3

4 4 1. INTRODUCTION manly nterested n, F s gven by an ordnary or a partal dfferental equaton, n whch case F often also contans an observaton operator that restrcts the soluton of the dfferental equaton to the quantty that can be measured, e.g., values at a part of the doman, at boundares or ponts. The varable d denotes the data or the measurements we are able to make. Often, these measurements are a corrupted verson of the outputs of the forward model. Ths error can be due to a model that does not fully descrbe the physcal phenomenon, t can be ntrnsc n the measurement process, or t can be due to roundoff error from a computer representaton of the measurements. Measurement errors, denoted by n n (1) are usually not known, but statstcal propertes of n (such as the mean and the varance) are often avalable. The nverse problem s to fnd the parameters p gven the (nosy) measurements d, havng knowledge over the forward operator F. A man dffculty s that n many applcatons of nterest, nverse problems are not well-posed n the sense of Hadamard, who defned the nverse problem of solvng d = F (p) to be well-posed, f the followng propertes are satsfed: (1) Exstence: For all data d (n an approprate data space), there exsts a parameter p of the problem (n an approprate parameter space). (2) Unqueness: For all (sutable) data d, the soluton p s unque. (3) Stablty: The soluton depends contnuously on the data,.e., small changes n the data d result n small changes n the parameter p. The problem (1) s called ll-posed f at least one of the above condtons s not satsfed. The man challenge n the numercal soluton of nverse problems s the stablty condton. If one wants to approxmate a problem whose soluton does not depend contnuously on the data by a numercal method as one does for well-posed problems, one has to expect the method to become unstable. There are two conceptually very dfferent approaches to solve llcondtoned nverse problems: Determnstc nverson s usually based on regularzaton methods that help to overcome the dffcultes due to the ll-posedness of nverse problems. These methods usually fnd a sngle parameter or mage p, whch solves (1) n an approprate sense. In ths class, we wll manly focus on ths determnstc approach for the soluton of nverse problems, dscuss regularzaton methods, ther nfluence on the reconstructon and numercal soluton algorthms. Bayesan nverson methods compute a probablty densty for the parameter p rather than a sngle soluton. Ths approach allows a

5 2. A DEBLURRING PROBLEM 5 flexble ntegraton of pror knowledge about p nto the soluton (.e., the probablty densty functon). Such a probablstc approach s often preferable n practcal problems snce ts soluton also quantfes the uncertantes n the reconstructon. However, Bayesan nverson s very costly and sometmes nfeasble, n partcular for the large-dmensonal problems, whch arse as dscretzatons of nverse problems wth partal dfferental equatons. In smple stuatons, connectons between the Bayesan and the determnstc approach can be made. For nstance, the choce of the pror n the probablstc approach s closely related to regularzaton methods n the determnstc approach. 2. A deblurrng problem Let us consder a deblurrng (or deconvoluton) problem as llustratve example. Even though n ths problem the parameters and the data are not connected through a dfferental equaton, t shares several features wth more complcated nverse problems that nvolve dfferental equatons. For smplcty, we consder a one-dmensonal blurrng operator gven by a Fredholm frst knd ntegral equaton. For a functon p : [, 1] R, we consder the operator (2) F (p) = d : [, 1] R defned by d(x) = 1 Here, the kernel k(x) s gven by k(x x )p(x ) dx for x 1. (3) k(x) = C exp( x2 ) wth C, γ >. 2γ2 The forward problem s the followng: Gven the source functon p and the kernel k, determne the blurred mage d. The assocated nverse problem s: Gven the kernel k and the blurred mage d, determne the orgnal mage p. To llustrate the ll-posedness of ths nverse problem, consder a perturbaton δp(x) := ε sn(2πωx) for p, where ε > and ω = 1, 2,.... The correspondng perturbaton for F (p + δp) s δd(x) = ε 1 k(x x ) sn(2πωx ) dx, whch converges to zero as ω 1. Hence, the rato between δp and δd 2 can become arbtrary large, whch shows that the stablty requrement for well-posedness cannot be satsfed. Fgure 2 llustrates the effect of the convoluton operator. Whle the parameter functon p contans jumps, the convolved data Kp s a smoothly 1 Ths s a consequence of the Remann Lebesgue lemma. 2 The norms for δp and δd can ether be the L or the L 2 -norm on (, 1).

6 6 1. INTRODUCTION varyng functon. Ths smoothng effect of the convoluton operator s partcularly obvous n the nterval [, 1/4], snce the small wave length varatons n p are averaged out n the convolved data. The amount of averagng depends on the wdth of the Gaussan, whch s controlled by the value γ n (3)..7.6 Gaussan kernel functon 1.8 paramter 1.8 data nosy data Fgure 1. Gaussan kernel k(x.5) wth γ =.5, C = 1/(γ 2π) as defned n (3) (left plot). A parameter functon p (mddle plot) and ts convoluton d (rght plot). Shown are the exact convoluton data n blue, and the nosy data n green. The dscretzaton of the convoluton operator uses N = 128 unknowns. Next, we dscretze the ntegral n (2) as needed for numercal computatons. For that purpose we use splt [, 1] nto N ntervals [kh, (k + 1)h], k =, N 1, where h = 1/N and N s a large nteger. Usng mdpont quadrature, the dscrete verson of (2) becomes (4) d = Kp wth the vectors d, p correspondng to the functon values of d, p at the mdponts of the ntervals, and a matrx K R N N. The entres of K are gven by K j = h C exp ( ), 1, j N. (( j)h)2 2γ 2 Snce the matrx K s nvertble, for gven (dscrete) data d one may smply compute the mage p as p = K 1 d. However, K becomes ncreasngly ll-condtoned as N becomes large and small nose n d can result n large errors n p, whch s a consequence of the ll-posedness of the deconvoluton problem SVD-based flterng. To analyze propertes of the system (4) we nclude a nose vector n,.e., we consder (5) d = Kp true + n. 3 To be precse, the dscrete system (4) s stable, but the stablty constant grows as N becomes larger and (4) becomes a better approxmaton to the unstable contnuous problem (2). The ll-posedness of ths lnear nverse problem s closely related to the fact that the matrx K s ll-condtoned.

7 2. A DEBLURRING PROBLEM 7 Here, p true are the true parameters we are tryng to reconstruct from the data d. The below analyss s based on the sngular value decomposton (SVD) of the matrx K. The SVD decomposton also allows to stablze the nverse problem by flterng. Whle ths regularzaton approach s llustratve and works well for moderate sze nverse problems, t cannot be appled for large-scale nverse problems, where computng an explct SVD decomposton s nfeasble Sngular value decomposton. For a real-valued matrx A R m n, there exst orthogonal matrces (6) U = [u 1,..., u m ] R m m and V = [v 1,..., v n ] R n n such that (7) A = Udag(σ 1,..., σ p )V T, wth p = mn{m, n}, wth σ 1 σ 2... σ p. The σ are the sngular values and the vectors u and v are the left and the rght sngular vectors, respectvely. If A R n n s symmetrc and postve defnte, the sngular values are all postve, they concde wth the egenvalues (.e., λ = σ for 1 n), and U = V. The columns u j of U are then orthonormal egenvectors of A,.e., { Au j = λ j u j, u T 1 f = j u j = δ j := for 1, j n. f j Moreover, the orthonormalty of the matrx U mples that U 1 = U T. Usng the propertes of the SVD and denotng by Λ = dag(λ 1,..., λ n ) the matrx wth egenvalues on the dagonal, we can fnd the nverse of K as K 1 = UΛ 1 U T and obtan from (5) that (8) K 1 d = UΛ 1 U T d = λ 1 (u T d)u = p true + λ 1 (u T n)u. From (8), t can be seen that nstablty arses for small egenvalues λ, snce (8) nvolves terms weghted by λ 1. The egenvalues for K R are shown n Fgure For the convoluton matrx K, as well as for many nverse problems wth PDEs, large egenvalues λ correspond to smooth egenfunctons, and small egenvalues correspond to oscllatory egenfunctons, as can be seen n Fgure 3. Thus, from (8) t follows that oscllatory components cannot relably be reconstructed from nosy data (.e., when n ) snce they correspond to small egenvalues. Often the nose ntroduced by round off error s large enough to render the explct nverson (8) useless Truncated SVD and Tkhonov flterng. As a remedy to the above descrbed problems, one can employ flter methods, whch remove or dampen the terms correspondng to the small egenvalues n (8). Flter functons

8 8 1. INTRODUCTION Fgure 2. Egenvalues of the dscrete convoluton operator K wth N = 128. All egenvalues are postve, the largest beng 1 and the smallest beng about Fgure 3. Egenvectors correspondng to the largest (top left), second largest (top rght), 1th largest (bottom left) and 1th largest (bottom rght) egenvalue. ω(λ ) for = 1,..., n are employed by modfyng (8) as follows: (9) p ω(λ 2 )λ 1 (u T d)u.

9 2. A DEBLURRING PROBLEM 9 A popular choce for a famly of flter functon s { (1) ω α (λ 2 1 f λ 2 α, ) := else, where α > s a regularzaton parameter. Usng ths flter, (9) smplfes to (11) p TSVD = λ λ 2 α 1 (u T d)u, that s, all terms correspondng to egenvalues that are smaller than the square root of α are dropped from the sum. Due to ths truncaton, ths method s known as truncated sngular value decomposton (TSVD). The parameter α controls where the sum s truncated and must be adjusted accordng to the nose level, see the dscusson n Secton 2.2. An alternatve famly of flter functons n (9) s (12) ω α (λ 2 ) = λ2 λ 2 + α, where, agan, α s a regularzaton parameter. Note that ω α (λ 2 ) s close to one when λ 2 α, and t s close to zero when λ 2 α. Thus, ths flter strongly dampens terms correspondng to the small egenvalues of K, whle lettng the terms for large egenvalues (almost) unchanged. It s called Tkhonov flter and uses a smoothed verson of the truncaton flter functon (1). It results n the fltered mage (13) p TIK = λ λ 2 + α(ut d)u. An advantage of the Tkhonov flter compared to the TSVD flter s that t can be computed wthout explct knowledge of the SVD of K. Ths s due to the fact that p TIK can also be found as soluton of the mnmzaton problem 1 (14) mn p 2 Kp d 2 + α 2 p 2, where denoted the Eukldan vector norm n R N. To show that the mnmzer of (14) s p TIK, one uses that the mnmzer s also characterzed by the normal equatons (15) p TIK = ( K T K + αi ) 1 K T d, and uses the SVD decomposton of K n (15). The queston arses f the regularzaton parameter α can be chosen such that the fltered solutons converge as the nose level goes to zero. For TSVD and Tkhonov flterng for the deblurrng problem, ths queston s answered next.

10 1 1. INTRODUCTION A determnstc error analyss. We consder fltered solutons of (5) denoted by (16) K 1 α d := p α = ω(λ 2 )λ 1 (u T d)u, where K α denotes the fltered convoluton matrx correspondng ether to TSVD or Tkhonov flterng wth flter parameter α. Dependng on the choce of α, an error e α n the reconstructon s commtted, namely (17) e α := p α p true = K 1 α (Kp true + n) p true =: e trunc α + e nose α, where the truncaton error due to the regularzaton e trunc α and the nose amplfcaton error e nose α are defned as e trunc α = K 1 ( α Kp true p true = ωα (λ 2 ) 1 ) (u T p true )u, e nose α = K 1 α n = ω α (λ 2 )λ 1 (u T n)u. Next, we show that for Tkhonov flterng and the TSVD, the parameter α can be chosen such that both errors converge to zero as the nose level δ := n goes to zero. We frst estmate the truncaton error: By defnton of the flter weght functons (1) and (12), t follows that ω α (λ 2 ) 1 as α. Ths mmedately mples that (19) e trunc α as α. Next we study the nose amplfcaton factor. By usng the explct form of the flter functons, t can be verfed that (2) ω α (λ 2 )λ 1 1 α for both, TSVD and Tkhonov flterng. Usng the orthonormalty of U, ths mples that e nose α 1 α (u T n)u = δ. α Thus, f we choose the flter parameter as α := δ p wth p < 2 we obtan (21) e nose α as δ. Combnng the requrements for the truncaton and the nose amplfcaton error, the choce α := δ p wth < p < 2 guarantees that e α as the nose level δ. Ths means that the TSVD and Tkhonov flters, together wth the above choce for the regularzaton parameter are convergent. A sgnfcant amount of research n nverse problems deals wth the computaton of

11 2. A DEBLURRING PROBLEM 11 rates for ths convergence n lnear and nonlnear nverse problems. Next, we consder practcal methods for choosng the regularzaton parameters Choce of the regularzaton parameter. As seen above, the choce of the regularzaton parameter α n ether TSVD or Tkhonov flterng s mportant. If α s small, the computaton of the parameter p s unstable as n the case wthout flterng. On the other hand, f α s too large, nformaton s lost n the fltered mage. In ths secton we dscuss methods to choose approprate flter (or regularzaton) parameters. Both methods are a posteror parameter choce methods,.e., they requre the soluton of several regularzed nverse problems to fnd an approprate value for α. Whle the L-curve crteron, whch s presented frst, does not requre knowledge of the nose level, the dscrepancy prncple presented afterwards s based on an estmate of δ := n The L-curve crteron. Choosng the flter parameter usng the L-curve crteron requres the soluton of nverse problems for a sequence of regularzaton parameters α. Then, for each α, the norm of the data msft (also called resdual) Kp α d s plotted aganst the norm of the regularzaton term p α n a log-log plot. Ths curve usually s found to be L-shaped and thus has an elbow,.e. a pont of greatest curvature. The L- curve crteron chooses the regularzaton parameter correspondng to that pont; see the left plot n Fgure 4 for an llustraton. The dea behnd the L-curve crteron s that ths choce for the regularzaton parameter s a good compromse between fttng the data and controllng the stablty of the parameters. A smaller α, whch correspond to ponts to the left of the optmal value n Fgure 4, only leads to a slghtly better data ft whle sgnfcantly ncreasng the norm of the parameters. Conversely, a larger α, correspondng to ponts to the rght of the optmal value, slghtly decrease the norm of the soluton, but they ncrease the data msft sgnfcantly. Provng convergence for ths parameter choce method s problematc and cannot be shown n all cases The dscrepancy prncple. The dscrepancy prncple, due to Morozov, chooses the regularzaton parameter to be the largest value of α such that the norm of the msft s bounded by the nose level n the data,.e., (22) Kp α d δ, where δ s the nose level. Here, p α denotes the parameter found ether usng a TSVD flter or Tkhonov regularzaton wth parameter α. Ths choce ams to avod overfttng of the data,.e., fttng the nose. The crteron s llustrated n the rght plot n Fgure 4. Convergence results and rates for the parameter when determned by the Morozov crteron as the nose level goes to zero are avalable. Next, we prove that for the dscretzed deblurrng problem such a regularzaton parameter α always exsts provded the nose level s less than the

12 12 1. INTRODUCTION p K*p d K*p d α Fgure 4. Choosng the regularzaton parameter α: The red dot on the L-curve (left plot), whch corresponds to the pont wth largest curvature, yelds the optmal flter/regularzaton parameter accordng to the L-curve crteron. For the dscrepancy crteron (rght plot), the optmal parameter corresponds to the ntersecton of the data msft curve wth the red lne ndcatng the nose level. norm of the data,.e., δ < d. For that purpose we defne the functon (23) D(α) := Kp α d Usng the form of the Tkhonov-regularzed parameter p α as gven n (13) and the egenvalue decomposton K = UΛU T, one obtans ( ) λ 2 Kp α d = λ 2 + α 1 (u T d)u, and due to the orthonormalty of U ths mples ( ) λ (24) D (α) = λ 2 + α 1 (u T d) 2. Ths shows that D(α) s contnuous n α, that D() = and that D s monotoncally ncreasng. Moreover, D(α) d as α. Thus, provded δ < d, there exsts an α such that D(α) = δ, as desred. Note that a smlar argument does not work for TSVD flterng, snce the functon D s not contnuous n α for the flter functon (1). Thus, n general, the optmal α accordng to the dscrepancy prncple wll satsfy (22) wth a strct nequalty Varatonal regularzaton methods. As dscussed above, the Tkhonov fltered soluton can also be found through the soluton of an optmzaton problem. Ths has the advantage that no explct SVD s requred. Moreover, such an optmzaton approach allows more flexblty n the choce of norms for the msft and the regularzaton, as s often desred n

13 2. A DEBLURRING PROBLEM 13 varatonal nverse problems,.e., nverse problems that nvolve dfferental equatons. To llustrate ths flexblty we consder the followng generalzaton of the optmzaton problem (14): 1 (25) mn p 2 Kp d 2 + R(p), wth a regularzaton functon R : R N R. Above we have dscussed the choce of R(p) = α/2 p 2. Alternatve choce are the squared dfference operator (26) R 2 (p) = α N 1 (p +1 p ) 2, wth p = (p 1,..., p N ), 2 whch s closely related to the squared gradent f p corresponds to the dscretzaton of a functon. The choce (26) favors the parameter p that has small dfferences between ts components. If the vector p orgnates from a dscretzed parameter functon, (26) expresses a preference for smooth parameter functons. An alternatve choce s to replace the sum of squares by a sum of absolute values N 1 (27) R 1 (p) = α p +1 p. Smlar to (26), the regularzaton (27) favors small dfferences. However, compared to (26) t puts less emphass on large values n the sum. For dscretzed functons, the choce (27) corresponds to total varaton regularzaton, whch s a popular regularzaton for nverse problems, n partcular n magng. Note that R 2 corresponds to the squared Eucldan (also called l 2 -norm) of the dfferences, whle R 1 corresponds to the l 1 norm of the dfferences. Snce R 1 s not dfferentable due to the absolute value, whch makes computng dervatves of (25), as requred by numercal optmzaton methods, challengng.