Augmented Tikhonov Regularization

Transcription

1 Augmented Tikhonov Regularization Bangti JIN Universität Bremen, Zentrum für Technomathematik Seminar, November 14, 2008

2 Outline 1 Background 2 Bayesian inference 3 Augmented Tikhonov regularization 4 Numerical results

3 Outline 1 Background 2 Bayesian inference 3 Augmented Tikhonov regularization 4 Numerical results

4 Linear inverse problem Hm = d, with H R n m ill-conditioned and the noisy data d Example d = d + ω noise vector Fredholm integral equations of the first kind (Tikhonov; Phillips) b a k(s, t)f (t)dt = g(s) inverse Laplace transform: k(s, t) = e st image deblurring

5 Linear inverse problem Hm = d, with H R n m ill-conditioned and the noisy data d d = d + ω noise vector Example Cauchy problem for Laplace equation (Hadamard, 1923) Ω: open bdd. domain with boundary Ω = Γ i Γ c Cauchy data: Dirichlet and Neumann b.c. on Γ c Cauchy problem: recover Dirichlet b.c. on Γ i applications: cardiography, i.h.c.p.

6 Linear inverse problem Hm = d, with H R n m ill-conditioned and the noisy data d d = d + ω noise vector Example Robin inverse problem for Laplace equation (Ingles 1997) Cauchy data: Dirichlet and Neumann b.c. on Γ c Robin inverse problem: recover Robin coefficient on Γ i applications: quenching process, corrosion detection, MOSFET semiconductor device

7 Ill-posedness (Hadamard, 1923) Existence of a solution not ensured Uniqueness of the solution violated Stable dependence on the data missing Tikhonov regularization (Tikhonov, 1960s) { } m η = arg min Hm d 2 m 2 + η Lm 2 2 ker(h) ker(l) = {0} uniqueness L: (often) discretized diff. operator smoothness Regularization parameter η plays an essential role! Data-fidelity term and regularization term can be rather general: impulsive noise, sparsity,...

8 Regularization parameter choice rules Discrepancy principle (Morozov, 1967) Hm η d 2 cδ estimated noise level mathematically rigorous: convergence and convergence rates require an estimate of the noise level very sensitive to under-estimation Quasi-optimality criterion (Tikhonov et al, 1979) η = arg min η η dm η dη no theoretical justification computationally expensive: sampling 2

9 Regularization parameter choice rules Generalized cross-validation (Golub-Heath-Wahba, 1979) G(η) = Hm η d 2 2 tr(i n HH # η ) 2 with H # η = (H T H + ηl T L) 1 H T existence of a minimizer not guaranteed often suffer from severe under-regularization computationally expensive

10 Regularization parameter choice rules L-curve criterion (Hansen, 1992) corner of L-curve L = { (log Hm η d 2, log Lm η 2 ) : η R +} existence of a corner not ensured nonconvergence results (Hanke, 1996; Vogel, 1996) computationally expensive, challenging to locate corner

11 Outline 1 Background 2 Bayesian inference 3 Augmented Tikhonov regularization 4 Numerical results

12 Rev. Thomas Bayes ( ) An essay towards solving a problem in the doctrine of chances, salient features uncertainty quantification flexible regularization

13 Fundamentals m and d are random variables. Goal: deduce the conditional distribution p(m d) (PPDF) of m on d. Bayes rule asserts p(m d) = p(d m)p(m) p(d m)p(m)dm p(d m)p(m) unimportant normalizing constant (a.k.a. Bayes factor) p(d) = p(d m)p(m)dm neglected building blocks likelihood: p(d m) data plays the role! prior distribution: p(m) Feature: The information is a prob. distr., and uncertainties of a specific inverse solution can be quantified!

14 Numerical exploration The PPDF lives in high-dim. space direct vis. impossible summarizing statistics of the PPDF posterior mean ˆm pm ˆm pm = E p(m d) [m] = mp(m d)dm R m maximum a posteriori (MAP) ˆm map marginal distribution p(m i d) ˆm map = arg max p(m d) m Caution: point estimates may not be representative!

15 Numerical strategies integration problems: no smoothness req., curse-of-dim. optimization methods: iterative, gradient type methods... Monte Carlo method Basic idea drawing i.i.d. samples {m i } N i=1 (nontrivial) sample approximation of the mean ˆm pm 1 N N m i, i=1 Traditional: importance sampling, accept-reject Modern: Markov chain Monte Carlo Metropolis-Hastings sampler, Gibbs sampler

16 Noise model additive noise model d i = (Hm) i + ξ i, i = 1, 2,..., n with ξ i being i.i.d. additive Gaussian random variables with mean zero and variance σ 2 = 1 τ, i.e. p(ξ i) = τ 2π e τ 2 ξ2 i p(d m) τ n 2 exp ( τ 2 d Hm 2 2 Remark: this noise model is not always suitable, e.g. multiplicative noise, impulsive noise, counting processes )

17 Prior model most challenging modeling task versatile tools: Markov random field, Gaussian process Markov random field A neighborhood system N is a collection of index sets verifying: i / N i ; j N i iff i N j m discrete MRF w.r.t. N if p(m i m i ) = p(m i m j, j N i ) Hammersley-Clifford theorem: p(m) exp( m i=1 V i(m)) V i (m) m i, m j, j N i German & German (1984, IEEE PAMI paper)

18 Markov random field Example: m lives on one-dim. equi. distr. lattice N i = {i 1, i + 1} V i (m) = λ 2 j N i (m j m i ) 2 p(m) λ m 2 exp ( λ 2 mt Wm ) with W = [w ij ] w ij = { 1, j Ni, 2, j = i. W = L T L with L being first-order difference operator p(m) λ m 2 exp ( λ 2 Lm 2 2) smoothness Remark: MRF can devise TV, L 1,...

19 PPDF Bayes rule PPDF p(m d) (with known τ and λ) ( p(m d) exp τ ) ( 2 d Hm 2 2 exp λ ) 2 Lm 2 2 Tikhonov regularization (MAP ˆm map ) ˆm map arg max m p(m d) = arg min m { } Hm d λτ 1 Lm 2 2 statistical basis: additive Gaussian noise with smoothness prior Regularization parameter η = λτ 1 plays the role of a regularization parameter parameters λ and τ nontrivial to determine!

20 Hierarchical modeling idea: let the data determine λ and τ! conjugate prior (Gamma distr.) for λ and τ p(λ) λ α 0 1 e β 0λ and p(τ) τ α 1 1 e β 1τ PPDF p(m, λ, τ d) p(m, λ, τ d) τ n 2 exp ( τ ) 2 d Hm 2 2 λ m 2 exp ( λ ) 2 Lm 2 2 λ α0 1 e β 0λ τ α1 1 e β 1τ

21 Outline 1 Background 2 Bayesian inference 3 Augmented Tikhonov regularization 4 Numerical results

22 Augmented Tikhonov functional MAP (m, λ, τ) map (m, λ, τ) map = arg max p(m, λ, τ d) = arg min J (m, λ, τ) (m,λ,τ) (m,λ,τ) with functional J (m, λ, τ) being J (m, λ, τ) = τ 2 Hm d λ 2 Lm β 0λ α 0 ln λ α 1 ln τ + β 1τ resembles Tikhonov method, but determines λ and τ automatically augmented Tikhonov regularization strictly biconvex w.r.t. m and (λ, τ) Theorem The functional J (m, λ, τ) has a finite minimizer.

23 Optimality system (with η = λτ 1 ) GSVD ( H T H + ηl T L ) m H T d = 0, 1 2 Lm β 0 α 0 1 λ = 0, 1 2 Hm d β 1 α 1 1 τ = 0. For matrix pair H R n m and L R p m, there holds ( ) Σ 0 H = U X 1, L = V(M 0 0 I p (m p) )X 1, m p where Σ = diag(σ 1,..., σ p ) and M = diag(µ 1,..., µ p ). The ratios γ i = σ i µ i are generalized singular values.

24 Variance estimate a-tikhonov estimate σ 2 (η) = Hmη d β 1 n+2α 1 2 GCV estimate V(η) = Hmη d 2 2 T (η) with T (η) = tr(i n HH # η ) α 1 1, β 1 0 σ 2 (η) V(η) T (η) n Lemma (Hansen,1993) If generalized s.v.s γ i relate as γ 2 i = c γ γ 2 i+1 (0 < c γ < 1), then n m + κ η 1 1 c γ T (η) n m + κ η c γ, where κ η is the number of γ 2 i less than η, i.e. γ 2 k η < γ γ 2 κ η+1.

25 The GCV estimate is only numerically verified. Theorem Let ȳ i = u T i d. Assume that there exist two constants 0 < c d < c γ < 1 such that γi 2 = c γ γi+1 2 and ȳ i 2 = c d ȳi+1 2. Moreover, PU d = 0, then V(η) satisfies [ ] ( ) K η 2 1 c γ σ c κη d 1 c d 2 1 c γc d ȳκ 2 η K η c γ where K η = n m + κ η. E[V(η)] [ K η c γ ] σ 2 0 +

26 Fix τ value at σ 2 0 η satisfies ( ) η Lm η β 0 = 2α 0 σ2 0. existence of a solution ensured! Lemma Assume the random vector ω satisfies ω c ω σ 0. There exists two constants c η,0 and c η,1 dependent on α 0 such that Theorem c η,0 σ 2 0 η c η,1 σ 2 0. For fixed β 0 and α 0 increasing like σ d 0 with 0 < d < 2, the a-tikhonov solution m converges to the generalized least-squares solution as the noise level σ 0 tends to zero.

27 Implications Hierarchical formulations with fixed α 0 and β 0 might fail for arbitrarily varying noise. Strategies to adapt α 0 are necessary. Choice of parameters variance estimate: α 1 1, β 1 0 convergence analysis: α 0 O(σ d 0 )(0 < d < 2), β 0 O( Lm )

28 biconvexity of J (m, λ, τ): alternating iterative algorithm (i) Set k = 0 and choose η 0. (ii) Solve for m k+1 by the Tikhonov regularization method { } m k+1 = arg min Hm d 2 m 2 + η k Lm 2 2. (iii) Update the parameter λ k+1 and τ k+1 by λ k+1 = α Lm k β, τ k+1 = 0 α Hm k+1 d β 1 and set η k+1 = λ k+1 τ 1 k+1. (iv) Check the stopping criterion. If not converged, set k = k + 1 and repeat from (ii).,

29 Theorem Let {(m k, λ k, τ k )} k be the sequence generated by the above algorithm. Then the sequence {J (m k, λ k, τ k )} k converges monotonically. Lemma For any η 0, the sequence {η k } k converges monotonically. Theorem The sequence {(m k, λ k, τ k )} k converges to a critical point (m, λ, τ ) of the functional J (m, λ, τ). Moreover, the convergence of the sequences {λ k } k and {τ k } k is monotonic.

30 Outline 1 Background 2 Bayesian inference 3 Augmented Tikhonov regularization 4 Numerical results

31 Phillips problem (FIE of 1st kind): Let φ(t) = [ 1 + cos( πt 3 )] χ t <3. The functions k, f, and g are respectively given by k(s, t) = φ(s t), f (t) = φ(t) and g(s) = (6 s ) ( ) πs 2 cos π sin( π 3 s ). The integration interval is [ 6, 6]. The problem is mildly ill-posed, with Cond(H) = The probability densities are calculated from 1000 Monte Carlo simulations.

32 Comparison of the regularization parameter and accuracy error. Observations AT works as well as QO, LC, and robust GCV is sensitive to noise realization

33 a-tikhonov estimate agrees well with GCV estimate, albeit slightly smaller GCV occasionally fails due to too small regularization parameter variance estimates

34 Convergence of the algorithm. Observations Variance estimate varies slowly during the iteration Converged value of λ is relatively independent of the noise level, and thus η σ 2 0 under-regularization

35 Convergence of the algorithm. Observations η k converges monotonically! AT under-regularizes for 1% noise!

36 Sensitivity analysis in case of 5% noise. β 0 σ λ η e 1e2 1.34e e e e0 1e0 1.36e e0 1.74e e-1 1e e e1 2.16e e-1 1e e e1 2.17e e-1 1e e e1 2.17e e-1 1e e e1 2.17e e-1 Observation Small β 0 yields practically identical results. Large β 0 under-regularizes.

37 Convergence of the algorithm (inverse Laplace transform). Observations The variance estimate converges within one iteration! The functional value increases with the noise level.

38 Summary Summary brief introduction to Bayesian inference propose an augmented Tikhonov functional mathematical analysis of a-tikhonov functional propose an efficient iterative algorithm Future research problems convergence rate analysis of the algorithm adaptive strategies

39 Further reading list 1 Gelman A, Carlin JB, Stern HS, Rubin DB. Bayesian Data Analysis (2nd edn). CRC, Vogel CR. Computational Methods for Inverse Problems. SIAM, Engl HW, Hanke M, Neubauer A. Regularization of Inverse Problems. Kluwer, Kaipio J, Somersalo E. Statistical and Computational Inverse Problems. Springer, Jin B, Zou J. Augmented Tikhonov regularization, in press. 6 Ito K, Jin B, Zou J. A new choice rule of regularization parameters in Tikhonov regularization, submitted.