Parameter Identification in Partial Differential Equations

Parameter Identification in Partial Differential Equations Differentiation of data Not strictly a parameter identification problem, but good motivation. Appears often as a subproblem. Given noisy observation of a function f, Martin Burger http://www.indmath.uni-linz.ac.at/people/burger Johannes Kepler Universität Linz Find its derivative Winter School Inverse Problems, Geilo 2 Differentiation of data Pointwise noise at measurement points identically normally distributed with mean 0 and variance δ. Differentiation of data Example: Law of large numbers yields Noise satisfies But error in derivative is large Winter School Inverse Problems, Geilo 3 Winter School Inverse Problems, Geilo 4

Differentiation of data Conclusion 1 Error: Arbitrarily large since k can be arbitrarily large Winter School Inverse Problems, Geilo 5 Winter School Inverse Problems, Geilo 6 Differentiation of data Additional information: Solution is smooth (e.g. twice differentiable) Regularization: e.g. smoothing by elliptic PDE Differentiaton of data Detailed estimate: Lecture notes, p. 7 Variational principle for elliptic PDEs: minimize Winter School Inverse Problems, Geilo 7 Winter School Inverse Problems, Geilo 8

Computerized Tomography (A nice link between Austria & Norway) Problem: reconstruct a spatial density f in a domain D from measurements of X-rays traveling through the domain. Computerized Tomography X-ray beam has intensity I Model: decay of itensity in a small distance t proportional to I, f, and distance X-rays travel along rays, parametrized by distance s from origin and its unit normal vector ω Limit t to zero Winter School Inverse Problems, Geilo 9 Winter School Inverse Problems, Geilo 10 Computerized Tomography Measurement: intensity at emitter and detector Computerized Tomography Integrate ODE Parameter identification problem for system of ordinary differential equations (first-order) Leads to inversion of the Radon transform Overdetermined for given f since initial and final value are known Winter School Inverse Problems, Geilo 11 Winter School Inverse Problems, Geilo 12

Computerized Tomography Radon Transform Computerized Tomography Radial symmetry, Exact inversion formula by Johann Radon 1917. SVD computed by McCormick et. al. in 1960s, Nobel Prize for Medicine in the 1970 Winter School Inverse Problems, Geilo 13 Winter School Inverse Problems, Geilo 14 Computerized Tomography Radial symmetry, Groundwater Filtration Identification of diffusivity of sediments from an observation of the piezometric head (describing the flow) Abel Integral equation! SVs decay with half speed as for differentiation Knowledge of diffusivity allows to draw conclusions about the structure of the groundwater Winter School Inverse Problems, Geilo 15 Winter School Inverse Problems, Geilo 16

Groundwater Filtration Mathematical model Groundwater Filtration Instead of exact data, we only know noisy measurement Diffusivity a, piezometric head u (measured), density of water sources and sinks f Typically the noise n(x) comes from identically normally distributed random variables From a law of large numbers this gives an estimate + Appropriate boundary conditions (e.g. u=0) Winter School Inverse Problems, Geilo 17 Winter School Inverse Problems, Geilo 18 Groundwater Filtration Consider 1D version of inverse problem, for simplicity with We can integrate the equation to obtain Groundwater Filtration Detailed estimate: Lecture notes, p. 9 Hence, if the derivative of u does not vanish, a is determined uniquely (Identifiability) Winter School Inverse Problems, Geilo 19 Winter School Inverse Problems, Geilo 20

Groundwater Filtration Some results (from Hanke, 1995) Groundwater Filtration Reconstructions for two different noise levels (1% and 0.1%) Exact phantom Winter School Inverse Problems, Geilo 21 Winter School Inverse Problems, Geilo 22 Groundwater Filtration Reconstruction along the diagonal Groundwater Filtration Reconstructions of piecewise constant parameter, no noise (nonuniqueness curve), fine grid Winter School Inverse Problems, Geilo 23 Winter School Inverse Problems, Geilo 24

Groundwater Filtration Reconstructions of piecewise constant parameter, no noise (nonuniqueness curve), hierarchical grids Electrical Impedance Tomography Application in Medical Imaging Set of electrodes placed around human chest, Response to various electrical impulses measured Picture from RPI-EIT Project Winter School Inverse Problems, Geilo 25 Winter School Inverse Problems, Geilo 26 Electrical Impedance Tomography Electrical Impedance Tomography Picture from RPI-EIT Project Pictures from RPI-EIT Project Winter School Inverse Problems, Geilo 27 Winter School Inverse Problems, Geilo 28

Electrical Impedance Tomography Mathematical model: Maxwell equations, reduced to potential equation Electrical Impedance Tomography Measurement: current density on the boundary for different voltage patterns u is electrical potential, f applied voltage pattern a denotes the (unknown) conductivity Idealized mathematical model: Dirichlet-Neumann map Winter School Inverse Problems, Geilo 29 Winter School Inverse Problems, Geilo 30 Electrical Impedance Tomography INVERSE CONDUCTIVITY PROBLEM In practice finite number of measurements Electrical Impedance Tomography To compute output, we have to solve solutions of N partial differential equations N typically very large Problem of extremely large scale Winter School Inverse Problems, Geilo 31 Winter School Inverse Problems, Geilo 32

Electrical Impedance Tomography Interesting special case: piecewise constant conductivities Semiconductor Devices Related problem to impedance tomography appears for semiconductor devices: inverse dopant profiling Real interest is the subset where a takes a value different from a 1 (e.g. a tumour) Winter School Inverse Problems, Geilo 33 Winter School Inverse Problems, Geilo 34 Inverse Dopant Profiling Identify the device doping profile from measurements Current-Voltage map Analogous definitions of voltage and current, but more complicated mathematical model Mathematical Model Stationary Drift Diffusion Model: PDE system for potential V, electron density n and hole density p in Ω (subset of R 2 ) Doping Profile C(x) enters as source term Winter School Inverse Problems, Geilo 35 Winter School Inverse Problems, Geilo 36

Boundary Conditions Boundary of Ω : homogeneous Neumann boundary conditions on Γ N and Device Characteristics Measured on a contact Γ 0 on Γ D : Outflow current density on Dirichlet boundary Γ D (Ohmic Contacts) Winter School Inverse Problems, Geilo 37 Winter School Inverse Problems, Geilo 38 Numerical Tests Test for a P-N Diode Numerical Tests Different Voltage Sources Real Doping Profile Initial Guess Winter School Inverse Problems, Geilo 39 Winter School Inverse Problems, Geilo 40

Numerical Tests Reconstructions with first source Numerical Tests Reconstructions with second source Winter School Inverse Problems, Geilo 41 Winter School Inverse Problems, Geilo 42 Polymer Crystallization Polymer Crystallization Mesoscale model for polymer crystallization ξ = degree of crystallinity u,v = surface densities T = temperature G = growth rate N = nucleation rate Winter School Inverse Problems, Geilo 43 Winter School Inverse Problems, Geilo 44

Polymer Crystallization Source term Polymer Crystallization Traditional way of determining nucleation rate as function of temperature: - Make separate experiment for each value of T - Count (by eyes!!!) the final number of crystals (typically >> 10 6 ). - Divide by volume Extremely expensive, extremely timeconsuming! Winter School Inverse Problems, Geilo 45 Winter School Inverse Problems, Geilo 46 Polymer Crystallization Idea: determine nucleation rate as function of temperature by single nonisothermal experiment Polymer Crystallization Results Measured data: temperature T at the boundary of the sample Degree of crystallinity at final time Winter School Inverse Problems, Geilo 47 Winter School Inverse Problems, Geilo 48

Polymer Crystallization Results Polymer Crystallization Results Winter School Inverse Problems, Geilo 49 Winter School Inverse Problems, Geilo 50 Polymer Crystallization Reconstruction error vs noise level Polymer Crystallization Reconstruction error vs noise level Winter School Inverse Problems, Geilo 51 Winter School Inverse Problems, Geilo 52

Polymer Crystallization Iteration numbers Polymer Crystallization Iteration numbers Winter School Inverse Problems, Geilo 53 Winter School Inverse Problems, Geilo 54 Iterative Regularization For nonlinear inverse problems, iterative regularization is of even higher importance as for linear ones We need to perform iteration to minimize nonlinear Tikhonov functional anyway Iterative Regularization Gradient of the least-squares functional Perform simple gradient descent: Start from least-squares functional Landweber iteration Winter School Inverse Problems, Geilo 55 Winter School Inverse Problems, Geilo 56

Iterative Regularization Where is the regularization parameter??!! Iterative Regularization Regularization by appropriate early stopping, e.g. discrepancy principle Each step of Landweber iteration is well-posed, so we obtain iterative regularization method Winter School Inverse Problems, Geilo 57 Winter School Inverse Problems, Geilo 58 Iterative Regularization Analysis of Landweber iteration: Hanke-Neubauer- Scherzer 1994, Scherzer 1995 Usual semi-convergence properties as for linear problems, but restriction on the nonlinearity of the problem (replacing regularity of ) Iterative Regularization Landweber iteration is forward Euler time discretization (with time step τ of the flow) Asymptotical regularization (Tautenhahn 1994) Iteration Winter School Inverse Problems, Geilo 59 Winter School Inverse Problems, Geilo 60

Iterative Regularization We can consider other time discretizations: Backward Euler Iterative Regularization We can consider other time discretizations: Semi-Implicit Euler Iterated Tikhonov Regularization (Hanke- Groetsch 1999, Groetsch-Scherzer 1999) Levenberg-Marquardt (Hanke 1995) Arbitrary Runge-Kutta methods possibly, even inconsistent ones (Rieder, 2005) Winter School Inverse Problems, Geilo 61 Winter School Inverse Problems, Geilo 62 Iterative Regularization Levenberg-Marquardt can be rewritten to Iterative Regularization General approach for Newton-type methods: Apply linear regularization to Tikhonov regularization of the linear Newtonequation Iteratively regularized Gauss-Newton (Kaltenbacher-Neubauer-Scherzer 1996, Kaltenbacher 1997) Newton-CG (Hanke, 1998) Winter School Inverse Problems, Geilo 63 Winter School Inverse Problems, Geilo 64

Iterative Regularization Total Variation Denoising Newton-Landweber (Kaltenbacher 1999) Broyden (Kaltenbacher, 1997) Multigrid / Discretization (Kaltenbacher, 2000-2003) In particular for parameter identification: Levenberg-Marquardt SQP (B-Mühlhuber 2002) Newton-Kaczmarcz (B-Kaltenbacher 2004) Winter School Inverse Problems, Geilo 65 Winter School Inverse Problems, Geilo 66 Total Variation Denoising TV-Images Basic problem in denoising: given noisy version g of image f, find optimal approximation u of f Basic problem in deblurring: given noisy version g of Kf, find optimal approximation u of f Winter School Inverse Problems, Geilo 67 Winter School Inverse Problems, Geilo 68

TV-Images What is optimal? Satisfy other criterion, e.g., minimization ROF-Model What is a suitable class of approximations? Discrepancy principle: allow images with with σ being the noise level: over a class of approximations Winter School Inverse Problems, Geilo 69 Winter School Inverse Problems, Geilo 70 ROF-Model Rudin-Osher-Fatemi 1989 (ROF): Minimize total variation ROF-Model Deblurring: Minimize total variation subject to subject to Winter School Inverse Problems, Geilo 71 Winter School Inverse Problems, Geilo 72

ROF-Model Acar-Vogel 1994: ROF is regularization method (wellposedness + convergence as σ to zero) Chambolle-Lions 1997: Solution unique, there exists Lagrange multiplier λ, problem equivalent to ROF-Model Acar-Vogel 1994: ROF is regularization method (wellposedness + convergence as σ to zero) Chambolle-Lions 1997: Solution unique, there exists Lagrange multiplier λ, problem equivalent to Winter School Inverse Problems, Geilo 73 Winter School Inverse Problems, Geilo 74 Total Variation Rigorous definition of total variation Total Variation BV includes discontinuous functions: 1D example Winter School Inverse Problems, Geilo 75 Winter School Inverse Problems, Geilo 76

Total Variation 1D example Total Variation Formal optimality for TV-Denoising Term on the right-hand side corresponds to mean curvature of level sets of u Winter School Inverse Problems, Geilo 77 Winter School Inverse Problems, Geilo 78 Duality Consider TV-regularization Duality Exchange inf and sup Solve inner inf problem for u Winter School Inverse Problems, Geilo 79 Winter School Inverse Problems, Geilo 80

Duality Remains sup problem for g Duality Introduce Rewrite equivalently Dual TV problem (Chambolle 2003) Dual space of BV Winter School Inverse Problems, Geilo 81 Winter School Inverse Problems, Geilo 82 Stair-Casing Lecture notes p.20 ROF-Model Meyer 2001 (and others before): ROF has a systematic error, since This means that jumps in the reconstructed image are smaller than jumps in the true image Winter School Inverse Problems, Geilo 83 Winter School Inverse Problems, Geilo 84

Noise Decomposition How to resolve the systematic error? Take some λ, and minimize TV-functional Noise Decomposition Minimize to obtain. Then take residual ( noise ) to obtain next iterate The second step may increase total variation! Winter School Inverse Problems, Geilo 85 Winter School Inverse Problems, Geilo 86 Iterated Total Variation Some Results Obtain new iterate by minimizing then decompose noise Need not change the code, just the data! Winter School Inverse Problems, Geilo 87 Winter School Inverse Problems, Geilo 88

Some Results Some Results Winter School Inverse Problems, Geilo 89 Winter School Inverse Problems, Geilo 90 Some Results Some Results Winter School Inverse Problems, Geilo 91 Winter School Inverse Problems, Geilo 92

Some Results Convergence Analysis So why does that work??? Expand fitting term, throw away constant terms: Winter School Inverse Problems, Geilo 93 Winter School Inverse Problems, Geilo 94 Convergence Analysis Optimality condition implies Convergence Analysis Hence, iterated TV is proximal point algorithm Write equivalent optimization problem Winter School Inverse Problems, Geilo 95 Winter School Inverse Problems, Geilo 96

Convergence Analysis The second term is called Bregman distance Convergence Analysis Example: quadratic case with yields Winter School Inverse Problems, Geilo 97 Iterated Tikhonov / Levenberg-Marquardt Method! Winter School Inverse Problems, Geilo 98 Inverse TV Flow Does the reconstruction depend on λ? To understand the dependence on the parameter, consider limit λ 0. Let λ N = T/N, set t k = k λ N. We compute the solution of the iterated TV flow with this specific parameter and set Inverse TV Flow We obtain a limiting flow, This flow is uniquely defined Formally, Linear interpolation for other times. Winter School Inverse Problems, Geilo 99 Winter School Inverse Problems, Geilo 100

Inverse TV Flow The inverse TV flow can be considered as an inverse scale space method (Scherzer-Weickert 99) Start with zero image, and gradually insert smaller and smaller scales. Noise should return in the end, this happened in experiments Inverse TV Flow The parameter λ can be interpreted as a time step for an implicit discretization of the inverse TV flow Should work well and be independent of the parameter, as long as λ is small enough. This can be observed in experiments, too. Winter School Inverse Problems, Geilo 101 Winter School Inverse Problems, Geilo 102 The algorithm can immediately be generalized to arbitrary linear and even nonlinear inverse problems: Find next iterate as minimizer of Winter School Inverse Problems, Geilo 103 Example: estimate diffusion coefficient q in from measurement of u Operator F: q u(q) Application: find material layers in groundwater modeling (piecewise constant) Winter School Inverse Problems, Geilo 104

Exact data Iterate 1 Exact data Iterate 2 Winter School Inverse Problems, Geilo 105 Winter School Inverse Problems, Geilo 106 Exact data Iterate 3 Exact data Iterate 4 Winter School Inverse Problems, Geilo 107 Winter School Inverse Problems, Geilo 108

Exact data Iterate 5 Exact data Iterate 6 Winter School Inverse Problems, Geilo 109 Winter School Inverse Problems, Geilo 110 Exact data Iterate 10 Exact data Iterate 20 Winter School Inverse Problems, Geilo 111 Winter School Inverse Problems, Geilo 112

Exact data Iterate 50 Exact data Iterate 100 Winter School Inverse Problems, Geilo 113 Winter School Inverse Problems, Geilo 114 Exact data Iterate 200 1% noise Iterate 1 Winter School Inverse Problems, Geilo 115 Winter School Inverse Problems, Geilo 116

1% noise Iterate 2 1% noise Iterate 3 Winter School Inverse Problems, Geilo 117 Winter School Inverse Problems, Geilo 118 1% noise Iterate 4 1% noise Iterate 5 Winter School Inverse Problems, Geilo 119 Winter School Inverse Problems, Geilo 120

1% noise Iterate 10 1% noise Iterate 50 Winter School Inverse Problems, Geilo 121 Winter School Inverse Problems, Geilo 122 10% noise Iterate 1 10% noise Iterate 2 Winter School Inverse Problems, Geilo 123 Winter School Inverse Problems, Geilo 124

10% noise Iterate 3 10% noise Iterate 4 Winter School Inverse Problems, Geilo 125 Winter School Inverse Problems, Geilo 126 10% noise Iterate 5 10% noise Iterate 10 Winter School Inverse Problems, Geilo 127 Winter School Inverse Problems, Geilo 128

Parameter Identification Implementation Aspects Landweber iteration: Discretize parameter Discretize state + state equation Compute gradient by adjoint method Try to use same kind of discretization for adjoint as for state equation ( Discretized Adjoint = Adjoint of Discretized Problem ) For multiple state equations compute gradients corresponding to each state equation immediately Winter School Inverse Problems, Geilo 129 Winter School Inverse Problems, Geilo 130 Implementation Aspects Landweber iteration with Black Box Solver: Discretize parameter Discretize state + state equation (determined by solver) If adjoint method not possible, try to compute gradients by finite differencing For multiple state equations compute gradients corresponding to each state equation immediately Implementation Aspects Quasi-Newton Methods (BFGS): Discretize parameter Discretize state + state equation (determined by solver) Compute gradients by adjoint method, use adjoint to update quasi-newton matrix Solve linear system for update Winter School Inverse Problems, Geilo 131 Winter School Inverse Problems, Geilo 132

Implementation Aspects Newton Methods (LM / IRGN / NCG): Discretize parameter Discretize state + state equation Never compute Newton-Matrix (NOT SPARSE)!! Implementation Aspects One shot methods: Discretize parameter + state + state equation + adjoint + adjoint equation Solve the full KKT system simultaneously, e.g. Try to use iterative method for computing the Newton step, only implement application of and its adjoint (two PDEs) Winter School Inverse Problems, Geilo 133 Winter School Inverse Problems, Geilo 134 Implementation Aspects One shot methods: KKT system is sparse and symmetric KKT system is not positive definite (no CG!) Use BiCGStab, MINRES, QMR or GMRES Apply appropriate preconditioning, note that there is a small parameter (α), acting like singular perturbation Preconditioning strategies: Battermann-Sachs 2000, Battermann-Heinckenschloss 2001, Ascher-Haber 2001-2003, B-Mühlhuber 2002, Griewank 2005 Kaczmarcz Methods For problems like impedance tomography, the operators are of the form Idea: perform multiplicative splitting (like in Gauss- Seidel), cyclic iteration over the N subproblems Winter School Inverse Problems, Geilo 135 Winter School Inverse Problems, Geilo 136

Kaczmarcz Methods Subproblem j: Landweber-Kaczmarcz (Natterer 1996, Kowar-Scherzer 2003) Kaczmarcz Methods Levenberg-Marquardt Kaczmarcz (B-Kaltenbacher 2004) State equation: Linear KKT: Winter School Inverse Problems, Geilo 137 Winter School Inverse Problems, Geilo 138 Kaczmarcz Methods Simple model problem: identify q in Numerical Solution Newton-Kaczmarz approach is equivalent to minimizing from measurements for in each step, subject to for different sources Winter School Inverse Problems, Geilo 139 Winter School Inverse Problems, Geilo 140

Numerical Solution Since we need another function to realize the H -1/2 -norm, corresponding KKT system is 4 x 4 Numerical Solution With boundary conditions and Winter School Inverse Problems, Geilo 141 Winter School Inverse Problems, Geilo 142 Numerical Results Numerical experiment with N=20 localized sources g j (5 on each side) Numerical Results Iterates 1, 2, 3, 4, 5, 6 Exact solution Constant initial value Winter School Inverse Problems, Geilo 143 Winter School Inverse Problems, Geilo 144

Numerical Results Iterates 10, 15, 20, 40, 60, 80 (1/2,3/4,1,2,3,4) Numerical Results Residuals and Error Winter School Inverse Problems, Geilo 145 Winter School Inverse Problems, Geilo 146 Numerical Results Numerical experiment with N=20 localized sources g j (5 on each side) Numerical Results Iterates 1, 6, 11, 16, 21, 41 Exact solution Constant initial value Winter School Inverse Problems, Geilo 147 Winter School Inverse Problems, Geilo 148

Numerical Results Iterates 80, 120, 160, 240, 320, 400 (4,6,8,12,16,20) Numerical Results Residuals and Error Winter School Inverse Problems, Geilo 149 Winter School Inverse Problems, Geilo 150