Parallelizing large scale time domain electromagnetic inverse problem

Transcription

1 Parallelizing large scale time domain electromagnetic inverse problems Eldad Haber with: D. Oldenburg & R. Shekhtman + Emory University, Atlanta, GA + The University of British Columbia, Vancouver, BC, Canada 14th February 2005

2 Outline The Electromagnetic Forward Problem Reduced Space Methods Examples Summary

3 Outline The Electromagnetic Forward Problem Reduced Space Methods Examples Summary

4 Outline The Electromagnetic Forward Problem Reduced Space Methods Examples Summary

5 Outline The Electromagnetic Forward Problem Reduced Space Methods Examples Summary

6 Outline The Electromagnetic Forward Problem Reduced Space Methods Examples Summary

7 Outline The Electromagnetic Forward Problem Reduced Space Methods Examples Summary

8 Mathematical Model Discretization Solving the linear systems Implementation The Electromagnetic Forward Problem Maxwell s equations Discretization Time stepping Solution of the system

9 Mathematical Model Discretization Solving the linear systems Implementation The Electromagnetic Forward Problem Maxwell s equations Discretization Time stepping Solution of the system

10 Mathematical Model Discretization Solving the linear systems Implementation The Electromagnetic Forward Problem Maxwell s equations Discretization Time stepping Solution of the system

11 Mathematical Model Discretization Solving the linear systems Implementation The Electromagnetic Forward Problem Maxwell s equations Discretization Time stepping Solution of the system

12 Mathematical Model Discretization Solving the linear systems Implementation The Electromagnetic Forward Problem Electromagnetic signals are governed by Maxwell s equations E µh t = 0 H + σe = 0 H(0, x) = H 0 (x)

13 Solving Maxwell s Equations Mathematical Model Discretization Solving the linear systems Implementation Conservative mimetic discretization for differential operators E da = E dl S Physical discretization of constitutive relations σe = J µh = B

14 Solving Maxwell s Equations Mathematical Model Discretization Solving the linear systems Implementation Conservative mimetic discretization for differential operators E da = E dl S Physical discretization of constitutive relations σe = J µh = B

15 Solving Maxwell s Equations Mathematical Model Discretization Solving the linear systems Implementation Conservative mimetic discretization for differential operators E da = E dl S Physical discretization of constitutive relations σe = J µh = B

16 Discretization - Time stepping Mathematical Model Discretization Solving the linear systems Implementation After discretization get the (very large) ODE u t = A(m)u u(0) = u 0 Use BE to obtain (I ka(m))u n+1 u n = 0 or B(m) 0 u 1 I B(m).. I B(m) u n = u

17 Discretization - Time stepping Mathematical Model Discretization Solving the linear systems Implementation After discretization get the (very large) ODE u t = A(m)u u(0) = u 0 Use BE to obtain (I ka(m))u n+1 u n = 0 or B(m) 0 u 1 I B(m).. I B(m) u n = u

18 Solving the linear systems Mathematical Model Discretization Solving the linear systems Implementation Systems highly ill-conditioned (due to null space of the curl). Roughly 10 6 d.o.f for every time step. Reformulate the system using potentials A φ, [Haber & Ascher 02] Multigrid preconditioner [Haber & Ascher 02, Ascher & Arulia 03]

19 Implementation Outline Mathematical Model Discretization Solving the linear systems Implementation F90 code Black Box MG for preconditioning (Thanks to D. Moulton) Coarse grain of the preconditioner

20 A Numerical Example Outline Mathematical Model Discretization Solving the linear systems Implementation

21 A Numerical Example Outline Mathematical Model Discretization Solving the linear systems Implementation

22 A Numerical Example Outline Mathematical Model Discretization Solving the linear systems Implementation

23 Formulation Regularization All At Once Reduced Space The Electromagnetic Inverse Problem Find the conductivity σ = e m given the measured data E or H Similar to inverse heat equation, ill-posedness. Need to regularize. Form as a constrained optimization problem.

24 Formulation Regularization All At Once Reduced Space The Electromagnetic Inverse Problem Find the conductivity σ = e m given the measured data E or H Similar to inverse heat equation, ill-posedness. Need to regularize. Form as a constrained optimization problem.

25 Formulation Regularization All At Once Reduced Space The Electromagnetic Inverse Problem Find the conductivity σ = e m given the measured data E or H Similar to inverse heat equation, ill-posedness. Need to regularize. Form as a constrained optimization problem.

26 Formulation Regularization All At Once Reduced Space The Electromagnetic Inverse Problem Find the conductivity σ = e m given the measured data E or H Similar to inverse heat equation, ill-posedness. Need to regularize. Form as a constrained optimization problem.

27 Mathematical Formulation Formulation Regularization All At Once Reduced Space min s.t 1 2 Qu d 2 + βr( m) B(m) 0 I B(m).. I B(m) u 1 u n = u d - observed data Q - Projection R( m) - Regularization

28 More about regularization Formulation Regularization All At Once Reduced Space Two types of regularizations Smoothness R( m) = 1 2 W m 2 Huber R( m) = ρ( m ) Allow to recover discontinuous models

29 The all at once approach Formulation Regularization All At Once Reduced Space Find the saddle point of the Lagrangian L = 1 2 Qu d 2 + βr( m) + λ (A(m)u q) Advantages Derivatives easy to calculate Only sparse matrices Fast algorithms for optimization

30 The Euler Lagrange Equations Formulation Regularization All At Once Reduced Space B(m) I 0 B(m) B(m) 0 I B(m).. I ( βr (m) + G 1,..., G n B(m) I B(m) u 1 u n ) λ = 0 λ 1 λ n = = Q (d Qu) u

31 The all at once approach Formulation Regularization All At Once Reduced Space Solve the Euler-Lagrange equations simultaneously using Newton s method A G δu Q Q A δλ = rhs G βr δm Major Disadvantage: Need to store (u, λ, m) Example - for 64 3 grid in space and 32 times need roughly 8 GB of RAM

32 Reduced Space Outline Formulation Regularization All At Once Reduced Space Eliminate u and λ by solving the forward and adjoint equations. Calculate the reduced gradient Approximate the reduced Hessian and solve for δm

33 Reduced Space Methods Formulation Regularization All At Once Reduced Space set set u 1 u n λ 1 λ n = = B(m) 0 I B(m).. I B(m) Calculate g r (m) = βr (m) + I 0 B(m) B(m) ( G 1,..., G n u I B(m) ) λ 1 Q (d Qu)

34 Reduced Space Methods: Cost Formulation Regularization All At Once Reduced Space COST/ITER = GRADIENT {}}{ (FORWARD + ADJOINT) + #(CGITER) (FORWARD + ADJOINT) }{{} INVERT HESSIAN

35 Outline Overview The EL equations Algorithm The Outer Iteration Reduced space methods are hard to parallelize. Main difficulty - Forward and adjoint are IVP. However, the inverse problem is a BVP in space-time

36 Outline Overview The EL equations Algorithm The Outer Iteration Reduced space methods are hard to parallelize. Main difficulty - Forward and adjoint are IVP. However, the inverse problem is a BVP in space-time

37 Outline Overview The EL equations Algorithm The Outer Iteration Reduced space methods are hard to parallelize. Main difficulty - Forward and adjoint are IVP. However, the inverse problem is a BVP in space-time

38 Outline Overview The EL equations Algorithm The Outer Iteration Reduced space methods are hard to parallelize. Main difficulty - Forward and adjoint are IVP. However, the inverse problem is a BVP in space-time

39 - Our focus Overview The EL equations Algorithm The Outer Iteration Local computational ability high but slow communications Look for algorithms that require little communication Use time decomposition to parallelize the problem

40 Overview The EL equations Algorithm The Outer Iteration The Euler Lagrange Equations - again B I B B 0 I B (I) B I (I) B B I I B I B B ( βr (m) + G 1,..., G n u 1 u n λ 1 λ n 0 = Q (d Qu) λ k u 0 0 = 0. u k ) λ =

41 Overview The EL equations Algorithm The Outer Iteration The Euler Lagrange Equations - again B B 0 I B I I B B I I B B I I B B ( βr (m) + G 1,..., G n I B u 1 u n λ 1 λ n 0 = Q (d Qu) λ k u 0 0 = 0. u k ) λ =

42 Algorithm Outline Overview The EL equations Algorithm The Outer Iteration partition the time into two macro-times [0, t k ], [t k, T] guess u(t k ) and λ(t k ) solve the nonlinear problem in parallel update u(t k ) and λ(t k ) Nonlinear block Jacobi iteration Need globalization strategy Converges linearly, but easily parallelizable

43 A different interpretation Overview The EL equations Algorithm The Outer Iteration Solve the inverse using an approximation to the true forward problem min φ(m, u) B 0 I B subject to I B I B I B u 1 u n u 0 = 0 u k 0 Update the approximation

44 Open Questions Outline Overview The EL equations Algorithm The Outer Iteration What are the true convergence properties of the method? Global convergence to a min How well should we solve each subproblem?

45 Open Questions Outline Overview The EL equations Algorithm The Outer Iteration What are the true convergence properties of the method? Global convergence to a min How well should we solve each subproblem?

46 Open Questions Outline Overview The EL equations Algorithm The Outer Iteration What are the true convergence properties of the method? Global convergence to a min How well should we solve each subproblem?

47 The outer iteration Outline Overview The EL equations Algorithm The Outer Iteration Need to solve a few problems for the appropriate regularization parameter Use a multiscale approach to get the regularization parameter Ascher & Haber 02 Use fields u k from previous regularization parameters

48 Examples Outline Examples Discretization size Solve time (forward) 1h, 1,2,4,8,16

49 Examples Outline Examples β ND Out Iter Av In Iter Rel Time Misfit Table: Experiments for 3D EM problem

50 Examples

51 Examples

52 Examples II Outline Examples Discretization size Solve time (forward) 1h, 1,2,4,8,16

53 Examples

54 Examples

55 Examples β ND Out Iter Av In Iter Rel Time Misfit Table: Experiments for 3D EM problem II

56 Conclusion Outline Examples Formulate and solve inverse Maxwell s equation in time Standard approach - sequential using time decomposition