Strongly Scalable Parallel Simulations of High-resolution Models in Computational Cardiology

Transcription

1 Strongly Scalable Parallel Simulations of High-resolution Models in Computational Cardiology Christoph Augustin, Gernot Plank in coop. G. Haase, M. Liebmann, O. Steinbach, G. Holzapfel, A. Neic, A. Prassl, T. Fastl, T. Eriksson, A. Crozier Medical University of Graz SFB Mathematical Optimization and Applications in Biomedical Science Modeling and Simulations in Biomechanics, September 15 th, / 23

2 Outline Electromechanical Modeling Parallel Strategies Configuration Numerical Examples Open Tasks and Perspectives 1 / 23

3 Outline Electromechanical Modeling Parallel Strategies Configuration Numerical Examples Open Tasks and Perspectives 2 / 23

4 Mechanics of Cardiovascular Tissues collagen fiber Find the displacement u such that Div FS(u, x) = 0 u(x) = u D (x) FS(u, x)n(x) = t N (x) matrix sheet-normal axis for x Ω, for x ΓD, for x ΓN. muscle fiber n0 S = Sp (u, x) + Sa (Vm, η, u, x) F = I + Grad u the deformation gradient Sp the passive 2nd Piola-Kirchhoff stress tensor1,2 sheet axis s0 f0 fiber axis Sa the active 2nd Piola-Kirchhoff stress tensor2,3 u D the prescribed displacement t N the prescribed traction n the normal vector Vm the transmembrane voltage and η state variables 1 Holzapfel and Ogden Philos. Trans. R. Soc. Lond. Ser. A, pp Eriksson et al Mathematics and Mechanics of Solids, pp Smith et al Acta Numer., pp C. Augustin Modeling and Simulations in Biomechan 3 / 23

5 Passive Stress Model Constitutive equation using the free-energy function Ψ S p = 2 Ψ(C), Ψ(C) = U(F)+Ψiso(C)+Ψaniso(C), Ψ (locally) convex C nearly incompressible: penalty with κ e.g., U(J) = κ 2 (J 1)2, J = det F isotropic components: ground matrix, elastin e.g., Ψ iso(c) = c 2 (J 2/3 tr(c) 3) anisotropic components 4,5 : fibers, sheets Ψ aniso(c) = a { exp[b(j 2/3 I f 1) 2 ] 1 } 2b invariant I f = Ff 0 Ff 0: stretch in fiber direction 4 Fung American Journal of Physiology, pp Eriksson, Gasser, Holzapfel, Ogden, Stiffness ds/d λ, kpa 600 Unloading Loading Stress S (F/A), kpa 4 / 23

6 Electical Activation in the Myocardium The Bidomain equations 6,7 describe the spread of cardiac electrical activity find (V m, φ e, η) such that (σ i + σ e)c 1 φ e = σ ic 1 V m, σ ic 1 V m = σ ic 1 φ e + β I m, V m I m = C m + I ion(v m, η, u), t η = f(vm, η) t Figure : Bidomain representation of cardiac tissue in 2D 6 Tung PhD thesis, 7 Vigmond et al Prog. Biophys. Mol. Biol., pp V m = φ i φ e the transmembrane voltage φ i, φ e intra- and extracellular potential η a vector of state variables σ i, σ e conductivity tensors I m(v m, η) transmembrane current flow I i, I e, I ion current densities β surface to volume ratio of cardiac cells Simplification: in our experiments we replace C 1 by the identity matrix 5 / 23

7 Active Stress Models Relaxed (+) (+) (-) Myosin (-) (+) (+) Contracted Actin + ATP + Ca 2+ Z disk Active stress is generated by electrical activation in the myocardium S a = S a(v m, η, u) I s f (f 0 f 0) with f 0 the myocyte fiber orientation s = 1 mathematical, s = 1 mechanical choice 2 Cell models to compute scalar-values stress term S a Weakly coupled electromechanics e.g. NPStress 9 : S a = ε(v m)(k Sa V m S a) Strongly Coupled Electromechanics e.g. Rice 10 : h(s a, S a, V m, η, λ, λ) = 0 8 Ambrosi and Pezzuto J. Elast., pp Nash and Panfilov Progress in Biophysics and Molecular Biology, pp Rice et al Biophysical Journal, pp / 23

8 Outline Electromechanical Modeling Parallel Strategies Configuration Numerical Examples Open Tasks and Perspectives 7 / 23

9 Motivation High-resolution geometries meshes with up to N = O(10 9 ) degrees of freedom direct solvers: solving time O(N 2 ), very high memory consumption iterative solvers: solving time O(N), lower memory consumption we require strongly scalable parallel algorithms using iterative solvers 8 / 23

10 Why do we need HR geometries? Easy to motivate in electrics: very small features influence wave propagation resolutions below 100 µm would be worthwhile i.e. cellular level, O(10 9 ) cells Harder to do so in mechanics: use same mesh as in electics no data mapping or mesh coarsening needed some phenomena of interest involve small-scale features infarcts and ischemic regions multiple tissue layers and thin structures as papillary muscles, heart strings, valves parallel framework is available improves computational time for smaller meshes as well 9 / 23

11 Global Problem FEM + Newton yields linearized system: K (u k ) u = F K(u k ) u k+1 = u k + u. p Decomposition: A i K i(u k i )Ai u i i=1 solve with algebraic multigrid method (AMG 11,12 ) Ω 10 / 23

12 Algebraic Multigrid Method FEM + Newton yields linearized system: Ω 1 Ω 3 Ω 2 Ω 4 K (u k ) u = F K(u k ) u k+1 = u k + u. p Decomposition: A i K i(u k i )Ai u i i=1 solve with algebraic multigrid method (AMG 11,12 ) Ω 5 Ω6 Ω 7 fine grid restriction base level prolongation presmoothing postsmoothing 11 Plank et al Biomedical Engineering, IEEE Transactions on, pp Neic et al IEEE Trans. Biomed. Eng., pp / 23

13 Finite Element Tearing and Interconnecting Ω 1 Ω 2 Ω 3 Ω 4 FETI: Finite element tearing and interconnecting Tearing: K 1,k u 1 K 1,k.... =. K p,k u p K p,k Ω 5 Ω 6 Ω 7 generally u i u j on the interface Γ C 13 Farhat and Roux Int. J. Numer. Methods Engrg., pp Klawonn and Rheinbach ZAMM Z. Angew. Math. Mech., pp Augustin, Holzapfel, and Steinbach Int. J. Numer. Meth. Engrg., pp / 23

14 Finite Element Tearing and Interconnecting Ω 5 Ω 1 Ω 2 Ω 3 Ω 4 Ω 6 Ω 7 FETI: Finite element tearing and interconnecting Interconnecting: with Lagrange multipliers λ and boolean matrices B i K 1,k B 1 u 1 K 1,k K p,k B p u =. p K p,k B 1 B p 0 λ 0 K i,k P a generalized inverse this yields p B ik i,k B i λ = P i=1 p B ik i,k f i. i=1 13 Farhat and Roux Int. J. Numer. Methods Engrg., pp Klawonn and Rheinbach ZAMM Z. Angew. Math. Mech., pp Augustin, Holzapfel, and Steinbach Int. J. Numer. Meth. Engrg., pp / 23

15 Software FETI: C. Pechstein, C. A. Cardiac electromechanics: CARP: G. Plank, E. Vigmond, C. A. PETSc: Krylov Solver Package with BoomerAMG (hypre) general AMG for various problems Parallel Toolbox (PT): M. Liebmann, A. Neic, G. Haase special AMG for electrics and mechanics Tarantula, ParaView, ParMetis, PaStiX, MUMPS, FEAP TIME++ S = S p + S a(v m, η, u, f) Start electrics Solver mechanics u 12 / 23

16 Outline Electromechanical Modeling Parallel Strategies Configuration Numerical Examples Open Tasks and Perspectives 13 / 23

17 Geometry - MR Images 16 from the Visible Heart Lab ( 14 / 23

18 Geometry - Smoothing 17 with K. Bredis and M. Holler (KFU Graz) 15 / 23

19 Myocardium Model Tetrahedral or hybrid meshes 18 Parameters 19, fiber directions 20 Decomposition with ParMETIS Computations on SuperMUC Leibniz Rechenzentrum Munich Nr 12 in Top500 List - June cores 18 Prassl et al IEEE Trans. Biomed. Engineering, pp Eriksson et al Mathematics and Mechanics of Solids, pp Bayer et al Ann. Biomed. Eng., pp / 23

20 Circulatory System - Pressure BC A to B: loading B to C: isovolumetric contraction C to D: ventricular ejection 16 D C D to E: isovolumetric relaxation E to B: refilling LV pressure (kpa) 3 0 E Challenges: A LV volume (ml) MRI images usually taken at point B unloading of the geometry needed in isovolumetric phases the cavity volumes have to stay constant pressure volume realtions in ejection and filling phase B 17 / 23

21 Fixed Location Boundary Conditions fix base (not physiological) contact problem 21 use bath from bid. model soft elastic material apply D-BC to bath heart covered by double layered membrane (pericard) space between layers is filled with fluid attached to diaphragm and pleura 21 Fritz et al Biomech Model Mechanobiol, 18 / 23

22 Fixed Location Boundary Conditions fix base (not physiological) contact problem 21 use bath from bid. model soft elastic material apply D-BC to bath heart covered by double layered membrane (pericard) space between layers is filled with fluid attached to diaphragm and pleura 21 Fritz et al Biomech Model Mechanobiol, 18 / 23

23 Fixed Location Boundary Conditions fix vessel in- and outlets heart covered by double layered membrane (pericard) space between layers is filled with fluid attached to diaphragm and pleura 21 Fritz et al Biomech Model Mechanobiol, 18 / 23

24 Outline Electromechanical Modeling Parallel Strategies Configuration Numerical Examples Open Tasks and Perspectives 19 / 23

25 Scaling for Electromechanics - AMG 0000 Assembling PC Setup Solver Monodomain Total nodes DOF tets 100 timesteps 5 NS each Number of processes [-] 20 / 23

26 GPU 21 / 23

27 Outline Electromechanical Modeling Parallel Strategies Configuration Numerical Examples Open Tasks and Perspectives 22 / 23

28 Open Tasks and Perspectives Ongoing work finish GPU assembling for mechanics TH-elements, dynamic model, adaptive timestepping,... block system solvers and preconditioning projections between fine and coarse mesh fitting parameters to experiments (Cardioproof project) Wish list contact problems, hemodynamics and FSI 23 / 23

29 Ambrosi, D and S Pezzuto (2012). Active stress vs. active strain in mechanobiology: constitutive issues. In: J. Elast , pp Augustin, CM, GA Holzapfel, and O Steinbach (2014). Classical and All-floating FETI Methods for the Simulation of Arterial Tissues. In: Int. J. Numer. Meth. Engrg. 99.4, pp Augustin, CM and G Plank (2013). Simulating the mechanics of myoardial tissue using strongly scalable parallel algorithms. In: Biomed Tech (Berl) i. Augustin, CM and O Steinbach (2013). FETI Methods for the Simulation of Biological Tissues. In: Domain Decomposition Methods in Science and Engineering XX, Lecture Notes in Computational Science and Engineering 91, pp Bayer, JD et al. (2012). A novel rule-based algorithm for assigning myocardial fiber orientation to computational heart models. In: Ann. Biomed. Eng. 40(10), pp Bols, J et al. (2011). A computational method to assess initial stresses and unloaded configuration of patient-specific blood vessels. In: 5th International conference on Advanced COmputational Methods in ENgineering (ACOMEN 2011). Université de Liège. Braess, Dietrich and Regina Sarazin (1997). An efficient smoother for the Stokes problem. In: Applied Numerical Mathematics 23.1, pp Eriksson, TSE et al. (2013). Influence of myocardial fiber/sheet orientations on left ventricular mechanical contraction. In: Mathematics and Mechanics of Solids 18, pp Farhat, C and FX Roux (1991). A method of finite element tearing and interconnecting and its parallel solution algorithm. In: Int. J. Numer. Methods Engrg. 32, pp / 23

30 Fritz, T et al. (2013). Simulation of the contraction of the ventricles in a human heart model including atria and pericardium : Finite element analysis of a frictionless contact problem. In: Biomech Model Mechanobiol. Fung, YC (1967). Elasticity of soft tissues in simple elongation. In: American Journal of Physiology 213, pp Gurev, V et al. (2011). Models of cardiac electromechanics based on individual hearts imaging data. In: Biomechanics and modeling in mechanobiology 10.3, pp Holzapfel, GA, TC Gasser, and RW Ogden (2000). A new constitutive framework for arterial wall mechanics and a comperative study of material models. In: J. Elasticity 61, pp Holzapfel, GA and RW Ogden (2009). Constitutive modelling of passive myocardium: a structurally based framework for material characterization. In: Philos. Trans. R. Soc. Lond. Ser. A , pp Kerckhoffs, RCP et al. (2007). Coupling of a 3D finite element model of cardiac ventricular mechanics to lumped systems models of the systemic and pulmonic circulation. In: Annals of biomedical engineering 35.1, pp Klawonn, A and O Rheinbach (2010). Highly scalable parallel domain decomposition methods with an application to biomechanics. In: ZAMM Z. Angew. Math. Mech. 90.1, pp Nash, MP and AV Panfilov (2004). Electromechanical model of excitable tissue to study reentrant cardiac arrhythmias. In: Progress in Biophysics and Molecular Biology 85, pp Neic, A et al. (2012). Accelerating cardiac bidomain simulations using graphics processing units. In: IEEE Trans. Biomed. Eng. 59.8, pp Pathmanathan, P and JP Whiteley (2009). A numerical method for cardiac mechanoelectric simulations. In: Ann. Biomed. Eng. 37.5, pp / 23

31 Plank, G, AJ Prassl, and CM Augustin (2013). Computational Challenges in Building Multi-Scale and Multi-Physics Models of Cardiac Electro-Mechanics. In: Biomed Tech (Berl). Plank, G et al. (2007). Algebraic multigrid preconditioner for the cardiac bidomain model. In: Biomedical Engineering, IEEE Transactions on 54.4, pp Prassl, AJ et al. (2009). Automatically Generated, Anatomically Accurate Meshes for Cardiac Electrophysiology Problems. In: IEEE Trans. Biomed. Engineering 56.5, pp Rice, JJ et al. (2008). Approximate Model of Cooperative Activation and Crossbridge Cycling in Cardiac Muscle Using Ordinary Differential Equations". In: Biophysical Journal 95.5, pp Rumpel, T and K Schweizerhof (2003). Volume-dependent pressure loading and its influence on the stability of structures. In: International Journal for Numerical Methods in Engineering 56.2, pp Smith, NP et al. (2004). Multiscale computational modelling of the heart. In: Acta Numer. 13, pp Tung, L (1978). A bi-domain model for describing ischemic myocardial D-C potentials. PhD thesis. MIT, Cambridge, MA. Vanka, SP (1986). Block-implicit multigrid solution of Navier-Stokes equations in primitive variables. In: Journal of Computational Physics 65.1, pp Vigmond, EJ et al. (2007). Solvers for the cardiac bidomain equations. In: Prog. Biophys. Mol. Biol , pp Westerhof, N, GIJS Elzinga, and P Sipkema (1971). An artificial arterial system for pumping hearts. In: Journal of applied physiology 31.5, pp / 23

32 Westerhof, N, J-W Lankhaar, and BE Westerhof (2009). The arterial windkessel. In: Medical & biological engineering & computing 47.2, pp / 23