A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP

Transcription

1 A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP Large-Scale Applications and Scalability for Problems in the Mechanics of Soft Biological Tissues in Arterial Wall Structures Andreas Fischle Forschungsseminar am Institut für Numerische Mathematik (joint work with A. Klawonn, O. Rheinbach, J. Schröder, and D. Brands) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

2 Overview Introduction Section 1 Introduction 2 Elasticity Theory of Large Deformations 3 A Scalable MPI-parallel Solver Environment Based on FEAP and FETI-DP 4 Scalability and Large Scale Simulations A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

3 Introduction Schematic Physiology of an Elastic Artery Figure: Schematic diagrams of a healthy human artery. Original image from Wikimedia Commons, user: BruceBlaus. Blausen Medical - Scientific and Medical Animations A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

4 Introduction Stained Slide of an Elastic Artery (Photomicrograph) Elastic fibers (black), collagen fibers (pink) and cell structure (Department of Anatomy, Kaohsiung Medical University, Taiwan) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

5 Introduction Anisotropic Models for Arterial Walls and FETI-DP Previous work (Groups of A. Klawonn/J. Strong transversal anisotropy in two fiber directions; quasi-incompressible; effects due to eigenstresses (have been ignored) Realistic material models for Adventitia, Media and plaque (FEAP, F77) Material parameters fitted to experimental data (Brands et al., 2008) Polyconvex modelling approach Meshed geometries reconstructed from patient-specific IVUS data A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

6 Introduction Anisotropic Models for Arterial Walls and FETI-DP Finite element simulations in a semi-parallel solver environment (Brands et al. 2008, Brinkhues et al. 2011, Balzani et al. 2012) Sequential assembly vs. MPI-parallel scalable linear solver (FETI-DP) Excellent start, but semi-parallel is not scalable (all components must scale!) Sequential assembly in FEAP was a showstopper... A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

7 Overview Elasticity Theory of Large Deformations Section 1 Introduction 2 Elasticity Theory of Large Deformations 3 A Scalable MPI-parallel Solver Environment Based on FEAP and FETI-DP 4 Scalability and Large Scale Simulations A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

8 Elasticity Theory of Large Deformations Finite Elasticity (Nonlinear Theory) Notation and terminology Consider locally orientation preserving deformation mappings ϕ : Ω ref R 3 Ω def := ϕ(ω ref ) R 3 Displacement u := ϕ id R 3, i.e., u(x) = ϕ(x) x Deformation gradient F := ϕ, det(f ) > 0 Right Cauchy-Green stretch tensor C := F T F A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

9 Elasticity Theory of Large Deformations Quasistatic Hyperelasticity (Calculus of Variations) Basic assumption of hyperelasticity stored strain energy potential Π(ϕ) := Ψ(F (x)) dx Ω ref induced by a local strain energy density Ψ(F ), where F := ϕ. Basic assumption of a quasistatic approximation The time-dependant mechanical system is near an equilibrium and the solutions can be obtained as local minimizers ϕ min,local of a potential. Goal: Computation of states of minimal deformation/strain energy (admissible) Π(ϕ) := Ψ(F (x)) dx ϕ min. Ω ref (dependency of Ψ on x, exterior forces and tractions neglected...) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

10 Elasticity Theory of Large Deformations Action of F := ϕ on Infinitesimal Cubes Column vectors of F := ϕ generate a parallelepipedon in the tangent space T ϕ(x) ϕ(ω), i.e., a deformed infinitesimal cube. Ψ(F ) is the associated local strain energy density. A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

11 Elasticity Theory of Large Deformations Isotropy and Invariants Isotropy of hyperelastic materials The energetic response of the material is direction-independent Characterization by group invariance: Q SO(3) : Ψ(F Q) = Ψ(F ). Objectivity + Isotropy Invariants Objectivity is invariance with respect to superimposed isometries: Q SO(3) : Ψ(Q F ) = Ψ(F ). Left and right invariance leads to principal invariants I 1 := tr (C), I 2 := tr (Cof (C)), I 3 := det (C). Representation: Ψ(F ) = Ψ(C) = Ψ(I 1, I 2, I 3 ). A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

12 Elasticity Theory of Large Deformations Anisotropy Characterization by invariance w.r.t. a group action Anisotropic materials show direction-dependent behavior (crystals) Material group G < SO(3), s. th. Q SO(3) Q G = Ψ(F Q) = Ψ(F ), and Q / G = Ψ(F Q) Ψ(F ). Mixed invariants for preferred directions (transversal isotropy) Structural tensors, e.g., M a := d a d a encode preferred directions d a, a = 1, 2 (e.g., muscle fiber directions of the artery) Associated mixed invariants J (a) 4 := tr(cm a ), J (a) 5 := tr(c 2 M a ), a = 1, 2 A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

13 Elasticity Theory of Large Deformations Polyconvexity, Anisotropy and Existence of Minimizers A type of generalized convexity A local strain energy Ψ is called polyconvex if there exists a convex function Ψ : R 3 3 R 3 3 R R 19 R such that F GL + (3) Polyconvexity in the isotropic case Ψ(F ) = Ψ(F, Cof (F ), det (F )). Polyconvexity implies existence of minimizers! (J. Ball 1977) The isotropic invariants I k, k = 1, 2, 3, are polyconvex and serve as building blocks for more complicated polyconvex energies Convexity of Ψ in F is a too strong requirement, i.e., unphysical Polyconvexity is not required by the physics... A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

14 Elasticity Theory of Large Deformations Polyconvexity, Anisotropy and Existence of Minimizers A type of generalized convexity A local strain energy Ψ is called polyconvex if there exists a convex function Ψ : R 3 3 R 3 3 R R 19 R such that F GL + (3) Ψ(F ; M (1), M (2),...) = Ψ(F, Cof (F ), det (F ) ; M (1), M (2),...). Polyconvexity in the anisotropic case Extension of polyconvex framework by structural tensors (J. Schröder and P. Neff 2003) Unfortunately J 5 := tr[c 2 M] is not polyconvex... But K 3 := I 1 J 4 J 5 is a polyconvex invariant (J. Merodio and P. Neff 2006) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

15 Elasticity Theory of Large Deformations A Glimpse of Functional Analysis Global existence theory of J. Ball (1977) Polyconvexity + Growth assumptions = Minimizers u W 1,p (2 p < but depends on the growth condition...) Regularity of solutions is hard to establish for n > 1 (stress concentration?) Whether minimizers solve the EL-equations is an open problem Polyconvexity allows for non-uniqueness / bifurcations (as required by the physics, e.g., plate buckling) Powerful framework, i.e., many material laws are polyconvex (but St.Venant-Kirchhoff is not!) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

16 Elasticity Theory of Large Deformations Example for an Arterial Wall Model Strain energy density Ψ a (Balzani, Neff, Schröder and Holzapfel, 2006) Ψ A (C, M 1, M 2 ) := c 1 ( I1 Notation and table of invariants 3 ) I 1/3 3 }{{} Neo-Hooke isotropic matrix material I 1 := tr(c), I 2 := tr(cof (C)), I 3 := det(c), C := F T F J (a) 4 := tr(cm a ), J (a) 5 := tr(c 2 M a ), K (a) 3 := I 1 J (a) 4 J (a) 5 Structural tensors for transversal isotropy in fiber directions d a, a = 1, 2: M a := d a d a, a = 1, 2 A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

17 Elasticity Theory of Large Deformations Example for an Arterial Wall Model Strain energy density Ψ a (Balzani, Neff, Schröder and Holzapfel, 2006) Ψ A (C, M 1, M 2 ) := c 1 ( I1 Notation and table of invariants 3 ) I 1/3 3 }{{} Neo-Hooke isotropic matrix material + 2 α 1 K (a) α2 3 2 a=1 }{{} Fiber contribution I 1 := tr(c), I 2 := tr(cof (C)), I 3 := det(c), C := F T F J (a) 4 := tr(cm a ), J (a) 5 := tr(c 2 M a ), K (a) 3 := I 1 J (a) 4 J (a) 5 Structural tensors for transversal isotropy in fiber directions d a, a = 1, 2: M a := d a d a, a = 1, 2 A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

18 Elasticity Theory of Large Deformations Example for an Arterial Wall Model Strain energy density Ψ a (Balzani, Neff, Schröder and Holzapfel, 2006) Ψ A (C, M 1, M 2 ) := c 1 ( I1 Notation and table of invariants 3 ) I 1/3 3 }{{} Neo-Hooke isotropic matrix material ) + ε 1 (I ε I ε 2 3 }{{} Volume penalty + 2 α 1 K (a) α2 3 2 a=1 }{{} Fiber contribution I 1 := tr(c), I 2 := tr(cof (C)), I 3 := det(c), C := F T F J (a) 4 := tr(cm a ), J (a) 5 := tr(c 2 M a ), K (a) 3 := I 1 J (a) 4 J (a) 5 A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

19 Elasticity Theory of Large Deformations Example for an Arterial Wall Model Strain energy density Ψ a (Balzani, Neff, Schröder and Holzapfel, 2006) Ψ A (C, M 1, M 2 ) := c 1 ( I1 3 ) I 1/3 3 }{{} Neo-Hooke isotropic matrix material ) + ε 1 (I ε I ε 2 3 }{{} Volume penalty + 2 α 1 K (a) α2 3 2 a=1 }{{} Fiber contribution Polyconvex, highly anisotropic and quasi-incompressible strain energy with a volumetric term due to S. Hartmann and P. Neff (2003) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

20 Elasticity Theory of Large Deformations Example for an Arterial Wall Model Strain energy density Ψ a (Balzani, Neff, Schröder and Holzapfel, 2006) Ψ A (C, M 1, M 2 ) := c 1 ( I1 3 ) I 1/3 3 }{{} Neo-Hooke isotropic matrix material ) + ε 1 (I ε I ε 2 3 }{{} Volume penalty + 2 α 1 K (a) α2 3 2 a=1 }{{} Fiber contribution The fiber contributions model the stiffening of collagen fibers embedded in the wall tissue under tension (power-law stiffening) Atherosclerotic plaque is modeled as an additional isotropic solid (Mooney-Rivlin) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

21 Elasticity Theory of Large Deformations Solution strategy: Newton s method in H 1 (Ω, R 3 ) via FEM Equilibria of forces correspond to critical points of d ϕ Π total (ϕ) }{{} static forces due to potentials f total }{{} applied forces = 0 Newton s method Finite element stiffness matrix K h arises from d 2 ϕπ total (ϕ) Symmetric, but may be indefinite (saddle points/bifurcations) (heuristically just a few negative or zero eigenvalues?) Pressure loads are follower loads: slight nonsymmetry Use iterative Krylov method for Newton corrections (Newton-Krylov) Avoid bad linearizations by adaptive incremental loading A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

22 Elasticity Theory of Large Deformations Finite Elements for Quasi-Incompressibility Elements Quadratic tetrahedral elements (TET10) F -approach: Incompressibility in the average per element Reduces volumetric locking (J. C. Simo 1998) Theory in F := θ 1/3 I 1/3 ) 3 F, i.e., det ( F = θ θ is coupled to det (F ) in an average/weak sense One Lagrange multiplier π per element, discontinuous, statically condensated D. Brands et al. (2008), A. Klawonn and O. Rheinbach (2010), S. Brinkhues et al. (2012), D. Balzani et al. (2012) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

23 Elasticity Theory of Large Deformations FETI-DP: A Non-Overlapping Domain Decomposition Method (Farhat et al. 2000) The Dual Primal Finite Element Tearing and Interconnecting Method Scalable iterative solution method for linear equation systems in FEM Geometric non-overlapping domain decomposition Ω ref = i=1,...,n into disjoint subdomains Ω i (with typical diameter H) Global interface Γ := i=1,...,n Ω i \ Ω Continuity constraints to guarantee a continuous solution on Γ: 1 Primal constraints via partial assembly (corners of subdomains) 2 Dual constraints via Lagrange multipliers λ (non-primal d.o.f. on Γ) Ω i A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

24 Elasticity Theory of Large Deformations The FETI-DP Master System Saddle point problem Block structure [ K B T B 0 ] [ ũ λ ] = [ f 0 ] K := K (1) BB K (1) ΠB K (1)T ΠB K (N) BB K (N) ΠB K (N)T ΠB K ΠΠ, f = f (1) B. f (N) B fπ. K can be efficiently inverted in parallel (block structure) Primal coupling provides a coarse space (required for scalability) Block-Gauß elimination of ũ leads to (dual) equation for λ: Fλ = d GMRES + Dirichlet preconditioner + Transformation of basis cf. Klawonn and Rheinbach (2010) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

25 Elasticity Theory of Large Deformations The FETI-DP Master System Saddle point problem Block structure [ K B T B 0 ] [ ũ λ ] = [ f 0 ] K := K (1) BB K (1) ΠB K (1)T ΠB K (N) BB K (N) ΠB K (N)T ΠB K ΠΠ, f = f (1) B. f (N) B fπ. Numerical scalability, i.e., estimates of the type κ(md 1 F) < C (1 + log(h/h))2 For linear elliptic problems, see e.g., Klawonn and Widlund (2006), Klawonn, Rheinbach, Widlund (2008), and others Reminder: Rate of convergence for conjugate gradients x x k A x x 0 A 2 ( ) k κ 1 κ + 1 A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

26 Elasticity Theory of Large Deformations Newton s Method and FETI-DP Newton-Krylov-FETI-DP A Newton-Krylov-FETI-DP solver uses FETI-DP for the iterative solution of the linear systems arising in the Newton method. Newton-like due to symmetric solver (memory savings) FETI-DP system for the n-th Newton iteration: F u (n) λ (n) = d u (n) Detail: F is created from a symmetric approximation K S := { Kij : for i j K ji : for i > j (in case of convergence this only affects the convergence rate) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

27 Overview Software Environment Section 1 Introduction 2 Elasticity Theory of Large Deformations 3 A Scalable MPI-parallel Solver Environment Based on FEAP and FETI-DP 4 Scalability and Large Scale Simulations A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

28 Software Environment FEAP (UC Berkeley, CA, Robert L. Taylor) A Finite Element Assembly Program (F77) Reads a problem description from an ASCII input card Designed for interactive use in a unix shell (command driven) Extensible via user interfaces (e.g., usolve.f, uasble.f) Popular in the mechanics community due to customizability (FEAP-JS) FEAP is one (if not the) most extensively tested implementation(s) (also on the conceptual level!) for nonlinear continuum mechanics (v1: Sept / F66). A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

29 Software Environment A Robust and Parallel Scalable FETI-DP Solver Implemented as a Krylov-method in PETSc (2.3.15) Developed by O. Rheinbach as part of his dissertation (2006) MPI-parallel and scalable to more than CPU cores (Inexact FETI-DP, A. Klawonn and O. Rheinbach, 2009) Transformation of basis approach for effective coarse spaces Highly tuned for numerical robustness, stability and performance Preconditioners originally developed for Laplace and linearized elasticity (A. Klawonn, O. Widlund et al.) Portable, Extensible Toolkit for Scientific Computation (PETSc, ANL) Powerful MPI-parallel C-library for HPC (á la LAPACK) Originates in the domain decomposition community (Barry Smith) Encapsulates many third-party libraries in a convenient way A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

30 Software Environment A Software Integration Challenge From A to B: semi-parallel to parallel scalable simulations A: Sequential assembly with FEAP as master process B: Parallel assembly and solution with FEAP as slave process FEAP was not designed for the scenario B... Adams et al., Gordon Bell Award, 2004, Athena (Prometheus) Similar usage of FEAP for parallel assembly (it s possible!) Adams et al. implementation still unknown and unavailable to us A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

31 Software Environment A Software Integration Challenge From A to B: semi-parallel to parallel scalable simulations A: Sequential assembly with FEAP as master process B: Parallel assembly and solution with FEAP as slave process FEAP was not designed for the scenario B... Transformation of FEAP into a software library Flow control: Simulate a user typing commands into FEAP Data exchange: FEAP input cards Callbacks using FEAP user extension interfaces Shared memory access A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

32 Software Environment FEAP Meets FETI-DP in Parallel fddp: FETI Domain-Decomposition Processor (C++) (w. O. Rheinbach) Sequential pre- and postprocessor computes decompositions for structured and unstructured problems stored in FEAP input card format (uses ParMetis) libfw: FEAP-Wrapper library (C/F77) Small wrapper library turning FEAP into a C-library Based on a patched version of FEAP 8.2 (feap-fw) (with D. Brands) mparfeap: Massively-Parallel FEAP (C++) (with O. Rheinbach) MPI-parallel Newton-Krylov-FETI-DP solver based on the FETI-DP solver developed by O. Rheinbach in his dissertation (2006) Vision: Extensible toolchain that makes Newton-Krylov-FETI-DP convenient to use for FEAP users. A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

33 Software Environment Collaboration Diagram for an Instance of mparfeap Object Management Nonlinear Solver (mparfeap) Command Interface Shared Memory Data Access Linear Solver Interface User Assembly Callback Interface libfw [FEAP] User Solver Callback Interface Assembly Object System Setup Command Interface Scalable Linear FETI-DP-Solver (PETSc-based) Control flow extracted from FEAP to mparfeap FEAP is controlled via the wrapper library libfw A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

34 Overview Scalability and Large Scale Simulations Section 1 Introduction 2 Elasticity Theory of Large Deformations 3 A Scalable MPI-parallel Solver Environment Based on FEAP and FETI-DP 4 Scalability and Large Scale Simulations A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

35 Scalability and Large Scale Simulations Eye Candy (Cray Uni-DUE) Cook s Membrane Artery Wall stresses on 512 CPU cores based on FEAP-JS and FETI-DP (using 12/24 CPU cores per compute node on Cray XT6m) Pressures in the physiological regime and above (up to 500 [mmhg]) Patient specific arterial wall geometry (IVUS) Scalability: Weeks melt to hours at ten-fold resolution (13 mio. d.o.f.) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

36 Scalability and Large Scale Simulations Eye Candy (Cray Uni-DUE) Cook s Membrane Artery Massively parallel simulations for structured geometries Simple tension tests on fully allocated Cray XT6m, i.e., up to 4096/4122 CPU cores, up to 134 mio. d.o.f. Too large for storing results and plotting... A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

37 Scalability and Large Scale Simulations Notions of Scalability for Parallel Algorithms Relevant quantities Problem sizes N i (increasing), i = 1, 2,... Number of CPUs P i (increasing), i = 1, 2,... Time to solution in [s]: T i (N i, P i ) Weak scalability (by example of scaling by a factor of 2) Problem size N i+1 = 2N i is doubled Number of CPUs P i+1 = 2P i is doubled Perfect weak parallel scalability: T i = T 1 is constant Parallel efficiency in [%]: E i := T 1 T i 100 [%] (perfect = 100%) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

38 Scalability and Large Scale Simulations Notions of Scalability for Parallel Algorithms Relevant quantities Problem sizes N i (increasing), i = 1, 2,... Number of CPUs P i (increasing), i = 1, 2,... Time to solution in [s]: T i (N i, P i ) Strong scalability (by example of scaling by a factor of 2) Problem size N i+1 = N i is (constant) Number of CPUs P i+1 = 2P i is (doubled) Perfect strong parallel scalability: T i+1 = 1 2 T i, i.e., (speed-up 2) Parallel efficiency in [%]: E i := T 1P 1 T i P i 100 [%] (perfect = 100%) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

39 Scalability and Large Scale Simulations FETI-DP Coarse Space for Arterial Wall Stresses FETI-DP Algorithm C (2002, Klawonn et al.) The primal coarse space is induced by enforcing continuity of u on all the subdomain vertices V and the continuity of all edge averages u E ik taken over all edges E in Γ. A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

40 Scalability and Large Scale Simulations Strong Scalability for Arterial Wall Stresses (Cray University of Duisburg-Essen) Cores Subdomains Problem Size Time Efficiency [# d.o.f.] [h:m:s] [s] :38: % :49: % :59: % Table: Strong scalability; 13 million d.o.f.; Ψ A Set-2; 12/24 cores per compute node; 512 FETI-DP subdomains; FETI-DP coarse space Algorithm C (Klawonn et al., 2002) FETI-DP-solver (due to Rheinbach, 2006) based on PETSc; Activated non-zero initial guess for the Lagrange multipliers λ in GMRES; Deactivated linear extrapolation for the displacement u. A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

41 Scalability and Large Scale Simulations Convergence History for an Artery Model (13 mio. d.o.f) Pressure kpa Newton Iterations Maximal interior normal pressure 500 [mmhg] [kpa] A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

42 Scalability and Large Scale Simulations Convergence History for an Artery Model (13 mio. d.o.f) FETI DP Iterations Log 10 FETI DP Cond. Est Total Newton Steps Total Newton Steps Maximal interior normal pressure 500 [mmhg] [kpa] A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

43 Scalability and Large Scale Simulations Weak Scalability for a Cuboid Geometry Scaling sequence for a 5% tension test FETI-DP Algorithm D E (2006, Klawonn and Rheinbach) The primal coarse space is generated by continuity of u E ik, i.e., by the continuity of the averages over all edges E in Γ. Linear extrapolation of the displacement is extremely helpful (communicated by J. Schröder) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

44 Scalability and Large Scale Simulations Weak Scalability Cuboid Problem Sizes Cores Global Problem Local Problem Coarse Problem Dual Problem [# d.o.f] [# d.o.f] [# d.o.f] [# d.o.f] Table: Weak scalability; Cuboid; H/h = 13; Problem sizes. A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

45 Scalability and Large Scale Simulations Weak Scalability Cuboid Linear Elasticity Cores FETI-DP Eff. FETI-DP Eff. Time Eff. Cond. [%] It. [%] [s] [%] s s s 76.7 Table: Weak scalability; Cuboid; H/h = 13; Linear elasticity; Numerical and parallel scalability; Tolerances atol Newton = 10 7 and rtol FETI-DP = 10 7 ; 12/24 CPU cores per compute node; Cray XT6m; FETI-DP coarse space Algorithm D E. A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

46 Scalability and Large Scale Simulations Numerical Weak Scalability Cuboid (Ψ B Set 3 Relaxed) Cores Newton Eff. FETI-DP Eff. FETI-DP Eff. FETI-DP Eff. It. (Σ) [%] It. (Σ) [%] It. ( ) [%] Cond. ( ) [%] Observations #Newton steps increases due to h 0 (why?) Adams et al. and Deuflhard indicate similar observations... Scalability of #FETI iterations in the average is 81% (Independent of number of Newton and load steps) Scalability of #FETI iterations in the total is 65% (Affected by the number of Newton and load steps) A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

47 Scalability and Large Scale Simulations Numerical Weak Scalability Cuboid (Ψ B Set 3 Relaxed) Cores Newton Eff. FETI-DP Eff. FETI-DP Eff. FETI-DP Eff. It. (Σ) [%] It. (Σ) [%] It. ( ) [%] Cond. ( ) [%] Conclusion It is possible that we have to expect an increase of Newton iterations until an asymptotic limit is reached (Deuflhard 2011?)... A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

48 Scalability and Large Scale Simulations Parallel Weak Scalability Cuboid Ψ B Set 3 Relaxed Cores FETI-DP Eff. FETI-DP Eff. GMRES Eff. GMRES Eff. Runtime Eff. (Σ) [s] [%] ( ) [s] [%] (Σ) [s] [%] ( ) [s] [%] [s] [%] Table: Weak scalability; Cuboid; H/h = 13; Timings and efficiences for parallel scalability; FETI-DP coarse space Algorithm D E (Klawonn and Rheinbach, 2006). Overall we achieve 59% efficiency. Lack of overall parallel scalability is, however, for a large part due to an increasing number of Newton iterations. Scalability of FETI-DP in the average ( ) excludes this effect and is 71%. This is quite close to linear elasticity with 76.7%. A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

49 Scalability and Large Scale Simulations Parallel Weak Scalability Cuboid Ψ B Set 3 Relaxed Cores FETI-DP Eff. FETI-DP Eff. GMRES Eff. GMRES Eff. Runtime Eff. (Σ) [s] [%] ( ) [s] [%] (Σ) [s] [%] ( ) [s] [%] [s] [%] Table: Weak scalability; Cuboid; H/h = 13; Timings and efficiences for parallel scalability; FETI-DP coarse space Algorithm D E (Klawonn and Rheinbach, 2006). Suppose that the Krylov method applied to the linearizations in the Newton scheme scales perfectly, i.e., with 100% efficiency. This is not sufficient for nonlinear scalability for the induced Newton-Krylov method if the number of Newton steps increases too much as h 0. A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

50 Scalability and Large Scale Simulations The End Thank you for your attention! A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

51 Scalability and Large Scale Simulations References I (Arterial Walls and FETI-DP) D. Brands, A. Klawonn, O. Rheinbach, and J. Schröder. Modelling and convergence in arterial wall simulations using a parallel FETI solution strategy. Comput. Methods Biomech. Biomed. Eng., 11(5): , D. Balzani, P. Neff, J. Schröder, and G. Holzapfel. A polyconvex framework for soft biological tissues. Adjustment to experimental data. Int. J. Solids Struct., 43(20): , D. Balzani. Polyconvex Anisotropic Energies and Modeling of Damage Applied to Arterial Walls. PhD thesis, Universität Duisburg-Essen, Fakultät für Ingenieurwissenschaften, D. Brands. Geometrical Modeling and Numerical Simulation of Heterogeneous Materials. PhD thesis, Universität Duisburg-Essen, Fakultät für Ingenieurwissenschaften, A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

52 Scalability and Large Scale Simulations References II (Polyconvex Models and Incompressibility) J. Schröder and P. Neff. Invariant formulation of hyperelastic transverse isotropy based on polyconvex free energy functions. Int. J. Solids Struct., 40: , J. Merodio and P. Neff. A note on tensile instabilities and loss of ellipticity for a fiber-reinforced nonlinearly elastic solid. Arch. Mecha., 58(3): , J. Schröder and P. Neff, editors. Poly-, Quasi-and Rank-one Convexity in Applied Mechanics. Number 516 in CISM International Centre for Mechanical Sciences. Springer, S. Hartmann and P. Neff. Polyconvexity of generalized polynomial type hyperelastic strain energy functions for near incompressibility. Int. J. Solids Struct., 40(11): , J. C. Simo. Numerical Analysis and Simulation of Plasticity. In P. G. Ciarlet and J. L. Lions, editors, Handbook of Numerical Analysis, number A Parallel 6. Newton-Krylov-FETI-DP Elsevier Science, Solver Based on FEAP / 36

53 Scalability and Large Scale Simulations References III (FETI-DP) C. Farhat, M. Lesoinne, and K. H. Pierson. A scalable dual-primal domain decomposition method. Numer. Linear Algebra Appl., 7: , A. Klawonn and O. B. Widlund. Dual-Primal FETI Methods for Linear Elasticity. Comm. Pure Appl. Math., 59: , A. Klawonn and O. Rheinbach. A parallel implementation of Dual-Primal FETI methods for three dimensional linear elasticity using a transformation of basis. SIAM J. Sci. Comput., 28(5): , A. Klawonn and O. Rheinbach. Highly scalable parallel domain decomposition methods with an application to biomechanics. ZAMM Z. Angew. Math. Mech., 90(1):5 32, O. Rheinbach. Parallel Scalable Iterative Substructuring: Robust Exact and Inexact FETI-DP Methods with Applications to Elasticity. PhD thesis, Universität Duisburg-Essen, Fakultät für Mathematik, A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36

54 Scalability and Large Scale Simulations References IV (Software) R. L. Taylor and the FEAP team. The FEAP homepage S. Balay, M. F. Adams, J. Brown, P. Brune, K. Buschelman, V. Eijkhout, W. D. Gropp, D. Kaushik, M. G. Knepley, L. C. McInnes, K. Rupp, B. F. Smith, and H. Zhang. PETSc Web page P. R. Amestoy, I. S. Duff, J.-Y. L Excellent, and J. Koster. A fully asynchronous multifrontal solver using distributed dynamic scheduling (MUMPS). SIAM J. Matrix Anal. Appl., 23(1):15 41, G. Karypis, K. Schloegel, and V. Kumar. ParMETIS - Parallel graph partitioning and sparse matrix ordering. Version 3.1. TR, University of Minnesota, Department of Computer Science and Engineering, A Parallel Newton-Krylov-FETI-DP Solver Based on FEAP / 36