High Performance Multiscale Simulation for Crack Propagation

Transcription

1 1 High Performance Multiscale Simulation for Crack Propagation Guillaume Anciaux, Olivier Coulaud and Jean Roman ScAlApplix Project HPSEC th August

2 2 Outline 1. Introduction Motivations State of the art 2. Our Approach Presentation of the method Coupling Algorithm Parallel algorithms and implementation 3. Results 1D & 2D case : wave propagations 2D case : crack 4.Conclusion

3 3 Introduction

4 4 Introduction : context Collaboration with the CEA DIF-DPTA - G. Zerah Goals : produce a tool for Impact on laser optics CEA 1. Better understanding of miscroscale phenomena 2. Reduce computing time of molecular dynamics simulations ScAlApplix 1. Analysis of coupling algorithms 2. HPC parallel scaling processes 3. Development of a framework for multiscale computation 4. Generics : use of legacy codes

5 5 Introduction : atomistic approach simulation with molecular dynamics tool: d2 x t M = V M x t 2 dt VM an empirical potential (one, two or three body interactions) fine description of the studied system. All structural phenomena are captured Teams : A. Nakano (Louisiana State University) H. Gao (Max Planck Institute for Metals Research)

6 6 Introduction : continuum approach Elastodynamics : ω ρ y = div T + b Ω Lagrangian formulation x T = TT Constitutive law for the material T = T( y) Discretization : P1 finite elements Variational problem energy minimization Allow to manipulate huge objects Well known technology y( x, t )

7 7 Difficulties and limitations of the methods MD limitations 1. Space scale : A 100 nm3 of silicon crystal contains near billion atoms 2. Time scale : Time step = femto-second(10-15 s.) 3. Boundary conditions (periodic) 4. Huge data volumes ~ 1 Terabyte per steps FE limitations [[ y]] 0 1. Discontinuities (Crack) Constitutive law not valid anymore near the fracture : the model needs to be extended (XFEM, ) 2. Needs a fine mesh to capture pertinent information.

8 8 Introduction : Multiscale approach (1) Idea : using advantages of both models Continuum model Reduce the size of the domain. Take into account complex boundary conditions. discrete model Near the discontinuities. How to couple this two models?

9 9 Introduction : Multiscale approach (2) Multi-scale approaches : Junction QC-method (Tadmor and al. 1996) Static simulations and T= 0 Macroscopic, Atomistic and Ab initio Dynamics (MAAD) (Abraham and al. 1998) Bridging : duplication of the data T. Belytschko (Bridging Method) Bridging Scale Method (Liu)

10 10 Introduction : Bridging approach Numerical difficulties : Avoiding that h = inter-atomic distance Different time/length scales mecanical wave reflexions Algorithmic difficulties 1. Need for smart data handling at the interface 2. Efficient computation of the FE shape functions in the overlapping zone 3. Decomposition of domain 4. Load balancing Problem 3 and 4 are tied together

11 11 Our approach

12 12 Discrete/continuum coupling The Bridging Method introduced by T. Belytschko & S. Xiao Idea : Imposing equivalent displacements on atom sites Introduction of Lagrangian multipliers Weighting the computations of the multipliers to keep predominances of each models α

13 13 Discrete/continuum coupling The constraint : with : Ai, j= i, gi y X i d i = y X i x i=0 J X i X j rhsi= J y J JMJ J X i x 1 i mi i, j i Ai = j Ai, j A is condensed on its diagonal Then the correction of the velocities : y I = y I t j j I M I I X j x I = x I t 1 i mi i A =rhs

14 14 Solution to algorithmic difficulties Needs to identify atoms in a given element Double loop O(Natoms x Nelements) Introduction of a grid Place atoms and elements in the grid. Map atoms to elements. Complexity O(Natoms x Nbox-elements)

15 15 Initialization of the bridging zone (2) Pre-computations of the shape functions for all atom sites The shape values are stored in an appropriate data structure Allowing, through atom/element mapping, accesses at constant time for any given atom site

16 16 Mapping the codes to processors Strategy : distinct processor sets for each model Molecular dynamics set Continuum mechanics set MD weighting FE weighting Coupling interaction

17 Diagram for the coupling model of parallel codes (SPMD) Initialization Initialization T ai Bridging Zone initialization 1a Position update T s Force computation c T i 1b Position update T s 2a 2b Force computation T s T s Computation of the Lagrange multipliers 3a Velocity update T s no Parallel Molecular dynamics T bi T c Velocity update T 3b s T = Tmax? yes no Parallel Continuum elasticity 17

18 Details on the computation of the Lagrange Multipliers Computation of RHS contribution Computation of RHS contribution T 1a c x i T 1b c J y J X i Summing the contributions T J y J X i x i =rhsi 2 c Solving the constraint system T 3c i =rhs i / Ai Solving the constraint system 3 c T Correcting the velocities 4a c T y new = y I + I t αi M I Correcting the velocities m λ ϕ (x j =1 j I Parallel Molecular dynamics i =rhs i / Ai j ) 4b c T d inew = d i t λi (1 αi ) mi Parallel Continuum elasticity 18

19 19 Constraint system data redistribution To illustrate the talk we consider the following distribution over processors : Atomic zone Bridging zone Continuum Zone

20 Constraint system data redistribution Case where the mapping of models is made onto two distinct sets of processors : both set of processor own a parallel vector of the Lagrangian unknowns. 20

21 Dynamical effects : atoms migrations management (1) Coherency in the redistribution scheme is maintained by protocol involving processors of the bridging zone 21

22 Dynamical effects : atoms migrations management (2) Coherency in the redistribution scheme is maintained by protocol involving communication between the two set. Standard MD atom migration Send new owner Send new position 22

23 23 Results

24 24 Wave Reflexions on th 1D Model Wave reflexions a caused by : Number of degree of freedom reducing Overlap zone impedance which depends on: Waves frequencies Waves phase 400 atoms 40 elements

25 25 Reproduction of the absorbsion results un ad ap te d im pe da nc e Initial condition d te ap ad im ce n da pe Imepdence depends on : overlapping size element size settings of the projection

26 Ang 2D Model : wave propagation 150 Ang Some numbers : Atoms : (Lennard Jones) Finite elements : nodes, elements Overlap zone atoms (~55%) 523 nodes (19%), 891 elements (~17%) Simulation : processors time steps 358 ms / time steps 62 % atoms 34 % elasticity 4% coupling

27 27 2D example : crack propagation Box of 600nmx800nm Numbers : atoms (Lennard Jones) nodes, elements Overlapping zone : atoms 912 nodes et elements Crack : ellipse 50Ang x 1Ang

28 Computational time repartition for a 2D sequential simulation 0,15%7,45% 0,37% 4,68% Atom Part Elast Part BuildRHS Correct Surface Effect Correcting Solving constraint 57,06% 30,29% optimal : 20MD/16FE 28

29 t im e in seconds for 100 sim ulat ion t im est eps Sim ulat ion t im es of t he different t asks on 36 processors ,5 3 Ts(c) Ts(a) 2,5 Tc 2 1,5 1 0, num ber of processors assigned t o m olecular dynam ics

30 30 Domain decomposition issues 4x5 vs 2x8 2x8 vs 2x10

31 31 num ber of processors assigned t o m olecular dy nam ics Overhead t im e due t o coupling on 36 processors during 100 t im est eps 4 8 Tc(4a,4b) 16 Tc(3) Tc(2) Tc(1a,1b) ,25 0,5 0,75 1 1,25 1,5 1,75 2 t im e in seconds 2,25 2,5 2,75 3

32 32 3D case of real size : under construction atoms elements atoms in the overlapping zone 8100 elements in the overlapping zone

33 33 Conclusion

34 34 Conclusion First version of the simulator Better understanding of the multi-scales problems Wave reflexion management Results in 2D Waves Crack Going to 3D simulations Enhancing the parallelism Molecular dynamics codes : limitation for the 3D cases Domain decomposition in boxes Take into account cost functions to map tasks to processors

35 35 Our simulator T. Belytschko Model 1D, 2D and 3D tests parallel version based on MPI communication paradigm C++ code Interfaced with : finite elements : libmesh molecular dynamics : Stamp (CEA), Lamps (Sandia) vizualisation and steering : EPSN (ScalApplix INRIA)