Fast Algorithms for Many-Particle Simulations on Peta-Scale Parallel Architectures

Transcription

1 Mitglied der Helmholtz-Gemeinschaft Fast Algorithms for Many-Particle Simulations on Peta-Scale Parallel Architectures Godehard Sutmann Institute for Advanced Simulation (IAS) - Jülich Supercomputing Centre (JSC) Research Centre Jülich g.sutmann@fz-juelich.de

2 Outline Many-Body simulations Methods for long range interactions in many-body systems Multigrid Method Barnes-Hut Tree Method Fast Multipole Method Coupling Methods to Codes 2

3 Outline Many-Body simulations Methods for long range interactions in many-body systems Multigrid Method Barnes-Hut Tree Method Fast Multipole Method Coupling Methods to Codes 3

4 Atomistic Models of Matter Galaxy Li F in water 4

5 Molecular dynamics Particle systems are described by a Hamiltonian Kinetic Energy Potential Energy Particle trajectories are generated by solving classical equations of motion 5

6 Long range interactions Typical form for a force field (AMBER / Sander) Classification of potentials Examples: Coulomb interactions (charge-charge, dipole-dipole) Gravitational potential 6

7 Long Range Interactions Potential energy calculations 7

8 Long Range Interactions Hierarchical potential energy calculations 8

9 Outline Many-Body simulations Methods for long range interactions in many-body systems Multigrid Method Barnes-Hut Tree Method Fast Multipole Method Coupling Methods to Codes 9

10 Methods for long range interactions Periodic systems Grid free: Plain Ewald summation O(N3/2) Fast multipole method (FMM) O(N) Tree-algorithms (Barnes-Hut) O(N log(n)) Grid based: Particle Mesh Ewald (PME) O(N log(n)) Particle-Particle Particle-Mesh Method (P3M) O(N log(n)) Open systems Grid free: Tree-algorithms (Barnes-Hut) O(N log(n)) Fast multipole method (FMM) O(N) Grid based: Multigrid based method (P3MG) O(N) List not complete 10

11 Classification of Algorithms Fully Hierarchical Hierarchical Splitting Methods complete subdivision of space complete subdivision of space continuous description from sources to far-field splitting between short- and longrange contributions hierarchical ordering of field contributions fast methods to solve long range contributions mesh-less description (often) grid-based description sources: charges, multipoles sources: pseudo-charges res.: physical potentials, fields res.: lr: pseudo-potentials / -fields sr: potentials + corrections expl: Fast Multipole Method Barnes-Hut Tree Method expl: P3M, SPME, multigrid 11

12 Why different methods? Multigrid Elliptic PDE solver (Poisson, Helmholtz) & non-linear (Poisson-Boltzmann) allows for dielectric shading prescription of desired boundary potentials (capacitors) Barnes-Hut Tree Method strongly inhomogeneous particle distributions very complex system geometries Fast Multipole Method high accuracy calculations full error control All methods are scalable on parallel architectures 12

13 Schematic differences of hierarchical methods Multigrid Barnes-Hut Tree method Fast multipole method 13

14 Acknowlegements FMM Holger Dachsel, Ivo Kabadshow Barnes-Hut Paul Gibbon, Lukas Arnold, Robert Speck, Mathias Winkel Multigrid Matthias Bolten, Bernhard Steffen Parallel Sorting Michael Hofmann (TU Chemnitz) Interface Lidia Westphal, Rene Halver Financial support from BMBF 14

15 Outline Many-Body simulations Methods for long range interactions in many-body systems Multigrid Method Barnes-Hut Tree Method Fast Multipole Method Coupling Methods to Codes 15

16 Multigrid for Poisson Equation Solve PDE on the finest grid with Dirichlet boundary conditions Use finite differences to approximate PDE Get an approximation for the potential from where an error ε and residuum r is obtained These obey the same differential equation Get correction to approximation: (repeat until convergence) 16

17 Multigrid for Poisson Equation Multigrid: solve for the error onto a hierarchy of coarse grids 2Lh Low frequency components on the fine grid get high frequency components on the coarse grid (fast error reduction) Forward MG: Restrict residuum Backward MG: Prolongate residuum e.g. V cycle 17

18 Complexity Natural complexity of long-range pair-wise interactions is O(N2) Multigrid solver reduces complexity of field calculations to O(N) niter fixed on each level 18

19 Convergence of Multigrid Test system of 4096 particles with (25+1)3 grid points Multigrid calculation 19

20 Convergence of Multigrid Test system of 4096 particles with (25)3 grid points Single grid calculation 20

21 Electrostatics: finite difference solution for point particles In molecular simulations, source terms are point charges, which create a potential, diverging at the origin Sampling problem on a discrete grid Resolution problem depending on grid spacing Large error close to the source term Also far field poisoned by discretization errors 21

22 Electrostatic Regularization Splitting of the potential into near- and far-field (analogon to Ewald summation) 22

23 Scaling 23

24 Multigrid Parallelization Exchange of neighbor grid points on each hierarchy level Data non-locality, examples Ng = 1283, i.e. (27)3, i.e. 7 MG-levels Np = 512, i.e. 83, i.e. 5 local levels (128,64,32,16,8) r ~ 2.5x104 Ng = 20483, i.e. (211)3, i.e. 11 MG-levels Np = 65536, i.e. 323, i.e. 7 local levels (2048,1024,512,256,128,64,32) r ~ 2x106 24

25 Parallel Fast Multipole Method Parallelization: Domain decomposition MPI (cartesian communicators) Asynchronous communications only nearest neighbor communication Strong scaling: N= point stencil Strong scaling: N= unknowns/core 25

26 Outline Many-Body simulations Methods for long range interactions in many-body systems Multigrid Method Barnes-Hut Tree Method Fast Multipole Method Coupling Methods to Codes 26

27 Tree Algorithms Organize physical space in a hierarchical way (relative to each particle) Interactions are weighted according to their importance Trees may be organized in different ways, e.g. 2d trees: binary-, quad-, oct-trees in 1,2,3 dimensions Principle is to sort particles into cells, where on the lowest level only one particle is inside a cell Original work by Barnes & Hut 27

28 Tree Algorithms: grouping criterion Interaction between individual particles and distant pseudo-particles (multipoles) Which criterion is adopted for grouping? If s/d < θ : internal structure of the pseudoparticle is ignored If s/d > θ : the node is resolved and the criterion is checked for every child At latest this procedure stops, when a leaf node is reached 28

29 Sorted list of keys maps 3D particle coordinates onto 1D space-filling curve Sort keys according to their size in ASCII representation This sequence will be a space filling curve (Morton, Z, Lebesque curve) finish start 29

30 PEPC Barnes-Hut algorithm implemented into PEPC (Pretty Efficient Parallel Coulomb-Solver) developed by Paul Gibbon (JSC) for plasma-laser interaction reduction of the n-body O(N2) complexity to O(N log N) OpenSource, freely available F90 and MPI part of ScaFaCoS library part of DEISA and PRACE benchmark suites 30

31 Scientific Application PEPC is highly flexible and comes in different flavours PEPC-E Coulomb interaction PEPC-B Magnetic fields laser-plasma interaction PEPC-V vortex fluid methods PEPC-G Newtonian gravitation extends to smooth particle hydrodynamics 31

32 Parallel Performance (pure MPI version) PEPC on IBM BG/P 32

33 Hybrid PEPC MPI + POSIX threads reduce metadata overhead communication thread + multiple worker threads overlap of communication and computation first successful runs on 300k cores of jugene system size N > 2x109 33

34 Outline Many-Body simulations Methods for long range interactions in many-body systems Multigrid Method Barnes-Hut Tree Method Fast Multipole Method Coupling Methods to Codes 34

35 Long range contributions For general orientations of coordinate vectors the expansion is Define 35

36 Extensions: rotation-based FMM Conventional FMM has complexity O(L4) with respect to the multipole length Costs can be reduced to O(L3) if Legendre polynomials are used 36

37 Fast electrostatics Fast Multipole Method FMM is subdivided into 5+1 passes Pass 0: determine optimal FMM parameter set Pass 1: form and shift multipole expansions Pass 2: transform distant multipole expansions Pass 3: shift Taylor like expansions Pass 4: calculate far field energy Pass 5: calculate near field energy 37

38 Fast Multipole Method Pass 0 Determine FMM parameters Seperation criteria ws Depth of the FMM tree Length of the multipole expansion L Optimisation of run time behavior Pass 4 Sum up far field energies Pass 5 Sum up near field energies 38

39 Fast electrostatics Fast Multipole Method Computational complexity: Every Pass consists of operations of O(N), > Fast Multipole Method has optimal scaling with number of particles! 39

40 Periodic Boundary Conditions Implementation straight forward (although involved) for 1d, 2d, 3d-periodicity Scaling of O(N) Full error control 40

41 FMM Parallelization 41

42 Parallel Fast Multipole Method Parallelization: Remote memory access (RMA) Point-to-point: One sided communication Put, Fence, Notify, Notifywait Jugene Contigous data transfer ARMCI / A1 Collective operations MPI Allgather, Allreduce, Barrier Juropa 42

43 Parallel Fast Multipole Method Weak scaling on Jugene and Juropa (Load: 86= particles/core) 43

44 Parallel Fast Multipole Method Strong scaling on Jugene (Load: N = 10243) 44

45 Parallel Fast Multipole Method Strong scaling on Juropa (Load: N = 10243) 45

46 FMM World record (Dec. 2010) After memory optimization: 50 Bytes / particle 512 Mbytes / core x cores / 50 Bytes / particle 3.0 x 1012 particles Largest simulation with N = particles = = 3.0 Trillion 46

47 Outline Many-Body simulations Methods for long range interactions in many-body systems Multigrid Method Barnes-Hut Tree Method Fast Multipole Method Coupling Methods to Codes 47

48 BMBF Project ScaFaCoS Scalable Library for Fast Coulomb Solver Coordinator: Forschungszentrum Jülich (IAS/JSC) University Partners: Bonn Chemnitz Stuttgart Wuppertal Research Centres: MPI Mainz (Polymer) Industry: BASF Ludwigshafen Cognis Düsseldorf (BASF) IBM Dresden Jülich Cognis Wuppertal SCAI Bonn IBM Chemnitz MPI BASF Stuttgart 48

49 BMBF Project ScaFaCoS Interfacing Programs Scalable Fast Coulomb Solvers liblr Particle2Grid Communication Ewald P3M FMM BHT P3MG Fast summation Long Range Interactions Multipole libsr S c h n i t t s t e l l e Communication Neighbor search Potential fcts. Short Range Interactions Raumf.Kurven Parallel Sorting Parallel NFFT, AFFT Parallel Multigrid Parallel Interface AnwendungsProgramm 1 (MD, MC, BD, ) AnwendungsProgramm N (MD, MC, BD, ) 49

50 Yes Insert essential solver-specific data into a new FCSHandle-object Call fcs_init to set method and MPI communicator Calls fcs_fmm_init Calls fcs_p3m_init Calls fcs_pp3mg_init Calls fcs_nfft_init Calls fcs_pepc_init Calls fcs_direct_init Calls fcs_pepc_tune Calls fcs_direct_tune Calls fcs_pepc_run Calls fcs_direct_run Insert optional solver-specific data into FCSHandle-object Call fcs_tune to calculate solver-specific parameters based on positions / charges Calls fcs_fmm_tune Calls fcs_p3m_tune Calls fcs_pp3mg_tune Calls fcs_nfft_tune No Call fcs_run to calculate a time step with the chosen solver Calls fcs_fmm_run Calls fcs_p3m_run Calls fcs_pp3mg_run Calls fcs_nfft_run Yes No Next time step with same configuration? Change solver? Calculate another time step at all? Call fcshandle_destroy to deallocate FCSHandle-object Simulation ends 50

51 Thank you for your Attention 51