Efficient Parallelization of Molecular Dynamics Simulations on Hybrid CPU/GPU Supercoputers

Transcription

1 Efficient Parallelization of Molecular Dynamics Simulations on Hybrid CPU/GPU Supercoputers Jaewoon Jung (RIKEN, RIKEN AICS) Yuji Sugita (RIKEN, RIKEN AICS, RIKEN QBiC, RIKEN ithes)

2 Molecular Dynamics (MD) 1. Energy/forces are described by classical molecular mechanics force field. 2. Update state according to equations of motion. dr i dt dp dt i p i m F Equation of motion i p r ( ) ( ) i i t t ri t t m p ( t t) p ( t) F t i i i Integration Long time MD trajectories are important to obtain thermodynamic quantities of target systems. Long time MD trajectory => Ensemble generation

3 Potential energy in MD using PME E k ( b b ) total b 0 bonds angles k a dihedrals 0 2 ( ) V [1 cos( n )] n 2 N1 N r r q q ij 2 r j 1i j 1 ij r ij r ij ij 0ij i j O(N) Total number of particles O(N) O(N) O(N 2 ) Main bottleneck in MD i j R ij 12 6 r 2 2 0ij r0ij qiq jerfc( rij ) exp( / 4 ) 2 k FFT( Q( )) r 2 ij r k ij rij k0 k Real space, O(CN) Reciprocal space, O(NlogN) 3

4 GENESIS MD software (Generalized Ensemble Simulation Systems) 1. Aims at developing efficient and accurate methodologies for free energy calculations in biological systems. 2. Efficient Parallelization - Suitable for massively parallel supercomputers, in particular, K. 3. Applicability for large scale simulation. 4. Algorithms coupled with different molecular models such as coarsegrained, all-atom, and hybrid QM/MM. 5. Generalized ensemble with Replica Exchange Molecular Dynamics. Ref : Jaewoon Jung et al. WIREs CMS, 5, (2015) Website :

5 Parallelization of the real space interaction: Midpoint cell method (1) Midpoint method : interaction between two particles are decided from the midpoint position of them. Midpoint cell method : interaction between two particles are decided from the midpoint cells where each particle resides. Small communication, efficient energy/force evaluations Ref : J. Jung, T. Mori and Y. Sugita, JCC 35, 1064 (2014)

6 Basic domain decomposition using the midpoint cell method (2) 1. Partitioning space into fixed size boxes, with dimension larger than the cutoff distance. 2. We need only information of neighbor space(domain) for computation of energies. 3. Communication is reduced by increasing process number. 4. Efficient for good parallelization and suitable for large system with massively parallel supercomputers.

7 Parallelization of FFT in GENESIS : Volumetric decomposition scheme in 3D FFT 1. More communications than existing FFT 2. MPI Alltoall communications only in one dimensional space (existing : communications in two/three dimensional space) 3. Reduce communicational cost for large number of processors

8 FFT in GENESIS (2 dimensional view) GENESIS (Identical domain decomposition between two space) NAMD, Gromacs (Different domain decomposition)

9 ns/day GENESIS performance on K GENESIS r1.1 GENESIS r1.0 NAMD GENESIS r1.0 GENESIS r1.1 GENESIS r1.2 NAMD 2.9 Gromacs Gromacs Number of Cores ApoA1 (92,224 atoms) on 128 cores STMV (1,066,628 atoms)

10 Why midpoint cell method for GPU+CPU cluster? 1. The main bottleneck of MD is the real space non-bonded interactions for small number of processors. 2. The main bottleneck of MD moves to the reciprocal space non-bonded interactions as we increase the number of processors. 3. When we assign GPU for the real space non-bonded interactions, the reciprocal space interaction will be more crucial. 4. The midpoint cell method with volumetric decomposition FFT could be one of good solutions to optimize the reciprocal space interactions due to avoiding communications before/after FFT. 5. In particular, the midpoint cell method with volumetric decomposition FFT will be very useful for massively parallel supercomputers with GPUs.

11 Overview of CPU+GPU calculations 1. Computation intensive work : GPU Pairlist Real space non-bonded interaction 2. Communication intensive work or no computation intensive work : CPU Reciprocal space non-bonded interaction with FFT Bonded interaction Exclusion list 3. Integration is performed on CPU due to file I/O.

12 Real space non-bonded interaction on GPU (1) - non-excluded particle list scheme Non-excluded particle list scheme is suitable for GPU due to small amount of memory for pairlist.

13 Real space non-bonded interaction on GPU (2) - How to make block/threads in each cell pair We make 32 thread blocks for efficient calculation on GPU by making blocks according to 8 atoms in cell I and 4 atoms in cell j.

14 Overview of GPU+CPU calculations with multiple time step integrator 1. In the multiple time step integrator, we do not perform reciprocal space interaction every step. 2. If reciprocal space interaction is not necessary, we assign subset of real space interaction on CPU to maximize the performance. 3. Integration is performed on CPU only.

15 Validation Tests (Energy drift) Machine Precision Integrator Energy drift CPU Double Velocity Verlet CPU Single Velocity Verlet CPU Double RESPA (4fs) CPU Single RESPA (4fs) CPU+GPU Double Velocity Verlet CPU+GPU Single Velocity Verlet CPU+GPU Double RESPA (4fs) CPU+GPU Single RESPA (4fs) CPU+GPU Single Langevin RESPA (8fs) CPU+GPU Single Langevin RESPA (10fs) Unit : kt/ns/degree of freedom 2fs time step with SHAKE/RATTLE/SETTLE constraints In the case of RESPA, the slow force time step is written in parentheses Our energy drift is similar to AMBER double and hybrid precision calculation.

16 1. MD program : GENESIS Benchmark condition 2. System : TIP3P water (22,000), STMV (1 million), Crowding system1 (11.7 million), and Crowding system 2 (100 million) 3. Cutoff : 12.0 Å 4. PME grid sizes : (STMV), (Crowding system1), and (Crowding system 2) 5. Integrator : Velocity Verlet (VVER), Respa (PME reciprocal every second step), and Langevin Respa (PME reciprocal every fourth step)

17 Acceleration of real space interactions (1) System : 9,250 TIP water molecules Cutoff distance : 12 Å Box size : 64 Å 64 Å 64 Å 1. 1 GPU increase the speed 3 times and 2 GPUs 6 times. 2. By assigning CPU as well as GPU when FFT on CPU is skipped, we can increase the speed up to 7.7 times.

18 Acceleration of real space interactions (2) Benchmark system (11.7 million atoms, Cutoff = 12.0 Å)

19 Comparison between real space and reciprocal space interactions STMV (1 million atoms) 11.7 million atoms 1. In both systems, the main bottleneck is the reciprocal space interactions irrespective of the number of processors. 2. Therefore, it is important to optimize the reciprocal space interaction when CPU+GPU clusters are used (=> Midpoint cell method could be best choice)

20 Comparison between TSUBAME and K STMV (1 million atoms) 11.7 million atoms 1. K has better parallel efficiency of reciprocal space interaction than TSUBAME. 2. Irrespective of the performance of reciprocal space interaction, TSUBAME shows better performance than K due to efficient evaluation of real space interaction on GPU.

21 Benchmark on TSUBAME VVER (1 million atoms) VVER (11.7 million atoms) RESPA (1 million atoms) RESPA (11.7 million atoms)

22 Performance on TSUBAME for 100 million atom systems Integrator Number of Nodes Time per step (ms) Simulation time (ns/day) VVER RESPA Langevin RESPA

23 Summary 1. We implemented MD for GPU+CPU clusters. 2. We assign GPUs for real space non-bonded interactions and CPUs for reciprocal space interactions, bonded interactions, and integrators. 3. We introduce a non-excluded particle list scheme for efficient usage of memory on GPU. 4. We also optimized the usage of GPUs and CPUs for multiple time step integrators. 5. Benchmark result on TSUBAME shows very good strong/weak scalability for 1 million, 11.7 million, and 100 million atoms systems.