Accelerating Three-Body Potentials using GPUs NVIDIA Tesla K20X

Transcription

1 Using a Hybrid Cray Supercomputer to Model Non-Icing Surfaces for Cold- Climate Wind Turbines Accelerating Three-Body Potentials using GPUs NVIDIA Tesla K20X GE Global Research Masako Yamada

2 Opportunity in Cold-Climate Wind Wind energy production > 285 GW/year and growing Cold regions favorable Lower human population Good wind conditions GW opportunity from ~$2million/MW installed Technical need Anti-icing surfaces 3-10% energy losses due to icing Shut-downs Active heating expensive VTT Technical Research Centre of Finland 13_wind_energy.jsp?lang=en 2

3 ALCC Awards million hours DOE ASCR Leadership Computing Challenge Awards Energy-relevant applications 1. Non-Icing Surfaces for Cold-Climate Wind Turbines Jaguar (Cray XK6) at Oak Ridge National Lab Molecular dynamics using LAMMPS 1 million mw water molecule droplets on engineered surfaces Completed >300 simulations Achieved >200x speedup from 2011 to 2013 >5x from GPU acceleration 2. Accelerated Non-Icing Surfaces for Cold-Climate Wind Turbines Titan (Cray XK7, hybrid) at Oak Ridge National Lab Time parallelization via Parallel Replica method Expected x faster results 3

4 Titan enables leadership-class study Size of simulation ~ 1 million molecules Droplet size >> critical nucleus size Mimic physical dimensions (*somewhat) Duration of simulation ~ 1 microsecond Nucleation is an activated process Freezing rarely observed in MD simulations Number of simulations ~ 100 s Study requires embarrassingly parallel runs Different surfaces, ambient temperatures, conductivity Multiple replicates required due to stochastic nature *million molecule droplet ~ 50nm diameter 4

5 Personal history with MD LOGO DRAFT Year Software/Language # of Molecules Hardware 1995 Pascal Few Desktop Mac 2000 C, Fortran90 Hundreds IBM SP, SGI O2K 2010 NAMD, LAMMPS 1000 s Linux HPC Present GPU-enabled LAMMPS Millions Titan

6 >200x overall speedup since Switched to mw water potential 3-body model is more expensive/complex than 2-body but Particle reduction at least 3x Timestep increase 10x No long-range forces 2. LAMMPS dynamic load balance 2-3x 3. GPU acceleration of 3-body model 5x 2011: 6 femtosecond/1024 CPU-second (SPC/E) 2013: 2 picoseconds/1024 CPU-second (mw) 6

7 1. mw water potential Stillinger Weber 3-body particle = one water molecule Introduced in 2009, Nature paper in 2011 Bulk water properties comparable or better than existing point-charge models Much faster than point-charge models Exemplary test case by authors: 180x faster than SPC/E GE production simulation: 40-50x faster than SPC/E asymmetric million molecule droplet on engineered surface; loaded onto 64 nodes SPC/E mw 7

8 2. LAMMPS dynamic load balance Introduced in 2012 Adjusts size of processor sub-domains to equalize number of particles 2-3x speedup for 1 million molecule droplets on 64 nodes (with user-specified processor mapping) No load balancing Default load balancing User-specified mapping 8

9 3. GPU-acceleration of 3-body potential See details W. Michael Brown and Masako Yamada Implementing Molecular Dynamics on Hybrid High Performance Computers Three-Body Potentials. Computer Physics Communications Computer Physics Communications, (2013) 9

10 Load 1 million molecules on Host/CPU + 1 million molecules 64 nodes Processor sub-domains correspond to spatial partitioning of droplet MPI tasks/node 1 core/paired-unit 10

11 Host Memory Per node ~ 15,000 molecules DRAFT Host AMD Opteron 6274 CPU Core0 Core1 Core2 Core3 Core4 Core5 Core6 Core7 Core8 Core9 Core10 Core11 Core12 Core13 Core14 Core15 Work iterm Work item Work item Work item Kernel Work item Work item Work item Work Group Processor 1 Processor 2. Processor 14 Accelerator NVIDIA Tesla K20X GPU Core 1 Core 2 Core 192 Private Private Private Local Memory Global Memory Work item = fundamental unit of activity 11

12 Parallelization in LAMMPS Host Time integration Thermostat/barostat Bond/angle calculations Statistics Accelerator 3-body potential Neighbor-lists 12

13 Generic 3-body potential U = i j i k>j φ p i, p j, p k r ij < r c, r ik < r c 0 otherwise Good candidate for GPU 1. Occupies majority of computational time 2. Can be decomposed into independent kernels/work-items p i i p j r ij r ik j k Stillinger-Weber MEAM Tersoff REBO/AIREBO Bond-order (0,0,0) p k r c = cutoff r α = neighbor skin 13

14 Redundant Computation Approach Atom-decomposition 1 atom 1 computational kernel only fewest operations (and effective parallelization) but shared memory access a bottleneck Force-decomposition 1 atom 3 computational kernels required redundant computations but reduced shared memory issues many work-items = more effective use of cores 14

15 Stillinger-Weber Parallelization U = φ 2 (r ij ) + φ 3 r ij, r ik, θ jik i j<i i j i k>j 2-body operations Atom i 3 kernels no data dependencies 3-body operations (r ij < r α ).AND. (r ik < r α ) ==.TRUE. update forces on i only 3-body operations (r ij < r α ).AND. (r ik < r α ) ==.FALSE. neighbor-of-neighbor interactions 15

16 Neighbor List 3-body force-decomposition approach involves neighbor-of-neighbor operations Requires additional overhead increase in border size shared by two processes neighbor list for ghost atoms straddling across cores GPU implementation not necessarily faster than CPU but less time spent in host-accelerator data transfer (note: neighbor lists are huge) 16

17 GPU acceleration benefit >5x speedup achieved in production water droplet of 1 million molecules on engineered surface (64 nodes) Not limited to Stillinger-Weber -- applicable to MEAM, Tersoff, REBO, AIREBO, Bond-order, etc. 17

18 Implementation 18

19 6 different surfaces Interaction potential developed at GE Global Research 19

20 Freezing front propagation Visualization of latent heat release 20

21 Bottom View Side View Visualizing crystalline regions Steinhardt-Nelson order parameter particle mobility DRAFT

22 Advanced visualization Mike Matheson, Oak Ridge National Lab Will include visuals/movies here 22

23 Next steps Quasi time parallelization using Parallel Replica Method Launch dozens of replicates simultaneously; monitor ensemble behavior Expected outcome: x faster results Analysis and application of simulation results 23

24 Credits Mike Brown (ORNL) GPU acceleration Paul Crozier (Sandia) dynamic load balancing Valeria Molinero (Utah) mw potential Aaron Keyes (Umich, Berkeley) Steinhardt-Nelson order parameters Art Voter/Danny Perez (LANL) Parallel Replica method Mike Matheson (ORNL) -- Visualization Jack Wells, Suzy Tichenor (ORNL) General Azar Alizadeh, Branden Moore, Rick Arthur, Margaret Blohm (GE Global Research) This research was conducted in part under the auspices of the Office of Advanced Scientific Computing Research, Office of Science, U.S. Department of Energy under Contract No. DEAC05-00OR22725 with UT- Battelle, LLC. This research was also conducted in part under the auspices of the GE Global Research High Performance Computing program. 24

25 References W. Michael Brown, W. M and Yamada, M. Implementing Molecular Dynamics on Hybrid High Performance Computers Three-Body Potentials. Computer Physics Communications Computer Physics Communications, (2013) Shi, B. and Dhir, V. K. Molecular dynamics simulation of the contact angle of liquids on solid surfaces. The Journal of Chemical Physics, 130, 3 (01/21/ 2009), ; Sergi, D., Scocchi, G. and Ortona, A. Molecular dynamics simulations of the contact angle between water droplets and graphite surfaces. Fluid Phase Equilibria, 332, 0 (10/25/ 2012), Oxtoby, D. W. Homogeneous nucleation: theory and experiment. Journal of Physics: Condensed Matter, 4, ), Plimpton, S. Fast Parallel Algorithms for Short-Range Molecular Dynamics. Journal of Computational Physics, 117, 1 (3/1/ 1995), Humphrey, W., Dalke, A. and Schulten, K. VMD: Visual molecular dynamics. Journal of Molecular Graphics, 14, 1 (2// 1996), Keys, A. S. Shape Matching Analysis Code. University of Michigan, City, 2011; Keys, A. S., Iacovella, C. R. and Glotzer, S. C. Characterizing Structure Through Shape Matching and Applications to Self-Assembly. Annual Review of Condensed Matter Physics, 2, 1 (2011/03/ ), ; Steinhardt, P. J., Nelson, D. R. and Ronchetti, M. Bond-orientational order in liquids and glasses. Physical Review B, 28, 2 (07/15/ 1983), Stillinger, F. H. and Weber, T. A. Computer simulation of local order in condensed phases of silicon. Physical Review B, 31, 8 (04/15/ 1985), Berendsen, H. J. C., Grigera, J. R. and Straatsma, T. P. The missing term in effective pair potentials. The Journal of Physical Chemistry, 91, 24 (1987/11/ ), Molinero, V. and Moore, E. B. Water Modeled As an Intermediate Element between Carbon and Silicon. The Journal of Physical Chemistry B, 113, 13 (2009/04/ ), ; Moore, E. B. and Molinero, V. Structural transformation in supercooled water controls the crystallization rate of ice. Nature, 479, 7374 (11/24/print 2011), Yamada, M., Mossa, S., Stanley, H. E. and Sciortino, F. Interplay between Time-Temperature Transformation and the Liquid-Liquid Phase Transition in Water. Physical Review Letters, 88, 19 (04/26/ 2002), Brown, W. M., Wang, P., Plimpton, S. J. and Tharrington, A. N. Implementing molecular dynamics on hybrid high performance computers short range forces. Computer Physics Communications, 182, 4 (4// 2011), Voter, A. F. Parallel replica method for dynamics of infrequent events. Physical Review B, 57, 22 (06/01/ 1998), R13985-R

26 Titan 299,008 Opteron cores (18688 nodes) 18,688 NVIDIA Tesla K20X GPU accelerators Each node 16 cores AMD Opteron CPUs 1 Tesla K20X GPU accelerator 2688 compute cores Gemini interconnect (ASIC, MPI messages) PCI-Express 2.0 bus LAMMPS one of the acceptance testing applications for Titan 26