Swift: task-based hydrodynamics at Durham s IPCC. Bower

Transcription

1 Swift: task-based hydrodynamics at Durham s IPCC Gonnet Schaller Chalk movie: Richard Bower (Durham) For the cosmological simulations of the formation of galaxies Bower Institute for Computational Cosmology Durham! 1

2 Contents: Introduction: cosmological simulations: aims and methods computational challenges load - (im) balance Swift: task-based hydrodynamics and gravity related talks/posters: keynote: Simon Portegies-Zwart: Massively-parallel GPU-accelerated galaxy simulation Poster: Karakasis et al Gadget on the Mike Several talks/posters on SPH (hydrodynamics scheme).e.g Guo: Exploring the Memory-Efficient Implementation Model for Incompressible Smoothed Particle Hydrodynamics(ISPH) 2

3 Introduction 3

4 d e rv e s ob l i M movie:volker Springel 4 m u i n l en Springel +05 Tom Theuns Virgo project

5 Smiling galaxies (because they contain dark matter) 5 SDSS J

6 Galaxy stellar mass function versus dark matter halo mass function Abundance galaxies Dark halos (const M/L) Stellar Mass 6 Halo Mass Bower+06

7 Physics of galaxy formation Aims: 2 pc How do galaxies form? How do they evolve? Which physical processes operate? x Basic paradigm (White & Rees, White & Frenk) Dark haloes form Cool(ed) gas forms discs Discs fragment to form stars 20 kpc x Multi-scale/complex/rich problem 200 Mpc 7

8 Galaxy stellar mass function versus dark matter halo mass function Abundance feedback from SNe? galaxies Dark halos (const M/L) feedback from AGN? Stellar Mass 8 Halo Mass Bower+06

9 Physics of supernova feedback Temperature density velocity pressure Gas column perpendicular to a galactic disc Creasey+13, 15 9 FLASH 30 Jupiter mass resolution

10 Current state of the art: Eagle simulations (Schaye +15) 10

11 Cosma 5 DataCentric 11 7 M CPU hours Tom Schaye Theuns +15

12 The Hubble Sequence 12 Schaye +15

13 Contents: Introduction: cosmological simulations: aims and methods computational challenges load - (im) balance Swift: task based hydrodynamics and gravity 13

14 Methods: algorithms and implementations gravity (from gas, stars and dark matter) hydrodynamics (gas accretion, gas cooling) Ramses (Teyssier) Gadget (Springel) Eulerian hydrodynamics (AMR codes) Highly optimised algorithms - minimise calculations! Lagrangian hydrodynamics (SPH codes) 14

15 Lagrangian hydrodynamics Gravity: density is found by summing weighted mass over neighbours distant force from mesh (FFTs) nearby force calculated from tree 15

16 Tree calculation (in 2 dimensions) 16

17 Contents: Introduction: cosmological simulations: aims and methods computational challenges load - (im) balance Swift: task based hydrodynamics and gravity 17

18 Build-up of dark matter halo typical dynamical time of a particle is 20 times longer than particle in a halo, and 1000 times that of a particle in a disc 18 Springel+08

19 Amount of memory per particle is large (many diagnostic properties stored, such as metallicity, star formation rate, etc), little computational time spent on interaction between two particles Runs require a lot of memory (and hence many cores/ nodes): currently 10 billion particle runs Lots of time spent in finding neighbours: leads to loadimbalance 19

20 Contents: Introduction: cosmological simulations: aims and methods computational challenges load - (im) balance Swift: task based hydrodynamics and gravity 20

21 Load (im) balance in gravity calculation (Gadget-2) total cpu time [s] tree walk tree walk + imbalance simulation time 21

22 Timings of pkdgrav on G-halo from Stadel Imbalance 22

23 Eagle reference run 4096 cores Cosma 5, Datacentric system (Durham) 23

24 Contents: Introduction: cosmological simulations: aims and methods computational challenges load - (im) balance Swift: task based hydrodynamics and gravity 24

25 Task based parallelism for SPH/Gravity Gonnet Schaller Chalk 25

26 Introduction This talk in a nutshell Speedup CosmoVolume (51M) 1 Parallel Efficiency CosmoVolume (51M) nr. cores Gadget number of cores 51M particle EAGLE box (z = 0.5) SPH-only simulation on the COSMA5 cluster. on future computers: gadget will become slower (Pete Beckman this morning) Pedro Gonnet: SWIFT: Task-based parallelism, hybrid shared/distributed-memory parallelism, and SPH simulations September 10th, /22

27 Introduction This talk in a nutshell Speedup CosmoVolume (51M) Swift Gadget nr. cores Parallel Efficiency CosmoVolume (51M) SWIFT Gadget nr. cores number of cores 64 Swift (task-based) Gadget 51M particle EAGLE box (z = 0.5) SPH-only simulation on the COSMA5 cluster. SWIFT: Strong scaling up to 1024 cores with 60% parallel efficiency Pedro Gonnet: SWIFT: Task-based parallelism, hybrid shared/distributed-memory parallelism, and SPH simulations September 10th, /22

28 computational domain choose cell to be larger than distance to neighbours h 28

29 Task type1 Task type1i i i j re-use particle data whenever possible in next task 29

30 QuickSched library Task-based parallelism Main concepts Shared-memory parallel programming paradigm in which the computation is formulated in an implicitly parallelizable way that automatically avoids most of the problems associated with concurrency and load-balancing. We first reduce the problem to a set of inter-dependent tasks. For each task, we need to know: I I Which tasks it depends on, Which tasks it conflicts with. Each thread then picks up a task which has no unresolved dependencies or conflicts and computes it. Pedro Gonnet: SWIFT: Task-based parallelism, hybrid shared/distributed-memory parallelism, and SPH simulations September 10th, /22

31 QuickSched library Task-based parallelism Main concepts Shared-memory parallel programming paradigm in which the computation is formulated in an implicitly parallelizable way that automatically avoids most of the problems associated with concurrency and load-balancing. We first reduce the problem to a set of inter-dependent tasks. For each task, we need to know: I I Which tasks it depends on, Which tasks it conflicts with. Each thread then picks up a task which has no unresolved dependencies or conflicts and computes it. Pedro Gonnet: SWIFT: Task-based parallelism, hybrid shared/distributed-memory parallelism, and SPH simulations September 10th, /22

32 QuickSched library Task-based parallelism Main concepts Shared-memory parallel programming paradigm in which the computation is formulated in an implicitly parallelizable way that automatically avoids most of the problems associated with concurrency and load-balancing. We first reduce the problem to a set of inter-dependent tasks. For each task, we need to know: I I Which tasks it depends on, Which tasks it conflicts with. Each thread then picks up a task which has no unresolved dependencies or conflicts and computes it. Pedro Gonnet: SWIFT: Task-based parallelism, hybrid shared/distributed-memory parallelism, and SPH simulations September 10th, /22

33 QuickSched library Task-based parallelism Main concepts Shared-memory parallel programming paradigm in which the computation is formulated in an implicitly parallelizable way that automatically avoids most of the problems associated with concurrency and load-balancing. We first reduce the problem to a set of inter-dependent tasks. For each task, we need to know: I I Which tasks it depends on, Which tasks it conflicts with. Each thread then picks up a task which has no unresolved dependencies or conflicts and computes it. Pedro Gonnet: SWIFT: Task-based parallelism, hybrid shared/distributed-memory parallelism, and SPH simulations September 10th, /22

34 QuickSched library Task-based parallelism Main concepts Shared-memory parallel programming paradigm in which the computation is formulated in an implicitly parallelizable way that automatically avoids most of the problems associated with concurrency and load-balancing. We first reduce the problem to a set of inter-dependent tasks. For each task, we need to know: I I Which tasks it depends on, Which tasks it conflicts with. Each thread then picks up a task which has no unresolved dependencies or conflicts and computes it. Pedro Gonnet: SWIFT: Task-based parallelism, hybrid shared/distributed-memory parallelism, and SPH simulations September 10th, /22

35 QuickSched library Task-based parallelism Main concepts Shared-memory parallel programming paradigm in which the computation is formulated in an implicitly parallelizable way that automatically avoids most of the problems associated with concurrency and load-balancing. We first reduce the problem to a set of inter-dependent tasks. For each task, we need to know: I I Which tasks it depends on, Which tasks it conflicts with. Each thread then picks up a task which has no unresolved dependencies or conflicts and computes it. Pedro Gonnet: SWIFT: Task-based parallelism, hybrid shared/distributed-memory parallelism, and SPH simulations September 10th, /22

36 QuickSched library Task-based parallelism Main concepts Shared-memory parallel programming paradigm in which the computation is formulated in an implicitly parallelizable way that automatically avoids most of the problems associated with concurrency and load-balancing. We first reduce the problem to a set of inter-dependent tasks. For each task, we need to know: I I Which tasks it depends on, Which tasks it conflicts with. Each thread then picks up a task which has no unresolved dependencies or conflicts and computes it. ParMETIS library distributes tasks Pedro Gonnet: SWIFT: Task-based parallelism, hybrid shared/distributed-memory parallelism, and SPH simulations September 10th, /22

37 MPI implementation uses asynchronous comms node 1 node 2 particles from neighbour cell on other node imported by comm task 37

38 QuickSched library tasks are distributed over resources to maximise throughput (and not spatially) 38

39 Sorting particles in cells cuts down on unnecessary neighbour testing 39

40 Introduction This talk in a nutshell Speedup CosmoVolume (51M) Parallel Efficiency CosmoVolume (51M) nr. cores 0.2 SWIFT Gadget nr. cores 51M particle EAGLE box (z = 0.5) SPH-only simulation on the COSMA5 cluster. SWIFT: Strong scaling up to 1024 cores with 60% parallel efficiency. 40 faster than GADGET. Pedro Gonnet: SWIFT: Task-based parallelism, hybrid shared/distributed-memory parallelism, and SPH simulations September 10th, /22

41 Node ID Swift tasks wall clock time (ms) 41

42 block ID Task plot for Barnes-Hut gravity tree purple=data movement time (ms) red, green, blue=different tasks big tasks to be moved

43 Timings of gravity part (Barnes-Hut) of code QuickSched implementation uses quadrupoles, Gadget uses monopoles Bonsai 2: 43

44 Contents: Introduction: cosmological simulations: aims and methods computational challenges load - (im) balance Swift: task based hydrodynamics and gravity Future: vectorisation of interaction kernels optimizing cache performance optimizing MPI performance self-tuning strategy to exploit hardware specifics 44

45 Swift: task-based hydrodynamics at Durham s IPCC t e n n o G S r e l l a h c k l a h C For the cosmological simulations of the formation of r galaxies e w o B Institute for Computational Cosmology Durham! movie: Richard Bower (Durham) 45