PIConGPU Bringing Large-Scale Laser Plasma Simulations to GPU Supercomputing

Transcription

1 PIConGPU Bringing Large-Scale Laser Plasma Simulations to GPU Supercomputing Michael Bussmann 1, Heiko Burau 1, René Widera 1, Florian Berninger 1, Axel Hübl 1, Thomas Kluge 1, Alexander Debus 1, Ulrich Schramm 1, Thomas E. Cowan 1, Felix Schmitt 2, Wolfgang Hönig 2, Guido Juckeland 2, Wolfgang Nagel 2 1 Helmholtz-Zentrum Dresden-Rossendorf 2 Zentrum für Informationsdienste und Hochleistungsrechnen Text optional: Institutsname Prof. Dr. Hans Mustermann Mitglied der Leibniz-Gemeinschaft

2 At HZDR we use high-power lasers to create beams of ions, electrons and x-rays. With the lasers we ionize matter and create a hot plasma. The electromagnetic fields in the plasma are strong enough to accelerate particles to relativistic energies. Parallel Simulations are needed to understand and optimize these acceleration processes. But we have to make them fast... Seite 2

3 Advanced Laser-driven Particle Sources Seite 3

4 W in s DRACO High-Power Laser 25 x seconds pulse duration 2 x Watts peak power 10 x worldwide power consumption Seite 4

5 Laser-driven Radiation HZDR Laser-driven electron beams Laser-driven proton beams Thomson scattering X-ray beams Seite 5

6 Application example: Hadron therapy of cancerous tissue X-rays Ions Effective Dose (%) Seite 6

7 Next generation tools to fight cancer? Why worry? Each one of us is carrying an unlicensed nuclear accelerator on his back. Dr. Peter Venkman, Ghost Busters Seite 7

8 The Particle-in-Cell Algorithm for Relativistic Plasmas Seite 8

9 The Particle-in-Cell (PiC) algorithm F Compute the Lorentz Force q E v B Integrate the Equation of Motion dp dt F Compute new Fields E t B t 1 J B 2 c ε0 E Compute Currents J uf(t,r,v)dv Seite 9

10 The Particle-in-Cell (PiC) algorithm in action Seite 10

11 The problem with particles that don t stay in cells Seite 11

12 Memory access conflicts caused by non-locality of particle data Threads Memory Access Physical Grid = Memory Layout Thread 1 Thread 2 Thread 3 Thread 4 Thread 5 Seite 12

13 PiConGPU Seite 13

14 Everything depends on data structures PART 1 cell-based data Super Cell Each thread in a thread block corresponds to one cell in the super cell ID 1 ID 2 ID 3 ID 4 ID 5 ID 6 ID 7 ID B,E,J ID 9 ID 10 ID 11 ID 12 ID 13 ID 14 ID 15 ID Seite 14

15 The super cell allows for cell-wise threading in one thread block 1 2 ID 1 local memory access ID 2 Magnetic Fields Electric Fields Currents Magnetic Fields Electric Fields Currents Parallel Execution 16 ID 16 Magnetic Fields Electric Fields Currents Seite 15

16 Everything depends on data structures PART 2 particle data Attribute Frame (the data structure formerly known as tile, pool,...) x 1 y 1 x 2 y 2 x 3 y 3 x 4 y 4 x 16 y 16 Each thread in a thread block is started for one particle in the frame z 1 z 2 z 3 z 4 z 16 ID 1 ID 2 ID 3 ID 4 ID 5 ID 6 ID 7 ID 8 m 1 m 2 m 3 m 4 m 16 ID 9 ID 10 ID 11 ID 12 Q 1 Q 2 Q 3 Q 4 Q 16 ID 13 ID 14 ID 15 ID Cell index tells you the particle s local cell index inside the super cell Seite 16

17 The super cell allows for cell-wise threading in one thread block x 1 1 ID 1 y 1 z 1 m 1 local memory access x 2 Q 1 2 ID 2 y 2 z 2 m 2 Q 2 Parallel Execution 3 ID 16 x 16 y 16 z 16 m 16 Q 16 Seite 17

18 Fill frames according to the particle position in the super cell Super Cell Attribute Frame x 1 y 1 x 2 y 2 x 3 y 3 x 4 y 4 x 15 y z 1 z 2 z 3 z 4 z m 1 Q 1 m 2 Q 2 m 3 Q 3 m 4 Q 4 m 15 Q If there are more particles than cells, just add frames! Seite 18

19 Associating tiles and super cells Super Cell Start of Frame List Attribute Frame x 1 x 2 x 3 x 4 x 16 x 17 x 18 x 19 y 1 y 2 y 3 y 4 y y 17 y 18 y 19 z 1 z 2 z 3 z 4 z 16 z 17 z 18 z m 1 m 17 m 2 m 18 m 3 m 19 m 4 m 16 Q 1 Q 17 Q 2 Q 18 Q 3 Q Q 4 Q 16 End of Frame List link frames Seite 19

20 Storage of cell data in global memory optimum alignment Super Cell B,E,J Global Memory optimum size Seite 20

21 Storage of particle data in global memory optimum alignment Attribute Frame Global Memory x 1 x 2 x 3 x 4 x 16 x 1 y 1 z 1 m 1 Q C 1 x 2 y 2 z 2 m 2 Q C 2 y 1 y 2 y 3 y 4 y 16 x 3 y 3 z 3 m 3 Q C 3 x 4 y 4 z 4 m 4 Q C 4 z 1 z 2 z 3 z 4 z 16 x 5 y 5 z 5 m 5 Q C 5 x 6 y 6 z 6 m 6 Q C 6 m 1 m 2 m 3 m 4 m 16 x 7 y 7 z 7 m 7 Q C 7 Q 1 Q 2 Q 3 Q 4 Q 16 x 8 y 8 z 8 m 8 Q C 8 x 9 y 9 z 9 m 9 Q C 9 x 10 y 10 z 10 m 10 Q C 10 x 11 y 11 z 11 m 11 Q C x 12 y 12 z 12 m 12 Q C 12 optimum size Seite 21

22 The particle push storing cell-data in shared memory Global Memory Thread Cell in Super Cell 1 B,E,J read ID 1 write 2 B,E,J read ID 2 write 3 B,E,J read ID 3 write Cell-wise threading Seite 22

23 The particle push pushing particles using stored cell data Attribute Frame Thread Cell in Super Cell x 1 y 1 z 1 1 push ID 1 read 1 B,E,J x 2 y 2 z 2 2 push ID 2 read 2 B,E,J x 3 y 3 z 3 2 push ID 3 read 2 B,E,J Particle-wise threading Seite 23

24 Current deposition atomicadd(float) is not so expensive (Fermi) Super Cell ID 5 2 J 3 J ID 9 6 J 7 J 6 J 7 J J 11 J 10 Seite 24

25 Beyond one GPU Seite 25

26 Encapsulation of all memory transfer functions GPU1 GPU2 cudamemcpy, MPI_ISEND asynchronous any data (particle, cell) + Local Domain Model a thread can assume that all the data it needs is in its local memory (SISD) Seite 26

27 How to add automatic memory transfer to your code (electric field) //######################################################## // Allocate Memory //######################################################## //Create layout GridLayout<DIM2> layout(dataspace<dim2>(1024,1024),dataspace<dim2>(16,16)); //Create memory GridBuffer<float2,DIM2> *field=new GridBuffer<float2,DIM2>(layout,FIELD_E,false); //define edges field->addexchange (Mask(TOP)+Mask(LEFT),DataSpace<DIM2>(1,1),false); Seite 27

28 Events help to direct kernel execution, data exchange, etc. EventTask tree 1 Depends on Parallel to Seite 28

29 Defining the algorithmic structure of a task graph 1 getsizefromdevice(); // memcpy from device EventTask e_split = gettransactionevent(); starttransaction( e_split ); // depend from e_split cudakernel_x(); // create a kernel task EventTask e1 = endtransaction(); starttransaction( e_split ); cudakernel_y(); synchronizewithneighbor(); // memcpy+mpi task EventTask e2 = endtransaction(); settransactionevent( e1+e2 ); // combine parallel work cudakernel_z(); Seite 29

30 How to add events to your code (electric field) //######################################################## // One iteration //######################################################## // Receive data needed for Computing FieldE EventTask eventfielde = this->fielde->getgridbuffer()->startreceive(); // Compute FieldE (1st kernel) this->fielde->updatee(*(this->fieldb)); // FieldE is ready and borders can be sent to neighbors EventTask eventfielde+=this->fielde->getgridbuffer()- >startsend(); //Join two events //Receive FieldB from neighbors EventTask eventfieldb=this->fieldb->getgridbuffer()- >startreceive(); Seite 30

31 What s the deal? Seite 31

32 Physics capabilities 3D3V relativistic particle-in-cell code Villasenor-Buneman charge conserving current deposition NGP / CIC / TSC macro-particle distribution functions Boris Push particle pusher Yee-scheme / Directional splitting Maxwell-Solver Seite 32

33 Strong and Weak Scaling on NVIDIA TESLA M2090 Seite 33

34 Laser-driven wakefield acceleration of electrons 64 NVIDIA Fermi GPUs 45 min 128 AMD cores 8 days 480x480x3070 Cells 2.8 x 10 9 macro particles PIConGPU ILLUMINATION Seite 34

35 picongpu.hzdr.de ccoe-dresden.de Seite 35