SENR: A Super-Efficient Numerical Relativity Code for the Age of Gravitational Wave Astrophysics. Zachariah B. Etienne Ian Ruchlin

Transcription

1 SENR: A Super-Efficient Numerical Relativity Code for the Age of Gravitational Wave Astrophysics Zachariah B. Etienne Ian Ruchlin in collaboration with Thomas W. Baumgarte

2 Moore's Law Is Slowing Down Intel CEO Brian Krzanich,, 2015: Our cadence today is closer to two and a half years than two. Intel CEO Brian Gordon Moore,, 2015: I I see Moore s law dying here in the next decade or so. Gordon Moore Computational Astrophysics needs to shift to smarter algorithms!

3 Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes AMR Adaptive Mesh Refinement (Most Popular Method in NR) BH dx BH dx 2 dx 4 dx 8 dx 16 dx, etc

4 Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes AMR Adaptive Mesh Refinement (Most Popular Method in NR) BH dx BH dx 2 dx 4 dx 8 dx 16 dx, etc

5 Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes BH dx

6 Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes Near-Spherical Object Highest res needed in radial dirn, need 1/3 1/10 points in angular directions Cost: Nr*Ntheta*Nphi ~ 1/100 Nr 3 1/10 Nr 3 Cartesian grid: need dx=dy=dz=dr. Cost: Nx*Ny*Nz ~ Nr 3 So far, spherical polar grid ~ x more efficient than Cartesian BH dy=dx=dr

7 Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes Near-Spherical Object Highest res needed in radial dirn, need 1/3 1/10 points in angular directions Cost: Nr*Ntheta*Nphi ~ 1/100 Nr 3 1/10 Nr 3 Cartesian grid: need dx=dy=dz=dr. Cost: Nx*Ny*Nz ~ Nr 3 So far, spherical polar grid ~ x more efficient than Cartesian BH dy=dx=dr What about dr along diagonal? Cube diagonal = 3*sidelength to get dr resolution in all directions, need to reduce dx,dy,dz,dt by 3 Since simulation cost ~1/dx 4, fitting the round peg in a square hole increases cost by another factor of ( 3) 4 =9x!

8 Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes Near-Spherical Object Highest res needed in radial dirn, need 1/3 1/10 points in angular directions Cost: Nr*Ntheta*Nphi ~ 1/100 Nr 3 1/10 Nr 3 Cartesian grid: need dx=dy=dz=dr. Cost: Nx*Ny*Nz ~ Nr 3 So far, spherical polar grid ~ x more efficient than Cartesian BH dy=dx=dr Inefficiencies so far: ~100-1,000x What about dr along diagonal? Cube diagonal = 3*sidelength to get dr resolution in all directions, need to reduce dx,dy,dz,dt by 3 Since simulation cost ~1/dx 4, fitting the round peg in a square hole increases cost by another factor of ( 3) 4 =9x!

9 Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes AMR Box Boundary is a Cube... but fields fall off radially! region outside orange circle is over-resolved by 2x Total volume of over-resolved region = 8-4/3 pi ~ 3.8 = about half the cube! So we gain by about another factor of 1.9x. BH AMR Box side- length = 2 Round Peg in Square Hole ~200-2,000x Costlier!

10 Baumgarte et al., Phys. Rev. D 87, (2012) arxiv: Idea: Move to Spherical Polar Coordinates! Cartesian Coords: Inefficient by ~200 2,000x, in computational cost ~100 1,000x inefficient in memory overhead

11 Baumgarte et al., Phys. Rev. D 87, (2012) arxiv: Idea: Move to Spherical Polar Coordinates! Cartesian Coords: Inefficient by ~200 2,000x, in computational cost ~100 1,000x inefficient in memory overhead Spherical-Polar Coords: Not a Panacea Coord singularities at r=0, sin(th)=0 num. instabilities Solved: PIRK! r = 0 focusing + high-resolution + CFL condition timestep reduced ~200 2,000x, even with cell-centering What can be done?

12 Baumgarte et al., Phys. Rev. D 87, (2012) arxiv: Idea: Move to Spherical Polar Coordinates! Cartesian Coords: Inefficient by ~200 2,000x, in computational cost ~100 1,000x inefficient in memory overhead Spherical-Polar Coords: Not a Panacea Coord singularities at r=0, sin(th)=0 num. instabilities Solved: PIRK! r = 0 focusing + high-resolution + CFL condition timestep reduced ~200 2,000x, even with cell-centering What can be done?

13 Improved Spherical Polar Coords. Problem: Focusing of gridpoints near r = 0 in Spherical Polar Coords + CFL condition 200 2,000x smaller timestep than Cartesian coords, despite being 100 1,000x more memory efficient Solutions: Magnify r coord near r = 0 ~10x timestep! Baumgarte et al., Phys. Rev. D 87, (2012) arxiv:

14 Improved Spherical Polar Coords. Problem: Focusing of gridpoints near r = 0 in Spherical Polar Coords + CFL condition 200 2,000x smaller timestep than Cartesian coords, despite being 100 1,000x more memory efficient Solutions: Baumgarte et al., Phys. Rev. D 87, (2012) arxiv: Magnify r coord near r = 0 ~10x timestep! Skip over angular points closest to r = 0 get another ~10x

15 Improved Spherical Polar Coords. Problem: Focusing of gridpoints near r = 0 in Spherical Polar Coords + CFL condition 200 2,000x smaller timestep than Cartesian coords, despite being 100 1,000x more memory efficient Solutions: Baumgarte et al., Phys. Rev. D 87, (2012) arxiv: Magnify r coord near r = 0 ~10x timestep! Skip over angular points closest to r = 0 get another ~10x Exploit memory efficiency move to GPU Get another ~100x speed-up! Per-GPU ~5 50x faster than Cartesian AMR

16 Improved Spherical Polar Coords. Problem: Focusing of gridpoints near r = 0 in Spherical Polar Coords + CFL condition 200 2,000x smaller timestep than Cartesian coords, despite being 100 1,000x more memory efficient Solutions: Baumgarte et al., Phys. Rev. D 87, (2012) arxiv: Magnify r coord near r = 0 ~10x timestep! Skip over angular points closest to r = 0 get another ~10x Exploit memory efficiency move to GPU Get another ~100x speed-up! Get gamers involved 1,000x speed-up

17 Improved Spherical Polar Coords. Problem: Focusing of gridpoints near r = 0 in Spherical Polar Coords + CFL condition 200 2,000x smaller timestep than Cartesian coords, despite being 100 1,000x more memory efficient Solutions: Baumgarte et al., Phys. Rev. D 87, (2012) arxiv: Magnify r coord near r = 0 ~10x timestep! Skip over angular points closest to r = 0 get another ~10x Exploit memory efficiency move to GPU Get another ~100x speed-up! Get gamers involved 1,000x speed-up Per-GPU ~5 50x faster than Cartesian AMR With gamers, 5,000 50,000x faster GW throughput

18 Current Literature Re: NR in Spherical Polar Coords Movie courtesy T. Baumgarte

19 Our Basic Strategy Numerical Relativity: strongly hyperbolic formalisms of GR Step 1: Solve scalar wave eq. in given coord system Demonstrate stability & convergence Step 2: Implement in new, easily-extensible NR (BSSN) code New coordinate systems Dynamical, co-rotating spherical polar coordinates Ideal for CCSNe; logarithmic radial coord beyond r = 0 Bispherical-like coords + co-rotation & linear rescaling Ideal for DNS + BHB simulations

20 Our Basic Strategy Numerical Relativity: strongly hyperbolic formalisms of GR Step 1: Solve scalar wave eq. in given coord system Demonstrate stability & convergence Step 2: Implement in new, easily-extensible NR (BSSN) code New coordinate systems Dynamical, co-rotating spherical polar coordinates Ideal for CCSNe; logarithmic radial coord beyond r = 0 Bispherical-like coords + co-rotation & linear rescaling Ideal for DNS + BHB simulations

21 Scalar Wave Evolutions in Rotating, Logarithmic-Radius Spherical Polar Coords

22 Scalar Wave Evolutions in Rotating, Logarithmic-Radius Spherical Polar Coords Exponential convergence with increased FD order

23 Our Basic Strategy Numerical Relativity: strongly hyperbolic formalisms of GR Step 1: Solve scalar wave eq. in given coord system Demonstrate stability & convergence Step 2: Implement in new, easily-extensible NR (BSSN) code New coordinate systems Dynamical, co-rotating spherical polar coordinates Ideal for CCSNe; logarithmic radial coord beyond r = 0 Bispherical-like coords + co-rotation & linear rescaling Ideal for DNS + BHB simulations

24 Scalar Wave Evolutions in Bispherical-Like Coordinates Optimal coordinate system for BHBs & DNSs: Spherical-polar coordinates near BHs/NSs Uniform spherical polar coords far from the binary system.

25 Scalar Wave Evolutions in Bispherical-Like Coordinates Optimal coordinate system for BHBs & DNSs: Spherical-polar coordinates near BHs/NSs Uniform spherical polar coords far from the binary system. TwoPunctures coords = great near BHs/NSs, but radial grid far from the binary compactified GWs under-resolved Uncompactify radial coords problem solved! Original TwoPunctures coordinates:

26 Scalar Wave Evolutions in Bispherical-Like Coordinates Exponential convergence with increased FD order

27 Movie: Bispherical-like Coordinate Improvement & Dynamics

28 Numerical Relativity (SENR) Code Extensive automatic code generation: Extensive automatic code generation: Arbitrary-order finite differencing Simple Mathematica interface for generating NR (BSSN) code in arbitrary coordinates: Simple Mathematica interface for generating NR (BSSN)

29 Numerical Relativity (SENR) Code Validated against industry-standard Kranc code for Cartesian coordinates (agrees( to roundoff error). Validated against industry-standard Kranc code for Validation against Baumgarte's spherical polar code: in progress. Validation against Baumgarte's spherical polar code: Extension to more sophisticated coords planned

30 Numerical Relativity (SENR) Code Validated against industry-standard Kranc code for Cartesian coordinates (agrees( to roundoff error). Validated against industry-standard Kranc code for Validation against Baumgarte's spherical polar code: in progress. Validation against Baumgarte's spherical polar code: Extension to more sophisticated coords planned Bottom Line: Rapid progress being made toward goals Rapid progress being made toward goals Codes maximize user-friendliness, rapid- prototyping of ideas, and extensibility Codes maximize user-friendliness, rapid- Join the open development! Join the open development!