SENR: A Super-Efficient Numerical Relativity Code for the Age of Gravitational Wave Astrophysics Zachariah B. Etienne Ian Ruchlin in collaboration with Thomas W. Baumgarte
Moore's Law Is Slowing Down Intel CEO Brian Krzanich,, 2015: Our cadence today is closer to two and a half years than two. Intel CEO Brian Gordon Moore,, 2015: I I see Moore s law dying here in the next decade or so. Gordon Moore Computational Astrophysics needs to shift to smarter algorithms!
Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes AMR Adaptive Mesh Refinement (Most Popular Method in NR) BH dx BH dx 2 dx 4 dx 8 dx 16 dx, etc
Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes AMR Adaptive Mesh Refinement (Most Popular Method in NR) BH dx BH dx 2 dx 4 dx 8 dx 16 dx, etc
Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes BH dx
Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes Near-Spherical Object Highest res needed in radial dirn, need 1/3 1/10 points in angular directions Cost: Nr*Ntheta*Nphi ~ 1/100 Nr 3 1/10 Nr 3 Cartesian grid: need dx=dy=dz=dr. Cost: Nx*Ny*Nz ~ Nr 3 So far, spherical polar grid ~ 10-100x more efficient than Cartesian BH dy=dx=dr
Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes Near-Spherical Object Highest res needed in radial dirn, need 1/3 1/10 points in angular directions Cost: Nr*Ntheta*Nphi ~ 1/100 Nr 3 1/10 Nr 3 Cartesian grid: need dx=dy=dz=dr. Cost: Nx*Ny*Nz ~ Nr 3 So far, spherical polar grid ~ 10-100x more efficient than Cartesian BH dy=dx=dr What about dr along diagonal? Cube diagonal = 3*sidelength to get dr resolution in all directions, need to reduce dx,dy,dz,dt by 3 Since simulation cost ~1/dx 4, fitting the round peg in a square hole increases cost by another factor of ( 3) 4 =9x!
Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes Near-Spherical Object Highest res needed in radial dirn, need 1/3 1/10 points in angular directions Cost: Nr*Ntheta*Nphi ~ 1/100 Nr 3 1/10 Nr 3 Cartesian grid: need dx=dy=dz=dr. Cost: Nx*Ny*Nz ~ Nr 3 So far, spherical polar grid ~ 10-100x more efficient than Cartesian BH dy=dx=dr Inefficiencies so far: ~100-1,000x What about dr along diagonal? Cube diagonal = 3*sidelength to get dr resolution in all directions, need to reduce dx,dy,dz,dt by 3 Since simulation cost ~1/dx 4, fitting the round peg in a square hole increases cost by another factor of ( 3) 4 =9x!
Case in Point: Enormous Inefficiencies Exist in Numerical Relativity Codes AMR Box Boundary is a Cube... but fields fall off radially! region outside orange circle is over-resolved by 2x Total volume of over-resolved region = 8-4/3 pi ~ 3.8 = about half the cube! So we gain by about another factor of 1.9x. BH AMR Box side- length = 2 Round Peg in Square Hole ~200-2,000x Costlier!
Baumgarte et al., Phys. Rev. D 87, 044026 (2012) arxiv:1211.6632 Idea: Move to Spherical Polar Coordinates! Cartesian Coords: Inefficient by ~200 2,000x, in computational cost ~100 1,000x inefficient in memory overhead
Baumgarte et al., Phys. Rev. D 87, 044026 (2012) arxiv:1211.6632 Idea: Move to Spherical Polar Coordinates! Cartesian Coords: Inefficient by ~200 2,000x, in computational cost ~100 1,000x inefficient in memory overhead Spherical-Polar Coords: Not a Panacea Coord singularities at r=0, sin(th)=0 num. instabilities Solved: PIRK! r = 0 focusing + high-resolution + CFL condition timestep reduced ~200 2,000x, even with cell-centering What can be done?
Baumgarte et al., Phys. Rev. D 87, 044026 (2012) arxiv:1211.6632 Idea: Move to Spherical Polar Coordinates! Cartesian Coords: Inefficient by ~200 2,000x, in computational cost ~100 1,000x inefficient in memory overhead Spherical-Polar Coords: Not a Panacea Coord singularities at r=0, sin(th)=0 num. instabilities Solved: PIRK! r = 0 focusing + high-resolution + CFL condition timestep reduced ~200 2,000x, even with cell-centering What can be done?
Improved Spherical Polar Coords. Problem: Focusing of gridpoints near r = 0 in Spherical Polar Coords + CFL condition 200 2,000x smaller timestep than Cartesian coords, despite being 100 1,000x more memory efficient Solutions: Magnify r coord near r = 0 ~10x timestep! Baumgarte et al., Phys. Rev. D 87, 044026 (2012) arxiv:1211.6632
Improved Spherical Polar Coords. Problem: Focusing of gridpoints near r = 0 in Spherical Polar Coords + CFL condition 200 2,000x smaller timestep than Cartesian coords, despite being 100 1,000x more memory efficient Solutions: Baumgarte et al., Phys. Rev. D 87, 044026 (2012) arxiv:1211.6632 Magnify r coord near r = 0 ~10x timestep! Skip over angular points closest to r = 0 get another ~10x
Improved Spherical Polar Coords. Problem: Focusing of gridpoints near r = 0 in Spherical Polar Coords + CFL condition 200 2,000x smaller timestep than Cartesian coords, despite being 100 1,000x more memory efficient Solutions: Baumgarte et al., Phys. Rev. D 87, 044026 (2012) arxiv:1211.6632 Magnify r coord near r = 0 ~10x timestep! Skip over angular points closest to r = 0 get another ~10x Exploit memory efficiency move to GPU Get another ~100x speed-up! Per-GPU ~5 50x faster than Cartesian AMR
Improved Spherical Polar Coords. Problem: Focusing of gridpoints near r = 0 in Spherical Polar Coords + CFL condition 200 2,000x smaller timestep than Cartesian coords, despite being 100 1,000x more memory efficient Solutions: Baumgarte et al., Phys. Rev. D 87, 044026 (2012) arxiv:1211.6632 Magnify r coord near r = 0 ~10x timestep! Skip over angular points closest to r = 0 get another ~10x Exploit memory efficiency move to GPU Get another ~100x speed-up! Get gamers involved 1,000x speed-up
Improved Spherical Polar Coords. Problem: Focusing of gridpoints near r = 0 in Spherical Polar Coords + CFL condition 200 2,000x smaller timestep than Cartesian coords, despite being 100 1,000x more memory efficient Solutions: Baumgarte et al., Phys. Rev. D 87, 044026 (2012) arxiv:1211.6632 Magnify r coord near r = 0 ~10x timestep! Skip over angular points closest to r = 0 get another ~10x Exploit memory efficiency move to GPU Get another ~100x speed-up! Get gamers involved 1,000x speed-up Per-GPU ~5 50x faster than Cartesian AMR With gamers, 5,000 50,000x faster GW throughput
Current Literature Re: NR in Spherical Polar Coords Movie courtesy T. Baumgarte
Our Basic Strategy Numerical Relativity: strongly hyperbolic formalisms of GR Step 1: Solve scalar wave eq. in given coord system Demonstrate stability & convergence Step 2: Implement in new, easily-extensible NR (BSSN) code New coordinate systems Dynamical, co-rotating spherical polar coordinates Ideal for CCSNe; logarithmic radial coord beyond r = 0 Bispherical-like coords + co-rotation & linear rescaling Ideal for DNS + BHB simulations
Our Basic Strategy Numerical Relativity: strongly hyperbolic formalisms of GR Step 1: Solve scalar wave eq. in given coord system Demonstrate stability & convergence Step 2: Implement in new, easily-extensible NR (BSSN) code New coordinate systems Dynamical, co-rotating spherical polar coordinates Ideal for CCSNe; logarithmic radial coord beyond r = 0 Bispherical-like coords + co-rotation & linear rescaling Ideal for DNS + BHB simulations
Scalar Wave Evolutions in Rotating, Logarithmic-Radius Spherical Polar Coords
Scalar Wave Evolutions in Rotating, Logarithmic-Radius Spherical Polar Coords Exponential convergence with increased FD order
Our Basic Strategy Numerical Relativity: strongly hyperbolic formalisms of GR Step 1: Solve scalar wave eq. in given coord system Demonstrate stability & convergence Step 2: Implement in new, easily-extensible NR (BSSN) code New coordinate systems Dynamical, co-rotating spherical polar coordinates Ideal for CCSNe; logarithmic radial coord beyond r = 0 Bispherical-like coords + co-rotation & linear rescaling Ideal for DNS + BHB simulations
Scalar Wave Evolutions in Bispherical-Like Coordinates Optimal coordinate system for BHBs & DNSs: Spherical-polar coordinates near BHs/NSs Uniform spherical polar coords far from the binary system.
Scalar Wave Evolutions in Bispherical-Like Coordinates Optimal coordinate system for BHBs & DNSs: Spherical-polar coordinates near BHs/NSs Uniform spherical polar coords far from the binary system. TwoPunctures coords = great near BHs/NSs, but radial grid far from the binary compactified GWs under-resolved Uncompactify radial coords problem solved! Original TwoPunctures coordinates:
Scalar Wave Evolutions in Bispherical-Like Coordinates Exponential convergence with increased FD order
Movie: Bispherical-like Coordinate Improvement & Dynamics
Numerical Relativity (SENR) Code Extensive automatic code generation: Extensive automatic code generation: Arbitrary-order finite differencing Simple Mathematica interface for generating NR (BSSN) code in arbitrary coordinates: Simple Mathematica interface for generating NR (BSSN)
Numerical Relativity (SENR) Code Validated against industry-standard Kranc code for Cartesian coordinates (agrees( to roundoff error). Validated against industry-standard Kranc code for Validation against Baumgarte's spherical polar code: in progress. Validation against Baumgarte's spherical polar code: Extension to more sophisticated coords planned
Numerical Relativity (SENR) Code Validated against industry-standard Kranc code for Cartesian coordinates (agrees( to roundoff error). Validated against industry-standard Kranc code for Validation against Baumgarte's spherical polar code: in progress. Validation against Baumgarte's spherical polar code: Extension to more sophisticated coords planned Bottom Line: Rapid progress being made toward goals Rapid progress being made toward goals Codes maximize user-friendliness, rapid- prototyping of ideas, and extensibility Codes maximize user-friendliness, rapid- Join the open development! http://tinyurl.com/senrcode Join the open development!