Lenstool-HPC. From scratch to supercomputers: building a large-scale strong lensing computational software bottom-up. HPC Advisory Council, April 2018

Transcription

1 LenstoolHPC From scratch to supercomputers: building a largescale strong lensing computational software bottomup HPC Advisory Council, April 2018 Christoph Schäfer and Markus Rexroth (LASTRO) Gilles Fourestey (SCITAS)

2 Gravitational lensing Einstein ring (credit: Nasa/Hubble)

3 Gravitational lensing Einstein ring (credit: Nasa/Hubble)

4 Gravitational lensing Light refraction caused by a distribution of matter according to Albert Einstein's general theory of relativity (1916) Article about star GR in 1936 (A. Einstein, Science) Fritz Zwicky posited in 1937 that the effect could allow galaxy clusters to act like lenses First observed in 1979 "Twin QSO" SBS Twin QSO (center), Credit: ESA/Hubble & NASA

5 Gravitational lensing Optical artefacts created by dense mass distributions Galaxies Dark matter Black holes Parametric Lens Model the ellipticity of the projected mass distribution ω the finite core radius 0 the normalized surface mass density (x,y) the lens position... Reverse engineer the lenses: Recompose faraway objects by computing the lenses mass Typical search space dimension: >1010 Using the vanilla version of lenstool requires months to find the optimal solution!

6 LenstoolHPC Motivations LenstoolHPC was developed: based on Lenstool (Pr. Kneib et al., from 1996 onwards), In 6 manmonth FTE, By two field scientists and one application expert, bottomup from scratch. No separation of concern: Field scientists define algorithmic constrains at every step Computer scientists provide the most optimized implementation on specific hardware extreme(ish) programming Performance is scaled bottomup: Focus on algorithms/kernels and data structures Performance scaling from core to full machine

7 Formalism Source s position on the source plane: 2D lensing potential: Example of gradient (SIS): The lens equation:

8 Strong Lensing Algorithm Step 0 Finite core radius Given a parametric model for all the lens types: normalized surface mass density Step 0: Compute all the gradients (~90% of TTS) DLP for each pixel of the image Mapping algorithms to the hardware: High performance data structures (SOA) Implicit and Explicit (handcoded) vectorization Ellipticity of the projected mass distribution Position

9 Strong Lensing Algorithm Step 1 Given a parametric model for all the lens types Step 0: Compute all the gradients unlensing Step 1a: unlensing (linear transformation) TLP lensing the green dots (images) to the Source plane (yellow dot) Compute the barycenter of the yellow dots Step 1b: relensing (nonlinear transformation) TLP Decompose the Image plane into triangles Lense the triangles to the Source plane If the lensed triangle includes the barycenter, a predicted image is found (red triangles in Image plane) relensing

10 Strong Lensing Algorithm Step 2 & 3 Given a parametric model for all the lens types Step 0: Compute all the gradients Step 1a: unlensing (linear transformation) lensing the green dots (images) to the Source plane (yellow dot) Compute the barycenter of the yellow dots Step 1b: relensing (nonlinear transformation) Decompose the Image plane into triangles Lense the triangles to the Source plane If the lensed triangle includes the barycenter, a predicted image is found (red triangles in Image plane) Step 2: (MPI) Compute Step 3: Pass the Chi2 to a Bayesian MCMC code (MPI) Restart with new set of parameters until close to reality

11 Performance scaling Strong Lensing Algorithm

12 Gradient Benchmark Results (Step 0) Gradient benchmark computation: 5000x5000 pixels image, 69 sources, 203 constraints AVX2* Code AVX512F* TTS Factor TTS Factor Lenstool s 1X 4.8s 1X LenstoolHPC AOS 0.8s 1.3X 5.6s 0.9X LenstoolHPC SOA 0.5s 2.0X 3.3s 1.4X LenstoolHPC SOA + DLP 0.2s 4.5X 0.4s 11.4X Performance on Broadwell: IACA: ~ 6 Flops/cycle Intel Advisor: ~25% of peak *AVX2: Broadwell Intel Xeon CPU E GHz, intel compilers 17 *AVX512F: Intel Xeon Phi CPU 1.30GHz, intel compilers 17

13 Distributed Grid Gradient Grid Gradient computation distribution (step 1): Images split into regular subdomains with MPI Subdomains are handled using OpenMP/CUDA

14 Grid Gradient Benchmark (Step 1) AVX2 Grid Gradient benchmark (TTS, in s) AVX v4 2695v4 (PizDaint) SKL Plat HT KNL (greina) P100 (greina) V100 lenstool (TLP) NA NA lenstoolhpc (SOA + TLP) NA NA lenstoolhpc (SOA + TLP + DLP/SIMT) Single node Grid Gradient benchmark computation: 6000x6000 pixels image, 69 sources, 203 constraints. SIMT TLP is giving the best bang for your bucks SOA alone gives a nice boost (and is mandatory for efficient DLP) DLP is getting better with wider vector sizes (avx512 is ~2x avx2). V100 is much faster than P100

15 Chi2 computation The Chi2 is computed by computing the distance between the original images and their computed unlensed/relensed projections from steps 1a and 1b The blue dots correspond to the same image in the source plane Each distance for the same source (in blue) are reduced to Rank 0 using MPI_Pack The Chi2 is computed on Rank 0

16 DaintGPU: Chi2 (Step 2) Strong Scaling Num. nodes Grid Gradient Comp Quadrant unlensing MPI reduction TTS Scalability of the Chi2 benchmark using a 8k x 8k image, 69 sources, 203 constraints on Piz Daint multicore, 1 MPI process and 18 threads per socket, in seconds

17 DaintMC: Chi2 (Step 2) Strong Scaling Num. nodes Grid Gradient Comp. Quadrant unlensing MPI reduction TTS Scalability of the Chi2 benchmark using a 8k x 8k image, 69 sources, 203 constraints on Piz Daint multicore, 1 MPI process and 18 threads per socket, in seconds This represents a 50X compared to Lenstool in 6 months FTE

18 Current Status and Next Steps Development: Code on c4science, with unit tests for each kernels (lensing, unlensing, Chi2 ) Large development project on CSCS Piz Daint Aries network tuning GPU tuning: lensing, unlensing and chi computation are (very) regular Development a parallel MCMC framework, could lead to a 500X speedup, e.g. Pi4u: (P. E. Hadjidoukas et al., ETHZ) Papers: High Performance Computing for gravitational lens modeling: single vs double precision on GPUs and CPUs Markus Rexroth, Christoph Schafer, Gilles Fourestey, JeanPaul Kneib To be submitted High Performance Strong Lensing Map Generation for Lenstool Christoph Schafer, Gilles Fourestey, JeanPaul Kneib In Preparation

19 Lensing Map Generation Maps based on second derivative of lensing potentials (Mass, Amplification, Shear) Used for calculation of statistical errors of the MCMC method Sampling of parameter space Compute average and standard deviation for every pixel Added to best prediction, gives asymmetric error bars Fast Map generation crucial Actual process takes months Grid Gradient 2 benchmark TTS, in s lenstool lenstoolhpc 1.3 Speedup Single node Grid Gradient benchmark computation: 4200x4200 pixels image, 201 individual lenses. x567 Lenstool: Intel(R) Xeon(R) CPU E GHz Lenstool HPC: P100

20 Brownie points Thanks to Pr. JeanPaul Kneib (LASTRO, EPFL), Pr. Jan Hesthaven and Dr. Vittoria Rezzonico (SCITAS, EPFL) Thanks to Colin McMurtrie and Hussein ElHarake from CSCS for their support using the CSCS test cluster

21 Questions?