High Performance Multigrid Software

Transcription

1 High Performance Multigrid Software Ulrike Meier Yang This work was performed under the auspices of the U.S. Department of Energy by under Contract DE-AC52-07NA Lawrence Livermore National Security, LLC

2 Time to Solution Motivation: Scalable Solvers Diag-CG Multigrid-CG scalable Number of Processors (Problem Size) 10 5 Multigrid solvers are essential components of LLNL simulation Multigrid solvers are optimal (O(N) operations) Scalable faster simulations better science! Concerns about scalability on future architectures: - higher communication cost - increased parallelism 2

3 Outline Hypre newest developments: Brief overview Efforts to reduce communication New elasticity interpolation Future plans XBraid: Overview Compressible Navier-Stokes application 3

4 The hypre Team Rob Falgout Tzanio Kolev Jacob Schroder Ulrike Yang Former Allison Baker Chuck Baldwin Guillermo Castilla Edmond Chow Andy Cleary Noah Elliott Van Henson Ellen Hill David Hysom Jim Jones Mike Lambert Barry Lee Jeff Painter Charles Tong Tom Treadway Panayot Vassilevski Deborah Walker 4

5 The best solver for a given application usually takes advantage of the setting structured grids, constant coefficients, FE discretization, etc. Traditional linear solver libraries take in only generic matrix-vector information Linear System Interfaces Linear Solvers PFMG,... FAC,... Split,... MLI,... AMG,... Data Layouts structured composite block-struc unstruc CSR 5

6 Current solver / preconditioner availability via hypre s linear system interfaces Data Layouts Structured Semi-structured Sparse matrix Matrix free System Interfaces Solvers Struct SStruct FEI IJ Jacobi P P SMG P P PFMG P P Split P SysPFMG P FAC P Maxwell P AMS, ADS P P P BoomerAMG P P P MLI P P P ParaSails P P P Euclid P P P PILUT P P P PCG P P P P GMRES P P P P BiCGSTAB P P P P Hybrid P P P P 6

7 New features in next hypre release (hypre b expected soon) New approaches in AMG with reduced communication Non-Galerkin AMG RD Falgout, JB Schroder, Non-Galerkin coarse grids for algebraic multigrid, SIAM Journal on Scientific Computing 36 (3), C309-C334, (Mult)-Additive AMG PS Vassilevski, UM Yang, Reducing communication in algebraic multigrid using additive variants, Numerical Linear Algebra with Applications 21 (2), , Interpolation for elasticity AH Baker, TV Kolev, UM Yang, Improving algebraic multigrid interpolation operators for linear elasticity problems, Numerical Linear Algebra with Applications 17 (2 3), ,

8 Algebraic Multigrid (AMG) iterative method for solving Ax=b commonly used as a preconditioner Setup phase: Solve phase: Select coarse grids smooth Define interpolation P (m),m 1,... Define restriction (m) (m)t R P Finest Grid restrict interpolate Define coarse-grid operators: (m 1) (m)t A P A (m) P (m) First Coarse Grid 8

9 AMG Communication patterns, 128 cores Performance degradation caused by increased communication complexity on coarser grids! 9

10 Goal: Replace the standard Galerkin coarse grid matrix, (m 1) (m)t A P A with a sparser approximation. Non-Galerkin AMG (m) P (m) Non-Galerkin coarse grid Let A g = P T AP Sparsify A g to yield A c for coarse grid Goal: less expensive method, especially in parallel Desire good spectral equivalence between A g and A c Heuristic targets: I A c A g 1 2 θ Can show this implies AMG convergence 10

11 Step 1: Choose appropriate sparsity pattern for coarse grid matrix. Algorithm Outline Leverage fine grid matrix graph to reproduce stencil patterns on coarse grid. Row-wise drop tolerance parameter controls information eliminated from each row. Remove symmetric edge Collapse Step 2: Form A (m+1) = R (m) A (m) P (m) and eliminate unwanted matrix entries through stencil collapsing approach. Preserves important near nullspace modes and spectral equivalence between the Galerkin and non- Galerkin operators. 11

12 Results: 3D Diffusion Timings on Vulcan IBM BG/Q, scaled up to 131,072 cores AMG convergence largely unchanged Comparison to best practices Galerkin AMG 12

13 Additive AMG Originally invented in the 80 s to increase parallelism in multigrid (Greenbaum, 1986, BPX, Bastian, Hackbusch, Wittum, 1998, ) Generally leads to increased number of iterations But: shows potential for reduced communication 13

14 Perform in parallel smooth smooth smooth smooth smooth smooth smooth smooth smooth solve solve 14

15 Time Proc id Performance profile of AMG solve cycle for 64 MPI tasks on Hera computation idle time MPI calls Cannot take advantage of parallelism, but Most communication generated by coarse grid operators, not by interpolation or restriction too little computation compared to communication on coarse grid, prohibiting overlap 15

16 Time Proc id Performance profile of AMG solve cycle for 64 MPI tasks on Hera computation idle time MPI calls Cannot take advantage of parallelism, but Most communication generated by coarse grid operators, not by interpolation or restriction too little computation compared to communication on coarse grid, prohibiting overlap Combine communication smooth smooth smooth solve 16

17 x 0 = 0, r 0 = b For k = 0,, l 1 (seq) x k = M 1 k r k r k+1 = (P k k+1 ) T r k A k x k x l = M l 1 r l x l x l + M l T r l A l x l For k = l 1,, 0 (seq) x k x k + P k k+1 x k+1 x k x k + M T k r k A k x k x 0 = 0, r 0 = b For k = 0,, l 1 (seq) k r k+1 = ( P k+1 ) T r k For k = 0,, l (parallel) x k = M 1 k r k x k x k + M T k r k A k x k simplified For k = l 1,, 0 (seq) k x k x k + P k+1 x k+1 k Both algorithms are equivalent for P k+1 = (I M T k A k )P k k+1. k Additive AMG: P k+1 k = P k+1 17

18 Multiplicative V-cycle Weighted Jacobi Additive V-cycle Weighted Jacobi Mult-Additive V-cycle with interpolation truncated to at most 8 elements per row Weighted Jacobi 18

19 seconds Axis Title solve times - 7pt solve times - 27pt mult-gs mult-l1j 9 mult-gs mult-l1j map8-gal 8 map8-gal 5 sp8-gal 7 sp8-gal no of cores no of cores 19

20 seconds Axis Title solve times - 7pt solve times - 27pt 7 10 mult-gs mult-gs mult-l1j map8-gal sp8-gal mult-ng-gs mult-ng-l1j map8-ng sp8-ng mult-l1j map8-gal sp8-gal mult-ng-gs mult-ng-l1j map8-ng sp8-ng no of cores no of cores 20

21 Definitions point or node: physical point of the grid unknown: function being approximated (e.g. component of displacement) Approaches Unknown-based: coarsen/interpolate only between variables of the same function (unknown) nodal-based: coarsen/interpolate in a nodal or pointwise fashion hybrid approach: nodal coarsening/ unknown interpolation 21

22 For effective AMG methods the smooth error vectors should be in the range of the interpolation operator P (near-null-space should be preserved on all coarse levels) For systems of PDEs the nullspace can contain more than just constant vectors, such as the rigid body modes (RBMs) in linear elasticity Idea: P = initial AMG interpolation (any) Augment P: P = P Q s.t. s F = P sc 1 with F s Q ij = P i ij and s c coarse grid restriction of s F For linear elasticity i P ij s j c, 2D, 1 new dof at each node P = P u 0 Q u 0 P v Q v 3D, 3 new dofs at each node 22

23 seconds Linear elasticity problem on a two dimensional beam domain The beam is fixed on one side and a volume force is applied U H H-GM Here E = 210, ν = 0.3 Quadratic finite elements, ~ 51,000 dofs per process no of processes 23

24 New release b soon: will contain communication-reducing approaches linear elasticity interpolation (requires adding rigid body modes info) Various bug fixes and added threading Add rectangular matrix structure to (semi-) structured interface Investigate transition to heterogeneous architectures 24

25 Description of Method R. Falgout, S. Friedhoff, Tz. Kolev, S. MacLachlan, and J. Schroder, Parallel Time Integration with Multigrid, to appear in SIAM J. Sci. Comput. A Fluid Dynamics application R. Falgout, A. Katz, Tz. Kolev, J. Schroder, A. Wissink and U. M. Yang, Parallel Time Integration with Multigrid Reduction for a Compressible Fluid Dynamics Application, submitted to J. Comp. Phys Our Team Veselin Dobrev Rob Falgout Tzanio Kolev Anders Petersson Jacob Schroder Ulrike Yang 25

26 Consider a system of ODEs of the form u t = f t, u t, u 0 = g 0, t 0, T. Let t i = iδt, i = 0,, N, with δt = T/N A general one-step method is now given by u i = Ф i u i 1 + g i, i = 1,, N This can also be expressed as an O(N), sequential direct method A u = I Ф 1 I Ф N I u 0 u 1 u N = g 0 g 1 g N = g We propose solving this system iteratively with a multigrid method Extend multigrid reduction (MGR, 1979) to the time dimension Coarsens only in time (non-intrusive) O(N), highly parallel 37

27 T 0 T 1 t 0 t 1 t 2 t 3 T = m t Relaxation is highly parallel t Alternates between F-points and C-points F-point relaxation = propagation of C-point value across time interval Define coarse-grid system with N = N/m grid points A u = I Ф,1 I Ф,N t N I u,0 u,1 u,n = where Ф,i should be at least as cheap to apply as Ф i F-point (fine grid only) C-point (form coarse grid) F-relaxation g,0 g,1 g,n = g, 38 6

28 1. Apply FCF-relaxation to A u = g. T 0 T 1 T = m t t 0 t 1 t 2 t 3 t t N F-point (fine grid only) C-point (form coarse grid) 2. Restrict fine grid approximation and residual to coarse grid u,i u mi, r,i g mi A(u) mi, i = 0,, N. 3. Solve A v = A u + r. 4. Compute coarse grid error approximation e = v u. 5. Add the error to the values of u at the C-points: u mi u mi + e,i 6. Correct u by applying an F-relaxation step Apply procedure to coarse grid (3) for multilevel method 39

29 Our code (XBraid) is agnostic to spatial decomposition and only parallelizes in time Serial time stepping Multigrid-in-time t (time) t (time) x (space) Parallelize in space only Store only one time step x (space) Parallelize in space and time Store several time steps 40

30 Full approximation scheme (FAS) formulation for nonlinear problems Non-intrusive approach with unchanged time discretization User provides time integrator MGRIT is optimal for simple parabolic problems (implicit and explicit) In practice, store and solve one space-time slab at a time MGRIT with F-relaxation and two levels is equivalent to parareal Two levels still requires a significant sequential solve Multigrid perspective proved useful for achieving the additional parallelism of a full multilevel method without sacrificing optimality Parallel time integration is only useful beyond some scale There is a crossover point, but we have already observed speedups around 10x 41

31 User defines two objects: App and Vector User also writes several wrapper routines: Phi, Init, Clone, Free, Sum, Dot, Write, BufPack, BufUnpack Coarsen, Restrict (optional, for spatial coarsening) Phi(app, tstart, tstop, accuracy, u, &rfactor) Advances vector u from time tstart to tstop Return value rfactor specifies a requested temporal refinement factor Code stores only C-points to minimize storage Consider relaxation over a processor s portion of the time interval Each proc starts with right-most interval to overlap comm/comp 1) Post receive 2) Compute and send 42

32 Use the serial Strand2D code*** Solves compressible Navier-Stokes (nonlinear) Problem is unsteady vortex shedding over a cylinder Implicit time stepping and a spatial FAS multigrid cycle for implicit solves Our tests use backward Euler 3 rd -order finite-differencing on Strand Grids for efficiency and accuracy *** A. J. Katz and D. Work, High-order flux correction/finite difference schemes for strand grids, 52nd Aerospace Sciences Meeting, American Institute of Aeronautics and Astronautics, Jan

33 Plot velocity magnitude Solution snapshots exhibiting unsteady vortex shedding t = t = t = t =

34 Strand2D Code: approximately 13,500 lines Not counting library code like LAPACK We added 129 lines to Strand2D 20 lines to facilitate file output in parallel 109 lines to enable restarting the code at a new time for a new state vector With little outside help, this took about 3-4 weeks If restarting had already been enabled, this would have been much shorter XBraid wrapper code is about 475 lines Includes main(), command line parsing, etc... Important code is much shorter 45

35 Plot velocity magnitude 5120 th time step (t=2.56s) After 13 XBraid iterations, accuracy is good Iteration 1 Iteration 5 Iteration 9 Iteration 13 46

36 Fix time domain with t final = 2.56s, then refine in time Linux cluster: Intel Sandybridge, InfiniBand QDR interconnect XBraid: F-cycles, FCF relaxation, coarsen by 5, relative tol of 1e-5 Strand: 24,960 d.o.f. spatial mesh 1280 time step case barely resolves unsteady behavior Num steps nprocs XBraid iters Run Time XBraid Run time serial Speedup min 37 min min 163 min min 655 min

37 50

38 51

39 XBraid added to Strand2D with relative ease XBraid enabled to run sequential Strand2D code on up to 4096 cores Achieved speedup up to ~8 Further theoretical investigation Add more features to the code Apply XBraid to unsteady 3D CFD production code to increase parallelism 52

40 This work was performed under the auspices of the U.S. Department of Energy by under Contract DE-AC52-07NA Partial support for this work was provided through Scientific Discovery through Advanced Computing (SciDAC) program funded by U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research (and Basic Energy Sciences/Biological and Environmental Research/High Energy Physics/Fusion Energy Sciences/Nuclear Physics) and by Applied Mathematics Program, DOE ASCR. 53