Using AmgX to accelerate a PETSc-based immersed-boundary method code

Transcription

1 29th International Conference on Parallel Computational Fluid Dynamics May 15-17, 2017; Glasgow, Scotland Using AmgX to accelerate a PETSc-based immersed-boundary method code Olivier Mesnard, Pi-Yueh Chuang, & Lorena A. Barba Mechanical and Aerospace Engineering, The George Washington University, United-States 1

2 PetIBM 2D & 3D incompressible Navier-Stokes equations Projection method: a block-lu decomposition (Perot, 1993) PETSc (Portable Extensible Toolkit for Scientific Computation) Immersed-boundary method + u ru = rp + 1 Re r2 u + R s r u = 0 >: u ( (s, t)) = R u (x) (x )dx x f ( (s, t)) ( x)ds 2

3 Immersed-boundary projection method (IBPM) Taira and Colonius (2007) Pressure and boundary forces gathered together Modified-Poisson system Q T B N Q 2 A G E T 4G 0 0 E q n+1 f 1 A = 0 1 r 0 A + u n+1 B 1 bc 1 bc 2 A 0 with Q G, E T f Block-LU decomposition: apple A 0 Q T Q T B N Q apple I B N Q 0 I q n+1 = r1 r 2 Aq = r 1 Q T B N Q = Q T q r 2 q n+1 = q B N Q (velocity system) (modified-poisson system) (projection step) 3

4 Application to 2D gliding snake Figures from Krishnan, A., Socha, J. J., Vlachos, P. P., & Barba, L. A. (2014). Lift and wakes of flying snakes. Physics of Fluids, 26(3), Enhanced lift at Re 2000 at AoA 35 deg AoA matches previous experimental data Can we observe lift-enhancement with more realistic 3D simulations? 4

5 Modified-Poisson system expensive to solve Conjugate-gradient Algebraic multigrid preconditioner (aggregation) 1 CPU node (16 cores - Dual 8-Core 2.6GHz Intel Xeon E5-2670) 80 time units (200,000 time steps) It takes about 90% of the simulation runtime! Q T B N Q 5

6 Nvidia AmgX library Various Krylov solvers Algebraic multigrid algorithms (classical and aggregation) Solve systems on multiple CUDA-capable GPU devices Available with a free license for non-commercial use for Accelerated Computing Developers Objective: use AmgX within PetIBM to reduce the time-to-solution of the Poisson system Problem: PETSc and AmgX have their own data structures 6

7 AmgXWrapper Interface between PETSc and AmgX Not specific to PetIBM 7

8 AmgXWrapper - Poisson system 8

9 PetIBM + AmgXWrapper Benchmark: 2D snake (Re=2000, AoA=35deg) 2.9M mesh-grid 1 CPU node: 12 CPU cores (2 Intel E5-2620) 1 GPU node: 1 CPU node (i.e., 12 CPU cores) + 2 K20 GPUs workstation: 6 CPU cores (1 Intel i7-5930k) + 2 K40c GPUs

10 Decoupled Immersed-boundary projection method Li and co-workers (2016) Decouple pressure field from Lagrangian forces 2-step block-lu decomposition 2 A G 3 E T 4G E 0 0 q n+1 f 1 A = 0 1 r 0 A + u n+1 B 1 bc 1 bc 2 A 0 appleā Ē T Ē 0 n+1 f = r1 r 2 with Ā appleā Ē T Ē 0 ; n+1 q n+1 First block-lu decomposition: appleā 0 Ē ĒĀ 1 Ē T apple A 0 G T G T A 1 G apple I Ā 1 Ē T 0 I apple I A 1 G 0 I n+1 Second block-lu decomposition: f q = = r1 r1 r 2 r 2 Aq = r 1 G T A 1 G = G T q + bc 2 q = q A 1 G EA 1 E T f = Eq q n+1 = q A 1 E T f u n+1 B 10

11 IBPM vs. decoupled method Benchmark: 2D snake (Re=2000, AoA=30deg) 2.9M meshgrid 200,000 time steps (80 time units) Cl Cd 11

12 IBPM vs. decoupled method Benchmark: 2D snake 2.9M meshgrid 200,000 time steps (80 time units) 1 CPU node (16 CPU cores) 1 GPU node (12 CPU cores & 2 K20 devices) 12

13 Flying snakes to the cloud CPU node on Colonial One Dual 8-Core 2.6GHz Intel Xeon E CPUs InfiniBand Microsoft Azure A9 Dual 8-Core 2.6GHz Intel Xeon E CPUs InfiniBand Ohio State University micro-benchmarks: 13

14 Flying snakes to the cloud Benchmark: Poisson system with 46M unknowns, Hypre BoomerAMG classical preconditioner PETSc CG 16 CPU cores per node 14

15 Flying snakes to the cloud Instance cores RAM disk sizes price A8 8 56GB 382GB $0.975/hr A-series A GB 382GB $1.95/hr A GB 382GB $0.78/hr A GB 382GB $1.56/hr Instance cores RAM disk sizes GPU price NC6 6 56GB 340GB 1 x K80 $0.90/hr NC-series NC GB 680GB 2 x K80 $1.80/hr NC GB 1,440GB 4 x K80 $3.60/hr NC24r GB 1,440GB 4 x K80 $3.96/hr 15

16 Flying snakes to the cloud Benchmark: 2D snake (Re=2000, AoA=35deg) 2.9M meshgrid 10,000 time steps $12.5 $

17 Flying snakes to the cloud Benchmark: 3D snake (Re=2000, AoA=deg) 46M meshgrid 1,000 time steps $15.5 $7.9 17

18 Conclusions Use AmgX in a PETSc-based code AmgXWrapper ( Fast decoupled immersed-boundary projection method PetIBM + AmgX to reduce cloud computing expenses Microsoft Azure Sponsorship ( 18