A Scalable, Parallel Implementation of Weighted, Non-Linear Compact Schemes

Transcription

1 A Scalable, Parallel Implementation of Weighted, Non-Linear Compact Schemes Debojyoti Ghosh Emil M. Constantinescu Jed Brown Mathematics Computer Science Argonne National Laboratory SIAM Annual Meeting Chicago, Illinois, July 7 11, 2014

2 Motivation Numerical Solution of Compressible Turbulent Flows Atmospheric flows, Aircraft and Rotorcraft wake flows Characterized by large range of length scales Convection and interaction of eddies Compressibility à Shock waves Shocklets Thin shear layers à High gradients in flow High order accurate finite-difference solver Shock-Turbulence Interaction (Stanford University) High spectral resolution for accurate capturing of smaller length scales Non-oscillatory solution across shock waves and shear layers Low dissipation errors for preservation of flow structures over large distances Shock-Turbulent B o u n d a r y L a y e r Interaction (Brandon Morgan, Stanford University and Nagi Mansour, NASA) Rising Thermal Bubble in Hydrostatically Balanced Atmosphere 2

3 Background Cell interfaces Cell centers 0 j j+1 Δx N j-1 j-1/2 j+1/2 Conservative finite-difference discretization of a Hyperbolic Conservation Law u t + f (u) x = 0; f '(u) R du j dt + 1 $ Δx f (x,t) f (x,t) % j+1/2 j1/2 = 0 Weighted Essentially N o n - O s c i l l a t o r y (WENO) Schemes Compressible Turbulent Flows Compact Finite- Difference Schemes Hybrid Compact- WENO Schemes Disadvantages: loss of spectral resolution near discontinuities Weighted Compact Non-Linear Schemes Disadvantage: Relatively Poor spectral resolution Compact-Reconstruction W E N O ( C RW E N O ) Schemes Ghosh Baeder, SIAM J. Sci. Comput.,

4 Compact-Reconstruction WENO (CRWENO) Schemes 1/2 3/2 j-1/2 j+1/2 j+3/2 N-1/2 N+1/2 0 1 j-1 j j+1 N-1 N N+1 General form of a conservative compact scheme: ( ) = B f jn,, f j,, f j+n A ˆf j+1/2m,, ˆf j+1/2,, ˆf j+1/2+m ( ) A [ ] ˆf = B [ ]f At each interface, r possible (r)-th order compact interpolations, combined using optimal weights c k to yield (2r-1)-th order compact interpolation scheme: r ( ) r c k A k ˆf j+1/2m,, ˆf r j+1/2+m = c k B k k=1 r k=1 ( f jn,, f j+n ) A 2r1 ( ˆf j+1/2m,, ˆf j+1/2+m ) = B 2r1 f jn,, f j+n ( ) Apply WENO algorithm on the optimal weights c k scale them according to local smoothness r r ω α k = k A k ˆf j+1/2m,, ˆf r r j+1/2+m = ω k B k ( f jn,, f j+n ) k=1 ( ) k=1 c k β k +ε ( ) ; ω p k = α k / α k k 4

5 5 th Order CRWENO Scheme (CRWENO5) j j+1/2 2 3 f j1/ f j+1/2 = 1 6 f j f j 1 3 f j1/ f j+1/2 = 5 6 f j f j f j+1/ f j+3/2 = 1 6 f j f j+1 c 1 = 2 10 c 2 = 5 10 c 3 = f + 6 j1/2 10 f + 1 j+1/2 10 f = 1 j+3/2 30 f + 19 j1 30 f + 10 j 30 f j+1 5

6 5 th Order CRWENO Scheme (CRWENO5) j j+1/2! " 2 3 ω ω f j1/ f j+1/2 = 1 6 f j f j 1 3 f j1/ f j+1/2 = 5 6 f j f j f j+1/ f j+3/2 = 1 6 f j f j+1 c 1 = 2 10 c 2 = 5 10 c 3 = 3 10 ω 1 ω 2 ω 3 $ % f +! 1 j1/2 3 ω (ω +ω ) $ " 2 f 3 j+1/2 + 1 % 3 ω 3 f j+3/2 = ω 1 6 f j1 + 5(ω +ω ) f j + ω 2 + 5ω 3 6 f j+1 6

7 Numerical Analysis Taylor Series Fourier analysis WENO5 1 6 f 60 x 6 j Δx 5 CRWENO5 1 6 f 600 x 6 j Δx 5 è WENO5 requires ~ 1.5 times more grid points per dimension to yield a solution of comparable accuracy as the CRWENO5 scheme. è Time step size limit is ~ 1.6 times smaller for CRWENO than WENO5 7

8 Preliminary Results doi: / doi: /s doi: / Periodic density wave advection Shu-Osher Problem Flow around Harrington Rotor WENO5 Lower absolute errors for smooth problems for the same order of convergence Resolution of small length scales improved Better preservation of f l o w f e a t u r e s ( s h e d vortices) CRWENO5 Vortex shedding from a flapping wing WENO5 CRWENO5 8

9 Scalable Parallel Implementation CRWENO5 needs a tridiagonal solution at each time-integration stage/step Treat sub-domain boundary as physical boundary Decouple system of equations across processors (biased compact or non-compact schemes at MPI boundaries) Drawback: Numerical properties of the scheme function of number of processors Good for small number of processors, numerical errors grow or spectral resolution falls as number of processors increase for same problem size Data Transposition Transpose pencils of data such that entire system of equations is collected on one processor. Huge communication cost Parallel Implementation of the Thomas Algorithm Pipelined Thomas Algorithm (PTA) (Povitsky Morris, JCP, 2000): Used a complicated static schedule to use idle times of processors to carry out computations Trade-off between computation communication efficiencies Parallel Diagonally Dominant (PDD) (Sun Moitra, NASA Tech. Rep., 1996): Solve a perturbed linear system that introduces an error due to assumption of diagonal dominance Other implementations of tridiagonal solvers not applied to compact schemes Increased mathematical complexity compared to the serial Thomas algorithm Existing approaches do not scale well! 9

10 Parallel Tridiagonal Solver ! b 0 c $! 0 a 1 b 1 c 1 a 2 b 2 c 2 Processor 0 a 3 b 3 c 3 b 5 c 5 a 5 a 6 b 6 c 6 Processor 1 a 7 b 7 c 7 b 9 c 9 a 9 a 10 b 10 c 10 Processor 2 a 11 b 11 c 11 b 13 c 13 a 13 a 14 b 14 c 14 Processor 3 a 15 b 15 c 15 b 17 c 17 a 17 a 18 b 18 c 18 Processor 4 a 19 b 19 c 19 a 20 b 20 a 4 c 4 b 4 a 8 c 8 b 8 a 12 c 12 b 12 " a 16 c 16 b 16 %" Processor 0 Processor 1 Processor 2 Processor 3 Processor 4 x 0 x 1 x 2 x 3 x 5 x 6 x 7 x 9 x 10 x 11 x 13 x 14 x 15 x 17 x 18 x 19 x 20 x 4 x 8 x 12 x 16 $! = % " Solution of reduced system is critical to scalability One row on each processor à Communication intensive r 0 r 1 r 2 r 3 r 5 r 6 r 7 r 9 r 10 r 11 r 13 r 14 r 15 r 17 r 18 r 19 r 20 r 4 r 8 r 12 r 16 $ %! " b 0 c 0 ˆb 1 c 1 ˆb 2 c 2 ˆb 3 c 3 Processor 3 Processor 4 b 5 c 5 a 5 ˆb 6 c 6 â 6 ˆb 7 â 7 c 7 Reduced Tridiagonal System Direct/Exact solutions to the reduced system (Cyclic Reduction / Recursive-Doubling Algorithm, Gather-and-Solve) - Do not scale well! Processor 0 Processor 1 b 9 c 9 a 9 ˆb 10 c 10 â 10 Processor 2 ˆb 11 â 11 c 11 $!a! 12 b12!c 12!a! 16 b16 % b 13 c 13 a 13 ˆb 14 c 14 â 14 ˆb 15 â 15 c 15 b 17 c 17 a 17 ˆb 18 c 18 â 18 Iterative Subs t r u c t u r i n g Method ˆb 19 c 19 â 19 ˆb 20 â 20!b 4!c 4!a 8! b8!c 8 10

11 Iterative Solution of the Reduced System Reduced system represents the coupling between the first interface of every subdomain through an approximation to a hyperbolic flux For large subdomain sizes, very diagonally dominant Diagonal dominance decreases as sub-domain size grows smaller (Number of processors increase for same problem size) N=65,536, Number of processors: Non-machine-zero elements of an arbitrarily chosen column of the inverse of the reduced system (N=1024) Number of Jacobi iterations required for an exact solution increases as subdomain size grows smaller. à Cost of the tridiagonal solver increases 11

12 Performance Analysis Governing Equations: Inviscid Euler Equations Platform: ALCF Vesta (IBM BG/Q) CRWENO5 scheme on N points per dimension WENO5 scheme on fn points per dimension On a single processor, CRWENO5 is f > 1 (f ~ 1.5 for smooth more efficient (error vs. wall time). solutions) WENO5 yields Cost of the tridiagonal solver increases as number of processors increases solutions of comparable accuracy/resolution with f times more points At a critical sub-domain size, the CRWENO5 becomes less efficient than the WENO5 scheme. Non-compact scheme, so almost ideal scalability expected. Same number of processors for the CRWENO5 scheme (on N points) and the WENO5 scheme (on fn points) Given p processors, is it faster to obtain a solution of given accuracy/ resolution with the WENO5 or CRWENO5 scheme? 12

13 Performance Comparison for a Smooth Problem Periodic Advection of a Sinusoidal Density Wave 1D WENO5 yields solutions with comparable accuracy with ~ 1.5 times as many points 2D WENO5 yields solutions with comparable accuracy with ~1.5 2 = 2.25 times as many points Time step size Δt is taken the same for the CRWENO5 scheme on N points and the WENO5 scheme on fn points because of the linear stability limit. Solutions are obtained after one cycle over the periodic domain with the SSP-RK3 scheme It is verified that the errors are exactly the same for the various number of processors considered à There are no parallelization errors (Number of Jacobi iterations are specified a priori to ensure this) Effect of Dimensionality The factor by which the grid needs for WENO5 needs to be refined to yield comparable solutions as CRWENO5 is f D (f times per dimension) Several tridiagonal systems are solved for multi-dimensional problems à Higher arithmetic density à Cost increases sub-linearly with number of systems 13

14 Performance Analysis for a Smooth Problem 1D Cases: Number of grid points and number of processors considered 64 (96) (192) (384) D Cases: Number of grid points and number of processors considered 64 2 (96 2 ) (192 2 ) (384 2 ) Note: Critical sub-domain size is insensitive to global problem size 14

15 Scalability Results for Benchmark Flow Problems Isentropic Vortex Convection Vortex convects 1000x its core radius Density Verified that WENO5 yields a solution of comparable accuracy on a grid with ~ x (~ 3.4x) more grid points ALCF/Mira (IBM BG/Q) (~ 8k to 500k cores) 8192x64x64 CRWENO5 8192x64x64 WENO x96x96 Strong Scaling: At very small subdomain sizes, CRWENO5 does not scale as well, yet is more efficient / has lower absolute walltime Weak Scaling: CRWENO5 shows excellent weak scaling 64 (4 3 ) points per processor 15

16 Conclusions Future Work Parallel Implementation on Distributed Memory Platforms Based on an iterative sub-structuring approach to the tridiagonal system of equations No parallelization-induced errors (however, need a priori estimate on the number of Jacobi iterations for the reduced system) Avoids collective communication à Excellent weak scaling Good strong scaling compared to a non-compact scheme; at very small subdomain sizes, retains higher parallel efficiency despite relatively poorer scaling Future Work (Practical and Interesting Applications) DNS of shock-turbulence interaction and shock-turbulent boundary layer interaction Apply the CRWENO5 scheme to benchmark atmospheric flow problems and compare solution and scalability with spectral element methods (popular in that community) Implement this implementation of the CRWENO5 scheme in an existing, validated Navier-Stokes solver and apply to practical flow problems 16

17 Thank you! Acknowledgements U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research Argonne Leadership Computing Facility Dr. Paul Fischer (Argonne National Laboratory) 17