S8241 VERSIONING GPU- ACCLERATED WRF TO Jeff Adie, 26 March, 2018 (Presented by Stan Posey, NVIDIA)

Size: px

Start display at page:

Download "S8241 VERSIONING GPU- ACCLERATED WRF TO Jeff Adie, 26 March, 2018 (Presented by Stan Posey, NVIDIA)"

Rudolph Wade
5 years ago
Views:

S8241 VERSIONING GPU- ACCLERATED WRF TO 3.7.

1 S8241 VERSIONING GPU- ACCLERATED WRF TO Jeff Adie, 26 March, 2018 (Presented by Stan Posey, NVIDIA) 1

2 ACKNOWLEDGEMENT The work presented here today would not have been possible without the efforts of NVIDIA applications engineers, particularly Carl Ponder and Alexey Romanenko. Their work on the original WRF GPU port provided groundwork for what follows 2

3 INTRODUCTION Motivation for this Work WRF is an important top 10 application in the HPC community As CPU performance plateaus, GPU accelerated computing becomes increasingly important for both overall scaling and performance/watt WRF is a challenging code to optimize, and a large effort was invested in the original OpenACC port of by NVIDIA However the release that followed, 3.7.1, included new features and performance improvements, and NVIDIA received several requests for this release This talk presents the challenges of updating a large community code with local optimizations, provides detailed results, and considers future work 3

4 WHAT IS WRF? Weather Research and Forecasting model Numerical Weather Prediction (NWP) for operational and research forecasting Open source, community code, mostly developed by NCAR, NOAA, and AFWA Original version released in 1999 Around 1 million lines of Fortran, using various revs from F77 to F2003 More than 50,000 users, including substantial number of operational weather centers Modular, with 2 dycores (only ARW in this work), and around 80 physics modules Add-on packages for forest fires, hydrology, atmospheric chemistry, data assimilation 4

5 WRF VERSIONING WRF generally has two releases each year: Major (feature) version release in April Minor (bugfix) version release in August Apr 18 Aug 14 Apr 17 Aug 14 Apr 8 Aug 12 Apr 17 Aug

6 GPU ACCELERATED WRF Previous Efforts Several early efforts targeting specific (physics) modules 2008: Michalakes & Vachhrajani s early efforts with WSM5 (8x speedup) 2009: Linford, et al accelerated KPP part of WRF-Chem) (5.5x speedup) 2011: Mielikainen, et al accelerated the RRTM module 2012: Mielikainen, et al improved WSM5 (200x speedup) 2013: SSEC showed speedups for multiple physics GTC2013 Accelerated versions of RRTMG for both SW and LW radiation included in WRF : Michalakes, Iacono/AER 6

GPU ACCELERATED WRF Current Efforts TempoQuest www.tempoquest.com WRF 3.8.

2x speedup 4nodes/8cpus/16gpus Source: TQI Abdi; 27 Dec 17 PSG cluster nodes: 2 CPUs, 16 cores each

7 GPU ACCELERATED WRF Current Efforts TempoQuest WRF Results for EM LES Case on PSG - 4 nodes 9.0x Higher Is Better GPU CPU Ideal - GPU 6.2x speedup 4nodes/8cpus/16gpus Source: TQI Abdi; 27 Dec 17 PSG cluster nodes: 2 CPUs, 16 cores each 4 x P100 GPUs CPU-only uses WRF trunk CPU-only MPI task each core CPU+GPU MPI task per GPU EM LES case mostly dycore computations 7

8 GPU ACCELERATED WRF Current Efforts NVIDIA OpenACC Approach OpenACC not CUDA Directives based Permits single Source tree for CPU and GPU Easier to maintain and support Initial efforts focused on WRF However, demand rising for support 8

9 THE CHALLENGE The main WRF source tree is ~1 million lines of code in ~1,200 source files Multiple contributors, multiple styles, multiple versions of Fortran (plus a little C) Porting from to included changes to 237 files and 143 k LoC (about 15% of the total codebase) Necessary to merge in all of the NVIDIA OpenACC work, and integrate with the to changes from the community 9

10 THE CHALLENGE Merging NVIDIA code and Community code Many WRF kernels contain insufficient work for good accelerated speedups on GPUs. NVIDIA Devtechs often modify these kernels to improve performance. Need to reapply (and sometimes adapt) that implementation when porting versions. NVIDIA Modified V3.6.1 Community Routine V3.6.1 Community Routine V3.7.1 NVIDIA Modified V

11 THE CHALLENGE Example: phys/module_bl_ysu.f, in ysu2d() real,dimension( its:ite, kts:kte ):: thx do k = kts,kte do i = its,ite thx(i,k) = tx(i,k)/pi2d(i,k) enddo enddo WRFV3.6.1(NCAR) real,dimension( its:ite, kts:kte,jts:jte ) :: thx $acc kernels do j = jts,jte do k = kts,kte do i = its,ite thx(i,k,j) = tx(i,k,j)/pi2d(i,k,j) enddo enddo enddo WRFV3.6.1(NV) real,dimension( its:ite, kts:kte ):: thx, thlix do k = kts,kte do i = its,ite thx(i,k) = tx(i,k)/pi2d(i,k) thlix(i,k) = (tx(i,k)- xlv*qx(i,ktrace2+k)/cp E6*qx(i,ktrace3+k)/cp)/pi2d(i,k) enddo enddo WRFV3.7.1(NCAR) real,dimension( its:ite, kts:kte,jts:jte ) :: thx, thlix $acc kernels do j = jts,jte do k = kts,kte do i = its,ite thx(i,k,j) = tx(i,k,j)/pi2d(i,k,j) thlix(i,k,j) = (tx(i,k,j)- xlv*qx(i,ktrace2+k,j)/cp E6*qx(i,ktrace3+k,j)/cp)/pi2d(i,k,j) enddo enddo 11 enddo WRFV3.7.1(NV)

12 METHODOLOGY Separate dycore from physics, port one then another Interface to physics packages is through the main solver As these interfaces did change from to Keep track of additional (new) variables, and determine if they need to be GPU resident or not, add to OpenACC data pragmas if so. Keep track of everything, package by package 12

13 METHODOLOGY You need a lot of pixels! Diff display NCAR NCAR Compile/Debug/Run NV NV 13

14 RESULTS GPU WRF V3.7.1 vs GPU WRF V3.6.1 Scaling over different model sizes GPU 371 GPU Smaller is better Small Medum Large 14

15 Elapsed Time (μs) RESULTS Dynamics - ConUS 2.5 km model, WRF V3.7.1 CPU GPU Smaller is better

16 Elapsed Time (s) RESULTS Overall Run - ConUS 2.5 km model, WRF V CPU GPU Smaller is better

17 Elapsed Time RESULTS Overall Run - ConUS 2.5 km model, WRF V3.7.1 Scaling Smaller is better Nodes 17

18 LESSONS LEARNED Yes, the effort is worth it Porting is far, far simpler than starting over Even with the pain of two diverging trees (community + NVIDIA) Order of magnitude less effort Need to get changes into the main tree as much as possible NVIDIA will unlikely port all physics packages WRF moving to physics suites, approved combos of packages Two initial suites: Tropical & ConUS 18

19 THE FUTURE Common Community Physics Packages NOAA funded Caps for the dycore, physics, etc Fixed routines: init, run, etc. Commented args: type, dim Can we push acceleration into these caps? Or into the CCPP driver? Need to work closely with the community 19

20 20

Acceleration of WRF on the GPU

Acceleration of WRF on the GPU Daniel Abdi, Sam Elliott, Iman Gohari Don Berchoff, Gene Pache, John Manobianco TempoQuest 1434 Spruce Street Boulder, CO 80302 720 726 9032 TempoQuest.com THE WORLD S FASTEST