NVIDIA HPC Update for Earth System Modeling. Stan Posey, HPC Program Manager, ESM Domain, NVIDIA (HQ), Santa Clara, CA, USA

Size: px

Start display at page:

Download "NVIDIA HPC Update for Earth System Modeling. Stan Posey, HPC Program Manager, ESM Domain, NVIDIA (HQ), Santa Clara, CA, USA"

Amberly French
5 years ago
Views:

1 NVIDIA HPC Update for Earth System Modeling Stan Posey, HPC Program Manager, ESM Domain, NVIDIA (HQ), Santa Clara, CA, USA

2 NVIDIA HPC UPDATE TOPICS OF DISCUSSION ESM PROGRESS WITH GPUS COSMO WRF ESCAPE/IFS ESM GROWTH IN HPC + AI 2

3 NVIDIA Company Update New HQ in Santa Clara, CA, USA FY17 Rev ~$7B USD GTC 2018 Mar NVIDIA Core Markets GPU Computing Computer Graphics Artificial Intelligence 3

4 NVIDIA Growth from Advancement of HPC and AI 120, ,000 2,200 55,000 Higher Ed 35% Software 19% Internet 15% Auto 10% Government 5% Medical 4% Finance 4% Manufacturing 4% X HPC Developers 25x Deep Learning Developers GPUs Power World s Leading Data Centers for HPC and AI: 4

5 NVIDIA Volta GPU Feature Comparisons V100 (2017) P100 (2016) K40 (2014) Double Precision TFlop/s x x 1.4 Single Precision TFlop/s x x 4.3 Half Precision TFlop/s 120 (DL) ~6x 21.2 n/a Memory Bandwidth (GB/s) x Memory Size 16GB 1.00x 16GB 1.33x 12GB Interconnect NVLink: Up to 300 GB/s PCIe: 32 GB/s NVLink: 160 GB/s PCIe: 32 GB/s 2.5x PCIe: 16 GB/s Power 300W 1.00x 300W 235W Volta Availability DGX-1: Q3 2017; OEM : Q

6 NVIDIA NVLink Fast Interconnect x86 Cluster Node: GPU-to-GPU Power Cluster Node: CPU-to-GPU-to-GPU 6

7 DOE CORAL Systems with Volta and NVLink LLNL Sierra 150PF in 2018 ORNL Summit 200PF in 2018 CAAR support from IBM and NVIDIA ~1/4x ~29x 27,600 GPUs ~1.7x 7

DOE ACME GPU-Accelerated Coupled Climate Model ACME: Accelerated Climate Modeling for Energy First fully accelerated climate model (GPU and MIC) Consolidation of DOE ESM projects from 7 into 1 DOE

8 DOE ACME GPU-Accelerated Coupled Climate Model ACME: Accelerated Climate Modeling for Energy First fully accelerated climate model (GPU and MIC) Consolidation of DOE ESM projects from 7 into 1 DOE Labs: Argonne, LANL, LBL, LLNL, ORNL, PNNL, Sandia Towards NH global Atm 12 km, Ocn 15 km, 80 year ACME component models and GPU progress Atm ACME-Atmosphere (NCAR CAM-SE fork) Dycore now in trunk, CAM physics started with OpenACC Ocn MPAS-O (LANL) LANL team at ORNL OpenACC Hackathon during 2015 Others published OpenACC progress Sea-Ice ACME-CICE (LANL) Land CLM (ORNL, NCAR) Cloud Superparameterization SAM (SBU, CSU) Land-Ice PISCEES (Multi-lab LLNL, Sandia) 8

9 NVIDIA HPC UPDATE TOPICS OF DISCUSSION ESM PROGRESS WITH GPUS COSMO WRF ESCAPE/IFS ESM GROWTH IN HPC + AI 9

10 NVIDIA Support for Earth System Modeling Domain Developer Relations: Stan Posey - Santa Clara, CA (HQ) sposey@nvidia.com Developer Technology: Carl Ponder, PhD Austin, TX, US WRF, MPAS-A, FV3, GEOS-5, COAMPS, GEM Jeff Larkin Oak Ridge, TN, US CAM-SE, all ACME component models Jeremy Appleyard, PhD Oxford, UK IFS, NEMO, UM/GungHo, GOcean Peter Messmer, PhD Zurich, CH IFS/ESCAPE, COSMO, ICON Akira Naruse, PhD Tokyo, JP JMA-GSM, ASUCA, NICAM... PGI Applications Eng: Dave Norton, Lake Tahoe, CA All models that use PGI compilers Business Alliances: Steve Rohm Charlotte, NC, US US East: NOAA, EC, DOE, DoD, NASA Greg Branch Boulder, CO, US US West: NOAA, NCAR, DOE, NASA Jeremy Purches - Bristol, UK ECMWF, UKMO, STFC, No. Europe Stefan Kraemer Würselen, DE DWD, DKRZ/MPI-M, MCH, Central Europe Frederic Pariente Toulouse, FR MF, IPSL, CNRS, CERFACS, So. Europe... Solution Architects: Jeff Adie Singapore WRF, MPAS, any ESM sales opportunity Several Others Worldwide Contact sposey@nvidia.com 10

11 Select NVIDIA ESM Highlights Since Multi-Core 6 WW ESM growth in GPU funded-development: NOAA, NCAR, ECMWF, DOE, DoD Large ESM-driven GPU systems (K80/P100): NOAA, ECMWF, CSCS, NIES, Others First ever GPU-based operational NWP: MeteoSwiss with COSMO (Since 2016) ~4x speedup with ~5x less energy vs. conventional CPU-only; New COSMO evaluations by Met s in DE, RU, IT DOE climate model ACME-Atm v1 production on TITAN using PGI OpenACC NCAR collaboration on MPAS with 2016 & 2017 GPU Hands-on Workshops (focus on GPU development of MPAS-A with KISTI) ECWMF selected NVIDIA as partner for ESCAPE exascale weather project NEMO Systems Team invitation for NVIDIA to join new HPC working group Following successful NVIDIA OpenACC scalability of NEMO for ORCA025 configuration (NEMO UGM 2014) New ESM opportunities, developing in new solution focus areas DL in climate and weather; BI for Ag and Actuary; Air quality monitoring (CN, KR); Commercial WRF start-up TQI ESM development teams with participation in global GPU Hackathons DOE/ACME, NRL/COAMPS, MPI-M/ECHAM6, ODU/FVCOM, NRL/HYCOM, NOAA GFDL radiation, RRTMGP 11

12 GPU Funded-Development Growing for ESM HPC Programs with Funding Specifically Targeted for GPU Development of Various ESMs SENA NOAA funding for accelerator development of WRF, NGGPS (FV3), GFDL climate, NMMB ESCAPE ECMWF-led EUC Horizon 2020 program for IFS; NVIDIA 1 of 11 funded partners ACME US DOE accelerated climate model: CAM-SE, MPAS-O, CICE, CLM, SAM, PISCEES, others AIMES Govt s from DE, FR, and JP for HPC (and GPU) developments of ICON, DYNAMICO, NICAM SIParCS NCAR academia funding for HPC (and GPU) developments of MPAS, CESM, DART, Fields AOLI US DoD accelerator development of operational models HYCOM, NUMA, CICE, RRTMG GridTools Swiss gov funding MCH/CSCS/ETH for accelerator-based DSL in COSMO, ICON, others NOTE: Follow each program LINK for details; Programs listed from top-down in rough order of newest to oldest start date 12

Important GPU-Based NWP System Deployments Cray CS-Storm, 760

Research with GPUs Cray CS-Storm, 192 x K80, 8 GPUs per node

Develop global model with 3km resolution, five-fold increase

research) 2-3x higher resolution for daily forecasts 14x more

13 Important GPU-Based NWP System Deployments Cray CS-Storm, 760 x P100, 8 GPUs per node NOAA To Improve NWP and Climate Research with GPUs Cray CS-Storm, 192 x K80, 8 GPUs per node MeteoSwiss Deploys World s 1 st Operational NWP on GPUs Develop global model with 3km resolution, five-fold increase from today s resolution NWP Model: FV3/GFS (also climate research) 2-3x higher resolution for daily forecasts 14x more simulation with ensemble approach for medium-range forecasts NWP Model: COSMO 13

NOAA NGGPS Motivated Several GPU Collaborations NGGPS NH Model Dycore Candidates (5) NGGPS final selection: FV3 Models with GPU developments: From:

14 NOAA NGGPS Motivated Several GPU Collaborations NGGPS NH Model Dycore Candidates (5) NGGPS final selection: FV3 Models with GPU developments: From: Next Generation HPC and Forecast Model Application Readiness at NCEP -by John Michalakes, NOAA NCEP; AMS, Phoenix, AZ, Jan 2015 NIM MPAS NEPTUNE/NUMA FV3 14

15 NOAA NIM Model and Scaling on Dense GPU Node Results by NOAA ESRL Source: NVIDIA GTC

16 NCAR MPAS Model and OpenACC Developments Results by NCAR presented at ISC17 BoF, 20 June, Frankfurt, DE Cloud Resolving Global Earth-System Models: HPC at Its Extreme MPAS-A Global grid P100 vs. BDW node P100 vs. BDW (dycore) resolution (2 x BDW sockets) (1 x BDW socket) v5.0 trunk 120 km 2.5x (actual) 5.0x (estimate) v5.0 trunk 60 km 2.7x (actual) 5.4x (estimate) v5.0 trunk 30 km n/a (exceeded memory of single GPU) Source: 16

US NRL NUMA Dycore and Scalability on TITAN Strong Scalability to

Scalability to 16,384 GPUs for 3 NUMA Numerical Methods (Explicit)

Computing with the Atmospheric Model NUMA -by Dr.

resolutions Weak scalability at 90% efficiency Sources: -

17 US NRL NUMA Dycore and Scalability on TITAN Strong Scalability to 4,096 GPUs for 3 NUMA Global Resolutions (Implicit Explicit) Weak Scalability to 16,384 GPUs for 3 NUMA Numerical Methods (Explicit) Multi-Core 6 Workshop 14 Sep 16, NCAR, Boulder, USA Towards Exascale Computing with the Atmospheric Model NUMA -by Dr. Daniel Abdi, Frank Giraldo, NPS Strong scalability for all grid resolutions Weak scalability at 90% efficiency Sources: Giraldo, et. al. SIAM CSE, Mar

18 NVIDIA HPC UPDATE TOPICS OF DISCUSSION ESM PROGRESS WITH GPUS COSMO WRF ESCAPE/IFS ESM GROWTH IN HPC + AI 18

Large Scale Climate Simulations with COSMO

3 PetaFLOPS 5320 x P100 GPUs Source: PASC

19 Large Scale Climate Simulations with COSMO - Oliver Fuhrer, et al, MeteoSwiss Strong Scaling to 4888 x P100 GPUs Piz Daint #3 Top PetaFLOPS 5320 x P100 GPUs Source: PASC 2017, Lugano, CH, Jun 2017; Contact Hannes Vogt, CSCS, hannes.vogt@cscs.ch 19

Total CPUs 192 Tesla K80 Total GPUs High GPU Density Nodes: 2 x CPU + 8 x

20 MeteoSwiss Weather Prediction Based on GPUs World s First GPU-Accelerated NWP Piz Kesch (Cray CS Storm) Installed at CSCS July x Racks with 48 Total CPUs 192 Tesla K80 Total GPUs High GPU Density Nodes: 2 x CPU + 8 x GPU > 90% of FLOPS from GPUs Operational NWP Mar 16 Image by NVIDIA/MeteoSwiss 20

2 KM) 8 per day, 24 hr forecast Before GPUs MeteoSwiss COSMO NWP Configurations During 2016 IFS from ECMWF 2 per day, 10 day forecast COSMO E (2.

21 MeteoSwiss and Operational COSMO NWP on GPUs MeteoSwiss COSMO NWP Configurations Since 2008 IFS from ECMWF 2 per day, 10 day forecast COSMO 7 (6.6 KM) 3 per day, 3 day forecast COSMO 2 (2.2 KM) 8 per day, 24 hr forecast Before GPUs MeteoSwiss COSMO NWP Configurations During 2016 IFS from ECMWF 2 per day, 10 day forecast COSMO E (2.2 KM) 2 per day, 5 day forecast COSMO 1 (1.1 KM) 8 per day, 24 hr forecast With GPUs New configurations of higher resolution and ensemble predictions possible owing to the performance-per-energy gains from GPUs X. Lapillonne, MeteoSwiss; EGU Assembly, Apr

22 NVIDIA HPC UPDATE TOPICS OF DISCUSSION ESM PROGRESS WITH GPUS COSMO WRF ESCAPE/IFS ESM GROWTH IN HPC + AI 22

23 GPU Developments for the WRF Model CUDA + OpenACC: TempoQuest Plans for commercial WRF-based software product NVIDIA providing standard engineering guidance Based on WRF and 3.8.1, ARW dycore \ OpenACC: NVIDIA + NCAR (guidance) Migrating routines to and Working towards unified memory capability PGI compiler continues to improve results Several P100 customer evaluations completed (Example in later slides) Potential for Full model WRF on GPUs Several months away, hybrid in near term P100/V100 GPU will improve hybrid vs. Kepler 23

24 TempoQuest USA Start-up Commercializing WRF TQI HQ in Boulder, 3 developers focus on CUDA + OpenACC Release candidate based on WRF ARW now under validation and QA NVIDIA providing engineering support 24

25 WRF Modules and Routines Available in OpenACC Project to implement OpenACC routines into full model WRF Several dynamics routines including all of advection Several physics schemes (10): Microphysics (4) Kessler, Morrison, Thompson, WSM6 Radiation (2) RRTM (lw), Dudhia (sw), RRTMG (modified AER version from trunk) Planetary boundary layer (2) YSU, GWDO Cumulus (1) Kain-Fritsch Surface layer (1) Noah Routines completed: Dynamics (11) dyn_em/module_advect_em.openacc.f dyn_em/module_bc_em.openacc.f dyn_em/module_big_step_utilities_em.openacc.f dyn_em/module_diffusion_em.openacc.f dyn_em/module_em.openacc.f dyn_em/module_first_rk_step_part1.openacc.f dyn_em/module_first_rk_step_part2.openacc.f dyn_em/module_small_step_em.openacc.f dyn_em/module_stoch.openacc.f dyn_em/solve_em.openacc.f dyn_em/start_em.openacc.f Physics (18) Other (8) phys/module_bl_gwdo.openacc.f phys/module_bl_ysu.openacc.f phys/module_cu_kfeta.openacc.f phys/module_cumulus_driver.openacc.f phys/module_microphysics_driver.openacc.f phys/module_microphysics_zero_out.openacc.f phys/module_mp_kessler.openacc.f phys/module_mp_morr_two_moment.openacc.f phys/module_mp_thompson.openacc.f phys/module_mp_wsm6.openacc.f phys/module_pbl_driver.openacc.f phys/module_physics_addtendc.openacc.f phys/module_physics_init.openacc.f phys/module_ra_rrtm.openacc.f phys/module_ra_sw.openacc.f phys/module_sf_noahlsm.openacc.f phys/module_sf_sfclayrev.openacc.f phys/module_surface_driver.openacc.f frame/module_dm.openacc.f frame/module_domain_extra.openacc.f frame/module_domain.openacc.f frame/module_domain_type.openacc.f frame/module_integrate.openacc.f share/mediation_integrate.openacc.f share/module_bc.openacc.f share/wrf_bdyin.openacc.f 25

26 WRF OpenACC Performance: Weather Service in JP Number of CPU Cores Number of CPU Cores + P100 GPUs WRF Domain (5km) WRF Grid = 661 x 711 x 51 (24 MM) Time step = 30 sec, 720 TS s MP = Morrison LSM = Noah PBL = YSU Rad = Dudhia + RRTM Comparisons on NVIDIA PSG Cluster 2x Intel Xeon E5-2698V3 Haswell CPUs (2.3GHz, 16 cores) 256 GB memory, 4x P100 GPUs, PGI

27 WRF OpenACC Performance: Weather Service in JP Number of CPU Cores 16 ~7400 (s) Number of CPU Cores + P100 GPUs ~15x with 16 GPUs WRF Domain (5km) WRF Grid = 661 x 711 x 51 (24 MM) Time step = 30 sec, 720 TS s MP = Morrison MP LSM = Noah PBL = YSU Rad = Dudhia + RRTM Comparisons on NVIDIA PSG Cluster 2x Intel Xeon E5-2698V3 Haswell CPUs (2.3GHz, 16 cores) 256 GB memory, 4x P100 GPUs, PGI

28 NVIDIA HPC UPDATE TOPICS OF DISCUSSION ESM PROGRESS WITH GPUS COSMO WRF ESCAPE/IFS ESM GROWTH IN HPC + AI 28

29 NVIDIA Member of ESCAPE Program on Exascale NWP O 1 O 3 29

length GPU-based Spectral Transform Approach: o o o For all vertical Layers 1D FFT along all

30 ESCAPE Development of Weather & Climate Dwarfs NVIDIA-Developed Dwarf: Spectral Transform - Spherical Harmonics Batched 1D FFT variable length Batched Legendre Transform (GEMM) variable length GPU-based Spectral Transform Approach: o o o For all vertical Layers 1D FFT along all latitudes GEMM along all longitudes NVIDIA-based libraries for FFT and GEMM OpenACC directives 30

31 Results of Multi-GPU Spectral Transform Dwarf Source: ESCAPE 2nd Dissemination Workshop, 5-7 Sep 2017, Poznań, Poland 31

32 NVIDIA HPC UPDATE TOPICS OF DISCUSSION ESM PROGRESS WITH GPUS COSMO WRF ESCAPE/IFS ESM GROWTH IN HPC + AI 32

33 Sample of Inside HPC Headlines Just Last Week... It s somewhat ironic that training for deep learning probably has more similarity to the HPL benchmark than many of the simulations that are run today - - Kathy Yelick, LBNL 33

34 Research Operational and Research Hardware Trends for Operational and Research HPC Organization Location Models Previous HPC Current HPC Size / Cost (M) / Date ECMWF Reading, UK IFS IBM Power Cray XC30 x PF / $65 / Jun 2013 Met Office Exeter, UK UM IBM Power Cray XC30 x86 16 PF / $120 / Oct 2014 DWD Offenbach, DE COSMO, ICON NEC SX-9 Cray XC30 - x86 2 PF / $23 / Jan 2013 MF Toulouse, FR Arpege, Arome NEC SX-9 Bull - x86 5 PF / $? / Nov 2012 NOAA NCEP Various, US GFS, HRRR/WRF IBM Power IBM idataplex - x86 Cray XC-40 - x86 5 PF / $50/yr / Oct 2015 Env Canada Montreal, CA GEM-YY, WRF IBM Power TBA ~2016 > 5 PF JMA Tokyo, JP GSM, ASUCA Hitachi Power TBA ~2016 > 5 PF DKRZ/MPI-M Hamburg, DE ICON, MPI-ESM IBM Power Bull - x86 3 PF / $35 / May 2014 NCAR Boulder, CO, US CESM, WRF, MPAS IBM idataplex SGI ICE XA - x PF / ~$60 / Jan 2016 NOAA ESRL Fairmont, WV, US FV3, MPAS, WRF Various Cray CS-Storm - x86 TBA ~

35 Use Cases for HPC + Artificial Intelligence A confluence of developments is driving this new wave of AI development. Computer power is growing, algorithms and AI models are becoming more sophisticated, and, perhaps most important of all, the world is generating once unimaginable volumes of the fuel that powers AI data. Billions of gigabytes every day, collected by networked devices... 35

36 HPC + AI for Weather and Climate Applications Challenges for HPC and AI in Weather and Climate AI in Weather Applications Yandex + Start-ups NERSC NOAA, MCH, others NCAR, KISTI, others 36

37 NWP Nowcasting Systems Apply Deep Learning 37

38 NWP Nowcasting Systems Apply Deep Learning 38

MCH Use of DL Models for Fog Forecast at Airports Background Unexpected fog can cause an airport to cancel or delay flights, sometimes having global effects on flight planning.

39 MCH Use of DL Models for Fog Forecast at Airports Background Unexpected fog can cause an airport to cancel or delay flights, sometimes having global effects on flight planning. Challenge While the weather forecasting model at MeteoSwiss work at a 2km x 2km resolution, runways at Zurich airport is less than 2km. So human forecasters sift through huge simulated data with 40 parameters, like wind, pressure, temperature, to predict visibility at the airport. Solution MeteoSwiss is investigating the use of deep learning to forecast type of fog and visibility at sub-km scale at Zurich airport. 39

NOAA Neural Network Study on Thompson MP https://ams.

40 NOAA Neural Network Study on Thompson MP 40

41 KISTI Weather Research Deploys Deep Learning KISTI Disaster Management HPC Research Group Deploys GPUs for Numerical and Deep Learning Models in the Prediction of Extreme Weather KISTI Schematic of HPC Systems with 60 x K40m GPUs Courtesy Dr. Minsu Joh, KISTI, May

42 KISTI Weather Research Deploys Deep Learning KISTI Disaster Management HPC Research Group Deploys GPUs for Numerical and Deep Learning Models in the Prediction of Extreme Weather KISTI to present this research at Climate Informatics (CI) 2017, Boulder, CO, Sep 2017 Courtesy Dr. Minsu Joh, KISTI, May

KISTI Weather Research Deploys Deep Learning More detail next slide KISTI Disaster Management HPC Research Group Deploys GPUs for Numerical and Deep Learning Models

43 KISTI Weather Research Deploys Deep Learning More detail next slide KISTI Disaster Management HPC Research Group Deploys GPUs for Numerical and Deep Learning Models in the Prediction of Extreme Weather KISTI to present this research at Climate Informatics (CI) 2017, Boulder, CO, Sep 2017 Courtesy Dr. Minsu Joh, KISTI, May

KISTI Weather Research Deploys Deep Learning KISTI Scientists use output from the MPAS numerical atmosphere model to train a deep learning model for

44 KISTI Weather Research Deploys Deep Learning KISTI Scientists use output from the MPAS numerical atmosphere model to train a deep learning model for improved forecast tracking of typhoons Courtesy Dr. Minsu Joh, KISTI, May 2017 KISTI Schematic of the workflow of a hybrid numerical plus deep learning model 44 44

45 Thank you and Questions? Stan Posey,

ACCELERATING WEATHER PREDICTION WITH NVIDIA GPUS

ACCELERATING WEATHER PREDICTION WITH NVIDIA GPUS Alan Gray, Developer Technology Engineer, NVIDIA ECMWF 18th Workshop on high performance computing in meteorology, 28 th September 2018 ESCAPE NVIDIA s