Exascale challenges for Numerical Weather Prediction : the ESCAPE project

Size: px

Start display at page:

Download "Exascale challenges for Numerical Weather Prediction : the ESCAPE project"

Amice Henry
5 years ago
Views:

1 Exascale challenges for Numerical Weather Prediction : the ESCAPE project O Olivier Marsden This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No

2 European Centre for Medium-Range Weather Forecasts Independent intergovernmental organisation established in 1975 with 19 Member States 15 Co-operating States 2

3 The success story of Numerical Weather Prediction: Hurricanes May be one of the best medium-range forecasts of all times! 3

4 NWP: Benefit of high-resolution Mean sea-level pressure AN 30 Oct 5d FC T3999 5d FC T1279 5d FC T639 Sandy 28 Oct 2012 Precipitation: NEXRAD 27 Oct 3d FC: Wave height Mean sea-level pressure 4d FC T639 4d FC T1279 4d FC T m wind speed 4

Type 98% from 60 different satellite instruments Observations physical parameters of atmosphere, waves, ocean Models Volume 200 million

5 What is the challenge? Observations Models Today: Tomorrow: Volume 20 million = 2 x million grid points 100 levels 10 prognostic variables = 5 x 10 9 Type 98% from 60 different satellite instruments Observations physical parameters of atmosphere, waves, ocean Models Volume 200 million = 2 x million grid points 200 levels 100 prognostic variables = 1 x Type 98% from 80 different satellite instruments physical and chemical parameters of atmosphere, waves, ocean, ice, vegetation Factor 10 per day Factor 2000 per time step 5

6 Fraction of Operational Threshold AVEC forecast model intercomparison: 13 km km Case: Speed Normalized to Operational Threshold (8.5 mins per day) IFS NMM-UJ FV3, single precision NIM FV3, double precision MPAS NEPTUNE 13km Oper. Threshold 1 0 Number of Edison Cores (CRAY XC-30) [Michalakes et al. 2015: AVEC-Report: NGGPS level-1 benchmarks and software evaluation] 6

7 Fraction of Operational Threshold AVEC forecast model intercomparison: 3 km km Case: Speed Normalized to Operational Threshold (8.5 mins per day) IFS NMM-UJ FV3 single precision FV3 double precision NIM NIM, improved MPI comms MPAS NEPTUNE 3km Oper. Threshold Advanced Computing Evaluation Committee (AVEC) to evaluate HPC performance of five Next Generation Global Prediction System candidates to meet operational forecast requirements at the National Weather Service through Number of Edison Cores (CRAY XC-30) 7

8 Technology applied at ECMWF for the last 30 years A spectral transform, semi-lagrangian, semi-implicit (compressible) hydrostatic model How long can ECMWF continue to run such a model? IFS data assimilation and model must EACH run in under ONE HOUR for a 10 day global forecast 8

9 IFS today (MPI + OpenMP parallel) IFS = Integrated Forecasting System 9

10 Predicted 2.5 km model scaling on a XC-30 Operational requirement 2 MW 6 MW (for a single HRES forecast) two XC-30 clusters each with 85K cores ECMWF require system capacity for October 29, to 20 simultaneous HRES forecasts

Numerical methods Code Adaptation - Architecture

weather Prediction at Exascale: Next generation IFS

algorithms Compute/energy efficiency diagnostics New

Testing in operational configurations *Funded by EC

High-Performance Computing Partners: ECMWF,

11 Numerical methods Code Adaptation - Architecture ESCAPE*, Energy efficient SCalable Algorithms for weather Prediction at Exascale: Next generation IFS numerical building blocks and compute intensive algorithms Compute/energy efficiency diagnostics New approaches and implementation on novel architectures Testing in operational configurations *Funded by EC H2020 framework, Future and Emerging Technologies High-Performance Computing Partners: ECMWF, Météo-France, RMI, DMI, Meteo Swiss, DWD, U Loughborough, PSNC, ICHEC, Bull, NVIDIA, Optalysys

Schematic description of the spectral transform method in the ECMWF IFS model FFT Grid-point space -semi-lagrangian advection -physical parametrizations -products of terms Inverse FFT

12 Schematic description of the spectral transform method in the ECMWF IFS model FFT Grid-point space -semi-lagrangian advection -physical parametrizations -products of terms Inverse FFT Fourier space Fourier space LT Spectral space -horizontal gradients -semi-implicit calculations -horizontal diffusion Inverse LT FFT: Fast Fourier Transform, LT: Legendre Transform 13

13 Schematic description of the spectral transform warf in ESCAPE Grid-point space d FFT Fourier space LT 100 iterations Time-stepping loop in dwarf1-atlas.f90 DO JSTEP=1,ITERS call trans%invtrans(spfields,gpfields) call trans%dirtrans(gpfields,spfields) ENDDO Inverse FFT Fourier space Inverse LT Spectral space FFT: Fast Fourier Transform, LT: Legendre Transform 14

14 GPU-related work on this dwarf Work carried out by George Mozdzynski, ECMWF An OpenACC port of a spectral transform test (transform_test.f90) Using 1D parallelisation over spectral waves Contrast with IFS which uses 2D parallelisation (waves, levels) About 30 routines ported, 280!$ACC directives Major focus on FFTs, using NVIDIA cufft library Legendre Transform uses DGEMM_ACC Fast Legendre Transform not ported (need working deep copy) CRAY provided access to SWAN (6 NVIDIA K20X GPUs) Latest 8.4 CRAY compilers Larger runs performed on TITAN Each node has 16 AMD Interlagos cores & 1 NVIDIA K20X GPU (6GB) CRESTA INCITE14 access Used CRAY compiler Compare performance of XK7/Titan node with XC-30 node (24 core Ivybridge) 15

15 msec per time-step 300 Tc km model Spectral Transform Compute Cost (40 nodes, 800 fields) XC TITAN LTINV_CTL LTDIR_CTL FTDIR_CTL FTINV_CTL 16

16 msec per time-step 700 Tc km model Spectral Transform Compute Cost (120 nodes, 800 fields) XC-30 TITAN LTINV_CTL LTDIR_CTL FTDIR_CTL FTINV_CTL 17

17 msec per time-step 1400 Tc km model Spectral Transform Compute Cost (400 nodes, 800 fields) XC-30 TITAN LTINV_CTL LTDIR_CTL FTDIR_CTL FTINV_CTL 18

Relative Performance Relative FFT performance NVIDIA K20X GPU (v2) v 24 core Ivybridge CRAY XC-30 node (FFT992) 1.60 1.40 1.20 1.00 0.80 0.60 0.40 0.20 0.

18 Relative Performance Relative FFT performance NVIDIA K20X GPU (v2) v 24 core Ivybridge CRAY XC-30 node (FFT992) T95 T159 T399 T1023 T1279 T2047 T T K20X GPU performance up to 1.4 times faster than 24 Ivybridge core XC-30 node 22

19 Time 0.50 Comparison of FFT cost for LOT size GPU ver 1 GPU ver FFT FFTW FFT length (latitude points) 24

20 What about MPI communications? Cost very much greater than compute for Spectral Transform test Tc3999 example follows XC-30 (Aries) is faster than XK7/Titan (Gemini) So made prediction for XC-30 comms with K20X GPU Potential for compute / communications overlap GPU compute while MPI transfers are taking place Not done (yet) 25

21 Tc3999, 400 nodes, 800 fields (ms per time-step) Tc3999 XC-30 TITAN XC-30+GPU Prediction LTINV_CTL LTDIR_CTL FTDIR_CTL FTINV_CTL MTOL LTOM LTOG GTOL HOST2GPU** GPU2HOST** ** included in comms (red) times 26

22 Spectral transforms experience OpenACC not that difficult, but Replaced ~10 OpenMP directives (high-level parallelisation) By ~280 OpenACC directives (low-level parallelisation) Most of the porting time spent on Strategy for porting IFS FFT992 interface (algor/fourier) Replaced by calls to new cuda FFT993 interface Calling NVIDIA cufft library routines Coding versions of FTDIR and FTINV where FFT992 and FFT993 both ran on same data to compare results Writing several offline FFT tests to explore performance Performance issues Used nvprof, gstats 27

23 Physics dwarf : CloudSC Work done by Sami Saarinen, ECMWF Adaptation of IFS physics cloud scheme (CLOUDSC) to new architectures as part of ECMWF Scalability programme Emphasis was on GPU-migration by use of OpenACC directives CLOUDSC consumes about 10% of IFS Forecast time Some 3500 lines of Fortran2003 before OpenACC directives Focus on performance comparison between - OpenMP version of CLOUDSC on Haswell -OpenACC version of CLOUDSC on NVIDIA K40 28

24 Problem parameters: Given 160,000 grid point columns (NGPTOT) Each with 137 levels (NLEV) About 80,000 columns fit into one K40 GPU Grid point columns are independent of each other So no horizontal dependencies here, but level dependency prevents parallelization along vertical dim Arrays are organized in blocks of grid point columns Instead of using ARRAY(NGPTOT, NLEV) we use ARRAY(NPROMA, NLEV, NBLKS) NPROMA is a (runtime) fixed blocking factor Arrays are OpenMP thread safe over NBLKS 29

25 Details on hardware, compilers, NPROMA: Haswell-node : 2.5GHz 2 x NVIDIA K40c GPUs on each Haswell-node via PCIe Each GPU equipped with 12GB memory with CUDA 7.0 PGI Compiler 15.7 with OpenMP & OpenACC O4 fast mp=numa,allcores,bind Mfprelaxed tp haswell Mvect=simd:256 [ -acc ] Environment variables PGI_ACC_NOSHARED=1 PGI_ACC_BUFFERSIZE=4M Typical good NPROMA value for Haswell~ For GPUs NPROMA up to 80,000 for max performance 30

26 OpenMP loop around CLOUDSC call: REAL(kind=8) :: array(nproma, NLEV, NGPBLKS)!$OMP PARALLEL PRIVATE(JKGLO,IBL,ICEND)!$OMP DO SCHEDULE(DYNAMIC,1) DO JKGLO=1,NGPTOT,NPROMA! So called NPROMA-loop IBL=(JKGLO-1)/NPROMA+1! Current block number ICEND=MIN(NPROMA,NGPTOT-JKGLO+1)! Block length <= NPROMA CALL CLOUDSC( 1, ICEND, NPROMA, KLEV, & END DO & array(1,1,ibl), &! ~ 65 arrays like this ) Typical values for!$omp END DO NPROMA in OpenMP implementation:!$omp END PARALLEL

27 OpenMP scaling (Haswell, in GFlops) 32

28 Development of OpenACC/GPU-version The driver-code with OpenMP-loop kept roughly unchanged GPU to HOST data mapping (ACC DATA) added OpenACC can (in most cases) co-exist with OpenMP Allows an elegant multi-gpu implementation CLOUDSC was pre-processed with acc_insert Perl-script Allowed automatic creation of ACC KERNELS and ACC DATA PRESENT / CREATE clauses to CLOUDSC In addition some minimal manual source code clean-up CLOUDSC performance on GPU needs very large NPROMA Lack of multilevel parallelism (only across NPROMA, not NLEV) 33

29 Driving OpenACC CLOUDSC with OpenMP!$OMP PARALLEL PRIVATE(JKGLO,IBL,ICEND) &!$OMP& PRIVATE(tid, idgpu) num_threads(numgpus) tid = omp_get_thread_num()! OpenMP thread number idgpu = mod(tid, NumGPUs)! Effective GPU# for this thread CALL acc_set_device_num(idgpu, acc_get_device_type())!$omp DO SCHEDULE(STATIC) DO JKGLO=1,NGPTOT,NPROMA! NPROMA-loop IBL=(JKGLO-1)/NPROMA+1! Current block number ICEND=MIN(NPROMA,NGPTOT-JKGLO+1)! Block length <= NPROMA!$acc data copyout(array(:,:,ibl),...) &! ~22 : GPU to Host!$acc& copyin(array(:,:,ibl))! ~43 : Host to GPU CALL CLOUDSC (... array(1,1,ibl)...)! Runs on GPU#<idgpu>!$acc end data END DO!$OMP END DO!$OMP END PARALLEL Typical values for NPROMA in OpenACC implementation: > 10,000 34

30 Sample OpenACC coding of CLOUDSC!$ACC KERNELS LOOP COLLAPSE(2) PRIVATE(ZTMP_Q,ZTMP) DO JK=1,KLEV DO JL=KIDIA,KFDIA ztmp_q = 0.0_JPRB ztmp = 0.0_JPRB!$ACC LOOP PRIVATE(ZQADJ) REDUCTION(+:ZTMP_Q, +:ZTMP) DO JM=1,NCLV-1 IF (ZQX(JL,JK,JM)<RLMIN) THEN ZLNEG(JL,JK,JM) = ZLNEG(JL,JK,JM)+ZQX(JL,JK,JM) ZQADJ = ZQX(JL,JK,JM)*ZQTMST ztmp_q = ztmp_q + ZQADJ ztmp = ztmp + ZQX(JL,JK,JM) ZQX(JL,JK,JM) = 0.0_JPRB ENDIF ENDDO PSTATE_q_loc(JL,JK) = PSTATE_q_loc(JL,JK) + ztmp_q ZQX(JL,JK,NCLDQV) ENDDO ENDDO!$ACC END KERNELS ASYNC(IBL) = ZQX(JL,JK,NCLDQV) + ztmp ASYNC removes CUDA-thread syncs 35

OpenACC scaling (K40c, in GFlops) 12 10 8 6 1 GPU 2

31 OpenACC scaling (K40c, in GFlops) GPU 2 GPUs 4 2 NPROMA

32 Timing (ms) breakdown : single GPU Other overhead Communication Computation Haswell NPROMA 37

33 Saturating GPUs with more work More threads here!$omp PARALLEL PRIVATE(JKGLO,IBL,ICEND) &!$OMP& PRIVATE(tid, idgpu) num_threads(numgpus * 4) tid = omp_get_thread_num()! OpenMP thread number idgpu = mod(tid, NumGPUs)! Effective GPU# for this thread CALL acc_set_device_num(idgpu, acc_get_device_type())!$omp DO SCHEDULE(STATIC) DO JKGLO=1,NGPTOT,NPROMA! NPROMA-loop IBL=(JKGLO-1)/NPROMA+1! Current block number ICEND=MIN(NPROMA,NGPTOT-JKGLO+1)! Block length <= NPROMA!$acc data copyout(array(:,:,ibl),...) &! ~22 : GPU to Host!$acc& copyin(array(:,:,ibl))! ~43 : Host to GPU CALL CLOUDSC (... array(1,1,ibl)...)! Runs on GPU#<idgpu>!$acc end data END DO!$OMP END DO!$OMP END PARALLEL 38

34 Saturating GPUs with more work Consider few performance degradation facts at present Parallelism only in NPROMA dimension in CLOUDSC Updating 60-odd arrays back and forth every time step OpenACC overhead related to data transfers & ACC DATA Can we do better? YES! We can enable concurrently executed kernels through OpenMP! Time-sharing GPU(s) across multiple OpenMP-threads About 4 simultaneous OpenMP host threads can saturate a single GPU in our CLOUDSC case Extra care must be taken to avoid running out of memory on GPU Needs ~ 4X smaller NPROMA : 20,000 instead of 80,000 39

Multiple copies of CLOUDSC per GPU (GFlops) 16

35 Multiple copies of CLOUDSC per GPU (GFlops) GPU 2 GPUs Copies

36 nvvp profiler shows time-sharing impact GPU is fed with work by one OpenMP thread only GPU is 4-way time-shared 41

37 Timing (ms) : 4-way time-shared vs. no T/S GPU is not time-shared GPU is 4-way time-shared Other overhead Communication Computation Haswell NPROMA

38 24-core Haswell 2.5GHz vs. K40c GPU(s) (GFlops) 18 T/S = GPUs timeshared Haswell 2 GPUs (T/S) 2 GPUs 1 GPU (T/S) 1 GPU

39 Conclusions CLOUDSC OpenACC prototype from 3Q/2014 was ported to ECMWF s tiny GPU cluster in 3Q/2015 Since last time PGI compiler has improved and OpenACC overheads have been greatly reduced (PGI 14.7 vs. 15.7) With CUDA 7.0 and concurrent kernels it seems time-sharing (oversubscribing) GPUs with more work pays off Saturation of GPUs can be achieved not surprisingly by help of multi-core host launching more data blocks onto GPUs The outcome is not bad considering we seem to be underutilizing the GPUs (parallelism just along NPROMA) 44

40 Thank You! This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No

Scalability Programme at ECMWF

Scalability Programme at ECMWF Picture: Stan Tomov, ICL, University of Tennessee, Knoxville Peter Bauer, Mike Hawkins, George Mozdzynski, Tiago Quintino, Deborah Salmond, Stephan Siemen, Yannick Trémolet