Scalability Programme at ECMWF Picture: Stan Tomov, ICL, University of Tennessee, Knoxville Peter Bauer, Mike Hawkins, George Mozdzynski, Tiago Quintino, Deborah Salmond, Stephan Siemen, Yannick Trémolet and Nils Wedi
Cores: very,, very simplified Operational application run = wall clock time x number of cores x 1/scaling factor Today s ECMWF ensemble (50M T639/T319 L91): = 3 hours x 9,000 cores x 1.0 Tomorrow s ECMWF ensemble (50M T1023/T639 L91): = 3 hours x 2.8 x 9,000 cores x 1.0 = 3 hours x 25,000 cores x 1.0 = 3 hours x 33,000 cores x 1.0/0.75 = 2 hours x 44,000 cores x 1.0/0.75/0.75 = 3 hours x 50,000 cores x 1.0/0.5 = 2 hours x 100,000 cores x 1.0/0.5/0.5 Notes: Global, convection resolving scales O(1 2 km)? Aerosols, trace gases, ocean, waves, sea ice coupling? 4D Var, EDA and ocean model scale the worst at present 10 day HRES at 2.5 km in 1 hour requires 250,000 Ivybridge cores (6MW) Scalability Programme
Examples: Compute and Archive Compute (communication): model time step of 30 seconds 10 day forecast model on 4,000,000 cores max. 1 hour wall clock 1 step needs to run in under 0.125 seconds by using 32 threads per task with 128.000 MPI tasks: a simple MPI_SEND from 1 task to all other 128K tasks will take an estimated 128k x 1 μsec = 0.128 seconds Global communications (+ memory limitations)? Archive*: EC Earth at 25km with 10 years/day on 5000 cores 25 member ensemble x 4 for e.g. calibration: 1,000,000 core experiment 25 year run over 2.5 days produces 60,000,000 core hours 250 Gb/compute month per member 6 Pb/day = 0.5 Tbit/s Data I/O rates, reliable management on disks for post processing and dissemination? (*Example courtesy Bryan Lawrence U Reading)
NWP: Benefit of high resolution Mean sea level pressure AN 30 Oct 5d FC T3999 5d FC T1279 5d FC T639 Sandy 28 Oct 2012 3d FC: Wave height Mean sea level pressure Precipitation: NEXRAD 27 Oct 10 m wind speed 4d FC T639 4d FC T1279 WWRP Open Science Conference 4d FC T3999 PB 08/2014 ECMWF
NWP: Benefit of high resolution 500 hpa geopotential height energy spectrum from non hydrostatic model integration T1279/T3999 (10 days) T7999 (1 12 hours)
Experiments with IFS: T2047L137 (10 km) RAPS12 (CY37R3, on HECToR), RAPS13 (CY38R2, on TITAN) 900 800 700 Forecast Days / Day 600 500 400 300 200 Critical time TITAN RAPS13 CRESTA OCT 13 TITAN RAPS13 CRESTA JUN 13 HECToR RAPS12 CRESTA HECToR RAPS12 Original start up, compiler Hector Titan Co arrays, MPI opt. 100 0 0 20000 40000 60000 80000 100000 120000 Number of Cores
Experiments with IFS: T3999L137 (5 km) Critical time start up, compiler Efficiency in %
ECMWF production workflow
ECMWF production workflow Data assimilation Model integration 12h EDA: 10 members, 2 outer loops, inner loops w/ iterations, 6h integrations, low resolution 6/12h 4DVAR: 3 outer loops, inner loops w/ iterations, 6h integrations, high/low resolution, wave coupling Observation DB incl. feedback, ML and PL output 10d HRES: 10d integrations, high resolution (radiation low resolution), wave coupling ML and PL output 15/32d ENS: 15/32d integrations, lower resolution (radiation low resolution), oceanwave coupling, (2 t steps ML and) PL output Data management Dissemination via RMDCN Post processing, archiving
ECMWF HPC utilization IBM P7 cluster A Total: 96% RD Model RD Data Reanalysis Member States Operations 12 UTC 12 UTC 6 UTC EDA 6h 4DVAR BC 12h 4DVAR HRES 10d FC ENS 15d FC
Scalability Programme Programme management Project: Data assimilation (OOPS) Control structure IFS integration Coupling Scripts Project: Numerical methods (PolyMitos) Data structures Discretization Algorithms Coupling Project: Data processing (HERMES) Profiling (I/O, post proc.) Grids, interpolation Formats, compression Visualization Project: IFS code adaptation (OAFS) Benchmarking Code optimization Accelerators Portability Project: Computer architecture support Cray phase 2 (CPU, accelerators) RAPS benchmarking I/O benchmarking Kernels
Model evolution with Scalability Programme 25 km 10 km 5 km 2 km Greenhouse/reactive gases Atmosphere Aerosols Land surface Waves Sea ice Ocean 10 6 Fully coupled atmosphere land sea ice ocean Fully coupled atmospheric physics chemistry Non hydrostatic model 10 4 10 2 2010 2015 2020 2025
Model evolution without Scalability Programme 25 km 10 km 10 km 5 km Greenhouse gases Atmosphere Aerosols Land surface Waves Sea ice Ocean 10 6 Fully coupled atmosphere land sea ice ocean Non hydrostatic model Fully coupled atmospheric physics chemistry 10 4 10 2 2010 2015 2020 2025