Exascale I/O challenges for Numerical Weather Prediction

Transcription

1 Exascale I/O challenges for Numerical Weather Prediction A view from ECMWF Tiago Quintino, B. Raoult, S. Smart, A. Bonanni, F. Rathgeber, P. Bauer, N. Wedi ECMWF tiago.quintino@ecmwf.int SuperComputing 16, Workshop on Exascale IO Challenges, Innovations and Solutions ECMWF November 28, 2016

2 ECMWF An independent intergovernmental organisa.on 21 Member States 13 Co-opera.ng States 2

3 Who are we and what do we do? Produce global weather forecasts Medium-Range, up to 15 days ahead Also monthly and seasonal forecasts Collect and store meteorological data on perpetual archive Reading, United Kingdom What do we have to achieve this? 260 staff, specialists and contractors State-of-the-art supercomputers and data handling system 3

4 Numerical Weather ECMWF 4

5 ECMWF s HPC Targets What do we do? Operations Time Critical Operational runs 2 hours from observation cut-off to deliver forecast products 10 day forecast twice per day, 00Z and 12Z Boundary Conditions 06Z and 18Z, monthly, seasonal, etc. Research Non Time Critical Improving our models Climate reanalysis, etc Tension HPC Facility Targets Capability, minimise the time to solution of Model runs Capacity, maximise the throughput of research jobs per day Time Critical vs. Non Time Critical Capacity vs. Capability Challenge: design our HPC system to optimise these goals, minimising TCO? 5

6 ECMWF s Production Workflow Global Observations Acquisition IFS Model Product Generation Dissemina tion Obs Fields Products Member States & Customers MARS Perpetual Archive 6

7 ECMWF HPC Job profile 7

8 Operational workload: Job allocation (1 cycle) 8

9 Le Petit Prince, Antoine de Saint-Exupéry 9

10 Operational workload: Job allocation (1 cycle) 10

11 Operational workload: Files opened (1 cycle) Target Files = # Users x # Steps x # Ranks 11

12 Operational workload: Input Read (1 cycle) 12

13 Operations workload: Output written (1 cycle) 13

14 ECMWF s Production Workflow IFS Model Product Generation ECPDS Member States & Customers Raw Output Fields 42TiB Read (70%) Products Parallel Filesystem Storage (Lustre) FDB PDB Time critical path MARS Perpetual Archive 14

15 Estimated Growth in Model IO km, 137 levels Time critical 21 TB/day written 22 Million fields 85 Million products 11 TB/day send to customers 2020 Increase: 2 horizontal, 1 upper air Time critical 128 TB/day written 90 Million fields 450 Million products 60 TB/day send to customers Non-time critical 100 TB/day archived 400 research experiments 400,000 jobs / day Non-time critical 1 PB/day archived 1000 research experiments 15

16 Big Data Challenge The 3 V s of Big Data Big Data is high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization. 3D Data Management: Controlling Data Volume, Velocity and Variety, D. Laney, Gartner,

17 V is for Volume: Observations Increase of satellite data usage 17

18 V is for Volume: Archive Deletion of 1 PB 18

19 V is for Velocity ECMWF s archive grows exponentially: Initial volume Time Volume of the archive Rate of growth r is around 0.5, which is a 50% increase per year The rate of added data also grows exponentially at the same rate! In 1995, the size of the archive was increasing at a rate of 14 TB/year Nov 2016, the size of the archive increases at a rate of 150 TB/day 19

20 V is for Variety 10T 1T X-MP/4 Y-MP/8 C90/12 C90/16 T106L19 T213L31 T106L16 VPP VPP T319L50 T319L31 VPP5000 IBM-P4 IBM-P5 IBM-P5+ IBM-P6 T319L60 T511L60 00Z EPS T799L91 Weekly Monthly T1279L91 Ensemble data assimilation 00Z 10 day FC Overlap, CalVal Vareps/Monthy 4d-Var Model errors 100G 4D-var increments SCDA 4D-Var DCDA Wave EFIs EPS PT levels DCDA Wave 10G Weekend EPS Extra fields, new gaussian grid End sensitivity Wave proba. SCDA Waves SCDA Forecast Wave 4V PT and PV levels SCDA Forecast EFIs Multi-Analysis 4D-Var Wave EPS SCDA Analysis Errors if FG, surface 1G 00Z Run 00Z Run Errors in FG Sensitivity Other centers Clusters EPS Waves FG Probabilities OI Tubes Errors in AN and FG NCEP EPS Ensemble means & stdev 1/3 growth is resolu9on increase 2/3 growth is increase of product types 100M FC Model levels SSTs Waves Chernobyl TOGA FC 3DVar 4DVar 12 Hour 4DVar DCDA Vareps/Monthy EDA FC Pressure levels 50 Members EPS EPS 15 days 10M

21 Meteorological Fields Fields: 6.6M grid points x 137 levels 904M values, per variable Daily Operational Cycle: 18M fields 27 TiB output Time critical window: 1h!!! 21

22 Archive growth versus HPC performances Log scale 22

23 Evolu.on of Forecast scores, comparison northern and southern hemispheres ~16km ~39km ~25km ~125km ~63km ~210km 23

24 Benefits of High Resolution AN 30 Oct Mean sea-level pressure 5d FC T3999 5d FC T1279 5d FC T639 Sandy 28 Oct 2012 Precipitation: NEXRAD 27 Oct 3d FC Wave height Mean sea-level pressure 4d FC T639 4d FC T1279 4d FC T

25 TCo1279 (~9km) a World s 6.6 Megapixel highest resolution camera global NWP today TCo1279 orography (12h forecast, hydrostatic, with deep convection parametrization, 450s time-step, 240 Broadwell nodes, ~0.75s per timestep) 25

26 TCo7999 (~1.25km) 256 Megapixel camera with predictive skill! (12 h forecast, hydrostatic, no deep convection parametrization, 120s time-step, 960 Broadwell nodes, ~10s per timestep) 26

27 History and Future of Resolution Upgrades Resolution Grid size Grid Points Field Size (in memory) T km 204 k 1.6 MB T km 524 k 4 MB T km 1.2 M 9.6 MB T km 2.1 M 16.8 MB Tco km 6.6 M 50.4 MB Tco km 16.1 M MB Tco km 64 M 490 MB Tco km 256 M 1909 MB The tendency of memory per core diminishing is likely to have serious implications on the post-processing workflows! 27

28 10-Year Challenge 30x more data sent to customers per day in critical path RMDCN Data acquisition Dissemination Internet 10x more observational data per day Forecast run Product generation Web services Internet 2000x more model data per time step 25x more forecast product data per day in critical path 100x more data archived per day Archive Data Handling System 28

29 Conclusions & Questions NWP has had I/O exponential growth for many years. What is different? Moving from compute centric to data centric paradigm Minimise data movement and bring compute to data Update our legacy codes and workflows to this new paradigm How to adapt upcomming technologies for complex workflows? Burst Buffers NVRAM Storage-side compute Object stores Can we move beyond the filesystem? How intrusive should that be? Interpreting scientific data as objects Challenges in data modelling and data curation 29