Computational Challenges in Big Data Assimilation with Extreme-scale Simulations

Transcription

1 May 1, 2013, BDEC workshop, Charleston, SC Computational Challenges in Big Data Assimilation with Extreme-scale Simulations Takemasa Miyoshi RIKEN Advanced Institute for Computational Science With many thanks to Y. Sato (JMA), UMD Weather-Chaos group, Data Assimilation Research Team

2 Data Assimilation (DA) Observations Numerical models Data Assimilation Vaisala Data assimilation best combines observations and a model, and brings synergy.

3 DA has an impact. SV w/ 4D-Var JMA operational system LETKF under development OBS FCST OBS FCST Miyoshi and Sato (2007) Using the same NWP model and observations. DA matters!

4 Expanding collaborations AFES JMA GSM Atmosphere Ocean OFES MOM CFES ROMS :Existing WRF-ROMS Mars GCM :Possible future expansion JMA MSM WRF LETKF Local Ensemble Transform Kalman Filter JAMSTEC Chem U.Tokyo Aerosol MRI Chem CPTEC Brazil GFS SPEEDY CO2 CAM Chemistry

5 Numerical Weather Prediction (NWP) Forecast Forecast Model Simulation Analysis Observation Analysis Analysis Observation True atmosphere (Unknown) time

6 Global Observing System Radar Aircraft Satellite Weather balloon Ship Buoy Surface station

7 Collecting the data World s effort! (no border in the atmosphere)

8 Collecting the data

9 We consider the evolution of PDF Analysis ensemble mean R Obs. Analysis w/ errors An approximation to KF with ensemble representations f f T f δxt1( δxt P 1) t1 m 1 FCST ensemble mean T=t0 T=t1 T=t2

10 Flow chart of DA (Best estimate) Initial State Simulation PDF represented by an ensemble Simulated State DA Sim-to-Obs DA conversion Sim-minus-Obs Observations

11 Flow chart of DA (Best estimate) Initial State Simulation PDF represented by an ensemble Simulated State DA Sim-to-Obs conversion Observations Sim-minus-Obs Broad-sense DA

12 Data size in NWP ~2TB/6h (~300TB/6h) Simulated State 28-km global mesh ~3GB 100 members for PDF ~300GB 7 time slots ~2TB DA ~2GB/6h (~1TB/6h) Size[GB/Month] Observations Size[GB/Month] Future: new satellites, world s radar data, etc. Extreme-scale Simulation: 3.5-km global mesh ~400GB 100 members ~40TB 7 time slots ~300TB Courtesy of JMA

13 Flow chart with current data size (Best estimate) ~300GB (100 members, 1 time level) Initial State Simulation ~300GB (100 members, 1 time level) DA ~300GB (100 members, 1 time level) ~200GB (100 members) Simulated State Sim-to-Obs conversion Sim-minus-Obs ~2TB (100 members, 7 time levels) ~2TB (100 members, 7 time levels) ~2GB Observations ~200GB (100 members)

14 Flow chart with exa-scale data size (Best estimate) ~40TB (100 members, 1 time level) Initial State Simulation ~40TB (100 members, 1 time level) DA ~40TB (100 members, 1 time level) ~100TB (100 members) Simulated State Sim-to-Obs conversion Sim-minus-Obs ~300TB (100 members, 7 time levels) ~300TB (100 members, 7 time levels) ~1TB Observations ~100TB (100 members) I/O intensive! Repetitions of I/O between separate programs Challenge in global data sharing among weather services

15 Strategy for fast I/O Computational challenge: I/O intensive! Repetitions of I/O between separate programs A strategy: It would be ideal to write files to RAM or fast-access memory device (~1PB required) An experiment: Timing of SPEEDY-model experiments File access Shared drive RAM Wall clock time (min.) Acceleration due to RAM access Using a Linux cluster (4 nodes, 32 cores) 2-month DA cycles Experiments with an intermediate atmospheric model (SPEEDY model) Almost pure computational time

16 How about parallel processing? Member 1 Member 100 Simulation ~0.4TB (1 time level) ~3TB (7 time levels) ~0.4TB Simulation ~3TB Ensemble simulations have ideal parallel efficiency. Sim-to-Obs conversion Sim-to-Obs conversion ~1TB Observations ~1.4TB ~1.4TB DA ~140TB LETKF is parallel efficient, requiring all-to-all comm. only twice.

17 An efficient architectural design Member 1 Member 100 ~0.4TB Simulation ~3TB Sim-to-Obs conversion ~0.4TB Simulation ~3TB Sim-to-Obs conversion Fast communication is important within each cluster; slower inter-cluster communication is acceptable. Cluster 1 Cluster 100 ~140TB DA LETKF requires inter-cluster communications only TWICE.

18 Other challenges of Big DA Transferring Big Data To assimilate Big Data into extreme-scale simulations, we need to collect them in an HPC. Can we apply a cloud approach? Exploring useful data e.g., live camera images may be useful for weather forecasting, but it is hard to collect, qc, and use them Archiving Extreme-scale DA produces at least ~1PB per day.