Big Ensemble Data Assimilation

October 11, 2018, WWRP PDEF WG, JMA Tokyo Big Ensemble Data Assimilation Takemasa Miyoshi* RIKEN Center for Computational Science *PI and presenting, Takemasa.Miyoshi@riken.jp Data Assimilation Research Team With many thanks to JMA UMD Weather-Chaos group JST CREST Big Data Assimilation project JAXA PMM Ensemble Data Assimilation project Japan s FLAGSHIP 2020 project RIKEN Data Assimilation Research Team

Who am I? http://data-assimilation.riken.jp/~miyoshi/ B.S. from Kyoto U JMA administration (2y) JMA NWP (1.25y) UMD (2y, M.S. and Ph.D.) JMA NWP (3.5y) UMD (4y) RIKEN (~6y) http://tedxsannomiya.com/en/ speakers/takemasa-miyoshi/

RIKEN Center for Computational Science (R-CCS) Japan s flagship institute for computational science Missions: 1) Development & operation of the Japanese flagship supercomputer 2) Center of Excellence for research on computational science

RIKEN Center for Computational Science (R-CCS) Japan s flagship institute for computational science Missions: 1) Development & operation of the Japanese flagship supercomputer 2) Center of Excellence for research on computational science Post-K is under development FLAGSHIP2020 project

A simulated study using the T30/L7 SPEEDY AGCM (Miyoshi, Kondo, Imamura 2014)

Advantage of large ensemble 100 samples (Miyoshi, Kondo, Imamura 2014) 10240 samples Sampling noise reduced High-precision probabilistic representation

20 Histogram, Q, lev=1, 1982/02/01 06Z 640 1.856N, 120.000E 40 1280 80 2560 160 5120 320 10240

20 Histogram, Q, lev=1, 1982/02/01 06Z 640 1.856N, 176.25E 40 1280 80 2560 160 5120 320 10240

20 Skewness, Ps, 1982/02/01 06Z 640 40 1280 80 2560 160 5120 320 10240

20 Kurtosis, Ps, 1982/02/01 06Z 640 40 1280 80 2560 160 5120 320 10240

Non-Gaussianity(KLD), Ps, 1982/02/01 06Z 20 640 40 1280 80 2560 160 5120 320 10240

Non-Gaussianity(KLD), Ps, 1982/02/01 06Z 20 640 40 1280 80 160 2560 >1000 members necessary for capturing Non-Gaussianity 5120 320 10240

RMSE Non-Gaussianity Surface-pressure RMSE (hpa) 20 (Kondo, Miyoshi 2016) Non-Gaussianity based on 10240 members Skewness Kurtosis 80 320 Frequency of Non-Gaussianity 10240 [%] Larger errors Non-Gaussian regions

A real-world study using the NICAM (Miyoshi, Kondo, Terasaki 2015) NICAM-LETKF (Terasaki et al. 2015)

Correlation patterns (Q at ~100 hpa) 40 members Kondo, Miyoshi (2015) Localized (σ=400km) This is what we use for EnKF with 40 members. 11/8 00UTC after a week cycling

Correlation patterns (Q at ~100 hpa) Kondo, Miyoshi (2015) 40 members 10240 members 11/8 00UTC after a week cycling

Correlation patterns (Q at ~100 hpa) Kondo, Miyoshi (2015) 40 members 10240 members FLOW-DEPENDENT 11/8 00UTC after a week cycling

With subsets of 10240 samples 20 640 Kondo&Miyoshi (2015) 40 1280 80 2560 160 5120 320 10240

To improve data assimilation 1-day fcst error = 5-day fcst error 80 members 10240 members w/o localization Implications to vertical localization for satellite data

Cover feature!

Only in 10 minutes!! (Courtesy of NICT) 17:30:16 17:32:16 17:34:16 17:36:16 17:38:16 17:40:16 17:42:16 17:44:16 10 km (height) 26

Phased Array Weather Radar (PAWR) 3-dim measurement using a parabolic antenna (150 m, 15 EL angles in 5 min) 100x more data! 10x more data in a 1/10 period 3-dim measurement using a phased array antenna (100 m, 100 EL angles in 30 sec) 27

Phased Array Radar (every 30 sec.) (Courtesy of NICT)

Pioneering Big Data Assimilation Era High-precision Simulations Future-generation technologies available 10 years in advance High-precision observations Mutual feedback

Revolutionary super-rapid 30-sec. cycle Obs data processing ~2GB Obs data processing ~2GB DA (4.5PFLOP) 380GB 3GB 30-sec. Ensemble forecasting (2.6PFLOP) 2.5TB DA (4.5PFLOP) 30-min. forecasting (1.6PFLOP) 380GB 3GB 30-sec. Ensemble forecasting (2.6PFLOP) 2.5TB D (4.5P 30-min. forecasting (1.6PFL -10 0 10 20 30 40 Time (sec.) 120 times more rapid than hourly update cycles

9/11/2014 morning, sudden rain 8:00 8:05 8:10 8:15 8:20 8:25 8:30 8:35 8:40 8:45 8:50 8:55

9/11/2014, sudden local rain

9/11/2014, sudden local rain >40,000 views #9 of RIKEN channel

9/11/2014, sudden local rain

1-km-mesh, 1000-member LETKF T skewness at z=3845 m (Ruiz et al. in prep.) skewness Even 30-second update shows strong non-gaussianity with 1000 members. contours: 30 dbz reflectivity

What do we expect with rapid updates? based on Lorenz-model exp. (Teramura&Miyoshi 2016) Obs interval = 0.08 Obs interval = 0.25 Obs interval = 0.50 Κ 4 of PCA1 Κ 4 of PCA1 Κ 4 of PCA1 Light tailed Heavy tailed Κ 4 : 4th order cumulant kurtosis Scatter diag. Scatter diag. Scatter diag.

What do we expect with rapid updates? based on Lorenz-model exp. (Teramura&Miyoshi 2016) Obs interval = 0.08 Obs interval = 0.25 Obs interval = 0.50 Κ 4 of PCA1 Κ 4 of PCA1 Κ 4 of PCA1 Heavy tailed Frequent obsmore Gaussian Κ 4 : 4th order cumulant kurtosis Light tailed Scatter diag. Scatter diag. Scatter diag.

1-km-mesh, 1000-member LETKF T skewness at z=3845 m (Ruiz et al. in prep.) skewness Even 30-second update shows strong non-gaussianity with 1000 members. 30-sec. update may not be fast enough! contours: 30 dbz reflectivity

Non-Gaussianity and data assimilation frequency Comparison of KLD for different assimilation frequencies At 05:15 UTC (15 minutes after the end of the spin-up) w/ 1000 members, 1-km mesh T at 600 hpa 5 min DA T at 600 hpa 2 min DA T at 600 hpa 1 min DA Contours: 30 dbz T at 600 hpa 30 sec DA (Ruiz et al. in prep.)

Non-Gaussianity and data assimilation frequency Comparison of KLD for different assimilation frequencies. T 5 min - rain T 2 min - rain w/ 1000 members, 1-km mesh T 1 min - rain T 30 sec - rain Averaged area > 30 dbz (Ruiz et al. in prep.) more Gaussian with faster cycles

(Ruiz et al. in prep.)

30-min forecast: 15:10L 15:40L D4_1KM (deterministic) OBS after QC 30-sec DA cycle D4_1KM (deterministic) Lien et al. (in prep.) 5-min DA cycle

30-min forecast: 15:40L 16:10L D4_1KM (deterministic) OBS after QC 30-sec DA cycle 30-sec. update certainly helps. D4_1KM (deterministic) Lien et al. (in prep.) 5-min DA cycle

20-min forecast: 15:30L OBS after QC 30 sec Lien et al. (in prep.) 5 min (4D) 5 min (1/10 data)

20-min forecast: 15:30L OBS after QC 30 sec Lien et al. (in prep.) 2 min (4D) 5 min (4D) 5 min (1/10 data) 2 min (1/4 data) 5 min (1/10 data)

20-min forecast: 15:30L OBS after QC 30 sec Lien et al. (in prep.) 1 min (4D) 2 min (4D) 5 min (4D) 5 min (1/10 data) 1 min (1/2 data) 2 min (1/4 data) 5 min (1/10 data)

Meteorological Satellite Center (MSC) of JMA Himawari-8: a new generation geostationary meteorological satellite frequent, colorful, precise ~50x more data Every hour (30 min in NH) Every 10 min. 16UTC 2 to 13UTC 3 April 2015 MTSAT-2 (VIS) Every 1 hour 16UTC 2 to 13UTC 3 April 2015 Himawari-8 (True Color) Every 10 minutes (Courtesy of JMA)

Typhoon Soudelor (2015) The strongest western north Pacific typhoon in 2015 captured well by Himawari-8 Japan 8/4 900hPa

Himawari-8 impact NoHim8 Him8 Observation Brightness Temperature (K)

Himawari-8 impact on intensity fcst weak strong

Every 10 min. vs. 30 min. DA 30 min. cycle 10 min. cycle Observation Brightness Temperature (K)

Intensity forecast (30 min. vs. 10 min.) weak strong Assimilating every 10-min. is essential.

Moisture intensity The inner-core moisture error causes the intensity forecast error (Emanuel and Zhang 2017). More moist Less uncertain Him8 DA reduces a dry bias and ensemble spread (uncertainty) of moisture.

An idea of merging two scales Motivated by Buehner (2012), we construct analysis increments at high (h) and low (l) resolutions separately. = + Miyoshi and Kondo (2013)

Results are promising. Experiments with the T30L7 SPEEDY model (Molteni, 2003) Global-average RMSE Regular localization (700 km) Dual localization (600-900 km) Mid-level U Low-level T Near-surface Q Surface pressure

Pushing the limits Big Data Big Simulations Big ensemble (10240 ensemble members) Rapid update (30-second update) High resolution (100-m mesh) Future NWP

Summary Scales Predictable range Update frequency