Building Ensemble-Based Data Assimilation Systems. for High-Dimensional Models

Transcription

1 47th International Liège Colloquium, Liège, Belgium, 4 8 May 2015 Building Ensemble-Based Data Assimilation Systems for High-Dimensional s Lars Nerger, Paul Kirchgessner Alfred Wegener Institute for Polar and Marine Research Bremerhaven, Germany

2 3 Components of an Assimilation System initialization time integration post processing mesh data/coordinates Observations obs. vector obs. operator obs. error state time DA method initialization analysis step ensemble transformation state observations Nerger, L., Hiller, W. Software for Ensemble-based DA Systems Implementation Lars Nerger, and Scalability. Paul Kirchgessner Computers and Building Geosciences Ensemble 55 DA (2013) Systems

3 Ensemble-based Kalman Filter Kalman filter: express probability distributions by mean and covariance matrix EnKF (Evensen, 1994): Use ensembles to represent probability distributions state estimate ensemble forecast initial sampling forecast analysis ensemble transformation observation time 0 time 1 time 2

4 Offline Coupling Separate Programs error Start Initialize generate mesh Initialize fields aaaaaaaaa Do i=1, nsteps Time stepper consider BC Consider forcing + Simple to implement - Inefficient: file reading/writing model restarts Assimilation program Start read ensemble files analysis step aaaaaaaa write model restart files a generic Post-processing Stop Stop For each ensemble state Initialize from restart files Integrate Write restart files Read restart files (ensemble) Compute analysis step Write new restart files

5 Ensemble Filter Analysis Step Ensemble of state vectors X Analysis operates on state vectors (all fields in one vector) Filter analysis 1. update mean state 2. ensemble transformation For localization: Local ensemble Local observations Vector of observations y Observation operator H(...) Observation error covariance matrix R

6 Online Coupling Single program initialization time integration post processing mesh data/coordinates Observations obs. vector obs. operator obs. error state time Generic Assimilation Core DA method initialization analysis step ensemble transformation state observations Explicit interface Indirect exchange (module/common)

7 Extending a for Data Assimilation Start Initialize init_parallel_da Start generate mesh Initialize fields Initialize generate mesh aaaaaaaaa Do i=1, nsteps aaaaaaaaa generate mesh Initialize fields Time stepper consider BC Consider forcing Post-processing Stop Initialize fields aaaaaaaaa Do i=1, nsteps Time stepper Time stepper consider BC consider BC Consider forcing Consider forcing Post-processing Start Initialize Init_DA Do i=1, nsteps Assimilate Extension for data assimilation plus: Possible model-specific adaption. ensemble forecast enabled by parallelization Stop Post-processing Stop NEMO: Euler time step after assimilation

8 2-level Parallelism Forecast Analysis Forecast Task 1 Filter Task 1 Task 2 Task 2 Task 3 Task 3 1. Multiple concurrent model tasks 2. Each model task can be parallelized! Analysis step is also parallelized! Configured by MPI Communicators

9 Framework Solution with Generic Filter Implementation Start init_parallel_da Initialize aaaaaaaaa Init_DA Do i=1, nsteps Time stepper Assimilate Post-processing Stop Subroutine calls or parallel communication Generic DA_Init Set parameters Initialize ensemble DA Error DA_Analysis Check time step Perform analysis Write results Dependent on model and observations Read ensemble from files Initialize state vector from model fields Initialize vector of observations Apply observation vector to a state vector multiply R-matrix with a matrix with assimilation extension Core-routines of assimilation framework Case specific callback routines

10 Assimilation Example with NEMO configuration medium size SANGOMA benchmark box-configuration SQB (SEABASS) wind-driven double gyre 1/12 o resolution 361x241 grid points, 11 layers Latitide (degree) Sea surface height Longitude (degree)

11 PDAF: A tool for data assimilation PDAF - Parallel Data Assimilation Framework " provide support for parallel ensemble forecasts " provide fully-implemented filter and smoother algorithms " makes good use of supercomputers (Fortran with MPI & OpenMP parallelization) " easily useable with (probably) any numerical model (coupled e.g. to NEMO, MITgcm, HBM, ADCIRC, FESOM) " allows for separate development of model and assimilation algorithms Open source: Code and documentation available at L. Nerger, Lars Nerger, W. Hiller, Paul Computers Kirchgessner & Geosciences Building Ensemble 55 (2013) DA Systems

12 Minimal changes to NEMO Add to mynode (lin_mpp.f90) just before init of myrank #ifdef key_use_pdaf CALL init_parallel_pdaf(0, 1, mpi_comm_opa) #endif Add to nemo_init (nemogcm.f90) at end of routine CALL init_pdaf() Add to stp (step.f90) at end of routine CALL assimilate_pdaf() For Euler time step after analysis step: Modify dyn_nxt (dynnxt.f90) Start init_parallel_pdaf Initialize generate mesh aaaaaaaaa Initialize fields init_pdaf Do i=1, nsteps Time stepper consider BC Consider forcing assimilate_pdaf Post-processing #ifdef key_use_pdaf IF((neuler==0.AND. kt==nit000).or. assimilate) #else Stop

13 Assimilation Example with NEMO - Observations SSH observations (m) Observations twin experiment Simulated satellite SSH (Envisat & Jason-1 tracks), 5cm error Temperature profiles on 3 o x3 o grid, 0.3 o C error Latitide (degree) Longitude (degree) Ensemble data assimilation Local ESTKF Assimilate each 48h Case-specific routines utilize mesh information from Fortran modules of NEMO Temperature observations upper layer (degc) Latitide (degree) Longitude (degree)

14 Parallel Performance Speedup of NEMO-PDAF SEABASS 1/12 o assimilation Ensemble size 32 State dimension ~ Speedup Parallel speedup of NEMO and assimilation NEMO NEMO+PDAF Ideal Speedup Speedup determined by speedup of NEMO Almost same speedup with assimilation Analysis step takes < 8% of total time (0.9s for largest case) Processor cores per model task Total number of processor cores

15 Very big test case Simulate a model Choose an ensemble state vector per processor: 10 7 observations per processor: Ensemble size: 25 2GB memory per processor time for analysis step [s] Timing of global SEIK analysis step N=50 N=25 Apply analysis step for different processor numbers " " Increase total state and obs. accordingly Very small increase in analysis time (~1%) processor cores State dimension: 1.2e11 Observation dimension: 2.4e9 Didn t try to run a real ensemble of largest state size (no model yet)

16 Summary Online coupling more efficient than offline coupling Generic model interface for online ensemble data assimilation Minimal changes to model code Parallelization allows for ensemble forecasts Data assimilation framework PDAF ( supports high-dimensional models Coding you own Ensemble Kalman filter usually not necessary

17 References Nerger, L., Hiller, W. (2013). Software for Ensemble-based Data Assimilation Systems - Implementation Strategies and Scalability. Computers and Geosciences, 55, Nerger, L., Hiller, W., Schröter, J. (2005). PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering, Use of high performance computing in meteorology : proceedings of the Eleventh ECMWF Workshop on the Use of High Performance Computing in Meteorology, Reading, UK, October 2004 / Eds.: Walter Zwieflhofer; George Mozdzynski, Singapore: World Scientific, 63-83