Short tutorial on data assimilation

Transcription

1 Mitglied der Helmholtz-Gemeinschaft Short tutorial on data assimilation 23 June 2015 Wolfgang Kurtz & Harrie-Jan Hendricks Franssen Institute of Bio- and Geosciences IBG-3 (Agrosphere), Forschungszentrum Jülich GmbH Centre for High-Performance Computing in Terrestrial Systems, Geoverbund ABC/J, Jülich

2 Data assimilation Optimal merging of (uncertain) model predictions with (uncertain) measurements. 23 June 2015 IBG-3: Agrosphere 1

3 Data assimilation Optimal merging of (uncertain) model predictions with (uncertain) measurements. Models are applied in a variety of earth system disciplines: Atmosphere Oceanography Land surface Surface water/ groundwater Glaciology Radiative transfer models Vegetation dynamics/ Biogeochemistry 23 June 2015 IBG-3: Agrosphere 1

4 Why are model predictions uncertain? Model structural errors: Richards equation in land surface models Soil respiration in land surface models: simple black-box concept 23 June 2015 IBG-3: Agrosphere 2

5 Why are model predictions uncertain? Model structural errors: Richards equation in land surface models Soil respiration in land surface models: simple black-box concept Parameter errors: Soil hydraulic parameters like saturated conductivity or porosity Ecosystem parameters like rooting depth 23 June 2015 IBG-3: Agrosphere 2

6 Why are model predictions uncertain? Model structural errors: Richards equation in land surface models Soil respiration in land surface models: simple black-box concept Parameter errors: Soil hydraulic parameters like saturated conductivity or porosity Ecosystem parameters like rooting depth Errors in model forcings: Precipitation Short wave radiation 23 June 2015 IBG-3: Agrosphere 2

7 Why are model predictions uncertain? Model structural errors: Richards equation in land surface models Soil respiration in land surface models: simple black-box concept Parameter errors: Soil hydraulic parameters like saturated conductivity or porosity Ecosystem parameters like rooting depth Errors in model forcings: Precipitation Short wave radiation Errors in initial conditions: Initial soil moisture content Carbon pools 23 June 2015 IBG-3: Agrosphere 2

8 Measurement data Provide information on model states Provide (indirectly) information on parameters and model forcings Data are more valuable if they provide information over larger spatial and temporal scales 23 June 2015 IBG-3: Agrosphere 3

9 Measurement data Provide information on model states Provide (indirectly) information on parameters and model forcings Data are more valuable if they provide information over larger spatial and temporal scales Important limitations Information is always incomplete: not everywhere, not always Random measurement errors (e.g., instrument precision) Systematic measurement errors (e.g., LAI from SMOS) Complicated relationship between what is measured and the quantity of interest (e.g., brightness temperature from SMOS and soil moisture content) 23 June 2015 IBG-3: Agrosphere 3

10 Bayes Law Posterior {}}{ p(x y) = Likelihood {}}{ p(x) p(y x) p(y) }{{} Evidence Prior {}}{ 23 June 2015 IBG-3: Agrosphere 4

11 Bayes Law Posterior {}}{ p(x y) = Likelihood {}}{ p(x) p(y x) p(y) }{{} Evidence Prior {}}{ Often a Gaussian distribution is assumed as prior: p(x) exp ( 1 2 (x µ) C 1 (x µ) ) The likelihood in case of a Gaussian assumption is given by: p(y x) exp ( 1 2 (y Hx) R 1 (y Hx) ) 23 June 2015 IBG-3: Agrosphere 4

12 Bayes Law Posterior {}}{ p(x y) = Likelihood {}}{ p(x) p(y x) p(y) }{{} Evidence Prior {}}{ Often a Gaussian distribution is assumed as prior: p(x) exp ( 1 2 (x µ) C 1 (x µ) ) The likelihood in case of a Gaussian assumption is given by: p(y x) exp ( 1 2 (y Hx) R 1 (y Hx) ) Kalman filter and variational DA follow as solutions 23 June 2015 IBG-3: Agrosphere 4

13 DA methods Inverse modelling / Variational DA Markov Chain Monte Carlo (MCMC) Ensemble Kalman Filter (EnKF) Particle Filter (PF) Ensemble Kalman Smoother (EnKS)/ Particle Smoother (PS) 23 June 2015 IBG-3: Agrosphere 5

14 DA methods Inverse modelling / Variational DA Markov Chain Monte Carlo (MCMC) Ensemble Kalman Filter (EnKF) Particle Filter (PF) Ensemble Kalman Smoother (EnKS)/ Particle Smoother (PS) Differ with respect to e.g.: Temporal treatment of observations Intrinsic assumptions Computational cost Uncertainty quantification 23 June 2015 IBG-3: Agrosphere 5

15 Observations in variational DA/ MCMC t 1 t 2 t 3 t 4 t 5 Measurements are processed in batch to update states/parameters for all time steps. 23 June 2015 IBG-3: Agrosphere 6

16 Observations in filters (EnKF/ PF) t 1 t 2 t 3 t 4 t 5 Incoming measurements are only used to update states/parameters at current time step. 23 June 2015 IBG-3: Agrosphere 7

17 Observations in smoothers (EnKS/ PS) t 1 t 2 t 3 t 4 t 5 Incoming measurements are used to update states/parameters at current and previous time steps. 23 June 2015 IBG-3: Agrosphere 8

18 DA methods Markov Chain Monte Carlo very general, but expensive Inverse modelling/ Variational DA Gaussian assumption, uncertainty estimates relatively poor Particle Filter Markovian assumption (sequential), expensive, uncertainty estimates relatively poor related to filter collapse Ensemble Kalman Filter Marcovian assumption (sequential), Gaussian assumption, efficient, better uncertainty estimates than for gradient based inverse Ensemble Kalman Smoother Gaussian assumption, but data over longer time period 23 June 2015 IBG-3: Agrosphere 9

19 Ensemble Kalman Filter t-1 t t+1 23 June 2015 IBG-3: Agrosphere 10

20 Ensemble Kalman Filter t-1 t t+1 23 June 2015 IBG-3: Agrosphere 10

21 Ensemble Kalman Filter t-1 t t+1 23 June 2015 IBG-3: Agrosphere 10

22 Ensemble Kalman Filter t-1 t t+1 23 June 2015 IBG-3: Agrosphere 10

23 Ensemble Kalman Filter t-1 t t+1 23 June 2015 IBG-3: Agrosphere 10

24 Ensemble Kalman filter - Analysis scheme Prediction equation i = M(x t 1, p i, q i ) + ωi t x t x = model states p = model parameters q = model forcings ω = model errors M = (non-linear) forward model t = time i = model realization i 23 June 2015 IBG-3: Agrosphere 11

25 Ensemble Kalman filter - Analysis scheme Measurement equation ỹ i = Hx i + ɛ i x = model states ỹ = simulation at observation point H = measurement operator ɛ = measurement error i = model realization 23 June 2015 IBG-3: Agrosphere 12

26 Measurement operator Model grid Measurement locations X 1 x 2 x 3 ỹ x 4 ỹ 2 = x 5 ỹ x 6 x 7 x 8 23 June 2015 IBG-3: Agrosphere x 9 13

27 Measurement operator Model grid Measurement locations not at cell centers X 1 x 2 x 3 ỹ x 4 ỹ 2 = x 5 ỹ x 6 x 7 x 8 23 June 2015 IBG-3: Agrosphere x 9 14

28 Measurement operator Model grid One remote sensing measurement X 1 x 2 x 3 ] [ [ỹ1 = 1 ] x x 5 x 6 x 7 x 8 23 June 2015 IBG-3: Agrosphere x 9 15

29 Ensemble Kalman filter - Analysis scheme Updating equation x + i = x t i + K (y ỹ i ) x = model states (predicted) x = model states (updated) K = Kalman gain y = measurement ỹ = simulation at observation point i = model realization 23 June 2015 IBG-3: Agrosphere 16

30 Ensemble Kalman filter - Analysis scheme Kalman gain K = C xỹ (HC xỹ + R) 1 K = Kalman gain C xỹ = covariance matrix of states and simulated measurements R = measurement error covariance matrix 23 June 2015 IBG-3: Agrosphere 17

31 Covariance matrix Model grid Measurement locations C x1 ỹ 1 C x1 ỹ 2 C x1 ỹ 3 C x2 ỹ 1 C x2 ỹ 2 C x2 ỹ 3 C x3 ỹ 1 C x3 ỹ 2 C x3 ỹ 3 C x4 ỹ 1 C x4 ỹ 2 C x4 ỹ 3 C xỹ = C x5 ỹ 1 C x5 ỹ 2 C x5 ỹ 3 C x6 ỹ 1 C x6 ỹ 2 C x6 ỹ 3 C x7 ỹ 1 C x7 ỹ 2 C x7 ỹ 3 C x8 ỹ 1 C x8 ỹ 2 C x8 ỹ 3 C x9 ỹ 1 C x9 ỹ 2 C x9 ỹ 3 23 June 2015 IBG-3: Agrosphere 18

32 Covariance matrix Model grid Measurement locations C x1 ỹ 1 C x1 ỹ 2 C x1 ỹ 3 C x2 ỹ 1 C x2 ỹ 2 C x2 ỹ 3 C x3 ỹ 1 C x3 ỹ 2 C x3 ỹ 3 C x4 ỹ 1 C x4 ỹ 2 C x4 ỹ 3 C xỹ = C x5 ỹ 1 C x5 ỹ 2 C x5 ỹ 3 C x6 ỹ 1 C x6 ỹ 2 C x6 ỹ 3 C x7 ỹ 1 C x7 ỹ 2 C x7 ỹ 3 C x8 ỹ 1 C x8 ỹ 2 C x8 ỹ 3 HC xỹ = Cỹ1 ỹ 1 Cỹ1 ỹ 2 Cỹ1 ỹ 3 Cỹ2 ỹ 1 Cỹ2 ỹ 2 Cỹ2 ỹ 3 Cỹ3 ỹ 1 Cỹ3 ỹ 2 Cỹ3 ỹ 3 C ɛ1 ɛ 1 C ɛ1 ɛ 2 C ɛ1 ɛ 3 R = C ɛ2 ɛ 1 C ɛ2 ɛ 2 C ɛ2 ɛ 3 C ɛ3 ɛ 1 C ɛ3 ɛ 2 C ɛ3 ɛ 3 C x9 ỹ 1 C x9 ỹ 2 C x9 ỹ 3 23 June 2015 IBG-3: Agrosphere 18

33 Kalman gain Kalman gain weigths model prediciton uncertainty and measurement uncertainty. For a scalar (one point): K = σ2 sim σ 2 sim +σ2 obs Model prediction uncertainty estimated by covariance matrix C xỹ (from the ensemble) Kalman gain matrix also determines how a measurement affects the surroundings and corrects surrounding states: Depending on the strength of spatial correlation, a measurement might correct the states in the neighbourhood strongly, or only weakly Spatial correlations depend on model physics, but also correlations of static parameters like land use or soil properties 23 June 2015 IBG-3: Agrosphere 19

34 EnKF update Filter performance dependent on: Relation between model uncertainty and measurement uncertainty Update frequency Ensemble size Amount of observations 23 June 2015 IBG-3: Agrosphere 20

35 Joint state-parameter update Measurement data can also be used to update model parameters jointly with the states 23 June 2015 IBG-3: Agrosphere 21

36 Joint state-parameter update Measurement data can also be used to update model parameters jointly with the states Parameters [ ] are appended to the state vector: s x = p x = state-parameter vector s = state vector p = parameter vector 23 June 2015 IBG-3: Agrosphere 21

37 Joint state-parameter update Measurement data can also be used to update model parameters jointly with the states Parameters [ ] are appended to the state vector: s x = p x = state-parameter vector s = state vector p = parameter vector Covariance matrix C xỹ then also contains covariances between states and parameters 23 June 2015 IBG-3: Agrosphere 21

38 Joint state-parameter update Measurement data can also be used to update model parameters jointly with the states Parameters [ ] are appended to the state vector: s x = p x = state-parameter vector s = state vector p = parameter vector Covariance matrix C xỹ then also contains covariances between states and parameters Measurements are used to update parameters indirectly 23 June 2015 IBG-3: Agrosphere 21

39 Data assimilation software 23 June 2015 IBG-3: Agrosphere 22

40 Data assimilation software Data Assimilation Research Testbed (DART) Parallel Data Assimilation Framework (PDAF) OpenDA 23 June 2015 IBG-3: Agrosphere 22

41 Data assimilation software Data Assimilation Research Testbed (DART) Parallel Data Assimilation Framework (PDAF) OpenDA Differ with respect to e.g.: Implementation strategy (Fortran, Java,...) Available filter methods Model coupling Parallelism Additional utilities (localization, covariance inflation, measurement operators,...) 23 June 2015 IBG-3: Agrosphere 22

42 Coupling to DA software User needs to link model with DA and provide certain functionality Allow ensemble propagation of different model realizations Extract state(-parameter) vector from model output Provide measurements and measurement operators Redirect updated states vector (parameters) to model input for next time step 23 June 2015 IBG-3: Agrosphere 23

43 Coupling to DA software Offline coupling Data transfer between model and DA module via input/output files Requires utilties to read/write the required input/output files Program could be proprietary (no source code needed) Easy to implement Performance degradation due to high I/O 23 June 2015 IBG-3: Agrosphere 24

44 Coupling to DA software Online coupling Data transfer between model and DA module via main memory DA module is wrapped around program Source code required Programming effort depends on model Usually faster than offline coupling 23 June 2015 IBG-3: Agrosphere 25

45 Lorenz 63 system dx dt dy dt dz dt = σ(y x) (1) = x(ρ z) y (2) = xy βz (3) x : convective flow y : horizontal temperature distribution z : vertical temperature distribution σ : viscosity / thermal conductivity ρ : temperature difference top/bottom β : width to height ratio of cell June 2015 IBG-3: Agrosphere 26 X Z Y