SPECTRAL AND MORPHING ENSEMBLE KALMAN FILTERS AND APPLICATIONS

Transcription

1 SPECTRAL AND MORPHING ENSEMBLE KALMAN FILTERS AND APPLICATIONS Jan Mandel, Jonathan D. Beezley, Loren Cobb, Ashok Krishnamurthy, University of Colorado Denver Adam K. Kochanski University of Utah Krystof Eben, Pavel Jurus, and Jaroslav Resler Czech Academy of Sciences 31st International Symposium on Forecasting, Praha, June 29, 2011

2 ACKNOWLEDGEMENTS This work was partially supported by the U.S. National Science Foundation grant AGS , National Institutes of Health grant 1RC1 LM , National Institute of Standards and Technology Fire Research grant 60NANB7D6144, and the Academy of Sciences of the Czech Republic project M A part of this work was done during visits of Jan Mandel and Jonathan Beezley at the Department of Nonlinear Modeling, Institute for Computer Science, Czech Academy of Sciences. The hospitality of the Institute during those visits is gratefully acknowledged.

3 EXAMPLE 1: WILDFIRE SIMULATION Simulation of FireFlux experiment (Clements et al., 2007) with WRF 3.3 and SFIRE Simulation setup by Adam Kochanski, visualization in VAPOR by Bedrich Sousedik.

4 EXAMPLE 2: AN EPIDEMIC IN SWEDISTAN Pandemic influenza in a population with no immunity. Human Development Index set to HDI = 0.30 (similar to Congo). No air traffic, dirt roads between cities. Initial epidemic has two epicenters. Entire country has only one IDP camp for the ill and displaced (homeless). No public health efforts except near the IDP camp. Author: Loren Cobb Funding: Nat l Defence College of Sweden and the US Joint Staff (J8 Directorate) Project: STRATMAS II Date: 2001

5 EXAMPLE 3: RAIN FORECASTING Project Medard, Department of Nonlinear Modeling, Institute for Informatics, Czech Academy of Sciences,

6 WHAT DO THESE THREE PROBLEMS HAVE IN COMMON? Strongly nonlinear Solutions exhibit coherent, moving features: fire line, epidemic wave, weather front Data assimilation: ingesting data while the simulation is running standard Bayesian amplitude corrections often give nonphysical solutions position corrections needed in addition to amplitude Also, in wildfire and epidemic, statistical variability causes secondary fires and epidemics, which keep growing, do not dissipate

7 DATA ASSIMILATION The state-space model: the model state is a probability density p(u) Simulation updates the probability distribution by mapping it forward in time. Data given with an error estimate, as data likelihood p(u d) : the probability density of the state u given data d Bayesian update of the state: panalysis(u) = const p(u d)p(u) If all is Gaussian, in particular p(u d) = const exp(-(d-hu) T R -1 (d-hu)/2), we get new state mean Uanalysis = U + QH T (HQH T +R) -1 (d-hu) This formula is also used in the non-gaussian case anyway (also justified as least squares or maximum likelihood) Now the game is to estimate the state covariance Q

8 COVARIANCE ESTIMATION The classical Kalman filter evolves the state covariance Q assuming the model is linear. Optimal statistical interpolation (OSI) takes a human guess for the covariance Q Ensemble Kalman filter (EnKF) is a confluence of two techniques: Ensemble forecasting a.k.a. scenarios - the ensemble members are independent simulations: if it rains in 50% of the members we say that the probability of rain is 50% Estimation of the state covariance Q from the ensemble (e.g., by sample covariance) But this gives the EnKF analysis covariance too small. Fix: randomly perturb the data, or deterministically move of ensemble members to ensure correct sample covariance (adjustment EnKF, square root EnKF,...)

9 SOME OF THE CHALLENGES Large state vector (easily, in GB), large ensembles needed (often, 100s) Covariance estimates the state consists of several random fields on a physical domain: the covariance function between locations drops off with their distance (sample covariance with a small sample does not; it needs tapering) but the covariance function also changes with the location in the domain; it needs localization Non-gaussian probability distributions - the Holy Grail of data assimilation here, the locations of coherent features may have approximately gaussian distribution but the value of the field at a fixed location surely does not

10 SO WHAT IS NEW AND DIFFERENT IN THIS TALK? Optimal statistical interpolation by Fast Fourier Transform (FFT) - needs just a laptop instead of a supercomputer. Automatic tapering. Morphing - position corrections, not just amplitude, makes distributions close to gaussian. FFT EnKF - approximate the covariance by the diagonal of the sample covariance in the frequency space. Then, small ensembles are enough. Wavelet EnKF - provides also an automatic localization.

11 COVARIANCE ESTIMATION BY THE FFT APPROXIMATION BY THE DIAGONAL IN THE FREQUENCY SPACE Given covariance Ensemble of 5 random functions Sample covariance FFT estimation

12 MORPHING Moving coherent features: need also position correction Replace the state by an extended state: deformation of a reference field + a residual by automatic registration: multiscale optimization, also related to advection field found in radar analysis run the EnKF on the extended states: probability distributions closer to gaussian recover ensemble members from the deformation and the residual fields basically, replace linear combinations by morphs: Intermediate states in the middle created automatically from a linear combination of deformation fields and residual fields tricky: the right kind of combination to avoid ghosting

13 OPTIMAL STATISTICAL INTERPOLATION BY FFT Use negative power of Laplace operator for the covariance Green s function drops off from the diagonal, cheap FFT implementation Combined with morphing, example: rain fields Forecast Data Standard analysis Analysis with morphing Duplicate storm center

14 OPTIMAL STATISTICAL INTERPOLATION WITH MORPHING AND MISSING RADAR DATA Forecast OPERA radar data Analysis The storm line direction as well as the overall precipitation intensity are interpolated between the forecast and the data

15 MORPHING ENKF FOR A WILDFIRE SIMULATION Forecast Truth Analysis - standard EnKF breaks down Analysis - morphing EnKF OK

16 COVARIANCE ESTIMATION BY WAVELETS Wavelets are basis functions localized both in space. We use orthogonal wavelets; wavelet transform of a function consists of the coefficients of its expansion in a wavelet basis. Wavelet EnKF uses diagonal approximation of covariance: Apply the wavelet transform to ensemble members Compute the diagonal of the sample covariance in the wavelet space Apply the inverse wavelet transform Diagonal approximation of covariance in frequency and wavelet space has been justified for weather fields in other data assimilation contexts (Deckmyn and Berre 2005,..) An example of a wavelet: Coiflet 25

17 ORTHOGONAL WAVELET BASIS Wavelet value Wavelet number Obtained by shift and argument scaling of a single function, the mother wavelet Organized in octaves (the argument scaling): multiresolution structure

18 Position correction by morphing EnKF Data assimilation by FFT and wavelet transforms Optimal statistical interpolation by FFT Spectral EnKF by FFT and wavelets FFT AND WAVELET COVARIANCE ESTIMATIONS, 2 VARIABLES Covariance estimation, 2 variables Covariance, sample of 1000 Variable 1, sample of 5 Variable 2, sample of 5 Covariance, sample of 5 FFT estimation, sample of 5 Wavelet estimation, sample of 5 Estimation by FFT results in a distribution that is homogeneous in space, smearing the distribution across the domain. Wavelet estimation keeps the spatial structure, while filtering out spurious long-distance correlations. Jan Mandel, Jonathan D. Beezley, and Loren Cobb Spectral and morphing ensemble Kalman filters

19 DATA ASSIMILATION FOR EPIDEMIC SPREAD The standard S-I-R model (susceptible, infected, recovered) with a spatial diffusion added to model infection spread in space Key: the probability distribution of cases in an epidemic distribution is Poisson. Use the Poisson distribution, not gaussian, also to randomize the data in the EnKF, then all falls in place. Go back to the EnKF roots: statistical ensemble for the forecast and uncertainty estimation, but use the assumed covariance i.e., optimal statistical estimation

20 Bayesian Tracking using OSI Time 10 Truth Forecast Analysis

21 Bayesian Tracking using OSI Time 20 Truth Forecast Analysis

22 Bayesian Tracking using OSI Time 30 Truth Forecast Analysis

23 Bayesian Tracking using OSI Time 40 Truth Forecast Analysis

24 Bayesian Tracking using OSI Time 50 Truth Forecast Analysis

25 CONCLUSION Morphing and spectral ensemble filters have the potential to improve data assimilation in problems with moving coherent features, such as firelines in a wildfire, epidemic waves, and rain fronts. Encouraging preliminary results for optimal statistical interpolation (basically, assumed covariance) and estimation from small ensembles (well under 10), compared to 100s for standard ensemble filters. Spectral filters provide automatic tapering (essentially, an estimation of the covariance distance), and localization (estimation how the covariance distance changes from place to place). In progress, and future plans: estimate the cross-covariances for the fire and the rain applications time series data at a point morphing for the epidemics operational data assimilations systems

26 THANK YOU FOR YOUR ATTENTION!

27 REFERENCES Clark, T. L., Coen, J. L., Latham, D., 2004: Description of a coupled atmosphere-fire model. International Journal of Wildland Fire, 13, Clements, C. B., Zhong, S., Goodrick, S., Li, J., Potter, B. E., Bian, X., Heilman, W. E., Charney, J. J., Perna, R., Jang, M., Lee, D., Patel, M., Street, S., and Aumann, G.: Observing the dynamics of wildland grass fires FireFlux A field validation experiment, Bull. Amer. Meteorol. Soc., 88, Deckmyn, A. and L. Berre, 2005: A wavelet approach to representing background error covariances in a limited-area model. Monthly Weather Review, 133, Nina Dobrinkova, Georgi Jordanov, and Jan Mandel, WRF-Fire Applied in Bulgaria, Numerical Methods and Applications, Ivan Dimov, Stefka Dimova, and Natalia Kolkovska, eds., vol of Lecture Notes in Computer Science, Springer, Berlin/Heidelberg, 2011, pp G. Evensen. Data assimilation: The ensemble Kalman filter. Springer, Berlin, 2007 Georgi Jordanov, Jonathan D. Beezley, Nina Dobrinkova, Adam K. Kochanski, Jan Mandel, and Bedrich Sousedik, Simulation of the 2009 Harmanli fire, 8th International Conference on Large-Scale Scientific Computation, June 2011, Sozopol, Bulgaria. Lecture Notes in Computer Science, to appear. Ashok Krishnamurthy, Loren Cobb, Jan Mandel, and Jonathan D. Beezley, Bayesian Tracking of Emerging Epidemics Using Optimal Statistical Interpolation, 2010 Joint Statistical Meetings, Vancouver, Canada, July 31- August 5, 2010, Jonathan D. Beezley, Jan Mandel, and Loren Cobb, Wavelet Ensemble Kalman Filters, IEEE IDAACS Proceedings, 2011, accepted. Jan Mandel, Jonathan D. Beezley, Loren Cobb, and Ashok Krishnamurthy, Data Driven Computing by the Morphing Fast Fourier Transform Ensemble Kalman Filter in Epidemic Spread Simulations, Procedia Computer Science, vol 1, , Jan Mandel, Jonathan D. Beezley, Janice L. Coen, and Minjeong Kim, Data Assimilation for Wildland Fires: Ensemble Kalman filters in coupled atmosphere-surface models, IEEE Control Systems Magazine 29, Issue 3, June 2009, Jan Mandel, Jonathan D. Beezley, and Adam K. Kochanski, Coupled atmosphere-wildland fire modeling with WRF-Fire version 3.3, Geoscientific Model Development Discussions (GMDD) 4, , 2011 Edward G. Patton and Janice L. Coen, WRF-Fire: A Coupled Atmosphere-Fire Module for WRF, Preprints of Joint MM5/Weather Research and Forecasting Model Users' Workshop, Boulder, CO, June 22-25, 2004, pp , NCAR.