Introduction to ensemble forecasting. Eric J. Kostelich

Transcription

1 Introduction to ensemble forecasting Eric J. Kostelich SCHOOL OF MATHEMATICS AND STATISTICS MSRI Climate Change Summer School July 21, 2008

2 Co-workers: Istvan Szunyogh, Brian Hunt, Edward Ott, Eugenia Kalnay, Jim Yorke and many others! Thanks to: Dave Kuhl Papers, preprints, and codes: eric MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 2 / 32

3 Principal papers Preprints: Initial papers: E. Ott et al., Tellus A 56 (2004), I. Szunyogh et al., Tellus A 57 (2005), Refined mathematical implementation: B. R. Hunt, E. K., I. Szunyogh, Physica D 230 (2007) Results with real data: I. Szunyogh, E.K. et al., Tellus A 60 (2008) MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 3 / 32

4 Recap from last time In a chaotic process, every point is sensitive Uncertainties in initial conditions grow exponentially (at least for awhile) The weather is chaotic (as far as anyone can tell) The uncertainty in the global weather vector roughly doubles every 2 days Forecast horizon: about 2 weeks MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 4 / 32

5 Relevant U. S. organizations The National Oceanographic and Atmospheric Administration (NOAA) is a division of the Department of Commerce The National Centers for Environmental Prediction (NCEP) is the division of NOAA responsible for developing and maintaining weather forecast models Spectrum of models: Global Forecast System (GFS), Regional Spectral Model (RSM), etc. Model data is distributed to local Weather Service offices, which generate public forecast products MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 5 / 32

6 Other important modeling efforts NASA develops and maintains its own forecast models International agreements to share forecasts and observations (NCEP, UK Met Office, ECMWF, Canada, Japan, Brazil, etc.) Research community: Weather Research and Forecasting model (WRF) NOAA and the U. S. Navy develop and maintain ocean models Private sector efforts: AccuWeather, airlines, etc. MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 6 / 32

7 What do we want to predict? The best long-term forecast is climatology (the mean is the maximum likelihood estimate) Prior to the mid 1960s, the starting initial condition was climatology The U. S. Weather Service defines normal as the average Example: in Phoenix, Arizona, tomorrow s weather will be sunny with 96% probability Exceptional weather often is of greatest interest MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 7 / 32

8 What is data assimilation? The process by which empirical measurements are incorporated into a forecast model to refine an estimate of the initial condition The distinction between variables and parameters is a matter of definition Operational weather forecast centers perform data assimilation steps 4 times per day (0Z, 6Z, 12Z, 18Z) Real-time constraints: NCEP allows 20 minutes MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 8 / 32

9 Measures of forecast quality One objective measure of goodness: forecast observations A 72-hour forecast today is as accurate as a 36-hour forecast in 1985 Holy grail: 7-day forecasts that are as accurate as 3-day forecasts are now MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 9 / 32

10 Many applications besides weather Controls (e.g., airplane autopilots) Ocean and climate models (obviously) Biological models (e.g., Tim Sauer & Steve Schiff) Parameter estimation MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 10 / 32

11 Some fundamental problems Naive approach: direct insertion Difficulty: there are usually many more grid points than available measurements Does not account for errors in the measurement Does not exploit correlations between nearby grid points The variables in the model are not necessarily the ones that can be easily measured MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 11 / 32

12 Example: Global Forecast System Principal variables in the GFS: natural logarithm of surface pressure virtual temperature divergence and vorticity of the wind field Principal measurements: barometric pressure sensible temperature relative humidity wind speed and direction satellite radiances (complicated!) MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 12 / 32

13 Typical 6-hour land surface dataset: 31,310 locations MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 13 / 32

14 Typical 6-hour surface marine dataset: 2,642 locations MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 14 / 32

15 Typical 6-hour satellite dataset: 53,842 locations MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 15 / 32

16 The observation space For these reasons, data assimilation is done in the observation space Given a vector of observations y, interpolate the model state x to the same locations The interpolation operator is denoted H The innovation is y H(x) MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 16 / 32

17 Basic idea: Weighted least squares Observations: y R p, y = H(x t ) + ε Observation errors: E(ε) = 0, E(εε T ) = R Model forecast ( background ): x R n, x b = x t + η Model errors: E(η) = 0, E(ηη T ) = P b Goal: minimize the objective function J(x) = [y H(x)] T R 1 [y H(x)]+(x x b ) T P 1 b (x x b) Minimization produces an analysis x a with associated covariance P a MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 17 / 32

18 Simplest assumptions The observation errors ε are normally distributed with mean 0 and covariance R Model errors similarly: N(0,P b ) When the underlying model is linear, it can be shown the the minimizer x a of J is unique, unbiased and has minimum variance among all linear estimators Weather models are linear enough over 6-hour intervals, but there is no guarantee of optimality MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 18 / 32

19 The dimensionality problem Must evaluate J(x) = [y H(x)] T R 1 [y H(x)]+(x x b ) T P 1 b (x x b) where y R p, x R n Current NCEP operations: p 1.75 million and n 3 billion We need R 1 (p p) and P 1 b (n n) MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 19 / 32

20 The computational complexity problem Inversion of a k k matrix is an O(k 3 ) algorithm If a matrix takes 1 sec to invert, then a matrix takes sec R is nearly diagonal if observation errors are mostly uncorrelated P b is not diagonal Computing P b (t + t) from P a (t) requires integration of the tangent linear model MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 20 / 32

21 Complexity reduction strategies Localization: Try to do the minimzation over smaller regions of the globe Estimate and precompute P 1 b : Assume that the forecast uncertainty is approximately constant from one day to the next. (Used in all current operational DA systems) Thin the observations and use only the most important ones MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 21 / 32

22 Each strategy has drawbacks Assuming P b constant ignores the errors of the day Generally regarded as one of the key impediments to better forecasts The result of sequential assimilation of observations depends on the order of processing Must assure continuity at the boundaries of the smaller regions MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 22 / 32

23 The Local Ensemble Transform Kalman Filter (LETKF) Addresses many of these problems Exploits the geometry of uncertainty in chaotic processes to lower the dimension but still account for errors of the day Assimilates all the data at once Uses localization and sets of observations that vary slowly in space to help assure continuity Permits efficient implementation on massively parallel computers MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 23 / 32

24 The geometry of forecast uncertainty The size of a typical high- or low-pressure system is about 1000 km 1000 km ( Texas) The GFS, when run at medium (T62) resolution, contains about 3000 grid-point variables in Texas-sized regions Suppose we run k statistically equivalent forecasts What are the singular values of the resulting 3000 k forecast matrix X F? MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 24 / 32

25 Correlation and dimensionality Over most Texas-sized regions, one solution looks much like another The columns of X F tend to be highly correlated MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 25 / 32

26 Correlation and dimensionality Over most Texas-sized regions, one solution looks much like another The columns of X F tend to be highly correlated so the SVD of X F yields a good rank-r approximation even when r k MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 25 / 32

27 Key empirical finding This was a key finding by D. J. Patil et al. PRL 86 (2001), GFS at T62 resolution: 3000 grid variables over typical Texas-sized region Typical ensemble of 100 k 200 forecasts generates a 3000 k forecast matrix X F whose first r singular vectors, 40 r 80, yield an excellent approximation of the forecast uncertainty MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 26 / 32

28 The ensemble dimension The ensemble dimension (E-dimension) of an n k matrix is E (s 1 + s s k ) 2 s s s2 k Measures the eccentricity of the ellipse of forecast uncertainty MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 27 / 32

29 Example: s 1 = 3.78, s 2 = 3.60, E dim = MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 28 / 32

30 Example: s 1 = 19.24, s 2 = 4.35, E dim = MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 29 / 32

31 Example: s 1 = 83.65, s 2 = 4.33, E dim = MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 30 / 32

32 The key idea behind the LETKF If the E-dimension is much less than the dimension of the overall space, then the distribution is flat The ensemble forecast uncertainty over a typical synoptic region resembles a pancake (at least for short intervals) Reduce the dimensionality of the problem by changing coordinates to the r-dimensional subspace containing most of the forecast uncertainty The dynamics reduces the uncertainty in the remaining directions MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 31 / 32

33 Next lecture Outline of the Kalman filter Mathematical details of how we accomplish the dimension reduction Results with operational models and real observations MSRI Lecture #2 E. Kostelich MATHEMATICS AND STATISTICS 32 / 32