Der SPP 1167-PQP und die stochastische Wettervorhersage Andreas Hense 9. November 2007
Overview The priority program SPP1167: mission and structure The stochastic weather forecasting Introduction, Probabilities and the Bayes-Theorem Stochastic dynamics of weather (and climate) Modelling stochastic dynamics Verification and Calibration Assimilation Examples 1
History: first round table discussions 2000 and 2001 submission preproposal November 2001 withdrawal February 2002 Elbe flooding Summer 2002 finalization of proposal end of 2002 submission in February 2003 2
approval by the senate of DFG in May 2003 for 6 years submission of individual proposals Oct. 2003, approval in January 2004 kick-off meeting in April 2004 second phase since April 2006, now halftime submission of last phase (2008-10) projects October 31st, 2007; review January 10th,2008
The goals from the initial proposal: Identification of physical and chemical processes responsible for the deficiencies in quantitative precipitation forecast Determination and use of the potentials of existing and new data and process descriptions to improve quantitative precipitation forecast Determination of predictability of weather forecast models by statisticodynamic analyses with respect to quantitative precipitation forecast 3
These goals are related to the following tasks Understanding of atmospheric processes relevant to precipitation formation has to be improved considerably. These processes have to be modeled in a more realistic manner. Initial distribution of the atmospheric water content in the three phases of vapor, liquid, and ice has to be improved by data that have not been used so far and new data. Their potential for quantitative precipitation forecast has to be quantified. 4
Methods of assimilating measurement data into atmospheric simulation models and application of these methods to data of any type for a statistico-dynamically consistent retrieval of the initial water distribution and, hence, optimum use of all data have to be improved. The stochastic character of precipitation has to be taken into account when using observational data, interpreting deterministic simulations, and developing alternative forecast strategies. 5
6
International Sister projects IHOP in USA CSIP in UK Quest-B in Belgium COPS-France MeteoSwiss/ETH Z urich University Vienna 7
Why did we include the stochastic aspect? for practical reasons to treat the physics of atmospheric dynamics correctly 8
Probabilities, probability densities and all those things.. o Observation space-time vector f model state space-time vector forecasts [ f, o] joint probability (density) of observations and model state [ o f] conditional probability (density) of observations given the model state 9
Why is the atmosphere a stochastic system? it is high-dimensional it is non linear 10
High - dimensionality total volume of the atmosphere: spherical shell thickness h 100km smallest unit volume V 0 1mm 3 = 10 9 m 3 within each unit volume about n i O(100) variables like Θ, p, ρ i, u, v, etc V 4πhR 2 Earth 4π10 5 (6.37 2 10 12 ) 5 10 19 m 3 V V 0 5 10 28 V n i V 0 10 30 11
We need about 10 30 numbers to characterize the actual state of the atmosphere T The observed state o is much smaller in dimension through a projection by measurements H: o = H( T ) + ɛ o detailed computations are impossible statistical physics: probability density description [ T ](t) with [( T )]d T = 1 12
The model approach high resolution global models: discrete spherical shell with about 2000 1000 100 grid points unit volumes 20km 2 1km within each unit volume about n i O(10) variables like Θ, p, ρ i, u, v, etc 13
We can treat numerically about 10 9 degrees-of-freedom to characterize the actual state f of high resolution atmospheric model the remaining d-o-f s in the state description ( T = f introduce observational or aleatoric uncertainty ɛ o a variety of true states T are possible for a given model state f the influence of the remaining d-o-f s ( 10 21 ) on the dynamics has to be parametrized introducing structural or epistemic uncertainty both have to be treated by probabilities 14
Additionally: atmosphere and its quasi-realistic model counterparts are nonlinear dynamical systems with embedded instabilities like convective instability inertial instability barotropic and baroclinic instability Nonlinearities and instabilities amplify small uncertainties (e.g. from the aleatoric part of initial conditions): The famous Lorenz butterfly (using the Lorenz (1984) model) 15
(Lorenz, 1984): Dynamics of two Rossbywaves (amplitudes y, z) of equal zonal wavelength in einem zonal symmetric zonal base flow (wavenumber zero) of strength x, forced by differential heating af. one of the Rossby waves is forced by orography or land-sea contrast (G). 16
Lorenz (1984) model: d dt x = ax y2 z 2 + af d y = xy y bxz + G dt d z = xz + bxy z dt 17
2.5 Zonaler Grundstrom 2 2 1.5 Wahrscheinlichkeitsdichte 1.5 1 0.5 0-1 -0.5 0 0.5 1 1.5 Zonaler Grundstrom 2 2.5 50 40 30 20 Zeit [d] 10 0 1 0.5 0-0.5-1 0 5 10 15 20 25 30 35 40 45 50 Zeit [d]
Working hypothesis The uncertainty in the atmospheric dynamic evolution can only be adequately described by probabilities or probability densities pdf. 18
Consequences for verification (Murphy and Winkler, 1987): any comparison of simulation f and observation o e.g. for verification has be done using the joint pdf [ o, f]. can be done with the Bayes-theorem using conditional probabilities [ o, f] = [ o f][ f] = [ f o][ o] or [ f o] }{{} posterior = 1 [ o] [ o f] }{{} likelihood [ f] }{{} prior note that o = ( o t, o t+1,..., o t+τ ) and similar for f 19
[ f] describes the stochastic dynamics of the forecast model [ o f] = [ o f, H, λ] models the measurement process, λ unknown parameters describe the difference between the actual measurement H and its discrete analog H 20
Modelling the stochastic dynamics E.g. for a single deterministic model forecast [ f t+τ f t+τ 1 ] = δ( f t+τ M t+τ,t+τ 1 ( f t+τ 1 )) But how to proceed with ensembles of deterministic forecasts or even stochastic forecasts d f = F ( f) dt d f = F ( f) + G( f) dt d dt W (t) }{{} Random Walk Idea: Probability is like a mass and can not be lost along trajectories in phase space: d dt ([ f]d f) = 0 21
For an ensemble of a deterministic model this leads to the Liouville equation t [ f] + F f [ f] = 0 For an ensemble of a stochastic model and a general noise model one gets the Kramers-Moyal equation, which simplifies under Gaussian noise to the Fokker-Planck Equation t [ f] + f ( F [ f]) 1 f (Q }{{} f [ f]) = 0 } 2 {{} Drift Diffusion
The Fokker-Planck equation A linear, but high-dimensional advection-diffusion equation. no analytical solution in general Numerical treatment up to 20 dimensions possible Nice theory but of almost no practical use 22
Two (three) methods for treating stochastic dynamics: Estimation of [f] instead of calculation well known Monte Carlo (ensemble) forecasting sample from the initial value density [ f t ]: a specific initial condition integrate each sample separately in the deterministic case as f t+1 M t+1,t ( f t ): ensemble forecasting in the stochastic case decide for a special calculus f t+1 f t = F dt + GdW (t) which determines the value of the last integral 23
Ito or Stratonovich type the discrete 1-d case (Milstein scheme) f t+1 f t = F (t) t + G(t) W (t) + 1 2 G(t) d df G(t)( W 2 α I,S t) W (t) Gaussian white noise with mean zero and variance t α I,S = 1 in case of Ito calculus α I,S = 0 in case of Stratonovich calculus less well known stochastic projection f t = k g k ( ζ) f k (t) combination of both
the propqpf project: the deterministic GME ensemble system 24
the propqpf project: a GME ensemble system 25
Verification, Calibration and Model Output Statistics Verification at lead time τ and stationarity assumption [ f t+τ o t+τ ] = [ o t+τ f t+τ ][ f t+τ ] [ o t+τ ] [ o t+τ f t+τ ] = [ o t+τ f t+τ, H, λ] models the measurement process, λ set of unknown parameters describing the difference between the actual measurement H and its discrete analog H λ are fixed by maximizing [ o t+τ f t+τ ] [ f t+τ o t+τ ] = argmax λ ( [ o t+τ f t+τ ][ f t+τ ] ) [ o t+τ ] 1 = [ o t+τ ] argmax λ([ o t+τ f t+τ, λ][ f t+τ ]) 26
model output statistics MOS (Glahn and Lowry, 1972)
A MOS example (T. Winkelnkemper, diploma thesis) multiple linear regression with Kalman updating of regression coefficients T max Stuttgart August 2003 27
if the set of free parameters λ is instead a part of the dynamical model [ f t+τ ] = [ f t+τ λ] we can calibrate the model by maximizing the posterior [ f t+τ o t+τ ] another type of calibration Bayesian Model Averaging BMA / dressing [ f t+τ o t+τ ] k w k ( o t+τ )[ f t+τ f (k) t+τ ] E{ k δ( f t+τ f (k) t+τ )} calibration is a necessary ingredient for ensemble forecasting 28
Data assimilation For weather ensemble prediction the probability density [ f t ] is not an unconditional probability density. It is conditioned on the observations at initial time [ f t+τ ] = [ f t o t ] Modern data assimilation systems (3d-var,4d-var) determine the most probable state of the model given the observations (with the additional assumption of a Gaussian density, see below) [ f t o t ]! = max 29
or [ f t o t ] [ o t f t ] [ f t ] }{{} First guess! = max (also used in remote sensing retrieval methods)
Theoretical consideration about the structure of the problem show that probabilistic treatment interrelates the verification, model output statistics, the calibration and the assimilation but is in its general form unusable: multidimensional integrals and optimization but allows one to boil down the problem and check the assumption necessary to arrive at a solvable problem Markov Chain Monte Carlo, stochastic fitting, Bayesian model averaging 30
Conclusions Stochastic weather forecasting and it s relatives are part of the SPP1167 mission Stochastic weather forecasting has its origin in the physics of atmospheric dynamics Stochastic weather forecasting can be structured through Bayes theorem A theoretical overview is necessary to assess the assumption for specific problems Avoid ad-hoc statistics 31
Announcement of a summerschool August/September 2008 in Potsdam jointly with SPP1167 PQP and SPP 1176 MetStroem on supercomputing and theoretical aspects of atmospheric dynamics organizer: Rupert Klein (Potsdam/Berlin), Nikos Nikiforakis (Oxford), Andreas Hense (Bonn) 32