A multi-model multi-analysis limited area ensemble: calibration issues

Transcription

1 - European Centre for Mediu-Range Weather Forecasts - Third International Workshop on Verification Methods Reading 31 Jan 2 Feb A ulti-odel ulti-analysis liited area enseble: calibration issues M. Marrocu CRS4, Parco Scientifico e Tecnologico POLARIS Edificio 1, Pula (CA), Italy P. A. Chessa Servizio Agroeteorologico della Sardegna, Viale Porto Torres 119, 07100, Sassari, Italy eail: arino@crs4.it eail: chessa@sar.sardegna.it

2 MUSE a Multiodel-ultianalysis enseble 4 LAMs : BOLAM - MM5 RAMS1 RAMS2 2 I.C & B.C.: AVN 12Z - ECMWF 12Z Area: 13.5W-34N / 24.5E-54.5N N 41 Spatial Resolution: 0.25 Fct tie range: +72h (by 6 h steps) 12.5 W Integration period: 15/10/2002 to 15/04/2003 (183 days) Thanks to C. Dessy, G. Ficca, C. Castiglia, I. di Piazza 12.5 W 10.5 W9 W 7.3 W6 W 4.3 W3 W 1.3 W E 3 E 4.8 E 6 E 7.8 E 9 E 11 E 13 E 15 E 17 E 19 E 21 E 23 E N 54 N 53 N 53 N 52 N 52 N 51 N 51 N 50 N 50 N 49 N 49 N 48 N 48 N 47 N 47 N 46 N 46 N 45 N 45 N 44 N 44 N 43 N 43 N 42 N 42 N 41 N 40 N 40 N 39 N 39 N N 37 N N 36 N N W9 W 7.3 W6 W 4.3 W3 W 1.3 W E 3 E 4.8 E 6 E 7.8 E 9 E 11 E 13 E 15 E 17 E 19 E 21 E 23 E OPERATIONAL IN MARCH

3 Measured data The calibration assessent is done for a continuous variable with a relatively siple PDF. Naely, the 2 teperature. For the 186 days, all 6-hourly easured data were collected fro 21 ground eteorological stations located in Sardinia. These stations were singled out fro the whole network (about 60 stations), because they were the sole having no issing data. 3

4 Spread-skill relationship ρ = 0.40 ρ = 0.19 Tax Tin NOTE. The variability of the spread-skill relationship across the forecast tie steps reflects on the RMSE of the deterinistic forecasts and on the calibration outcoes. 4

5 Why calibrate? The enseble is under-dispersive and the single forecasts are clearly not equi-probable. Calibration should reduce the under-dispersion, provide a suitable weight for each eber and, hopefully, increase the sharpness of the resulting distribution. 5

6 Calibration ethods - 1 Bayesia Model Averaging (BMA) p( o 1 = f1,..., f ) = w G ( o f ) K where G( o f ) =Ν ( a + b f, σ 2 ) w and σ are estiated by axiu likelihood and in a further step the variance is refined iniizing the Continuous Ranked Probability Score, CRPS = K { F ( z) H ( z t ) } dz 1 K + 2 j j j= 1, over the training period. F(z) is the Cuulative Distribution Function of G while H is the Heaviside function. Ref.: A. E. Raftery et al. - MWR

7 Calibration ethods - 2 Enseble odel output statistics (EMOS) The EMOS PDF is expressed as: N ( α + β f βk f K;γ + δ S ) (enseble spread) the coefficients are calculated iniizing the CRPS over the training period Modified enseble odel output statistics (EMOS + ) CRPS iniization iterated: after each step odels associated to negative are drop out fro the next iteration. The process stops when all β i left are positive. Id est: enseble retains only forecasts providing a skilful contribution. β i Ref.: T. Gneiting et al. - MWR

8 Dressing kernel Calibration ethods - 3 The covariance of the stochastic values to be added to the dynaical forecasts f, is calculated in a way that renders the, seasonally averaged, variance of the dressed enseble and that of the observation, indistinguishable. That is to say that: η with F dress = f +η then T η η = T 2 ( fi oi ) ( fi oi ) σ i (saple ean and variance of true forecast PDF) (observations) The eans are taken over all forecast-observation occurrences in the training period. The nuber of perturbations to be added to each dynaical forecast was set to 32. Ref.: Wang and Bishop - QJRM

9 Training period The training period is a sliding-window varying fro tie step to tie step.to define it, a few quantities used to evaluate the calibration quality (the rank histogra, the PDF s coverage and width, the RMSE for the related deterinistic forecasts) were used. In practice the chosen interval length is such that a longer training period do not bring any iproveent on the calibration scores. In this case this happens between 60 and 90 days. In the following results are based on a 90 days training period. In order to test the robustness of the techniques and its independence fro the training set, all the calculation were also accoplished swapping training and testing periods. The final results did not change. 9

10 Calibration: rank histogras (+66h) raw eos and eos + dressing ba 10

11 Calibration: rank histogras (+72h) raw eos and eos + dressing ba 11

12 Calibration: rank histogras (all steps) Root ean square error with respect central outliers to perfect intervals calibration 12

13 Calibration: coverage and width 13

14 Calibration: coverage and width 14

15 BMA weights 15

16 Expectation values The expectation value of the PDFs for BMA, EMOS and EMOS +, and the dressed enseble ean are deterinistic forecasts on their own. For instance for BMA is: Scores like RMSE and MAE have been calculated for all of the and d copared to the likes of: each enseble eber, the unbiased enseble ean and the super-enseble. Why so? The hope was to unveil a behaviour so good to gain for free, and for a syste which inherently lacks it, a reference (control) forecast directly fro the calibration ethod. µ BMA = K = 1 w ( a + b f ) 16

17 Deterinistic forecasts 17

18 Conclusions Calibration for 2 teperature works well both with BMA and DRESSING. (Easy the extension to teperature at pressure levels and to other continuous variables as MSLP, geopotential, etc.) BMA shows ore consistent results than DRESSING across the forecast tie steps, especially for the external intervals (outliers). Moreover, BMA weights are directly interpretable in ters of probabilities. Deterinistic scores for the expectation values of calibration ethods, the dressed enseble ean, the unbiased enseble ean and the super-enseble are siilar. All of the outperfor, on average, the best odel. Therefore, once a calibration ethod is chosen, it is argued that the expectation value can be used as reference/control forecast for the enseble. 18

19 Future work Calibration is going to be ipleented on MUSE (needs a good aount of coputer power) SPITLOMS: a ECMWF special project (SAR CRS4 Italian MetService) aied at exploring the potential of longer and ore structured training periods. Calibration for wind and precipitation is going to be shortly addressed (need a careful analysis of the underlying PDF and probably, for precipitation, longer training sets). 19