Forecast jumpiness in the ECMWF, UK Met Office and NCEP ensemble systems

Size: px

Start display at page:

Download "Forecast jumpiness in the ECMWF, UK Met Office and NCEP ensemble systems"

Dinah Dawson
5 years ago
Views:

1 Forecast jumpiness in the ECMWF, UK Met Office and NCEP ensemble systems How often do forecast jumps occur in the models? Ervin Zsoter (Hungarian Meteorological Service) In collaboration with Roberto Buizza & David Richardson (ECMWF)

2 Outline Introduction case study of a jumpy forecast period Inconsistency measures and experiments with the EPS control and EPS mean Inconsistency statistics Relationship between forecast inconsistency/jumpiness and forecast error Recent experiments with the EPS distribution Summary

3 A prominent example UTC All Z500 forecasts verifying at UTC

4 A prominent example UTC Difference of consecutive Z500 forecasts verifying on UTC

5 A prominent example UTC Difference of consecutive Z500 forecasts verifying on UTC

6 A prominent example UTC Difference of consecutive Z500 forecasts verifying on UTC

7 A prominent example UTC Difference of consecutive Z500 forecasts verifying on UTC

8 A prominent example UTC Difference of consecutive Z500 forecasts verifying on UTC

9 A prominent example UTC Difference of consecutive Z500 forecasts verifying on UTC

10 A prominent example UTC Zigzagging Forecasts verifying at z

Zigzagging Other examples Forecasts verifying at 20080223 12z Forecasts verifying at

11 Zigzagging Other examples Forecasts verifying at z Forecasts verifying at z Forecasts verifying at z Forecasts verifying at z

12 General questions How do the EPS-mean forecasts (representing the whole EPS distribution) and the single forecasts (high resolution deterministic & EPS-control) relate in terms of day-to-day consistency/inconsistency? If the EPS-control forecast jumps, how closely does the EPS-mean forecast follow the control forecast? What is the relationship between forecast inconsistency/jumps and forecast error?

13 Framework of the forecast inconsistency experiments We have investigated the behaviour of the EPS control vrs. EPS mean regarding day-today consistency based on the operational ECMWF, Met Office and NCEP EPS for January 2008 to August 2010 (~ 2.5 years) The inconsistency indices were computed based on consecutive (12 and 24-hour apart) runs all verifying at the same date. Forecasts starting at 00z and 12z were also considered. We processed all possible forecast ranges from T+12 (compared with T+24) to T+348 (compared with T+360) The inconsistency behaviour was investigated on 4 parameters: z500, z1000, t500 and t850 The inconsistency between two forecasts was computed for an area based on the grid point values inside the area, using a unified 1*1 degree grid We used four areas over Europe (and two control areas with the same size in North America) to asses the sensitivity of the inconsistency indices to the area Seasonal and annual differences were also investigated

14 Inconsistency Index For two forecasts (f(d,t) with run date d and forecast step t), δ-hour apart, verifying on the same date (d+t), over an area (Σ): difference between the two fields, ΙΝC Σ [f(d, t),δ] = 0.5 d [f(d, t),f(d δ, t + δ)] { std [f(d, t)]+ std [f(d δ, t + δ)] } Σ Σ Forecast f(d,t) can be any of the four forecast versions: EPS control or EPS mean of ECMWF, Met Office or NCEP (for ECMWF the high resolution forecast is also considered) We tried to account for the fact that the EPS mean fields can be much smoother than the corresponding EPS control fields by normalising the differences (both EPS mean and EPS control) by the Standard Deviation of the forecast fields grid values inside the area Therefore the EPS-control forecasts, normally showing more details than the EPSmean over an area, will have in general larger normalising factor which should compensate the larger field difference Σ averaged over the area (Σ) Average of the standard deviations of the values in the two fields inside the area (Σ)

15 Inconsistency index time series T+60 Parameter: Z500 Area: Northwest Europe [50N, 20W, 65N, 20E] Sample period: 1 January February 2008 T+60 T+72 Correlation = 0.95

16 Inconsistency index time series T+156 Parameter: Z500 Area: Northwest Europe [50N, 20W, 65N, 20E] Sample period: 1 January February 2008 T+156 T UTC z Correlation = 0.73

17 Inconsistency index time series T+348 Parameter: Z500 Area: Northwest Europe [50N, 20W, 65N, 20E] Sample period: 1 January February 2008 T+348 T+360 Correlation = 0.31

18 Inconsistency Index Inconsistency index statistics Parameter: Z500, Area: Northwest Europe, Sample period: 16 January August 2010 (~2.5 years) Period average inconsistency index (A-INC) ECMWF control ECMWF ens-mean ECMWF highres UKMO control UKMO ens-mean NCEP control NCEP ens-mean Correlation Inconsistency correlation (CORR) EC control / EC ens-mean UK control / UK ens-mean NC control / NC ens-mean EC control / UK control EC control / NC control EC ens-mean / UK ens-mean EC ens-mean / NC ens-mean Forecast setp (hours) A-INC = average of the inconsistency index values over a sampling period EPS-mean is increasingly more consistent than EPS-control for longer lead times EPS-control and high resolution forecast inconsistencies are the same for ECMWF Forecast setp (hours) CORR = correlation between two inconsistency index time series over a sampling period The correlation drops back from ~1 to ~0.3 by day 15 between inconsistency index time series of EPScontrol and ESP-mean of the same forecast system Correlations are significantly lower between different forecast systems

19 Forecast jumps / flip-flops We define the significant/large inconsistencies, i.e. forecast jumps (flips), as inconsistencies (INC) over 2/3 of the period average inconsistency (2/3*A-INC) with both positive and negative signs There is a flip threshold for each forecast step and for each forecast versions (e.g. for EPScontrol dashed lines, for EPS-mean dotted lines on the diagram) EPS-control flip EPS-mean flip Parallel flip Forecasts verifying at z When a forecast makes more than one consecutive flips with alternating signs (ups and downs) we call these events flip-flops, flip-flop-flips, etc. When two forecast versions are investigated in combination, and they make flips, flip-flops, etc. in phase, we call these events parallel flips, parallel flip-flops, etc., (e.g. the EPS control and the EPS mean on the diagram)

Forecast jump statistics Parameter: Z500, Area: Northwest Europe, Sample period: 16 January 2008 31 August 2010 (~2.

throughout the 15-day forecast period In the contrary, the two and three consecutive alternating jumps occur clearly less often

control It is also important to notice that the EPS-mean and the control forecasts from one system (the ECMWF, Met Office or

20 Forecast jump statistics Parameter: Z500, Area: Northwest Europe, Sample period: 16 January August 2010 (~2.5 years) Flips Flip-flops Flip-flop-flips The frequency of single jumps is the same for both the EPScontrol and the EPS-mean throughout the 15-day forecast period In the contrary, the two and three consecutive alternating jumps occur clearly less often in the EPS-mean, confirming that the EPSmean is not only more consistent on average but also zigzags less often than the control It is also important to notice that the EPS-mean and the control forecasts from one system (the ECMWF, Met Office or NCEP) follow each other more closely than the control or the EPS-mean of two different ensemble systems follow each other (red curves vrs. purple curves)

21 Inconsistency measure sensitivity A-INC Parameters Inconsistency Index A-INC Areas Northwest Europe Northest North-America Small Northwest Europe Large Northwest Europe Southeast Europe Southwest North-America 0.6 A-INC Seasons Control forecasts Forecast setp (hours) 30 % Frequency of parallel flips Parameters / Areas / Seasons 20 NWEU T850 NWEU Z500 NWEU Z500 Summer NWEU Z500 Winter 10 NWEU-small Z500 NWEU-large Z500 SEEU Z500 NENA Z Forecast setp (hours)

22 Forecast error statistics The relationship between forecast error and inconsistency/jumpiness was investigated on seasonal subsets of the ~2.5 years sample for the usual parameters and areas The benchmark was the relationship between error and EPS spread (SPREAD setup) We investigated the EPS-control, the EPS-mean and for ECMWF also the high resolution control forecasts Three inconsistency/jumpiness measures were considered: FLIPS: When inconsistency/jumpiness is characterised by the number of consecutive zigzagging jumps ( flips), as defined with the Inconsistency Index (INC) RDIFF: When only the RMS difference between forecast fields (without normalisation) is considered RDIFFSUM: When the most recent RMS differences between consecutive forecasts are summed up (we considered the last 4) As the FLIPS setup provided 4 subgroups ( flips), we also created 4 subgroups for RDIFF, RDIFFSUM and SPREAD. The subgroup generation was based on the distribution of the values in the sample, providing subgroups (with equal size in all three versions) with small, medium small, medium large and large rdiff, rdiffsum and spread values

23 FLIPS Forecast error statistics RDIFF error stds mean errors error stds mean errors RDIFFSUM SPREAD error stds mean errors error stds mean errors Statistics for the ECMWF EPS-control forecasts, for Northwest Europe, Z500 and JJA (9 months)

24 Forecast error statistics The relationship between error and jumpiness (FLIPS setup) seems to be week as the error characteristics in subgroups of 0, 1, 2 or 3 consecutive flips are relatively similar. Forecasts having increasing number of consecutive flips seem to show, on average, only a bit larger error In contrary, when we stratify the forecasts into four subgroups based on the level of EPS spread (SPREAD setup, small, medium small, medium large and large), we get a solid relationship, i. e. forecasts with larger spread are, on average, more likely to produce larger forecast errors However, the inconsistency/error relationship gets significantly stronger if we stratify the sample based on the RMS difference between forecast fields (RDIFF setup) and not on the consecutive number of flips Some further improvement seems to be achievable if the RMS difference between consecutive forecasts are summed (RDIFFSUM setup) The superiority of the EPS spread as an indicator of forecast error is more pronounced for ESP-mean and less for EPS-control (and high resolution forecast for ECMWF) Similar conclusions can be drawn for the UKMO and NCEP EPS and for all investigated parameters, areas and seasons

25 Inconsistency experiments with the EPS distribution We modified the Inconsistency Index in order to be able to apply it to the whole EPS distribution, not just to the EPS-mean The field difference operator was replaced by the difference delivered by the continuous ranked probability score (CRPS) between two fieldsets ΙΝC Where: 2 2 Σ [f(d, t),δ] = CRPS= sign 0.5 x= x= 2 CRPS [f(d, t),f(d δ, t + δ)] { std [f(d, t)]+ std [f(d δ, t + δ)] } (F f Σ Σ (x) F o (x)) The modified inconsistency becomes the original index for single forecasts (control, mean, high resolution) The forecast error computed with the (default) CRPS also gives back the error, based on the original field difference for single forec. The definition of any other inconsistency measure remains unchanged Σ x= (F x= f (x) F o (x)) 2 CRPS difference with artificial sign between the two fieldsets, averaged over the area (Σ) Average of the standard deviations of the values in the two fields inside the area (EPS-mean std is used for the EPS)

26 Inconsistency Index Inconsistency index statistics Parameter: Z500, Area: Northwest Europe, Sample period: 16 January August 2010 (~2.5 years) Period average inconsistency index (A-INC) EPS ECMWF ens-mean ECMWF ensemble UKMO ens-mean UKMO ensemble NCEP ens-mean NCEP ensemble EPS-mean Correlation Inconsistency correlation (CORR) EC control / EC ens-mean EC control / EC ensemble EC ens-mean / EC ensemble EC control / UK control Forecast setp (hours) A-INC = average of the inconsistency index values over a sampling period The whole EPS is represented by smaller inconsistency for all lead times especially for longer leads Forecast setp (hours) CORR = correlation between two inconsistency index time series over a sampling period As one would expect the correlation is very high between the EPS and the EPS mean inconsistencies The manner of day-to day change is the same, but the difference values are lot lower for the EPS

27 Forecast jump statistics Parameter: Z500, Area: Northwest Europe, Sample period: 16 January August 2010 (~2.5 years) 60 Flips 25 Flip-flops % % EC control EC ens-mean EC ensemble EC control / EC ens-mean EC control / EC ensemble EC control / UK control Forecast setp (hours) Forecast setp (hours) The frequency of single jumps is clearly lower for the EPS than for either the EPS-control or the EPS-mean throughout the 15-day forecast period For flip-flops and flip-flop-flips the advantage of the EPS-mean over the EPS-control is further improved by the EPS The lower frequencies also appear in the parallel jumps between the EPS-control and EPS compared to the EPS-control vrs. EPS-mean

28 We defined an inconsistency index for a given area and also some other related indices to investigate the inconsistency behaviour of the ECMWF, UK Met Office and NCEP EPS forecasts with special emphasis on forecast jumpiness. We found that: Summary - 1 The EPS-mean provides more consistent forecasts than the EPS-control from about T+72, and this benefit gradually increases with forecast range to day-15 An other aspect of the higher consistency of the EPS-mean is that although the flip frequencies (single jumps) are very similar for both the control and the EPS-mean, the zigzagging occurs clearly less frequently with the EPS-mean Although the inconsistency index is highly dependent on the area size, the sensitivity seems to be moderate on the parameters and rather small on the seasons. Besides, the number of expected jumps (all kinds) is rather similar with all investigated sample versions The relationship between the EPS-control and the EPS-mean of the same ensemble system is much stronger than the relationship between forecasts of two different ensemble systems This suggests that the forecast uncertainty is not sufficiently well sampled in either EPS. This could be because the ensemble size is not large enough, or because the EPS perturbations do not adequately represent the initial and model uncertainties This also seems to suggest that (e.g.) the 1-2% parallel flip-flop-flip frequency (control and mean zigzagging together) found for Z500 is likely to be higher than an optimal frequency

29 Summary - 2 We also found that: The connection between forecast inconsistency and forecast error is week if we represent inconsistency by consecutive flips. However, the connections seems to be lot stronger by considering inconsistency as an RMS type of difference between fields The best indicator of forecast error is still the ensemble spread, which should be seen as a further evidence of the superiority of the EPS system compared to the lagged ensembles of the latest available forecasts We also found that: The inconsistency in the EPS gets smaller if we consider the whole distribution not just the EPSmean. This is highlighted by the smaller Inconsistency Index and more importantly the significantly reduced number of flips (flip-flops, etc.) Plans for the future: Explore further the possibilities with the CRPS definition of the inconsistency measures We would like to extend the experiments to the multi-model versions of the investigated EPS systems or other EPS-s available within TIGGE

30 Thank you! Further reading Zsoter, E., R. Buizza, D. Richardson, Jumpiness of the ECMWF and the UK Met Office EPS control and ensemble-mean forecasts. Mon. Wea. Rev., 137,

Forecast Inconsistencies

Forecast Inconsistencies How often do forecast jumps occur in the models? ( Service Ervin Zsoter (Hungarian Meteorological ( ECMWF ) In collaboration with Roberto Buizza & David Richardson Outline Introduction