An evaluation of the skill of ENSO forecasts during

Transcription

1 An evaluation of the skill of ENSO forecasts during Tony Barnston and Mike Tippett IRI Lead:

2 Nino.4 anomaly forecasts at 4 leads All Models Nino.4 anomaly Dynamical Nino.4 anomaly Statistical

3 MME Mean by Model Type, month lead. 2 OBS ALL 2 STAT DYN DYN models called for 2/8 La Nina onset too early

4 MME Mean by Model Type, month lead 2 OBS STAT. ALL 2 DYN DYN models did well for the 2/8 La Nina

5 Selected Dynamical Models, month lead 2 OBS GMAO GMAO ECMWF ECMWF CFS 2 CFS

6 Selected Statistical Models, month lead 2 OBS MARKOV CCA CA LIM CLIPER LIM CA CLIPER CCA MARKOV

7 Selected Dynamical Models, month lead 2 OBS. GMAO GMAO.. CFS LDEO 2.. CFS LDEO

8 Selected Statistical Models, month lead. 2 OBS MARKOV CLIPER CLIPER CA LIM CCA MARKOV CA CCA LIM

9 Lead OBS Lead OBS Lead OBS Forecasts and observations All Models Dynamical Statistical

10 We forecast conditions that occur earlier than the intended target period Lag Correlation: Fct vs Obs Correl Leads Lag (months)

11 Mean bias (fcst obs) All Models Dynamical Statistical

12 All Models Seasonality of mean bias (fcst obs) Dynamical Statistical

13 Statistical significance of skill is defined using the null hypothesis: The forecast is a sample of 9 running forecasts drawn from a sequence of 9 running observations spanning the same seasons, drawn from random years from the period The covariance of the forecasts spanning 9 running month periods is taken into account by using the observations for the same seasons. We generate years of climatological forecasts (same length as our time series of real time forecasts), score them and repeat many times. Then we determine percentiles of the actual score against this no skill background distribution. We use 9%ile to qualify for significance.

14 6 6 All Models Squared error (fcst obs) 2 when significant Dynamical 6 Statistical Significance threshold depends on magnitude of ENSO signal

15 All Models Squared error skill score: 2 ( fct obs ) 2 obs Dynamical (anomaly) when significant Statistical Significance threshold depends on magnitude of ENSO signal

16 All Models Seasonality of squared error (fcst obs) 2 when significant Dynamical Statistical

17 All Models 2 ( fct obs ) 2 obs Seasonality of squared error skill score when significant Dynamical anomaly Statistical

18 All Models Seasonality of correlation (fcst,obs) when significant Dynamical Statistical

19 All Models Continuous ranked probability skill score when significant Dynamical Statistical Constant spread, based on skill

20 All Models Seasonality of continuous ranked probability skill score when significant Dynamical Statistical

21 Squared error (f o) 2 Ensemble variance Squared error Squared error Ensemble variance Climo of ensemble variance

22 Statistical vs. dynamical forecast and forecast error scatterplots statistical statistical errors dynamical dynamical errors

23 Climatology of model variance J F M A M J J A S O N D Climatology of squared error J F M A M J J A S O N D Climatology model variance Climatology squared error

24 .9 Correlation Skill, all seasons, by lead time ALL DYN..6 STAT x x x x x o o o o Lead (months)

25 2.8.6 Standard Deviation Ratio (forecast/obs), Dyn Stat DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ lead-: Dyn.4.2 lead : Dyn lead-: Stat.8 lead : Stat.6.4.2

26 Correlation Skill (all multi model mean), DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ lead=

27 Correlation Skill (dynamical multi model mean), DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ lead=

28 Correlation Skill (statistical multi model mean), DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ lead=

29 .2 RMSE (SDs), all seasons, by lead time STAT ALL 9 99 DYN o o o o x x x x x Lead (months)

30 RMSE Skill (SDs), all multi model mean, DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ lead=.2

31 RMSE Skill, (SDs), dynamical multi model mean, DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ lead=.2

32 RMSE Skill (SDs), statistical multi model mean, DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ lead=

33 Dynamical Models: Correlation, all seasons.9.8 NCEP CFS ECMWF JMA. UKMO.6 LDEO L E A D T I M E (M O N T H S)

34 Statistical Models: Correlation, all seasons MARKOV.9 UCLA TCD CLIPER CCA.8 FSU REGR. CCA.6. MARKOV.4 UCLA TCD FSU REGR L E A D T I M E (M O N T H S)

35 Dynamical Models: RMSE (SDs), all seasons UKMO.6 JMA LDEO NCEP.4 CFS ECMWF.2 JMA NCEP CFS l l2 l l4 l l6 l l8 l9 L E A D T I M E (M O N T H S)

36 Statistical Models: RMSE (SDs), all seasons.4.2 CCA.8 UCLA TCD CLIPER MARKOV FSU REGR.6 CCA MARKOV.4 CLIPER UCLA TCD FSU REGR L E A D T I M E (M O N T H S)

37 Conclusions Our ENSO prediction skill is not much different this decade from how it was in the previous two decades. Decadal variations in ENSO prediction skill appears to be a stronger function of decadal variability of ENSO amplitude than of improvements in our models and/or prediction methodologies. For the first time, we see dynamical models delivering slightly more skillful ENSO predictions than statistical models, in the mean. This comes largely because of better performance in predicting the onset of the La Nina of 2 8, even though onset was predicted earlier than observed.