An evaluation of the skill of ENSO forecasts during

An evaluation of the skill of ENSO forecasts during 22 29 Tony Barnston and Mike Tippett IRI Lead: 2 4 6 8 9

Nino.4 anomaly forecasts at 4 leads All Models 22 4 6 8 9 Nino.4 anomaly Dynamical 22 4 6 8 9 Nino.4 anomaly Statistical 22 4 6 8 9

MME Mean by Model Type, month lead. 2 OBS ALL 2 STAT DYN DYN models called for 2/8 La Nina onset too early 22 2 24 2 26 2 28 29

MME Mean by Model Type, month lead 2 OBS STAT. ALL 2 DYN DYN models did well for the 2/8 La Nina 22 2 24 2 26 2 28 29

Selected Dynamical Models, month lead 2 OBS GMAO GMAO ECMWF ECMWF CFS 2 CFS 22 2 24 2 26 2 28 29

Selected Statistical Models, month lead 2 OBS MARKOV CCA CA LIM CLIPER LIM CA CLIPER CCA MARKOV 2 22 2 24 2 26 2 28 29

Selected Dynamical Models, month lead 2 OBS. GMAO GMAO.. CFS LDEO 2.. CFS LDEO 22 2 24 2 26 2 28 29

Selected Statistical Models, month lead. 2 OBS MARKOV CLIPER CLIPER CA LIM CCA MARKOV CA CCA LIM. 2 22 2 24 2 26 2 28 29

Lead 8 6 4 2 OBS Lead 8 6 4 2 OBS Lead 8 6 4 2 OBS Forecasts and observations 2 4 6 8 9 All Models Dynamical Statistical

We forecast conditions that occur earlier than the intended target period Lag Correlation: Fct vs Obs Correl Leads 4 2 2 4 Lag (months)

Mean bias (fcst obs) 2 4 6 8 9 All Models Dynamical Statistical

All Models Seasonality of mean bias (fcst obs) Dynamical Statistical

Statistical significance of skill is defined using the null hypothesis: The forecast is a sample of 9 running forecasts drawn from a sequence of 9 running observations spanning the same seasons, drawn from random years from the period 9 22. The covariance of the forecasts spanning 9 running month periods is taken into account by using the observations for the same seasons. We generate years of climatological forecasts (same length as our time series of real time forecasts), score them and repeat many times. Then we determine percentiles of the actual score against this no skill background distribution. We use 9%ile to qualify for significance.

6 6 All Models Squared error (fcst obs) 2 when significant Dynamical 6 Statistical Significance threshold depends on magnitude of ENSO signal

All Models Squared error skill score: 2 ( fct obs ) 2 obs Dynamical (anomaly) when significant Statistical Significance threshold depends on magnitude of ENSO signal

All Models Seasonality of squared error (fcst obs) 2 when significant Dynamical Statistical

All Models 2 ( fct obs ) 2 obs Seasonality of squared error skill score when significant Dynamical anomaly Statistical

All Models Seasonality of correlation (fcst,obs) when significant Dynamical Statistical

All Models Continuous ranked probability skill score 2 4 6 8 9 2 4 6 8 9 when significant Dynamical Statistical Constant spread, based on skill

All Models Seasonality of continuous ranked probability skill score when significant Dynamical Statistical

Squared error (f o) 2 Ensemble variance 2 4 6 8 9 Squared error Squared error Ensemble variance Climo of ensemble variance

Statistical vs. dynamical forecast and forecast error scatterplots statistical statistical errors dynamical dynamical errors

Climatology of model variance J F M A M J J A S O N D Climatology of squared error J F M A M J J A S O N D Climatology model variance Climatology squared error

.9 Correlation Skill, all seasons, by lead time 22 29.8 ALL DYN..6 STAT x x x x x o 98 99..4 9 99 o o o..2. 9 99 2 2 4 6 8 9 Lead (months)

2.8.6 Standard Deviation Ratio (forecast/obs), 22 29 Dyn Stat DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ lead-: Dyn.4.2 lead : Dyn lead-: Stat.8 lead : Stat.6.4.2

Correlation Skill (all multi model mean), 22 29.2.8.6.4 DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ lead= 2 4 6.2 8.2.4 9.6

Correlation Skill (dynamical multi model mean), 22 29.2.8.6 DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ lead= 2 4.4.2 6 8.2.4 9.6

Correlation Skill (statistical multi model mean), 22 29.2.8.6.4 DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ lead= 2 4 6.2 8.2.4 9.6

.2 RMSE (SDs), all seasons, by lead time 22 29.8 STAT ALL 9 99 DYN o o o o x x x x x 98 99.6.4.2 9 99 2 2 4 6 8 9 Lead (months)

RMSE Skill (SDs), all multi model mean, 22 29.8.6 DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ.4.2.8.6.4 9 8 6 4 2 lead=.2

RMSE Skill, (SDs), dynamical multi model mean, 22 29.8.6 DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ.4.2.8.6.4 9 8 6 4 2 lead=.2

RMSE Skill (SDs), statistical multi model mean, 22 29.8.6 DJF JFM FMA MAM AMJ MJJ JJA JAS ASO SON OND NDJ.4.2 8 9.8.6.4.2 6 4 2 lead=

Dynamical Models: Correlation, all seasons.9.8 NCEP CFS ECMWF JMA. UKMO.6 LDEO..4..2. 2 4 6 8 9 L E A D T I M E (M O N T H S)

Statistical Models: Correlation, all seasons MARKOV.9 UCLA TCD CLIPER CCA.8 FSU REGR. CCA.6. MARKOV.4 UCLA TCD FSU REGR..2. 2 4 6 8 9 L E A D T I M E (M O N T H S)

Dynamical Models: RMSE (SDs), all seasons.4.2.8 UKMO.6 JMA LDEO NCEP.4 CFS ECMWF.2 JMA NCEP CFS 2 4 6 8 9 l l2 l l4 l l6 l l8 l9 L E A D T I M E (M O N T H S)

Statistical Models: RMSE (SDs), all seasons.4.2 CCA.8 UCLA TCD CLIPER MARKOV FSU REGR.6 CCA MARKOV.4 CLIPER UCLA TCD FSU REGR.2 2 4 6 8 9 L E A D T I M E (M O N T H S)

Conclusions Our ENSO prediction skill is not much different this decade from how it was in the previous two decades. Decadal variations in ENSO prediction skill appears to be a stronger function of decadal variability of ENSO amplitude than of improvements in our models and/or prediction methodologies. For the first time, we see dynamical models delivering slightly more skillful ENSO predictions than statistical models, in the mean. This comes largely because of better performance in predicting the onset of the La Nina of 2 8, even though onset was predicted earlier than observed.