Regional Production Quarterly report on the daily analyses and forecasts activities, and verification of the ENSEMBLE performances December 2015 January 2016 February 2016 Issued by: METEO-FRANCE Date: 04/05/2016 REF.: CAMS50_2015SC1_D50.4.2-2016Q1_201605 CAMS50_2015SC1_D50.4.4-2016Q1_201605 CAMS50_2015SC1_D50.5.1-2016Q1_201605
This document has been produced in the context of the Copernicus Atmosphere Monitoring Service (CAMS). The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of CAMS on behalf of the European Union (Delegation Agreement signed on 11/11/2014). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and liability. For the avoidance of all doubts, the European Commission and the European Centre for Medium-Range Weather Forecasts has no liability in respect of this document, which is merely representing the authors view. Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances
Quarterly report on the daily analyses and forecasts activities, and verification of the ENSEMBLE performances December 2015 January 2016 February 2016 METEO-FRANCE (M. Pithon, M. Plu, J. Parmentier, J. Arteta, S. Guidotti, N. Assar) Date: 04/05/2016 REF.: CAMS50_2015SC1_D50.4.2-2016Q1_201605 CAMS50_2015SC1_D50.4.4-2016Q1_201605 CAMS50_2015SC1_D50.5.1-2016Q1_201605 Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances
Contents: 1. Executive Summary...4 2. The ENSEMBLE...5 Product portfolio...5 Availability statistics...5 3. Verification report...8 Verification of NRT forecasts...8 Verification of NRT analyses...13 Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances
1. Executive Summary The Copernicus Atmosphere Monitoring Service (CAMS, www.copernicusatmosphere.eu) is establishing the core global and regional atmospheric environmental service delivered as a component of Europe's Copernicus programme. The regional forecasting service provides daily 4-days forecasts of the main air quality species and analyses of the day before, from 7 state-of-the-art atmospheric chemistry models and from the median ensemble calculated from the 7 model forecasts. The regional service also provides posteriori reanalyses using the latest validated observation dataset available for assimilation. This report covers the deliverables related to Near Real Time Production (NRT) for the ENSEMBLE: D50.4.2-2016Q1, D50.4.4-2016Q1, D50.5.1-2016Q1, for the quarter December 2015 January 2016 February 2016. Verification is done against in-situ surface observations. They are described in the report D50.1.2-2016Q1, that will be delivered shortly. The verification of analyses is done against non assimilated observations. During this quarter, the ENSEMBLE analyses and forecasts were produced on time everyday. The ENSEMBLE forecasts included at least 5 members everyday. The ENSEMBLE analyses included at least 5 members 99% of the time. During this quarter, the ENSEMBLE scores reach their target (Key Performance Indicators) for every pollutants (O 3, NO 2, PM 10, forecasts and analyses). Compared to past winter, the ENSEMBLE ozone forecasts show an improvement of the KPI score. For NO 2 and PM 10 forecasts, the RMSE values are lower by about 1 µg.m -3 than during past winter. These are the most noticeable improvements compared to past winter. Regarding the ENSEMBLE analyses, the diurnal patterns of the scores are similar for the analyses and for the forecasts, but the analyses show a systematic improvement of the scores for all pollutants. On-going model developments in every individual model, concerning the implementation or refinement of secondary aerosol formation, and also the assimilation of new observed species at the surface, will help to improve ENSEMBLE scores in the future. Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 4
2. The ENSEMBLE Product portfolio Name Forecast Analysis Description Forecast at surface, 50m, 250m, 500m, 1000m, 2000m, 3000m, 5000m above ground Available for users at 6:45 UTC for D0-D1 8:30 UTc for D2-D3 Species O 3, NO 2, CO, SO 2, PM 2.5, PM 10, NO, NH 3, NMVOC, PANs, Birch pollen at surface during season Analysis at the surface 11:30 UTC for the day before O 3, NO 2 Time span 0-96h, hourly 0-24h for the day before, hourly Availability statistics The statistics below describe the ratio of days for which the ENSEMBLE model outputs were available on time to be included in the ENSEMBLE fields (analyses and forecasts) that are computed at METEO-FRANCE. They are based on the following schedule for the provision at METEO-FRANCE of: - forecasts data before: 05:30 UTC for D0-D1 (up to 48h), 07:30 UTC for D2-D3 (from 49h to 96h) - analyses data: before 11:00 UTC These schedules have been set to meet the IT requirements for ENSEMBLE products (no later than 8 UTC for 0-48h, 10 UTC for 49-96h and 12 UTC for analyses). Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 5
Indicators: Schedule compliance base quarter DJF ENSEMBLE Fcst ENSEMBLE Ana Number of model forecasts contributing to ENSEMBLE Dec 2015 100% 100% N= 7: 84% D0 D1 D2 D3 100% 100% N= 7: 84% 100% 100% N= 7: 52% 100% 100% N= 7: 29% Jan 2016 N = 7: 94% N = 7: 87% N = 7: 77% N = 7: 77% Feb 2016 N = 7: 86% N = 7: 83% N = 7: 79% N = 7: 69% Number of model analyses contributing to ENSEMBLE Dec 2015 Jan 2016 N= 7: 58% N>=5: 97% N = 7: 81% Feb 2016 N = 7: 90% Comments During this quarter, all 7 models have contributed to the ENSEMBLE forecast calculation in 86% of cases, for the first two day of forecasts (D0, D1). The lower score for the next two days (64 % of cases with all 7 models for D2-D3) is improving at the second half of the quarter, thanks to the earlier provision of CHIMERE results. Regarding analyses, except one day, the ENSEMBLE has always been calculated with at least 5 members. During this quarter, in 76% of cases, all 7 models have effectively been contributing. 1 The poor figures for the December analyses were mainly due to the late provision of LOTOS-EUROS; this issue was definitively solved mid December. 1 Statistics for analysis are calculated for O 3 pollutant which is the only one provided by all models, at the moment. Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 6
During this quarter, the following issues have been encountered by the ENSEMBLE production system: Date Problem description (origin, effects) 08/12/2015 Internet issue at ECMWF: internet dissemination, ec access, Web services affected 11/12/2015 Issue related to the switch of MATCH and EURAD-IM on METEO-FRANCE operational telecommunication system Impact on production Impact on ENSEMBLE analysis: 2 models missing Impact on ENSEMBLE analysis: MATCH results missing Ensemble calculation with less than 5 models Date Duration (in days) Missing models 08/12/2015 1 day: Analysis ENSEMBLE EURAD-IM, LOTOS-EUROS, MATCH due to Internet problem at ECMWF (EURAD-IM and MATCH missing) and late arrival of LOTOS-EUROS Total number of days 1 analysis run Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 7
3. Verification report This verification report covers the period December 2015 January 2016 February 2016. The ENSEMBLE skill scores are successively presented for four pollutants: ozone, NO 2, PM 10 and PM 2.5. The skill is shown for the entire forecast horizon from 0 to 96h (hourly values), allowing to evaluate the entire diurnal cycle and the evolution of performance from day 0 to day 3. The forecasts and the analyses cover a large European domain (25 W-45 E, 30 N-70 N). The statistical scores that are reported are the root-mean-square error, the modified mean bias and the correlation. The surface observations that are acquired by METEO-FRANCE and used for verification are described in D50.1.2-2016Q1, that will be delivered shortly. Verification of NRT forecasts The following figures present, for each pollutant (ozone, NO 2, PM 10, PM 2.5 ): - in the upper-left panel, the root-mean square error of daily maximum (for ozone and NO 2 ) or of daily mean (PM 10 ) for the first-day forecasts with regards to surface observations, for every quarter since DJF2014/2015, a target reference value is indicated as an orange line, - in the upper-right panel, the root-mean square error of pollutant concentration forecasts with regards to surface observations as a function of forecast term, - in the lower-left panel, the modified mean bias of pollutant concentration forecasts with regards to surface observations as a function of forecast term, - in the lower-right panel, the correlation of pollutant concentration forecasts with regards to surface observations as a function of forecast term. The graphics show the performance of the ENSEMBLE (black curves). Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 8
ENSEMBLE forecasts: ozone skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 The RMSE of the daily maximum of ozone ENSEMBLE forecasts reaches the target this quarter. This score has improved by about 2 µg.m -3 compared to past winter. The bias is always positive as a function of the forecast hour and it is around 0 during the afternoon. The RMSE peaks at 19.5 µg.m -3 during night-time and gets as down as 15 µg.m -3 during daytime. The RMSE and correlation scores are similar as during past winter. There is a perceptible tendency to increasing RMSE and decreasing correlation with forecast day. Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 9
ENSEMBLE forecasts: NO 2 skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 The RMSE of the daily maximum of NO 2 ENSEMBLE forecasts reaches the target this quarter. This score as a similar value as during past winter. The ENSEMBLE bias is always negative, following a diurnal cycle with the highest absolute values during daytime and the lowest ones during night-time. The RMSE oscillates twice per day between 9.5 and 18 µg.m -3, which is lower than during past winter. The correlation is similar than during past winter and tends to decrease with forecast day. Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 10
ENSEMBLE forecasts: PM 10 skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 The RMSE of the daily mean of PM 10 ENSEMBLE forecasts reaches the target this quarter. This score as a similar value as during past winter. The ENSEMBLE bias is always negative, but close to 0 in the late night. The RMSE oscillates between 10 µg.m -3 and 15 µg.m -3, which is lower by about 1 µg.m -3 than during past winter. The correlation is similar than during past winter and tends to decrease with forecast day. Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 11
ENSEMBLE forecasts: PM 2.5 skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 The RMSE of the daily mean of PM 2.5 ENSEMBLE forecasts reaches the target this quarter. This scores is lower than during past winter. The bias is mostly negative. The RMSE is minimum during daytime (around 8.5 µg.m -3 ) and peaks at 11 µg.m -3 during the night. The correlation tends to decrease with forecast day. Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 12
Verification of NRT analyses The following figures present, for each pollutant (ozone, NO 2, PM 10 ): - in the upper-left panel, the root-mean square error of daily maximum (for ozone and NO 2 ) or of daily mean (PM 10 ) for the analyses (solid line) and for the first-day forecasts (dashed line) with regards to surface observations, for every quarter since DJF2014/2015, a target reference value is indicated as an orange line, - in the upper-right panel, the root-mean square error of pollutant concentration of the analyses (solid line) and of the first-day forecasts (dashed line), with regards to surface observations as a function of forecast term, - in the lower-left panel, the modified mean bias of pollutant concentration forecasts of the analyses (solid line) and of the first-day forecasts (dashed line), with regards to surface observations as a function of forecast term, - in the lower-right panel, the correlation of pollutant concentration of the analyses (solid line) and of the first-day forecasts (dashed line), with regards to surface observations as a function of forecast term. The graphics show the performances of the ENSEMBLE (black curves). The superposition of the analysis scores (solid lines) and of the forecast scores (dashed lines) computed over the same observation dataset is helpful to assess the added value of data assimilation. Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 13
ENSEMBLE analyses: ozone skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 The RMSE of the daily maximum of ozone ENSEMBLE analyses reaches the target this quarter. The diurnal patterns of the scores are similar for the analyses and for the forecasts, but the analyses show a clear improvement compared to the forecasts: by about 0.8 for the mean bias, 4 µg.m -3 for the RMSE and 0.18 for the correlation. Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 14
ENSEMBLE analyses: NO 2 skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 The RMSE of the daily maximum of NO 2 ENSEMBLE analyses reaches the target this quarter. The diurnal patterns of the scores are similar for the analyses and for the forecasts, but the analyses show a clear improvement compared to the forecasts. The benefit of the assimilation is the higher during night-time. Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 15
ENSEMBLE analyses: PM 10 skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 ENSEMBLE production of PM 10 analyses has not been implemented yet. Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 16
Analysis of ENSEMBLE performances for the quarter The ENSEMBLE scores reach their target this quarter for every pollutant (O 3, NO 2, PM 10, forecasts and analyses). The ozone RMSE and correlation scores are similar as during past winter, but the RMSE of daily ozone maximum has improved by about 2 µg.m -3 compared to past winter. For the correlation of all pollutant forecasts, there is a perceptible decreasing tendency with forecast day. The performance of ENSEMBLE ozone, NO 2 and PM 10 forecasts are similar compared to past winter, except for the ozone daily maximum which has improved significantly. It was the first winter during which scores for the analyses have been reported. The ENSEMBLE analyses for O 3 and NO 2 perform significantly better than the ENSEMBLE forecasts for all the scores. The impact of assimilation of NO 2 is the highest during night-time. On-going model developments in every individual model, concerning the implementation or refinement of secondary aerosol formation, and also the assimilation of new observed species at the surface, will help to improve ENSEMBLE scores in the future. Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances 17