Regional Production, Quarterly report on the daily analyses and forecasts activities, and verification of the CHIMERE performances

Transcription

1 ECMWF COPERNICUS REPORT Copernicus Atmosphere Monitoring Service Regional Production, Quarterly report on the daily analyses and forecasts activities, and verification of the CHIMERE performances September October November 2017 Issued by: METEO-FRANCE / G. Collin Date: 13/02/2018 Ref: CAMS50_2015SC2_D CHIMERE_201802_Daily_Analyses_Report_v1 CAMS50_2015SC2_D CHIMERE_201802_Daily_Forecasts_Report_v1 CAMS50_2015SC2_D CHIMERE_201802_NRT_Verification_Report_v1

2 This document has been produced in the context of the Copernicus Atmosphere Monitoring Service (CAMS). The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of CAMS on behalf of the European Union (Delegation Agreement signed on 11/11/2014). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and liability. For the avoidance of all doubts, the European Commission and the European Centre for Medium-Range Weather Forecasts has no liability in respect of this document, which is merely representing the authors view.

3 Contributors INERIS F. Meleux A. Ung L. Rouïl METEO-FRANCE M. Pithon M. Plu V. Petiot M. Joly J. Arteta G. Collin N. Assar CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 3 of 22

4 Table of Contents 1. The CHIMERE model Product portfolio Availability statistics Indicators Problems encountered Use of observations for data assimilation Use of observations September Use of observations October Use of observations November Verification report Verification of NRT forecasts CHIMERE forecasts: ozone skill scores against data from representative sites CHIMERE forecasts: NO 2 skill scores against data from representative sites CHIMERE forecasts: PM10 skill scores against data from representative sites CHIMERE forecasts: PM2.5 skill scores against data from representative sites Verification of NRT analyses CHIMERE analyses: ozone skill scores against data from representative sites CHIMERE analyses: NO 2 skill scores against data from representative sites CHIMERE analyses: PM10 skill scores against data from representative sites CHIMERE analyses: PM2.5 skill scores against data from representative sites Analysis of the CHIMERE performances over the quarter 21 CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 4 of 22

5 Executive summary The Copernicus Atmosphere Monitoring Service (CAMS, atmosphere.copernicus.eu/) is establishing the core global and regional atmospheric environmental service delivered as a component of Europe's Copernicus programme. The regional forecasting service provides daily 4-day forecasts of the main air quality species and analyses of the day before, from 7 state-of-the-art atmospheric chemistry models and from the median ensemble calculated from the 7 model forecasts. The regional service also provides posteriori reanalyses using the latest validated observation dataset available for assimilation. This report covers the D , D and D deliverables related to the CHIMERE Near Real Time Production (NRT), for the quarterly period ending November 30 th, Verification is done against in-situ surface observations; these are described in the D report covering the same period. The verification of analyses is done against non-assimilated observations. During this quarter, CHIMERE production strengthened its reliability for forecasts. The D0-D3 forecasts provision reached 99% of availability wich is excellent. However, availability figures only reached 63% for for analyses. This delay is due to a higher computing time than expected for analysis runs combined with resource contention in this time frame. An optimisation of the analysis production was implemented successfully in mid-december to reduce these delays for the next quarters. During this quarter, the performances for forecasts were relatively stable compared to the previous quarters. Nevertheless, two issues were identified. The persistence of the fire emissions and certainly the way CHIMERE distribute the fire emissions vertically led to an accumulation of PM concentration over Portugal responsible for anomalies in the CHIMERE RMSE during the afternoon of the first day of forecast. Regarding the analysis, an issue detected at the last production report was corrected during the upgrade period (End of November) and thus didn t manage to get better the scores for the analysis during this quarter. At the beginning of this quarter, an update of the list of stations used for the verification of forecasts and of analyses was done, and a one-hour time shift was applied to the hour of validity of observations. Thus, if any improvement of the CHIMERE performance since last year, it is hardly possible to disentangle it from the effect of the new observation dataset that is used for assimilation and for evaluation. Consistently with this change of dataset, we note in particular an improvement of RMSE for all pollutants, an increase of PM bias (from negative to less negative or to positive), and a degradation of NO2 correlation. CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 5 of 22

6 1. The CHIMERE model 1.1 Product portfolio Item Forecast Analysis Description Forecast at surface, 50m, 250m, 500m, 1000m, 2000m, 3000m, 5000m above ground Analysis at the surface, 50m, 250m, 500m, 1000m, 2000m, 3000m, 5000m above ground Available for users at 6:00 UTC 09:45 UTC for the day before Species O 3, NO 2, CO, SO 2, PM2.5, PM10, NO, NH 3, NMVOC, PANs, Birch pollen at surface during O 3, PM10, NO 2, CO*, SO 2 *, PM2.5*, NO*, NH 3 *, NMVOC*, PANs* season Time span 0-96h, hourly 0-24h for the day before, hourly * Non-assimilated species 1.2 Availability statistics The statistics below describe the ratio of days for which the CHIMERE model outputs were available on time to be included in the ENSEMBLE fields (analyses and forecasts) that are computed at Meteo-France. They are based on the following schedule for the provision at Meteo-France of: Forecasts data before: 05:30 UTC for D0-D1 (up to 48h), 07:30 UTC for D2-D3 (from 49h to 96h); Analyses data: before 11:00 UTC. These schedules were set to meet the IT requirements for ENSEMBLE products (no later than 8 UTC for 0-48h, 10 UTC for 49-96h and 12 UTC for analyses) Indicators Availability_model_Forecast Quarterly basis Availability_model_Analysis Quarterly basis D0: 99% D1: 99% D2: 99% D3: 99% D: 63% CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 6 of 22

7 1.2.2 Problems encountered The following issues were encountered by the CHIMERE production system: Date Problem description Impact on production 06, 08, 12,16,23,26,27/09/2017 Data production delayed due to congestion of HPC nodes because of runtime of tasks too high. 07/10/2017 Data production delayed due to congestion of HPC nodes because of runtime of tasks too high. From 10 to 12, 20,21,24,28,29/10/ , 03, 06, 07, 08, 12, 13, 14, 17, 18, 19, 20, 21,22,24,25, 28, 29, 30/11/2017 Data production delayed due to congestion of HPC nodes because of runtime of tasks too high. Data production delayed due to congestion of HPC nodes because of runtime of tasks too high.. Analysis data arrived too late for ENS calculation. Analysis data arrived too late for ENS calculation. Analysis data arrived too late for ENS calculation. Analysis data arrived too late for ENS calculation. 27/11/2017 Forecast production crash D0-D3 Forecast data missing for ENS calculation 1.3 Use of observations for data assimilation Please see the next three pages. CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 7 of 22

8 1.3.1 Use of observations September 2017 Day O 3 NO 2 NO SO 2 C0 PM10 PM CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 8 of 22

9 1.3.2 Use of observations October 2017 Day O 3 NO 2 NO SO 2 C0 PM10 PM CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 9 of 22

10 1.3.3 Use of observations November 2017 Day O 3 NO 2 NO SO 2 C0 PM10 PM CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 10 of 22

11 2. Verification report This verification report covers the quarterly period ending November 30 th, The CHIMERE skill scores are successively presented for four pollutants: ozone, NO2, PM10 and PM2.5. The skill is shown for the entire forecast horizon from 0 to 96h (hourly values), allowing to evaluate the entire diurnal cycle and the evolution of performance from day 0 to day 3. The forecasts and the analyses cover a large European domain (25 W-45 E, 30 N-70 N). The statistical scores that are reported are the root-mean-square error, the modified mean bias and the correlation. The surface observations that are acquired by Meteo-France and used for verification are described in D covering the same period. 2.1 Verification of NRT forecasts The following figures present, for each pollutant (ozone, NO2, PM10 and PM2.5): In the upper-left panel, the root-mean square error of daily maximum (for ozone and NO2) or of daily mean (PM10 and PM2.5) for the first-day forecasts with regard to surface observations, for every quarter since DJF2014/2015, a target reference value is indicated as an orange line; In the upper-right panel, the root-mean square error of pollutant concentration forecasts with regard to surface observations as a function of forecast term; In the lower-left panel, the modified mean bias of pollutant concentration forecasts with regard to surface observations as a function of forecast term; In the lower-right panel, the correlation of pollutant concentration forecasts with regard to surface observations as a function of forecast term. The graphics show the performance of CHIMERE (black curves) and of the ENSEMBLE (blue curves). CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 11 of 22

12 2.1.1 CHIMERE forecasts: ozone skill scores against data from representative sites The CHIMERE RMSE for the daily ozone maximum is below the KPI with a better RMSE than it was one year ago for SON2016. It remains higher than the Ensemble RMSE. The hourly RMSE shows the same diurnal cycle for CHIMERE and the ENSEMBLE, but CHIMERE has higher values than the ENSEMBLE (5 µg/m3 during daytime and until 7 µg/m3 in the early morning). According to the MMB, both forecasts overestimate the observations with a similar time profile. They have similar correlations showing a slow decrease along with the forecast time. An overall comment is that the best performance occurs for hours in the early afternoon usually when ozone concentration are the highest. CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 12 of 22

13 2.1.2 CHIMERE forecasts: NO 2 skill scores against data from representative sites Like for the past quarters, CHIMERE has a RMSE for the daily maximum below the KPI and close to the Ensemble RMSE. During night-time, CHIMERE has a higher RMSE (2 µg/m3) than the Ensemble whereas during daytime the values are very close. The temporal profiles for both RMSE are almost similar. The MMB indicates that both forecasts underestimate the NO2 concentrations during daytime and overestimate during night-time. Correlations are the same for both forecasts and show a slow decrease along with the forecast time. CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 13 of 22

14 2.1.3 CHIMERE forecasts: PM10 skill scores against data from representative sites The performances of CHIMERE and the ENSEMBLE are very close over the three last quarters looking at the RMSE for PM10 daily mean concentrations, and both forecasts are well below the KPI. The MMB shows that the ENSEMBLE underestimates the PM10 concentrations whereas CHIMERE mostly overestimates. The CHIMERE RMSE is slightly different to the ENSEMBLE RMSE with usually 1-2 µg/m3 of difference. An unusual degradation of the CHIMERE RMSE occurs during the afternoon of the first day compared to the Ensemble. Thereafter the scores are again closed to the Ensemble ones. The CHIMERE correlation is slightly below the ENSEMBLE one, and both show a decrease together with the forecast time. CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 14 of 22

15 2.1.4 CHIMERE forecasts: PM2.5 skill scores against data from representative sites The performances of CHIMERE and the ENSEMBLE are very close over the last quarters looking at the RMSE for PM2.5 daily mean concentrations. The MMB shows that the ENSEMBLE slightly underestimates, whereas the CHIMERE MMB overestimates. The RMSE for CHIMERE and the ENSEMBLE are very close and with the same temporal profile. It s worthy to notice the unusual values of the RMSE on the afternoon of the first day with values even higher than for PM10. From day two the CHIMERE RMSE is back to values close to the Ensemble. We suspect the dust-fire episode over Portugal which occurred in October to be responsible for such scores. Especially the fire emissions which are only active for that day with a temporal profile highly correlated to this sharp RMSE increase. The correlation of the ENSEMBLE is better than the CHIMERE correlation and both show a decrease over the forecast time with a similar rate. CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 15 of 22

16 2.2 Verification of NRT analyses The following figures present, for each pollutant (ozone, NO2, PM10 and PM2.5): In the upper-left panel, the root-mean square error of daily maximum (for ozone and NO2) or of daily mean (PM10) for the analyses (solid line) and for the first-day forecasts (dashed line) with regard to surface observations, for every quarter since DJF2014/2015, a target reference value is indicated as an orange line; In the upper-right panel, the root-mean square error of pollutant concentration of the analyses (solid line) and of the first-day forecasts (dashed line), with regard to surface observations as a function of forecast term; In the lower-left panel, the modified mean bias of pollutant concentration forecasts of the analyses (solid line) and of the first-day forecasts (dashed line), with regard to surface observations as a function of forecast term; In the lower-right panel, the correlation of pollutant concentration of the analyses (solid line) and of the first-day forecasts (dashed line), with regard to surface observations as a function of forecast term. The graphics show the performances of CHIMERE (black curves) and of the ENSEMBLE (blue curves). The superposition of the analysis scores (solid lines) and of the forecast scores (dashed lines) computed over the same observation dataset is helpful to assess the added value of data assimilation. CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 16 of 22

17 2.2.1 CHIMERE analyses: ozone skill scores against data from representative sites A bug was fixed during the last period of upgrade in November. The goal was to correct an issue in the observation dataset processing which was initially bypassed to produce SON. This bug explains that very low differences were observed during this quarter between analysis and forecast. The slight differences are due to more precise meteorological data considered in the analysis production than in the forecasts CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 17 of 22

18 2.2.2 CHIMERE analyses: NO 2 skill scores against data from representative sites A bug was fixed during the last period of upgrade in November. The goal was to correct an issue in the observation dataset processing which was initially bypassed to produce SON. This bug explains that very low differences were observed during this quarter between analysis and forecast. The slight differences are due to more precise meteorological data considered in the analysis production than in the forecasts CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 18 of 22

19 2.2.3 CHIMERE analyses: PM10 skill scores against data from representative sites A bug was fixed during the last period of upgrade in November. The goal was to correct an issue in the observation dataset processing which was initially bypassed to produce SON. This bug explains that very low differences were observed during this quarter between analysis and forecast. The slight differences are due to more precise meteorological data considered in the analysis production than in the forecasts. A more significant gap occurs for RMSE on the afternoon certainly related to the fire emissions. To explain this, we can suppose that the fire parameters (like the persistence) are more representative in the analysis day and led to better PM contributions than in the forecast day. In other words, the accumulation of PM (from fire emissions) higher in the forecast day than in the analysis day degrades the scores CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 19 of 22

20 2.2.4 CHIMERE analyses: PM2.5 skill scores against data from representative sites A bug was fixed during the last period of upgrade in November. The goal was to correct an issue in the observation dataset processing which was initially bypassed to produce SON. This bug explains that very low differences were observed during this quarter between analysis and forecast. The slight differences are due to more precise meteorological data considered in the analysis production than in the forecasts CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 20 of 22

21 2.3 Analysis of the CHIMERE performances over the quarter The performances are in line with the previous quarters if we considered the scores relatively to the Ensemble ones to avoid misinterpretations due to the upgrade of the stations used for evaluation. All CHIMERE forecasts are below the KPI and are usually close to the Ensemble. The time profile of the scores are very similar between Ensemble and CHIMERE. For ozone the differences are significative looking at the RMSE whereas the values are very close for correlation. The NO2 is the pollutant for which CHIMERE and ENSEMBLE have the closest scores. Regarding particulate matter, we noticed an issue with the RMSE of PM10 and PM2.5 in the afternoon of the first day forecast. A significant increase of RMSE seems to be due to the effect of the fire emissions during the event which occurred over Portugal in October, The persistence of two days makes this effect restricted to the first day of forecast. Otherwise, CHIMERE tends to overestimate the PM concentrations when the Ensemble underestimates. The CHIMERE PM correlations are slightly below the ENSEMBLE correlations. For analysis, we had noticed an issue at the last production report for JJA An investigation led to a bug identification. The bug was fixed during the period of upgrade for the operational production. Regarding this analysis production few comments were provided for the analysisforecast comparison. CAMS50_2015SC2 CHIMERE Production Report SON2017 Page 21 of 22

22 ECMWF - Shinfield Park, Reading RG2 9AX, UK Contact: info@copernicus-atmosphere.eu atmosphere.copernicus.eu copernicus.eu ecmwf.int