ECMWF COPERNICUS REPORT Copernicus Atmosphere Monitoring Service Regional Production, Quarterly report on the daily analyses and forecasts activities, and verification of the CHIMERE performances June July August 2016 Issued by: METEO-FRANCE / S. Guidotti Date: 28/10/2016 Ref: CAMS50_2015SC1_D50.3.1.2.CHIMERE_201610_Daily_Analyses_Report_v1 CAMS50_2015SC1_D50.3.2.2.CHIMERE_201610_Daily_Forecasts_Report_v1 CAMS50_2015SC1_D50.5.1.1.CHIMERE_201610_NRT_Verification_Report_v1
This document has been produced in the context of the Copernicus Atmosphere Monitoring Service (CAMS). The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of CAMS on behalf of the European Union (Delegation Agreement signed on 11/11/2014). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and liability. For the avoidance of all doubts, the European Commission and the European Centre for Medium-Range Weather Forecasts has no liability in respect of this document, which is merely representing the authors view.
Regional Production, Quarterly report on the daily analyses and forecasts activities, and verification of the CHIMERE performances June July August 2016 INERIS F. Meleux A. Ung L. Rouïl METEO-FRANCE M. Pithon M. Plu J. Parmentier J. Arteta S. Guidotti N. Assar Date: 28/10/2016 Ref: CAMS50_2015SC1_D50.3.1.2.CHIMERE_201610_Daily_Analyses_Report_v1 CAMS50_2015SC1_D50.3.2.2.CHIMERE_201610_Daily_Forecasts_Report_v1 CAMS50_2015SC1_D50.5.1.1.CHIMERE _201610_NRT_Verification_Report_v1 CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 3 of 22
Table of Contents 1. The CHIMERE model 6 1.1 Product portfolio 6 1.2 Availability statistics 6 1.2.1 Indicators 6 1.2.2 Problems encountered 7 1.3 Use of observations for data assimilation 7 1.3.1 Use of observations June 2016 8 1.3.2 Use of observations July 2016 9 1.3.3 Use of observations August 2016 10 2. Verification report 11 2.1 Verification of NRT forecasts 11 2.1.1 CHIMERE forecasts: ozone skill scores against data from representative sites 12 2.1.2 CHIMERE forecasts: NO 2 skill scores against data from representative sites 13 2.1.3 CHIMERE forecasts: PM10 skill scores against data from representative sites 14 2.1.4 CHIMERE forecasts: PM2.5 skill scores against data from representative sites 15 2.2 Verification of NRT analyses 16 2.2.1 CHIMERE analyses: ozone skill scores against data from representative sites 17 2.2.2 CHIMERE analyses: NO 2 skill scores against data from representative sites 18 2.2.3 CHIMERE analyses: PM10 skill scores against data from representative sites 19 2.2.4 CHIMERE analyses: PM2.5 skill scores against data from representative sites 20 2.3 Analysis of the CHIMERE performances over the quarter 21 CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 4 of 22
Executive summary The Copernicus Atmosphere Monitoring Service (CAMS, atmosphere.copernicus.eu/) is establishing the core global and regional atmospheric environmental service delivered as a component of Europe's Copernicus programme. The regional forecasting service provides daily 4-day forecasts of the main air quality species and analyses of the day before, from 7 state-of-the-art atmospheric chemistry models and from the median ensemble calculated from the 7 model forecasts. The regional service also provides posteriori reanalyses using the latest validated observation dataset available for assimilation. This report covers the D50.3.1.2, D50.3.2.2 and D50.5.1.1 deliverables related to the CHIMERE Near Real Time Production (NRT), for the quarterly period ending August 31 st, 2016. Verification is done against in-situ surface observations; they are described in the D50.1.1.2 report covering the same period. The verification of analyses is done against non-assimilated observations. During this quarter, the CHIMERE production was affected by compute server failures. The D3 forecasts provision only reached 35% of availability: results arrived behind the set time-limit, making it impossible to take them into account for the ENSEMBLE. To address this issue, the production hours of the D3 ENSEMBLE calculation will be shifted 30mn later; this will allow the CHIMERE data to be considered. Sometimes, the overloading of the servers also caused model crashes. The availability of the first 3 days forecasts reached, however, 83% in average and 87% for analysis results. CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 5 of 22
1. The CHIMERE model 1.1 Product portfolio Item Forecast Analysis Description Forecast at surface, 50m, 250m, 500m, 1000m, 2000m, 3000m, 5000m above ground Analysis at the surface Available for users at 6:00 UTC 09:45 UTC for the day before Species O 3, NO 2, CO, SO 2, PM2.5, PM10, NO, NH 3, NMVOC, PANs, Birch pollen at surface during season O 3, PM10, NO 2, CO*, SO 2 *, PM2.5*, NO*, NH 3 *, NMVOC*, PANs* Time span 0-96h, hourly 0-24h for the day before, hourly * Non-assimilated species 1.2 Availability statistics The statistics below describe the ratio of days for which the CHIMERE model outputs were available on time to be included in the ENSEMBLE fields (analyses and forecasts) that are computed at METEO- FRANCE. They are based on the following schedule for the provision at METEO-FRANCE of: Forecasts data before: 05:30 UTC for D0-D1 (up to 48h), 07:30 UTC for D2-D3 (from 49h to 96h); Analyses data: before 11:00 UTC. These schedules were set to meet the IT requirements for ENSEMBLE products (no later than 8 UTC for 0-48h, 10 UTC for 49-96h and 12 UTC for analyses). 1.2.1 Indicators Availability_model_Forecast Quarterly basis Availability_model_Analysis Quarterly basis D0: 86% D1: 80% D2: 84% D3: 35% D: 87% CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 6 of 22
1.2.2 Problems encountered The following issues were encountered by the CHIMERE production system: Date Problem description Impact on production 01/06, 02/06, 16/06, 20/06, Overloading of compute Analysis and Forecast (D0-D3) 24/06, 13/07, 14/07, 22/07, servers that caused system data missing for ENS 05/08, 27/08, 28/08, crashes. calculation. 30/08/2016 1.3 Use of observations for data assimilation Please see the next three pages. CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 7 of 22
1.3.1 Use of observations June 2016 Day O 3 NO 2 NO SO 2 C0 PM10 PM2.5 1 7,606 7,943 5,204 2 7,741 8,151 5,394 3 7,866 8,199 5,336 4 7,448 7,778 4,941 5 7,615 7,870 5,136 6 7,466 8,055 5,268 7 2,344 2,599 1,653 8 7,575 8,056 5,132 9 7,774 8,205 5,240 10 7,750 8,128 5,155 11 7,204 7,681 5,007 12 7,070 7,478 4,836 13 7,559 8,059 5,292 14 7,819 8,120 5,315 15 7,164 7,457 4,838 16 5,046 5,322 3,552 17 6,619 6,912 4,858 18 6,658 6,862 4,874 19 5,421 5,800 4,024 20 5,745 5,971 4,381 21 7,759 8,163 5,245 22 7,951 8,333 5,279 23 7,986 8,291 5,237 24 7,705 8,141 5,372 25 7,445 7,689 5,208 26 7,635 7,426 5,163 27 7,809 8,008 5,056 28 7,919 8,104 5,208 29 8,251 8,163 5,518 30 311 203 205 CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 8 of 22
1.3.2 Use of observations July 2016 Day O 3 NO 2 NO SO 2 C0 PM10 PM2.5 1 0 0 0 2 0 0 0 3 0 0 0 4 3,509 3,922 2,475 5 5,506 6,043 3,792 6 2,971 3,302 2,034 7 7,625 7,891 5,145 8 7,923 8,283 5,388 9 7,654 7,996 5,241 10 7,498 7,737 5,180 11 8,154 8,012 5,643 12 7,939 8,178 5,451 13 4,554 4,828 3,172 14 7,731 7,991 5,227 15 205 208 92 16 0 0 0 17 7,735 8,003 5,347 18 228 252 159 19 0 0 0 20 3,627 3,848 2,610 21 7,501 5,279 5,222 22 7,651 8,028 5,339 23 7,395 7,813 5,236 24 166 164 77 25 0 0 0 26 255 259 100 27 7,355 7,784 5,117 28 7,647 7,949 5,130 29 195 198 95 30 7,771 7,780 5,478 31 7,657 7,827 5,351 CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 9 of 22
1.3.3 Use of observations August 2016 Day O 3 NO 2 NO SO 2 C0 PM10 PM2.5 1 7,536 7,861 5,229 2 7,779 8,096 5,211 3 7,706 7,884 93 4 7,718 7,929 5,122 5 201 194 5,240 6 7,373 7,555 5,201 7 7,360 7,453 5,129 8 200 193 95 9 7,501 7,570 5,126 10 7,587 7,731 5,142 11 7,697 7,829 5,218 12 7,898 8,119 5,477 13 7,814 7,974 5,371 14 6,891 207 4,419 15 208 0 102 16 0 0 0 17 0 7,894 0 18 7,408 7,885 5,065 19 7,891 8,156 5,490 20 7,819 8,127 5,440 21 7,778 8,012 5,543 22 7,833 8,284 5,528 23 7,678 8,039 5,196 24 7,379 7,820 5,045 25 7,970 8,307 5,370 26 8,023 8,302 5,529 27 7,907 8,165 5,402 28 7,845 8,097 5,354 29 4,205 4,460 2,931 30 0 0 0 31 428 308 95 CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 10 of 22
2. Verification report This verification report covers the quarterly period ending August 31 st, 2016. The CHIMERE skill scores are successively presented for four pollutants: ozone, NO 2, PM10 and PM2.5. The skill is shown for the entire forecast horizon from 0 to 96h (hourly values), allowing to evaluate the entire diurnal cycle and the evolution of performance from day 0 to day 3. The forecasts and the analyses cover a large European domain (25 W-45 E, 30 N-70 N). The statistical scores that are reported are the root-meansquare error, the modified mean bias and the correlation. The surface observations that are acquired by METEO-FRANCE and used for verification are described in D50.1.1.2 covering the same period. The availability of both NRT analyses and NRT forecasts were lower than expected during this quarter. Actions have been taken at INERIS in order to increase the reliability of this production. This is however a heavy task, consisting in moving all production toward a new production system with a higher level of monitoring; this work will take at least 6 months. Note that in order to minimise the impact of shorts delays, it was decided to delay the ENSEMBLE computation (30 minutes), and it remains within the deadline. 2.1 Verification of NRT forecasts The following figures present, for each pollutant (ozone, NO 2, PM10, PM2.5): In the upper-left panel, the root-mean square error of daily maximum (for ozone and NO 2 ) or of daily mean (PM10 and PM2.5) for the first-day forecasts with regards to surface observations, for every quarter since DJF2014/2015, a target reference value is indicated as an orange line; In the upper-right panel, the root-mean square error of pollutant concentration forecasts with regards to surface observations as a function of forecast term; In the lower-left panel, the modified mean bias of pollutant concentration forecasts with regards to surface observations as a function of forecast term; In the lower-right panel, the correlation of pollutant concentration forecasts with regards to surface observations as a function of forecast term. The graphics show the performance of CHIMERE (black curves) and of the ENSEMBLE (blue curves). CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 11 of 22
2.1.1 CHIMERE forecasts: ozone skill scores against data from representative sites Compared to previous quarters, CHIMERE has a daily maximum RMSE slightly above the ENSEMBLE one and if ENS remains below the KPI, it is not the case for CHIMERE RMSE which is slightly above the KPI. If we look further at the RMSE, it is noticeable that CHIMERE has a higher RMSE than the ENSEMBLE with a maximum difference of 4 µg/m3 occurring in the morning and a minimum difference of 2 µg/m3 in the early afternoon, usually when O 3 daily peak occurs. The daily time profile of the RMSE is similar for ENS and CHI and shows a low decrease of the performances along the forecast term. The profile for the bias look very similar for ENS and CHI and very close. CHIMERE correlation is a bit lower than the ENSEMBLE correlation, with a daily time profile slightly different especially in the afternoon. Both correlations depict a continuous decrease along the forecast time. CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 12 of 22
2.1.2 CHIMERE forecasts: NO 2 skill scores against data from representative sites Like for the previous quarters, CHIMERE has a RMSE close to the ENSEMBLE and both are still below the KPI value. Looking at the MMB, the score spotlights that the underestimation of CHIMERE is higher than the ENSEMBLE around midday, and that CHIMERE has a low bias during night time. Time profiles are similar for both, showing a significant increase of the underestimation during morning hours. The CHIMERE RMSE is a little bit higher than the ENSEMBLE RMSE. Correlation of CHIMERE is similar to the ENSEMBLE correlation, except in the early afternoon where a low difference occurs. The time evolution of the scores are very similar for both forecasts, and it shows a decrease of the performance with the forecast term. CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 13 of 22
2.1.3 CHIMERE forecasts: PM10 skill scores against data from representative sites The performances of CHIMERE and the ENSEMBLE are very close over the last quarters for the PM10 daily mean concentrations RMSE, and both are well below the KPI. The MMB shows that CHIMERE and ENSEMBLE are very close and they mostly underestimate the observations. The CHIMERE RMSE is similar to the ENSEMBLE, with a slight difference in the morning which becomes higher from D2. From D2, CHIMERE shows a higher RMSE as well in the afternoon. The CHIMERE correlation is slightly below the ENSEMBLE one and both decrease with the same magnitude along the forecast time. CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 14 of 22
2.1.4 CHIMERE forecasts: PM2.5 skill scores against data from representative sites CHIMERE has a similar RMSE than the ENSEMBLE for the PM2.5 daily mean concentrations over the last quarters. The MMB shows a lower underestimation in CHIMERE than in the ENSEMBLE. The CHIMERE RMSE is slightly higher than the ENSEMBLE RMSE. The CHIMERE correlation is slightly below the ENSEMBLE one. The time variation of the scores looks the same for CHIMERE and the ENSEMBLE, with a decrease of the performances along with the forecast terms. CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 15 of 22
2.2 Verification of NRT analyses The following figures present, for each pollutant (ozone, NO 2, PM10 and PM2.5): In the upper-left panel, the root-mean square error of daily maximum (for ozone and NO 2 ) or of daily mean (PM10) for the analyses (solid line) and for the first-day forecasts (dashed line) with regards to surface observations, for every quarter since DJF2014/2015, a target reference value is indicated as an orange line; In the upper-right panel, the root-mean square error of pollutant concentration of the analyses (solid line) and of the first-day forecasts (dashed line), with regards to surface observations as a function of forecast term; In the lower-left panel, the modified mean bias of pollutant concentration forecasts of the analyses (solid line) and of the first-day forecasts (dashed line), with regards to surface observations as a function of forecast term; In the lower-right panel, the correlation of pollutant concentration of the analyses (solid line) and of the first-day forecasts (dashed line), with regards to surface observations as a function of forecast term. The graphics show the performances of CHIMERE (black curves) and of the ENSEMBLE (blue curves). The superposition of the analysis scores (solid lines) and of the forecast scores (dashed lines) computed over the same observation dataset is helpful to assess the added value of data assimilation. CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 16 of 22
2.2.1 CHIMERE analyses: ozone skill scores against data from representative sites CHIMERE analyses have RMSE for this quarter (like for the previous one) with similar values than for the ENSEMBLE. Regarding the other scores, it is worth noting that the assimilation reduces significantly the gap between CHIMERE RMSE and the ENSEMBLE RMSE compared to the scores established from the raw outputs, especially during night times. A difference of 2 µg/m3 remains during daytime. The MMBs are very close and shows that both analyses have low bias and that the assimilation process led to a decrease of the overestimation. The correlation of CHIMERE is very close to the ENSEMBLE correlation and they are both better in AN than in FC. CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 17 of 22
2.2.2 CHIMERE analyses: NO 2 skill scores against data from representative sites This is the first quarter of evaluation of the CHIMERE NO 2 analyses. The RMSE of the daily maximum is lower for CHIMERE analyses than for the ENSEMBLE, showing a significant added-value of the assimilation process in the performance of the CHIMERE outputs. The MMB of CHIMERE is very low with a flat profile when forecasts and ENSEMBLE AN depict an underestimation during daytime. It is also shown by the hourly RMSE, which is better in CHIMERE AN than in ENS AN. The correlation of the analyses is very similar, with higher values in the morning than in the afternoon. CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 18 of 22
2.2.3 CHIMERE analyses: PM10 skill scores against data from representative sites CHIMERE analyses have RMSE for this quarter (like for the previous one) for the daily mean concentrations which are slightly better than for the ENSEMBLE. Regarding the other scores, it is worth noting that the CHIMERE assimilation process led to a significant improvement of the PM10 surface concentrations, compared to the raw forecasts. The assimilation process leads to a significant improvement of the CHIMERE FC RMSE. The MMB shows that CHIMERE AN has almost no bias. The correlation of CHIMERE is a bit higher than the ENSEMBLE correlation, but values around 0.5 show that it is still perfectible. CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 19 of 22
2.2.4 CHIMERE analyses: PM2.5 skill scores against data from representative sites There is no PM2.5 assimilation in CHIMERE so far, so here is just a comparison of the performances of CHIMERE D0 with D-1. The scores for D-1 are a bit better than the forecast, due to the use of more recent meteorological fields in the run. The time profiles are very similar for both and consistent. Compared to the ENSEMBLE analyse scores, CHIMERE values are less good, but differences are not so important except for correlation. CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 20 of 22
2.3 Analysis of the CHIMERE performances over the quarter The scores depicted in this report are in line with scores calculated for the previous quarter for all pollutants, meaning a stability of the CHIMERE performances, for the forecasts as well as for the analyses. The CHIMERE FC scores for ozone remain perfectible according to the significant differences in the scores between ENSEMBLE and CHIMERE. For NO 2 and PMs, CHIMERE FC behaviour over this quarter is very close to the ENSEMBLE, even if most of the time the CHIMERE performance are lower than the ENSEMBLE performances. Still for NO 2 and PM10, the results are in good agreement with the last year results. The PM2.5 scores show a similar temporal variability than the PM10, but with better scores. The assimilation process significantly improved the O 3 forecasts and reduced the gap between CHIMERE and the ENSEMBLE. For ozone, it is worth noting that CHIMERE analyses depict scores which are similar to those of the ENSEMBLE AN. The assimilation process involved in CHIMERE is also very efficient for NO 2 and PM10, leading to improvements which make CHIMERE AN better than the ENSEMBLE analyses. CAMS50_2015SC1 CHIMERE Production Report JJA2016 Page 21 of 22
ECMWF - Shinfield Park, Reading RG2 9AX, UK Contact: info@copernicus-atmosphere.eu atmosphere.copernicus.eu copernicus.eu ecmwf.int