Regional Production Quarterly report on the daily analyses and forecasts activities, and verification of the CHIMERE performances December 2015 January 2016 February 2016
This document has been produced in the context of the Copernicus Atmosphere Monitoring Service (CAMS). The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of CAMS on behalf of the European Union (Delegation Agreement signed on 11/11/2014). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and liability. For the avoidance of all doubts, the European Commission and the European Centre for Medium-Range Weather Forecasts has no liability in respect of this document, which is merely representing the authors view. Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances
Quarterly report on the daily analyses and forecasts activities, and verification of the CHIMERE performances December 2015 January 2016 February 2016 INERIS (F. Meleux, A. Ung, L. Rouïl) METEO-FRANCE (M. Pithon, M. Plu, J. Parmentier, J. Arteta, S. Guidotti, N. Assar) Date: 04/05/2016 REF.: CAMS50_2015SC1_D50.3.2.CHIMERE-2016Q1_201605 CAMS50_2015SC1_D50.3.4.CHIMERE-2016Q1_201605 CAMS50_2015SC1_D50.5.1.CHIMERE-2016Q1_201605 Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances
Contents: 1. Executive Summary...4 2. The CHIMERE model (INERIS)...5 Product portfolio...5 Availability statistics...5 Use for observations for data assimilation...7 3. Verification report...10 Verification of NRT forecasts...10 Verification of NRT analyses...15 Analysis of CHIMERE performances for the quarter...19 Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances
1. Executive Summary The Copernicus Atmosphere Monitoring Service (CAMS, www.copernicusatmosphere.eu) is establishing the core global and regional atmospheric environmental service delivered as a component of Europe's Copernicus programme. The regional forecasting service provides daily 4-days forecasts of the main air quality species and analyses of the day before, from 7 state-of-the-art atmospheric chemistry models and from the median ensemble calculated from the 7 model forecasts. The regional service also provides posteriori reanalyses using the latest validated observation dataset available for assimilation. This report covers the deliverables related to Near Real Time Production (NRT) for CHIMERE: D50.3.2-2016Q1, D50.3.4-2016Q1, D50.5.1-2016Q1, for the quarter December 2015 January 2016 February 2016. Verification is done against in-situ surface observations; they are described in the report D50.1.2-2016Q1, that will be delivered shortly. The verification of analyses is done against non assimilated observations. A significant improvement was done during this quarter related to the robustness of the CHIMERE production, but as it has been set-up in January/February the impact on the reliability of the CHIMERE production is not complete yet. Nevertheless, the availability of CHIMERE files (forecasts and analyses) during this quarter is higher than in the past. Regarding the quality of the performances, the main point is that the sharp decrease of the CHIMERE ozone forecast quality of the last quarter is now over. Scores for all pollutants are now common and for most of them close to the ENSEMBLE values. The CHIMERE analysis processing provides an overall improvement of the CHIMERE forecasts. Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 4
2. The CHIMERE model (INERIS) Product portfolio Name Forecast Analysis Description Forecast at surface, 50m, 250m, 500m, 1000m, 2000m, 3000m, 5000m above ground Analysis at the surface Available for users at 6:00 UTC 09:45 UTC for the day before Species O 3, NO 2, CO, SO 2, PM 2.5, PM 10, NO, NH 3, NMVOC, PANs, Birch pollen at surface during season O 3, NO 2, CO, SO 2, PM 2.5, PM 10, NO Time span 0-96h, hourly 0-24h for the day before, hourly Availability statistics The statistics below describe the ratio of days for which the CHIMERE model outputs were available on time to be included in the ENSEMBLE fields (analyses and forecasts) that are computed at METEO-FRANCE. They are based on the following schedule for the provision at METEO-FRANCE of: - forecasts data before: 05:30 UTC for D0-D1 (up to 48h), 07:30 UTC for D2-D3 (from 49h to 96h) - analyses data: before 11:00 UTC These schedules have been set to meet the IT requirements for ENSEMBLE products (no later than 8 UTC for 0-48h, 10 UTC for 49-96h and 12 UTC for analyses). Indicators: Availability_model_Forecast Quarterly basis Availability _model_analysis Quarterly basis D0: 99% D1: 99% D2: 88% D3: 73% D: 97% Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 5
During this quarter, the following issues have been encountered by the CHIMERE production system: Date From 01/12/2015 to 17/12/2015 and 21/12/2015 From 23/12/2015 to 25/12/2015 and from 27/12/2015 to 29/12/2015 Problem description (origin, effects) Production delay due to a high number of forecast on our computing system. Since then, this number has been significantly reduced to meet the deadline required for the CHIMERE forecast delivery 15/01/2016 Crash of the computing system From 15/02/2016 to 16/02/2016 Production delay due to instability of the computing system 20/02/2016 Production delay due to instability of the computing system From 23/02/16 to 24/02/2016 Analysis missing due to unavailability of observations Impact on production Last days (D2-D3) forecast results arrive too late for participating to ENSEMBLE calculation. ENSEMBLE with 6 members for D3 forecasts and sometimes D2 forecasts No analysis and forecast result available for ENSEMBLE calculation D3 forecast results arrive too late to contribute to ENSEMBLE D3 forecast results arrive too late to contribute to ENSEMBLE No analysis result available for ENSEMBLE calculation Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 6
Use for observations for data assimilation Day Use of observation for CHIMERE December O 3 NO 2 NO SO 2 CO PM 10 PM 2.5 1 7,268 5,022 2 7,304 5,334 3 7,618 5,390 4 7,813 5,555 5 7,518 5,260 6 7,538 5,700 7 7,447 5,218 8 7,451 5,321 9 7,912 5,612 10 7,857 5,322 11 7,831 5,482 12 7,851 5,648 13 7,284 4,972 14 3,184 5,512 15 7,878 5,339 16 7,367 4,654 17 3,332 5,499 18 7,581 5,619 19 125 81 20 7,220 5,541 21 3,459 5,481 22 7,743 5,335 23 7,869 5,350 24 7,617 5,140 25 7,409 5,237 26 750 392 27 0 0 28 653 416 29 858 577 30 952 588 31 955 569 average 5,601 4,231 Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 7
Day Use of observation for CHIMERE January O 3 NO 2 NO SO 2 CO PM 10 PM 2.5 1 1,400 986 2 1,686 1,179 3 262 170 4 3,453 2,903 5 5,213 2,662 6 7,502 5,151 7 1,361 776 8 6,739 4,337 9 1,359 775 10 4 0 11 2,360 1,435 12 5,493 2,464 13 8,040 5,128 14 7,964 5,217 15 3,342 1,843 16 2,320 1,490 17 2,239 1,408 18 8,105 5,368 19 8,120 5,403 20 8,268 5,419 21 8,190 5,536 22 8,173 5,501 23 8,056 5,267 24 7,831 5,219 25 8,212 5,425 26 4,018 2,778 27 7,774 5,216 28 8,055 5,358 29 8,181 5,389 30 7,770 5,192 31 7,930 5,114 average 5,465 3,552 Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 8
Day Use of observation for CHIMERE February O 3 NO 2 NO SO 2 CO PM 10 PM 2.5 1 7,890 5,146 2 2,433 1,421 3 7,983 5,286 4 7,972 5,369 5 8,004 5,470 6 8,036 5,333 7 7,923 5,357 8 7,915 5,318 9 7,773 4,962 10 7,536 5,133 11 7,774 5,232 12 7,162 5,012 13 6,950 4,959 14 7,191 5,070 15 7,581 5,164 16 7,657 5,257 17 7,207 4,941 18 4,215 2,759 19 11 0 20 7,194 4,757 21 7,435 4,617 22 7,476 4,722 23 6,406 4,132 24 7,389 4,980 25 7,437 4,961 26 5,528 3,715 27 174 0 28 173 0 29 8 0 average 6,153 4,106 Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 9
3. Verification report This verification report covers the period December 2015 January 2016 February 2016. The CHIMERE skill scores are successively presented for four pollutants: ozone, NO 2, PM 10 and PM 2.5. The skill is shown for the entire forecast horizon from 0 to 96h (hourly values), allowing to evaluate the entire diurnal cycle and the evolution of performance from day 0 to day 3. The forecasts and the analyses cover a large European domain (25 W-45 E, 30 N-70 N). The statistical scores that are reported are the root-mean-square error, the modified mean bias and the correlation. The surface observations that are acquired by METEO-FRANCE and used for verification are described in D50.1.2-2016Q1, that will be delivered shortly. Verification of NRT forecasts The following figures present, for each pollutant (ozone, NO 2, PM 10, PM 2.5 ): - in the upper-left panel, the root-mean square error of daily maximum (for ozone and NO 2 ) or of daily mean (PM 10 ) for the first-day forecasts with regards to surface observations, for every quarter since DJF2014/2015, a target reference value is indicated as an orange line, - in the upper-right panel, the root-mean square error of pollutant concentration forecasts with regards to surface observations as a function of forecast term, - in the lower-left panel, the modified mean bias of pollutant concentration forecasts with regards to surface observations as a function of forecast term, - in the lower-right panel, the correlation of pollutant concentration forecasts with regards to surface observations as a function of forecast term. The graphics show the performance of CHIMERE (black curves) and of the ENSEMBLE (blue curves). Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 10
CHIMERE forecasts: ozone skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 Compared to previous quarters, CHIMERE recovers a RMSE close to the ENSEMBLE one and below the KPI. If we take a closer look at the RMSE, it is noticeable that CHIMERE has a higher RMSE than the ENSEMBLE, with a maximum difference of 7 µg/m3 occurring in the morning and a minimum difference of 3 µg/m3 in the early afternoon, usually when O 3 daily peak occurs. The daily time profile of the RMSE is similar for ENSEMBLE and CHIMERE and stable along the forecast days. The profile for the bias look very similar for ENSEMBLE and CHIMERE and consistent with the RMSE. CHIMERE overestimates the observations more than the ENSEMBLE. CHIMERE correlation is very close to the ENSEMBLE correlation, and both depict a continuous decrease along the forecast time. Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 11
CHIMERE forecasts: NO 2 skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 As for the two previous quarters, CHIMERE has a RMSE close to the ENSEMBLE but with a growing gap. The NO 2 RMSE is still below the KPI value. Looking at the MMB, the score spotlights that the underestimation of CHIMERE is higher than the ENSEMBLE. This result does not impact the RMSE and correlation of CHIMERE, which are similar to the ENSEMBLE scores even if some differences may occur less than 2 µg/m3 for the RMSE and with a more pronounced gap just before noon, for the correlation. The time varying evolution of the scores shows a decrease of the performance for RMSE and correlation and stability for MMB. Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 12
CHIMERE forecasts: PM 10 skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 CHIMERE depicts a similar RMSE than the ENSEMBLE for the PM 10 daily mean concentrations over the 4 last quarters. The MMB shows a higher underestimation in CHIMERE than in the ENSEMBLE. The CHIMERE RMSE is also slightly above the ENSEMBLE RMSE, and the temporal evolution shows a slight increase of the score for both. The CHIMERE correlation is 0.1 below the ENSEMBLE one, and both have a decrease with the same magnitude along the forecast time. Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 13
CHIMERE forecasts: PM 2.5 skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 CHIMERE has a similar RMSE than the ENSEMBLE for the PM 2.5 daily mean concentrations over the 4 last quarters. The MMB shows a higher underestimation in CHIMERE than in the ENSEMBLE. The CHIMERE RMSE is also slightly above the ENSEMBLE RMSE. The CHIMERE correlation is slightly below the ENSEMBLE one and it has a faster decrease of the scores along the forecast time. The daily time profiles are very similar for both forecasts. The results of the PM 2.5 are similar to the PM 10 results with better values for all scores. Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 14
Verification of NRT analyses The following figures present, for each pollutant (ozone, NO 2, PM 10 ): - in the upper-left panel, the root-mean square error of daily maximum (for ozone and NO 2 ) or of daily mean (PM 10 ) for the analyses (solid line) and for the first-day forecasts (dashed line) with regards to surface observations, for every quarter since DJF2014/2015, a target reference value is indicated as an orange line, - in the upper-right panel, the root-mean square error of pollutant concentration of the analyses (solid line) and of the first-day forecasts (dashed line), with regards to surface observations as a function of forecast term, - in the lower-left panel, the modified mean bias of pollutant concentration forecasts of the analyses (solid line) and of the first-day forecasts (dashed line), with regards to surface observations as a function of forecast term, - in the lower-right panel, the correlation of pollutant concentration of the analyses (solid line) and of the first-day forecasts (dashed line), with regards to surface observations as a function of forecast term. The graphics show the performances of CHIMERE (black curves) and of the ENSEMBLE (blue curves). The superposition of the analysis scores (solid lines) and of the forecast scores (dashed lines) computed over the same observation dataset is helpful to assess the added value of data assimilation. Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 15
CHIMERE analyses: ozone skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 CHIMERE analysis has RMSE for this quarter, with similar values than for the ENSEMBLE with a slight improvement compared to the previous quarter. Regarding the other scores, it is worth noting that the assimilation reduces significantly the gap between CHIMERE and the ENSEMBLE compared to the scores established from the raw outputs. The assimilation process leads to a significant improvement of the RMSE which is better than the ENSEMBLE RMSE, the MMBs are very close whereas the correlation of the CHIMERE is better than the ENSEMBLE correlation. Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 16
CHIMERE analyses: NO 2 skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 No NO 2 assimilation in CHIMERE yet. It is foreseen very soon. Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 17
CHIMERE analyses: PM 10 skill scores against data from representative sites, period December 2015 - January 2016 - February 2016 The impact of the assimilation stage provides positive effects on the quality of the representation of the PM 10 surface concentrations. It is worth noting that the performances of the analyses are less good at the end of the day, maybe due to a decrease of observations availability. The MMB shows that the assimilation process provides overestimation of the PM 10 surface concentrations whereas the CHIMERE forecasts tend to underestimate. Compared to the previous quarter the RMSE calculated from the daily mean concentrations shows a stability of the performances. Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 18
Analysis of CHIMERE performances for the quarter The main point of the CHIMERE performance for this quarter is the ozone production which is back to a level close to the ENSEMBLE. Nevertheless, the CHIMERE scores for ozone show up differences which indicates that CHIMERE is still perfectible for this period of the year when ozone concentrations are usually at their lowest level. For NO 2 and PMs, CHIMERE behaviour over this quarter is very close to the ENSEMBLE even if most of the time the CHIMERE performance are lower that the ENSEMBLE performances. Still for NO 2 and PM 10, the results are in good agreement with the last year results. The PM 2.5 scores show similar temporal variabilities that the PM 10, but with better scores. The assimilation process improved significantly the O 3 forecasts, and reduced the gap between CHIMERE and the ENSEMBLE. For ozone, it is worth noting that CHIMERE analyses depict scores which are better that the ENSEMBLE. The PM 10 analysis also leads to a significant improvement of the PM 10 surface concentration, compared to the CHIMERE raw outputs. Qr. report on daily analyses and forecasts activities, verification of the CHIMERE performances 19