Regional Production, Quarterly report on the daily analyses and forecasts activities, and verification of the MATCH performances

Transcription

1 ECMWF COPERNICUS REPORT Copernicus Atmosphere Monitoring Service Regional Production, Quarterly report on the daily analyses and forecasts activities, and verification of the MATCH performances September October November 2016 Issued by: METEO-FRANCE / S. Guidotti Date: 31/01/2017 Ref: CAMS50_2015SC2_D MATCH_201701_Daily_Analyses_Report_v1 CAMS50_2015SC2_D MATCH_201701_Daily_Forecasts_Report_v1 CAMS50_2015SC2_D MATCH_201701_NRT_Verification_Report_v1

2 This document has been produced in the context of the Copernicus Atmosphere Monitoring Service (CAMS). The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of CAMS on behalf of the European Union (Delegation Agreement signed on 11/11/2014). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and liability. For the avoidance of all doubts, the European Commission and the European Centre for Medium-Range Weather Forecasts has no liability in respect of this document, which is merely representing the authors view.

3 Contributors SMHI L. Robertson R. Bergström E. Engström METEO-FRANCE M. Pithon M. Plu J. Parmentier J. Arteta S. Guidotti N. Assar CAMS50_2015SC2 MATCH Production Report SON2016 Page 3 of 24

4 Table of Contents 1. The MATCH model Product portfolio Availability statistics Indicators Problems encountered Use of observations for data assimilation Use of observations September Use of observations October Use of observations November Verification report Verification of NRT forecasts MATCH forecasts: ozone skill scores against data from representative sites MATCH forecasts: NO 2 skill scores against data from representative sites MATCH forecasts: PM10 skill scores against data from representative sites MATCH forecasts: PM2.5 skill scores against data from representative sites Verification of NRT analyses MATCH analyses: ozone skill scores against data from representative sites MATCH analyses: NO 2 skill scores against data from representative sites MATCH analyses: PM10 skill scores against data from representative sites MATCH analyses: PM2.5 skill scores against data from representative sites Analysis of the MATCH performances over the quarter 23 CAMS50_2015SC2 MATCH Production Report SON2016 Page 4 of 24

5 Executive summary The Copernicus Atmosphere Monitoring Service (CAMS, atmosphere.copernicus.eu/) is establishing the core global and regional atmospheric environmental service delivered as a component of Europe's Copernicus programme. The regional forecasting service provides daily 4-day forecasts of the main air quality species and analyses of the day before, from 7 state-of-the-art atmospheric chemistry models and from the median ensemble calculated from the 7 model forecasts. The regional service also provides posteriori reanalyses using the latest validated observation dataset available for assimilation. This report covers the D , D and D deliverables related to the MATCH Near Real Time Production (NRT), for the quarterly period ending November 30 th, Verification is done against in-situ surface observations; they are described in the D report covering the same period. The verification of analyses is done against non-assimilated observations. During this quarter, production reliability was good with 100% delivery of D0-D1 forecasts, 99% for D2-D3 forecasts and 95% for analyses. The main failures occurred in October and were related to transfer issues. The MATCH forecasts have improved for ozone with higher correlations than for the JJA period and show better modified mean bias (MMB) than the ENSEMBLE forecasts. Both the MATCH model and the ENSEMBLE overestimate the ozone concentrations during the morning hours (the MMB peak at ~08-09 UTC). For NO 2 MATCH forecasts perform rather similar to the ENSEMBLE, while PM10 and PM2.5 are underestimated. This PM deficit will have high priority for investigation during the first part of The MATCH analyses perform well in terms of reducing bias for all components including PM. Most significant improvement by the analyses is seen for PM10 and PM2.5. CAMS50_2015SC2 MATCH Production Report SON2016 Page 5 of 24

6 1. The MATCH model 1.1 Product portfolio Item Forecast Analysis Description Forecast at surface, 50m, 250m, 500m, 1000m, 2000m, 3000m, 5000m above ground Analysis at the surface Available for users at 3:00 UTC 10:00 UTC for the day before Species O 3, NO 2, CO, SO 2, PM2.5, PM10, NO, NH 3, NMVOC, PANs, Birch pollen at surface during season O 3, NO 2, CO, SO 2, PM2.5, PM10, NO*, NH 3 *, NMVOC*, PANs* Time span 0-96h, hourly 0-24h for the day before, hourly * Non-assimilated species 1.2 Availability statistics The statistics below describe the ratio of days for which the MATCH model outputs were available on time to be included in the ENSEMBLE fields (analyses and forecasts) that are computed at METEO- FRANCE. They are based on the following schedule for the provision at METEO-FRANCE of: Forecasts data before: 05:30 UTC for D0-D1 (up to 48h), 07:30 UTC for D2-D3 (from 49h to 96h); Analyses data: before 11:00 UTC. These schedules were set to meet the IT requirements for ENSEMBLE products (no later than 8 UTC for 0-48h, 10 UTC for 49-96h and 12 UTC for analyses) Indicators Availability_model_Forecast Quarterly basis Availability_model_Analysis Quarterly basis D0: 100% D1: 100% D2: 98% D3: 99% D: 95% CAMS50_2015SC2 MATCH Production Report SON2016 Page 6 of 24

7 1.2.2 Problems encountered The following issues were encountered by the MATCH production system: Date Problem description Impact on production 03/10/2016 MATCH analysis crash. MATCH Analysis results non available for ENS calculation. 04/10/2016 MATCH analyses late: system failure at SMHI (authentication errors between machines). 07/10/2016 Some MATCH forecast steps missing (from 49 to 71): failure in the transfer process. A fix has been added. 11/10/2016 MATCH forecast step 73 missing. Incomplete file transferred. Some checks have been added. MATCH Analysis results non available for ENS calculation. MATCH forecast results (D2) non available for ENS calculation. MATCH forecast results (D3) non available for ENS calculation. 27/10/2016 MATCH forecast step 72 missing. MATCH forecast results (D2) non available for ENS calculation. 20/11/2016 MATCH analyses crash. MATCH Analysis results non available for ENS calculation. 24/11/2016 MATCH analysis: delay in provision of results to METEO- FRANCE. Number of checks increased to avoid this problem. 30/11/2016 MATCH analysis results provided late (issue related to upgrade and restart of server). MATCH Analysis results non available for ENS calculation. MATCH Analysis results non available for ENS calculation. 1.3 Use of observations for data assimilation Please see the next three pages. CAMS50_2015SC2 MATCH Production Report SON2016 Page 7 of 24

8 1.3.1 Use of observations September 2016 Day O 3 NO 2 NO SO 2 C0 PM10 PM ,493 3,145 1,752 1,059 2, ,432 11,530 5,680 2,761 8,338 3, ,477 10,954 5,712 2,136 7,965 2, ,714 10,205 5,146 2,827 7,300 2, ,227 11,150 5,714 2,852 7,656 2, ,729 12,980 6,151 2,799 9,111 3, ,330 12,164 5,940 2,722 8,438 2, ,475 10,127 5,356 2,700 7,037 2, ,980 9,544 5,187 2,689 6,706 2, ,866 9,428 5,168 2,570 6,674 2, ,076 9,758 5,191 2,719 6,801 2, ,896 9,770 5,320 2,617 6,839 2, ,783 9,635 5,208 2,709 6,844 2, ,296 5,304 2,280 3,023 2,280 1, ,007 13,225 5,875 3,088 10,162 3, ,114 9,991 5,107 3,121 7,622 3, ,791 8,231 4,034 2,832 5,857 2, ,088 13,232 5,736 3,263 10,050 3, ,328 13,508 5,650 3,265 10,222 4, ,412 13,436 6,183 3,099 10,235 4, ,294 11,048 4,732 2,604 7,649 3, ,021 2,952 2,156 4,800 2, ,179 5,982 3,077 1,870 4,038 2, ,615 6,461 3,430 2,180 4,161 2, ,919 6,858 3,575 2,248 4,306 2, ,329 7,290 3,570 2,074 4,610 2, ,362 8,598 4,387 2,425 5,951 2, ,649 9,411 4,677 2,764 6,787 2, ,763 12,769 5,661 3,097 9,407 3,771 CAMS50_2015SC2 MATCH Production Report SON2016 Page 8 of 24

9 1.3.2 Use of observations October 2016 Day O 3 NO 2 NO SO 2 C0 PM10 PM ,555 12,286 5,718 3,056 9,209 3, ,279 13,377 5,900 3,233 9,792 3, ,355 13,426 6,139 3,282 9,991 3, ,417 13,532 6,037 3,125 9,997 3, ,602 12,591 5,934 3,181 9,439 3, ,896 10,661 5,361 3,162 7,925 3, ,732 10,439 3,312 3,053 8, ,734 10,598 2,880 2,926 7, ,472 6,336 2,712 1,968 3,408 2, ,400 12,078 5,931 3,210 9,495 3, ,788 10,402 5,620 3,127 8,085 3, ,721 10,473 5,876 3,018 8,365 3, ,387 10,234 5,112 2,856 8,227 3, ,382 10,037 5,671 2,696 7,733 3, ,345 10,116 5,342 2,656 7,699 3, ,462 10,325 5,483 2,821 8,065 3, ,500 10,476 5,546 2,748 7,889 3, ,177 11,006 5,615 2,964 8,150 3, ,887 11,012 5,616 2,990 8,421 3, ,220 3,120 2,624 4,416 2, ,110 10,241 5,200 2,390 7,625 3, ,241 10,463 5,468 2,558 8,449 3, ,690 2,520 2,431 3,648 1, ,729 11,069 5,988 2,680 8,714 3, ,801 11,314 6,036 2,521 8,702 3, ,304 14,417 2,976 2,966 11, ,890 14,128 6,771 3,133 10,470 4, ,608 13,936 6,741 3,093 10,431 4, ,472 14,819 7,130 3,024 10,962 4,730 CAMS50_2015SC2 MATCH Production Report SON2016 Page 9 of 24

10 1.3.3 Use of observations November 2016 Day O 3 NO 2 NO SO 2 C0 PM10 PM ,621 4,320 2,915 5,736 3, ,800 14,888 6,903 3,287 11,116 4, ,673 8,012 6,937 1,848 6,801 2, ,060 14,197 6,981 2,672 10,980 4, ,156 14,189 6,841 2,615 11,075 4, ,172 14,145 6,674 2,869 10,810 4, ,331 14,496 3,456 2,954 5,136 2, ,329 14,834 6,974 2,849 11,223 4, ,050 14,285 4,056 2,844 10,704 4, ,062 11,990 3,576 2,508 9, ,027 12,073 6,429 2,696 8,754 3, ,636 11,693 3,504 2,713 8, ,733 11,838 6,275 2,787 8,492 3, ,784 7,200 3,408 1,824 3,768 2, ,707 12,235 5,859 2,247 9,473 4, ,867 13, , ,502 13,081 6,495 2,089 10,377 4, ,725 11,320 6,218 2,470 8,962 4, ,055 13,932 6,634 2,538 11,087 4, ,869 13,690 6,571 2,313 10,904 4, ,794 13,639 6,462 2,479 10,916 4, ,388 14,268 6,652 2,765 11,141 4, ,853 12,682 6,068 2,411 9,703 4, ,740 13,914 6,483 2,244 10,808 4,808 CAMS50_2015SC2 MATCH Production Report SON2016 Page 10 of 24

11 2. Verification report This verification report covers the quarterly period ending November 30 th, The MATCH skill scores are successively presented for four pollutants: ozone, NO 2, PM10 and PM2.5. The skill is shown for the entire forecast horizon from 0 to 96h (hourly values), allowing to evaluate the entire diurnal cycle and the evolution of performance from day 0 to day 3. The forecasts and the analyses cover a large European domain (25 W-45 E, 30 N-70 N). The statistical scores that are reported are the rootmean-square error, the modified mean bias and the correlation. The surface observations that are acquired by METEO-FRANCE and used for verification are described in D covering the same period. 2.1 Verification of NRT forecasts The following figures present, for each pollutant (ozone, NO 2, PM10 and PM2.5): In the upper-left panel, the root-mean square error of daily maximum (for ozone and NO2) or of daily mean (PM10 and PM2.5) for the first-day forecasts with regards to surface observations, for every quarter since DJF2014/2015, a target reference value is indicated as an orange line; In the upper-right panel, the root-mean square error of pollutant concentration forecasts with regards to surface observations as a function of forecast term; In the lower-left panel, the modified mean bias of pollutant concentration forecasts with regards to surface observations as a function of forecast term; In the lower-right panel, the correlation of pollutant concentration forecasts with regards to surface observations as a function of forecast term. The graphics show the performance of MATCH (black curves) and of the ENSEMBLE (blue curves). CAMS50_2015SC2 MATCH Production Report SON2016 Page 11 of 24

12 2.1.1 MATCH forecasts: ozone skill scores against data from representative sites The RMSE of the daily maximum concentration of surface ozone in MATCH for this period (~16 µg/m 3 ) is lower than for the two preceding 3-month periods, and it is now below the reference target value. The diurnal variation of the RMSE is similar for MATCH as for the ENSEMBLE except during the morning hours (08 11 UTC), when the RMSE is clearly higher than for the ENSEMBLE. The model bias is small during the afternoon, evening and night hours (the modified mean bias, MMB, is below ~0.22 from about UTC; the Ensemble median has a clearly higher MMB than MATCH during the dark hours UTC). For MATCH the MMB rises sharply at about 06 UTC, and peaks at 08 UTC at a similar level as the ENSEMBLE. After about 09 UTC the bias drops rapidly both in MATCH and the ENSEMBLE. The correlation in MATCH for the SON period is markedly higher than for the previous (JJA) period. However, for most hours, the correlation in MATCH is clearly lower than in the Ensemble median during night and morning hours (~00 11 UTC) the difference in the correlation coefficient is about CAMS50_2015SC2 MATCH Production Report SON2016 Page 12 of 24

13 0.1. The MATCH correlation exhibits a distinctive dip during the morning hours (05 09 UTC) and a rapid rise after 09 UTC a second large drop in correlation occurs from about UTC. Earlier MATCH model evaluations (with a model set-up similar to the one used in CAMS) against observational data from the EMEP monitoring network have shown substantially higher correlation coefficients than this CAMS evaluation the evaluation against EMEP sites (for the SON-period, 2012) gave correlation coefficients about 0.1 higher than the ones shown in the figure above, during the hours with the lowest correlation (21 11 UTC). The reason for the large difference in correlation is not known it will be investigated during the model development work in CAMS50_2015SC2 MATCH Production Report SON2016 Page 13 of 24

14 2.1.2 MATCH forecasts: NO 2 skill scores against data from representative sites Both MATCH and the ENSEMBLE are well below the target reference value (25 µg/m 3 ) for RMSE of the daily maximum NO 2 concentration. The modified mean bias for NO 2 is negative for all hours except 05 UTC. MATCH underestimates NO 2 more than the ENSEMBLE does during daytime (08 20 UTC) and less during the dark hours (21 07 UTC). Both MATCH and the ENSEMBLE underestimate NO 2 most around noon (09 14 UTC) this could possibly indicate too rapid photo-dissociation of NO 2 in the models, but there may be other explanations for the underestimation, e.g. related to relatively coarse model resolution and/or the handling of NOx-emissions. The correlation for NO 2 is a bit lower for MATCH than for the ENSEMBLE (r in MATCH vary from ~ , is about lower for MATCH). CAMS50_2015SC2 MATCH Production Report SON2016 Page 14 of 24

15 2.1.3 MATCH forecasts: PM10 skill scores against data from representative sites The PM10 results for MATCH are very different for the SON period compared to the previous 3-month period (JJA). The RMSE for the daily mean PM10 concentrations for MATCH are still relatively close to the ENSEMBLE and well below the target reference value (18 µg/m 3 ), but the modified mean bias is much worse for the SON period than for JJA for SON MATCH underestimates PM10 substantially (MMB in the range %; for JJA the MMB was about %). RMSE also became worse for SON than JJA. The correlation improved slightly but MATCH still has a rather low correlation (r ~ for the first 18 hours of the forecast). CAMS50_2015SC2 MATCH Production Report SON2016 Page 15 of 24

16 2.1.4 MATCH forecasts: PM2.5 skill scores against data from representative sites The deterioration of the SON results compared to JJA is similar for PM2.5 as for PM10. The average modified mean bias for PM2.5 was close to zero for JJA but is about -70% for SON. The Ensemble median underestimates PM2.5 by about 12 32%. The very large underestimation for PM2.5 indicates that the poor model results noted above for PM10 are likely (mainly) due to problems with the fine particle fraction. The correlation for PM2.5 is higher than for PM10 (r ~ , for the first 24 hours, for MATCH, and about units higher for the Ensemble median). The large drop in modelled PM2.5 and PM10, compared to the previous 3-month period, indicates that some major model change (possibly an error/bug) may have been introduced in the MATCH model lately. Another possible explanation is that an earlier bug in the model handling of SOxemissions, which was fixed in the middle of July 2016, may have partially compensated the effects of another (unknown) bug in the model. The problems with the large negative bias for PM in the CAMS- CAMS50_2015SC2 MATCH Production Report SON2016 Page 16 of 24

17 version of MATCH need to be investigated urgently this will have highest priority during the first part of CAMS50_2015SC2 MATCH Production Report SON2016 Page 17 of 24

18 2.2 Verification of NRT analyses The following figures present, for each pollutant (ozone, NO 2, PM10 and PM2.5): In the upper-left panel, the root-mean square error of daily maximum (for ozone and NO2) or of daily mean (PM10) for the analyses (solid line) and for the first-day forecasts (dashed line) with regards to surface observations, for every quarter since DJF2014/2015, a target reference value is indicated as an orange line; In the upper-right panel, the root-mean square error of pollutant concentration of the analyses (solid line) and of the first-day forecasts (dashed line), with regards to surface observations as a function of forecast term; In the lower-left panel, the modified mean bias of pollutant concentration forecasts of the analyses (solid line) and of the first-day forecasts (dashed line), with regards to surface observations as a function of forecast term; In the lower-right panel, the correlation of pollutant concentration of the analyses (solid line) and of the first-day forecasts (dashed line), with regards to surface observations as a function of forecast term. The graphics show the performances of MATCH (black curves) and of the ENSEMBLE (blue curves). The superposition of the analysis scores (solid lines) and of the forecast scores (dashed lines) computed over the same observation dataset is helpful to assess the added value of data assimilation. CAMS50_2015SC2 MATCH Production Report SON2016 Page 18 of 24

19 2.2.1 MATCH analyses: ozone skill scores against data from representative sites The MATCH analyses are below the reference value of RMSE of daily maximum and have stayed so for the latest 4 trimesters. The morning peak of positive MMB in the MATCH forecasts (as also seen in the ENSEMBLE forecasts) is more strongly reduced in the MATCH analyses than in the ENSEMBLE analyses even though not removed entirely. For this SON period the RMSE and corrections are not improved to the level of the ENSEMBLE as was seen in the JJA statistics. The improvement of RMSE is mainly seen where the bias is most strongly corrected. CAMS50_2015SC2 MATCH Production Report SON2016 Page 19 of 24

20 2.2.2 MATCH analyses: NO 2 skill scores against data from representative sites Both the MATCH forecasts and analysis are below the reference value for RMSE of the daily maximum for NO 2. The MMB is improved dramatically during daytime the large dip, with minimum at noon, seen in the forecast model is removed in the analysis. The RMSE is improved by about 2-5 ug/m 3, while the correlation is just slightly improved and stays about 0.15 lower than the correlation of the ENSEMBLE analyses and the MATCH analyses also shows lower correlation than the ENSEMBLE forecasts. The pattern seen for RMSE and correlation is very much the same as for JJA. CAMS50_2015SC2 MATCH Production Report SON2016 Page 20 of 24

21 2.2.3 MATCH analyses: PM10 skill scores against data from representative sites The MATCH forecasts and analyses are clearly below the reference value of RMSE for daily maximum for PM10. The MMB for the MATCH analyses is positive in contrast to the MATCH forecasts and the ENSEMBLE forecasts and analyses that all show negative MMB. For RMSE the MATCH analyses are comparable with the ENSEMBLE analyses. The RMSE is about 10 µg/m3 that is an increase from 8-9 µg/m3 in JJA. The correlation is slightly better for the SON period than for JJA. CAMS50_2015SC2 MATCH Production Report SON2016 Page 21 of 24

22 2.2.4 MATCH analyses: PM2.5 skill scores against data from representative sites The MATCH analyses overestimate PM2.5 concentrations but efficiently correct the negative bias in the MATCH forecasts. The analyses reduce the RMSE during morning hours and late afternoon, while the changes in RMSE are small during the rest of the day. The MATCH analyses improve the correlation in comparison to the MATCH forecasts substantially but do not fully reach the correlation of the ENSEMBLE analyses. CAMS50_2015SC2 MATCH Production Report SON2016 Page 22 of 24

23 2.3 Analysis of the MATCH performances over the quarter The MATCH forecasts have improved for ozone with higher correlations than for the JJA period and show better modified mean bias (MMB) than the ENSEMBLE forecasts. Both the MATCH model and the ENSEMBLE overestimate ozone during the morning hours, which could be investigated further. For NO 2 MATCH forecasts perform rather similar to the ENSEMBLE while PM10 and PM2.5 are underestimated. This PM deficit will have high priority for investigation during first part of The MATCH analyses perform well in terms of reducing bias for all components including PM. Most significant improvement by the analyses is seen for PM10 and PM2.5. CAMS50_2015SC2 MATCH Production Report SON2016 Page 23 of 24

24 ECMWF - Shinfield Park, Reading RG2 9AX, UK Contact: info@copernicus-atmosphere.eu atmosphere.copernicus.eu copernicus.eu ecmwf.int