On the use of the intensity-scale verification technique to assess operational precipitation forecasts

METEOROLOGICAL APPLICATIONS Meteorol. Appl. 5: 45 54 (28) Published online in Wiley InterScience (www.interscience.wiley.com).49 On the use of the intensity-scale verification technique to assess operational precipitation forecasts Gabriella Csima and Anna Ghelli 2 Hungarian Meteorological Service, Budapest, Hungary 2 ECMWF, Reading, UK ABSTRACT: The article describes the attempt to include the intensity-scale technique introduced by Casati et al. (24) into a set of standardized verifications used in operational centres. The intensity-scale verification approach accounts for the spatial structure of the forecast field and allows the skill to be diagnosed as a function of the scale of the forecast error and intensity of the precipitation events. The intensity-scale method has been used to verify two different resolutions of the European Centre for Medium-Range Weather Forecasts (ECMWF) operational quantitative precipitation forecast (QPF) over France, and to compare the performance of the ECMWF and the Hungarian Meteorological Service operational model (ALADIN) forecasts, run over Hungary. Two case studies have been introduced, which show some interesting insight into the spatial scale of the error. The distribution of daily skill score for an extended period of time is also presented. The intensity-scale technique shows that the forecasts in general exhibit better skill for large-scale events, and lower skill for small-scale and intense events. In the paper, it is mentioned how some of the stringent assumptions on the domain over which the method can be applied, and the availability of the matched forecasts and observations, can limit its usability in an operational environment. Copyright 28 Royal Meteorological Society KEY WORDS verification; quantitative precipitation; intensity-scale; wavelet decomposition; binary error image Received 25 September 27; Revised December 27; Accepted 2 January 28. Introduction The variability and the discontinuous nature of quantitative precipitation forecasts make their verification challenging. Traditional verification scores for example, continuous verification scores like RMSE, BIAS, or categorical verification scores such as POD, FAR, FBI do not fully account for the unique characteristic of precipitation. The first group of verification scores is sensitive to discontinuities, noise and outliers, and the second group is sensitive to the bias and the base rate of the event. A complete review of the methods can be found in Wilks (995) and Jolliffe and Stephenson (23). Only a very small number of these verification methods is used by operational centres to verify their quantitative precipitation forecasts (QPF). A recent survey by the Royal Meteorological Society has shown that most national meteorological centres use some kind of verification but very few of the verification results are actually passed onto those making decisions (Mailier et al., 26). Holland et al. (27) have presented an attempt to build a standardized verification package which comprises basic verification methods. The challenge is building standardized packages that are flexible enough to be able to Correspondence to: Gabriella Csima, Hungarian Meteorological Service, Budapest, Kitaibel P., 24 Hungary. E-mail: csima.g@met.hu include new verification methods tested in less standardized environments like research institutes. The present paper describes an attempt to introduce the alternative intensity-scale approach for verification of spatial precipitation forecasts, described by Casati et al. (24), in the standard verification package of two operational centres. The technique allows the skill to be diagnosed as a function of the spatial scale of the forecast error and intensity of the precipitation events. This makes it easier to evaluate mesoscale and convective features (such as fronts or convective cells), drizzle and intense events. The technique is used to compare the performance of quantitative precipitation forecasts for two different resolutions of the European Centre for Medium-Range Weather Forecasts (ECMWF) model, and to assess the performance of ECMWF and the Hungarian Meteorological Service operational model (ALADIN) forecasts over Hungary. In Section 2, a short summary of the new verification method is given. Section 3 introduces the models used and precipitation datasets, and in Section 4, verification results for two case studies are presented, as well as an assessment of the models for an extended period of time. The conclusions are drawn in Section 5. 2. The intensity-scale verification method Forecast and observed precipitation analysis fields were first pre-processed, to obtain more reliable data before Copyright 28 Royal Meteorological Society

46 G. CSIMA AND A. GHELLI verification. All non-zero precipitation values have been adjusted by adding a very small amount of uniformly distributed noise. This adjustment helps compensate for the discretization effects caused by the finite precision storage of the precipitation rate values. To reduce the skewness of the distribution the precipitation rate values have been normalized by performing a logarithmic (base 2) transformation (Casati et al., 24). Thresholds are used to convert the forecast (Y) and analysis (X) into binary images I Y and I X. Forecasts and analyses are transformed into binary images on a rain/norain basis for the rainfall rate thresholds. The difference between binary forecast and binary analysis is defined as the binary error image (Z = I Y I X ). The binary error image is decomposed into the sum of components at different spatial scales by performing a two-dimensional discrete Haar wavelet decomposition. The mean squared error (MSE) of the binary image is given by the average of all the squared differences over all the pixels in the domain. The MSE of the binary error image is equal to where MSE = L MSE l () i= MSE l = Z 2 l (2) is the MSE of the lth spatial scale component of the binary error. For each precipitation rate threshold, the binary MSE skill score can be calculated, relative to the MSE of a random forecast: SS = MSE MSE random (3) MSE best MSE random where MSE best = is the MSE associated with a perfect forecast, and MSE random is the MSE associated with a random forecast calculated from the bias and the base rate at each threshold. 3. The models and datasets The technique has been applied to two ECMWF forecast datasets in order to compare how the resolution can affect the precipitation forecast. Changes in horizontal and spatial resolutions have generally improved QPF and this can be seen in the verification comparison project carried out by WGNE (Ebert et al., 23). The intensityscale method is herewith applied to see whether more detailed information can be gathered on the spatial scale of the errors. The ECMWF QPF datasets cover the period January to July 25 and have a resolution of about 4 km (corresponding to the spectral truncation T5) and a resolution of about 25 km (corresponding to the spectral truncation T799). The T5 forecast is able to capture synoptic systems of several hundred kilometres, while the T799 should provide insight in mesoscale synoptic systems. A complete description of the ECMWF model is available from the ECMWF (Thépaut et al., 25) and the reader is referred to Untch et al. (26) for a comprehensive review of the changes that were included in the T799 datasets. The ALADIN model is used operationally by the Hungarian Meteorological Service. It is a spectral, hydrostatic limited area model (Horányi et al., 996), initialized by running a 3D-variational assimilation for the upper air and optimal interpolation for the surface variables. The model is run at a resolution of 8 km on an area which covers Europe and up-scaled, following a procedure similar to that applied to the observations, to the resolution of the ECMWF model (25 km). The month of August 25 has been considered for the Hungarian verification because of the high number of wet events (27 rainy days in the area). For both ECMWF and ALADIN precipitation forecasts, the value at the grid points represents a gridbox areal quantity (Skelly and Henderson-Sellers, 996). Therefore, to guarantee that verifications are fair to the models, the observed precipitations have been up-scaled to the model resolution. A 24-h accumulated observation field (hereafter referred to as precipitation analysis) has been constructed using the data from the high-resolution network of rain gauges covering the European territory. Ebert et al. (23) describes different strategies used at meteorological centres to make use of rain gauge data. Figure. T5 (a) and T799 (b) grid points in the area used in the verification.

INTENSITY-SCALE VERIFICATION TO ASSESS PRECIPITATION FORECASTS 47 At the ECMWF, the observed values are up-scaled to the corresponding grid of the model. The up-scaling technique is described in Ghelli and Lalaurette (2) and Cherubini et al. (22). It is a simple average of the data available in each grid-box centred on a grid point. The intensity-scale technique is based on the assumption that in the domain of interest, the matched forecastobservation is available for all the grid points, it is therefore important to have an analysis of precipitation for the area. This assumption is rather stringent, as it forces the operational verification to be applied only to those domains for which one can guarantee the full spatial description of the forecast-observed field. The comparison between the two resolutions of ECMWF model has been performed over a square area of 32 32 (T799) and of 6 6 points (T5) over France (Figure ), while the comparisons between ECMWF and ALADIN have been carried out on the area shown in Figure 2. The domain in Figure 2 is represented by 32 32 points (grid spacing 25 km). 4. Results 4.. A case study over France The case study examines the passage of a frontal zone on the 22nd and 23rd of January. A deepening low pressure area moved very slowly from the Atlantic Ocean towards the Mediterranean (Figure 3). The forecast for the range t + 36 captures the situation well, but earlier forecasts (not shown) verified on the same day show both a time shift and an intensity error. Figure 4 depicts the forecast 44 46 48 5 5 2 25 Figure 2. The squared (inner box) area used for the comparison of the ECMWF and the ALADIN forecasts. Resolution: 25 km; 32 32 pixels; points all around: rainfall gauges of the Danube Tisza catchment area. This figure is available in colour online at www.interscience.wiley.com/ma Figure 3. ECMWF analysis of 7 hpa relative humidity (shaded), mean sea level pressure (solid lines) and 85 hpa temperature (dashed lines) for 23rd January 25, UTC.

48 G. CSIMA AND A. GHELLI (b) and the observed analysis (a). The displacement of the front is about 2 4 km to the south, mainly in the western part of the territory. The intensity-scale skill plot (Figure 5) shows that for thresholds of 4 6 mm 24 h, representative of the frontal zone, the skill is negative for spatial scales between and 4 km. Figure 6 is an example of the binary error for rainfall above 8 mm 24 h (a), while the bottom panel is a (a) 44 45 46 47 48 49 5 (b) 44 45 46 47 48 49 5 ECMWF obs (France (32x32), 23..25) mm d 64 32 6 8 4 2 /2 /4 /8 2 4 6 8 mm d ECMWF fc (T799, TS=84, France (32x32), 23..25) 64 32 6 8 4 2 /2 /4 /8 2 4 6 8 Figure 4. (a) Observed 24 h accumulated precipitation for 22nd January 25, 6 UTC, to 23rd January 25, 6 UTC, and (b) the corresponding T799 forecast started on the 2th January 25, UTC, for the range t + 84. This figure is available in colour online at www.interscience.wiley.com/ma Scale (km) 25 5 2 4 8 Intensity Scale Skill Score (T799, TS=84, France (32x32), 23..25) /8 /4 /2 2 4 8 6 32 64 Threshold (mm day ).5.5.5 2 2.5 3 3.5 4 4.5 5 5.5 6 Figure 5. Two-dimensional plot of skill scores as functions of threshold and spatial scale for the T799 accumulated precipitation forecast started on 2th January 25, UTC, for the range t + 84. This figure is available in colour online at www.interscience.wiley.com/ma

INTENSITY-SCALE VERIFICATION TO ASSESS PRECIPITATION FORECASTS 49 (a) ECMWF binary error image (T799, TS=84, France (32x32), 23..25) 44 45 46 47 48 49 5 2 4 6 8 (b) ECMWF binary error cont. (T799, TS=84, France (32x32), 23..25) 44 45 46 47 48 49 5 false alarm hits correct rej. misses 2 4 6 8 Figure 6. Binary error for the T799 accumulated precipitation forecast started on the 2th January 25, UTC, for the t + 84 range. The top panel (a) shows the precipitation threshold is 8 mm 24 h and the bottom panel (b) is the contingency table. This figure is available in colour online at www.interscience.wiley.com/ma graphical representation of a contingency table for the selected period: the hits in light grey, the misses in black, the correct negative in white and the false alarms in dark grey. Both binary error and contingency images show that the forecast moves the frontal zone too far south. All the mother wavelet components obtained from the two-dimensional discrete Haar wavelet decomposition of the binary error field for the same case study with rainfall rate of 8 mm 24 h are shown in Figure 7. The five mother wavelet components (st-5th) refer to the error spatial scale of 25, 5,, 2 and 4 km, respectively. The st mother wavelet component retains details of the error on the smallest spatial scale (25 km) which, compared to Figure 6 for a threshold above 8 mm 24 h, indicates a poor definition of the frontal edges. Subsequent components lose the details and the macro error on the 4 km scale is the southward displacement of the frontal zone. Sample size becomes small for thresholds above 32 mm 24 h, therefore no conclusions are drawn for such precipitation amounts. 4.2. A case study over Hungary This case study looks at a severe weather event over Central Europe. The interactions between a mid-latitude cyclone centred over the Baltic Sea and a deepening low pressure area moving from southwest to the northeast of Europe contributed to severe weather conditions. Heavy showers occurred in some parts of Europe, while northern Hungary was mainly under the influence of the deep Baltic cyclone on th and th August 25. Figure 8 depicts the observed analysis for the th August 25 (a) and the ECMWF forecast for the range t + 3 verifying on the same day (b). The forecast shows a spatially and temporally displaced front with an area of precipitation exceeding 6 mm 24 h that was not observed. The spatial displacement of the front is about 2 km to the southeast. The intensity-scale skill score plot (Figure 9) shows that at thresholds 2 8 mm 24 h, representative of the frontal zone, the negative skills are for spatial scales of 2 km. For higher thresholds the sample size becomes too small and robust conclusions cannot be made. 4.3. Verification over extended periods of time The T799 and T5 performances on monthly and seasonal scales are analysed for the period January to July 25. The skill score is calculated for the whole period and for a rainy season (January March). The distribution of the daily skill score is presented using boxand-whisker plots (McGill et al., 978). The lower and the upper lines of the box (shaded) are the 25th and the 75th percentiles of the sample. The distance between the top and bottom of the box is the interquartile range. The line in the middle of the box is the sample median. The

5 G. CSIMA AND A. GHELLI st Mother wavelet comp. 3rd Mother wavelet comp. 5 49 48 47 46 45 44 2 4 6 8 2 4 6 8 2nd Mother wavelet comp. 4th Mother wavelet comp. 5 49 48 47 46 45 44 5 49 48 47 46 45 44 2 4 6 8 5 49 48 47 46 45 44 2 4 6 8 5 49 48 47 46 45 44 5th Mother wavelet comp. 2 4 6 8.5.3...3.5 Figure 7. Mother wavelet components obtained for the two-dimensional discrete Haar wavelet decomposition of the binary error image for the T799 accumulated precipitation forecast started on the 2th January 25, UTC, for the range t + 84, at the precipitation threshold 8 mm 24 h. This figure is available in colour online at www.interscience.wiley.com/ma (a) 44 46 48 5 6 8 2 22 24 more 28 64 32 6 8 4 2 (b) 44 46 48 5 6 8 2 22 24 more 28 64 32 6 8 4 2 Figure 8. Observed 24 h accumulated precipitation for th August 25, 6 UTC, to th August 25, 6 UTC (a) and the corresponding ECMWF forecast started on the th August 25, UTC, for the range t + 3 (b). This figure is available in colour online at www.interscience.wiley.com/ma whiskers are lines extending above and below the box. They show the extent of the rest of the sample (unless there are outliers). By default, an outlier is a value that is more than.5 times the interquartile range away from the top or bottom of the box. The circles are the outliers. The notches in the box indicate the confidence interval about the median of the sample. A side-by-side comparison of two notched box plots provides a graphical way to

INTENSITY-SCALE VERIFICATION TO ASSESS PRECIPITATION FORECASTS 5 Scale (km) 25 5 2 4 8 Intensity Scale SS (ECMWF_, TS:3 6, LARGE area (32x32),.8.25) 2 4 8 6 32 64 Threshold.5.5.5 2 2.5 3 3.5 4 4.5 5 5.5 6 Figure 9. Two-dimensional plot of skill scores as functions of threshold and spatial scale for the ECMWF accumulated precipitation forecast started on the th August 25, UTC, for the range t + 3. This figure is available in colour online at www.interscience.wiley.com/ma (a) COMPARING MODELS (2 mm/day threshold, TS: 36-2, (6 x 6), WHOLE. 25) (b) COMPARING MODELS (4 mm/day threshold, TS: 36-2, (6 x 6), WHOLE. 25) Skill score 4 3 2 Skill score 4 3 2 4 4 8 8 6 6 32 32 64 64 4 4 8 8 6 6 32 32 64 64 Spatial scale (km) Spatial scale (km) Figure. Box-and-whisker plots of skill versus error spatial scale for the ECMWF T5 (light) and the T799 (dark), up-scaled to the spatial resolution of the T5 dataset, for the period January July 25 for two thresholds: (a) 2 mm 24 h and (b) 4 mm 24 h, for the range t + 36. determine whether the two groups have significantly different medians. The width of the box represents the size of the sample. Implicitly, the result represented by a very narrow box is not significant. This graphical representation of the daily scores describes the variability in the sample. In order to make the comparisons easier to interpret, the T799 dataset has been up-scaled to that of the T5, and the scores have been computed on the same square area comprising 32 32 points. Figure shows the skill score for the forecast range t + 36 as a function of the spatial scale for the 2 and 4 mm 24 h threshold. Both datasets behave similarly for the smallest spatial scale. If the spatial scale is doubled, both the forecasts show better scores and the spread is also reduced considerably. It is interesting to note that if the spatial scale is further increased, the skill scores for the two resolutions (T5 and T799) is similar but the spread of the T5 is slightly less than the spread of the T799 as in the case of outliers. This is in agreement with the study by Cherubini et al. (22), which states that better scores can be obtained by enlarging the grid-box, that is, increasing the spatial scale of the model. Each individual dataset does indeed show that skills get better while increasing the spatial scale. In particular, both models have negative skills at the smallest spatial scale, and positive skills at larger scales. During the period considered in the study there were not many events with thresholds of 6 mm 24 h and above, thus rendering general conclusions for such precipitation thresholds difficult because of sample size problems. A comparison of the two models at twice their grid length for the 4 mm 24 h threshold can be seen in Figure for the whole period. The verification for the whole period shows the median higher for

52 G. CSIMA AND A. GHELLI skill score -.5 -. -.5..5. T5_8km T799ds_8km model_spatial scale Figure. Box-and-whisker plots of skill versus error spatial scale at 8 km for the ECMWF T5 and the T799 (up-scaled to the spatial resolution of the T5 dataset) for the period January July 25 for the 4 mm 24 h threshold, for the range t + 36. COMPARING MODELS ( 8 km spatial scale, TS: 36-2, ( 6 x 6 ), WHOLE. 25 ) Skill score 4 3 2 /8 /8 /4 /4 /2 /2 2 2 4 4 8 8 6 6 32 32 64 64 Threshold (mm day) Figure 2. Box-and-whisker plots of skill versus precipitation for the ECMWF T5 (light) and the T799 (dark), up-scaled to the spatial resolution of the T5 dataset, for the period January July 25 for the 8 km spatial scale, for the range t + 36. The skill is nearly constant for the lower thresholds, and then falls off rapidly at 8 mm 24 h, which might be due to the fact that 8 mm 24 h events are intense events, generally associated to small-scale (convective) features, and which are related to processes which are less predictable than large-scale events. the T799 and the spread lower, thus indicating fewer cases of bad scores. This result applies to all thresholds below 6 mm 24 h, and to all the time steps verified. Results for the shorter period do not show significant differences between the two models. Figure 2 shows the decrease of the skill score as the threshold is increased for the period January July, indicating the ability of the model to go beyond the yes/no-rain discrimination. The ECMWF and the ALADIN daily performance distributions for the month of August 25 are shown in Figure 3 for the models with the smallest scale (25 km, a) and a larger scale (5 km, b) for the forecast range t + 3. Both ECMWF and ALADIN forecasts show an increase of skill when the spatial scale is increased, except in the case of large thresholds, when the number of cases is rather small, thus rendering general conclusions for such precipitation threshold difficult because of sample size problems. Significant differences can be seen in Figure 3(a) (at 25 km spatial scale) between the two models: ECMWF shows better skills than ALADIN for

INTENSITY-SCALE VERIFICATION TO ASSESS PRECIPITATION FORECASTS 53 (a) 25 km spatial scale (b) 5 km spatial scale Skill score Skill score 2 2 3 ECMWF ALADIN 3 ECMWF ALADIN 2 2 4 4 8 8 6 6 32 32 64 64 Threshold 2 2 4 4 8 8 6 6 32 32 64 64 Threshold Figure 3. Box-and-whisker plots for the ECMWF (light) and the operational ALADIN (dark) UTC forecasts for August 25, at the (a) 25 km and (b) 5 km spatial scale, for the range t + 36. all the thresholds considered. Similar results are valid for the 5 km spatial scale (Figure 3(b)). 5. Conclusions The intensity-scale method has been used to verify the ECMWF operational quantitative precipitation forecast for two different horizontal resolutions: 4 km (T5) and 25 km (T799), and to compare the performance of ECMWF and ALADIN forecasts. The evaluation was performed over France, for the period from January to July 25 for different forecast ranges, and over Hungary for the month of August 25. Monthly and seasonal plots of daily performance were analysed, as well as two case studies. The T799 forecasts have been upscaled to the T5 resolution domain. In this fashion, the wavelet filter, which constitutes the main ingredient of the intensity-scale method, was applied on a common spatial domain. This enables the definition of the same scales for the two different resolution forecasts, and makes comparisons more easy to interpret. The case studies for both France and Hungary show some interesting insights into the spatial scale of the error. In both cases the main issue with the forecast has been the shift of the frontal zone, which was well described by the negative skill in the pertinent scales. However, this kind of graphical information does not suit the operational verification as the latter are more concerned with the average behaviour of the model over areas, rather than single day forecast. The distribution of daily skill scores shown using the box-and-whisker plot for the designated periods is more in line with the interest of the two operational centres. The results show that the models have negative skills at the smallest scale, but skill improves when considering larger spatial scales. As for other scores, the intensityscale skill score decreases as the precipitation threshold is increased and this is due to the poor ability of the model to go beyond just the yes/no-rain discrimination. A major challenge faced in this study has been the generation of a precipitation analysis to guarantee the availability of observations at each grid point of the forecast domain. This rather stringent request related to the intensity-scale approach has led to the limitation of the verification areas. Moreover, the intensity-scale method is based on the Haar wavelet decomposition, which is designed for a square domain defined by a grid 2 l 2 l imposing further conditions on the verification areas. However, Casati (27) introduce a waveletbased verification method which allows the application of the Haar wavelet filter in the presence of missing observations over the forecast domain, and which does not require a square spatial domain dimensioned as a power of 2. This method applied to the thresholded binary observations and forecast fields, can alleviate the intensity-scale technique from the above-discussed limiting constraints. Acknowledgements The authors wish to acknowledge the help of Barbara Casati in commenting on a draft of the paper. References Casati B, Ross G, Stephenson D. 24. A new intensity-scale approach for the verification of spatial precipitation forecasts. Meteorological Applications : 4 54. Casati B. 27. Verification techniques for spatial forecasts. Oral presenation at the Third International Workshop on Verification Methods: Reading, UK, January 27. http://www.ecmwf.int/newsevents/ meetings/workshops/27/jwgr/index.html. Cherubini T, Ghelli A, Lalaurette F. 22. Verification of precipitation forecasts over the Alpine region using high-density observing network. Weather and Forecasting 7: 238 249. Ebert EE, Damrath U, Wergen W, Baldwin ME. 23. The WGNE assessment of short-term quantitative precipitation forecasts. Bulletin of the American Meteorological Society 84: 48 492. Ghelli A, Lalaurette F. 2. Verifying precipitation forecasts using upscaled observations. ECMWF Newsletter 87: 9 7. Holland L, Fowler T, Brown B, Nance L. 27. Designing a stateof-the-art verification system. Oral presentation at the Third International Workshop on Verification Methods, PDF available from: http://www.ecmwf.int/newsevents/meetings/workshops/27/jwgv/ index.html. Horányi A, Ihász I, Radnóti G. 996. ARPEGE/ALADIN: a numerical weather prediction model for Central-Europe with the participation of the Hungarian Meteorological Service. Idojárás : 277 3.

54 G. CSIMA AND A. GHELLI Jolliffe IT, Stephenson DB. 23. Forecast Verification: A Practitioner s Guide in Atmospheric Science. John Wiley and Sons: Chichester, 24pp. Mailier PJ, Jolliffe IT, Stephenson DB. 26. Quality of weather forecasts, Review and Recommendations. Available from Royal Meteorological Society. McGill R, Tukey JW, Larsen WA. 978. Variations of box plots. American Statistician 32(): 2 6. Skelly WC, Henderson-Sellers A. 996. Grid box or grid point: what type of data do GCM deliver to climate impacts researchers? International Journal of Climatology 6: 79 86. Thépaut J-N, Andersson E, Beljaars A, Hortal M, Janssen P. 25. Two new cycles of the IFS: Cycle 26r3 and Cycle 28r. ECMWF Newsletter 2: 26 35. Untch A, Miller M, Hortal M, Buizza R, Jansen P. 26. Towards a global mesoscale model: the high-resolution system T799L92 and T399L62 EPS. ECMWF Newsletter 8: 6 3. Wilks DS. 995. Statistical Methods in the Atmospheric Sciences: An Introduction. Academic Press: San Diego, CA.