Verification Marion Mittermaier

Transcription

1 MOSAC November 2012 PAPER 17.6 Verification Marion Mittermaier Crown Copyright

2 1. Introduction Whilst it has been relatively easy to show an increase in skill when comparing forecasts from models with greater than 30 km to less than 20 km horizontal resolutions, it has been somewhat more difficult to do so for horizontal resolutions further improving from greater than10 km to socalled convection-permitting grid resolutions, e.g. 4 km or less (e.g. Mass et al. 2002). This has been especially true for precipitation where the forecast detail is realistic but inaccurate. Lorenz (1969) pointed out that output from km-scale models may not be accurate at the grid-scale. This inaccuracy introduces the so-called double-penalty effect where the traditional approach to forecast verification of precise matching of forecasts to point observations means that even small displacement errors in features get penalized, and closeness is not rewarded. This has led to a multitude of new verification methods being developed to assess precipitation spatially. Ebert (2008) made the first attempt at categorizing these new methods. One group of methods have been referred to as neighbourhood methods, where either a single observation (SO) or observation neighbourhood (NO) is used to compare to a forecast neighbourhood (NF). All these new methods were primarily developed to produce objective verification results that showed the benefit of convection-permitting model forecasts, because the traditional methods and metrics failed to match subjective assessment results. Many of these new methods rely on a gridded truth, e.g. radar accumulations, which are not without issues. This will be discussed further in Section 2. For many parameters the main source of truth is still single site synoptic observations, where representativeness remains a factor. A model grid box now represents a very small area and for some variables such as cloud the model often resembles a binary response rather than a range of values. This was demonstrated by Mittermaier (2012) in assessing synoptic cloud observations and forecasts from four different MetUM configurations. Our forecast approach is becoming increasingly probabilistic and focused on the early detection of changes of weather type and significant events that lead to downstream impacts. Methods which can help detect whether forecasts can discriminate and detect a significant signal and how far in advance, are becoming increasingly important. Some thoughts will be offered in Section 3. Finally a short round-up of other relevant research highlights are given in Section Benefit of km-scale NWP Routine verification of deterministic Numerical Weather Prediction (NWP) forecasts from the convection-permitting 4 km (UK4) and near-convection-resolving 1.5 km (UKV) configurations of the Met Office Unified Model (MetUM) has typically shown that it is hard to consistently prove that the higher resolution near-convection-resolving model is more skilful. The UK index has long been considered the benchmark by which we measure model performance. Six surface parameters are considered: 2 m temperature, 10 m wind speed, total cloud amount (TCA), cloud base height (CBH), visibility and hourly precipitation. The time series of the 6 individual UK index components for UK4 vs UKV is shown in Fig. 1. A positive (negative) bar indicates the UKV is better (worse). For temperature the trends are seasonal with indications that the raw UKV temperatures are better in the colder months. For wind the score is uniformly negative and this is largely because the UKV has a lot more detailed structure in the wind fields which is often interpreted as noise by a metric based on the root-mean-square error which rewards smoothness. Until recently cloud was considerably worse in the UKV but model upgrades have essentially turned this skill deficit around such that now the differences are marginal. The signal for CBH is also not conclusive. For visibility there are hints that the UKV is better in the cold season, whereas for hourly precipitation the UKV forecasts verified at point locations suffered severely from a combination of representativeness and double penalty issues. Crown Copyright

3 Figure 1: History of the 6 individual UK index components comparing the UK4 and UKV Precipitation at least can (and should) also be assessed against a gridded truth such as radar accumulations. The realism of the spatial distribution of high-resolution precipitation forecasts is one of the key benefits when assessing forecasts subjectively. At the Met Office the spatial assessment of precipitation (using the Fractions Skill Score) has been part of our routine verification suite since Mittermaier et al. (2011) summarised a 5-year assessment between the soon-to-be retired 12 km NAE and the UK4. Unfortunately the long-term monitoring of Crown Copyright

4 precipitation forecast skill against radar accumulations is not straightforward as any changes in the forecast bias are not the only influence on verification metrics. The radar baseline is not as stable in time as that provided by gauges. Radar hardware changes (through component failure or upgrade), gauge calibrations, and other artefacts can result in considerable fluctuations. Disentangling the reasons for changes in the verification signal therefore becomes virtually impossible at times, and not all change is attributable to the model alone. Nevertheless, the comparison against radar is extremely valuable when comparing model configurations against each other in a relative sense, as they are all compared against the same observations set. Therefore the comparisons have been expanded to cross-compare the UK4, UKV and the 4-km global downscaler referred to as the UK4 extended or UK4X. Results are shown in Fig 2. These schematics show a range of thresholds for 6-h precipitation totals for 5 accumulation windows for validity times (VT) ranging from t+3h to t+36h. See the caption for further details. So subjectively we can determine that the UKV is better than the UK4. Objectively we can also show that the UKV precipitation forecasts are better than other models using a spatial verification method and a gridded truth (radar). What about the other parameters? We typically do not have gridded independent data sets to compare against, so standard synoptic observations are still the only viable means for routine assessment. Figure 2: The coloured tables show the percentage difference of verification times that the higher-resolution model had a better score than the lower-resolution model. That is, they show [the percentage of VTs the higher-resolution model scored better] minus [the percentage of VTs the lower-resolution model scored better]. If the lower-resolution model has more VTs with a better score, the cell is shaded red. If the higherresolution model has more superior VTs, the cell is green. Grey cells are either equally matched or statistically insignificant. Stronger shaded cells are more statistically significant. Bright green implies results significant at least at the 5% level. Mittermaier (2012) describes a proposed framework for verifying a variety of surface parameters from km-scale NWP against surface observations by using forecast neighbourhoods centred on the observing site. The framework is inherently probabilistic in that it uses the Brier Score, Ranked and Continuous Ranked Probability Scores (See Appendix) and a range of forecast neighbourhood sizes emulating coarser MetUM configurations. The method enables a 3-way comparison of: the impact of the neighbourhood size over a single nearest grid point or direct model output (DMO) for quantifying the relative skill gained through using a forecast neighbourhood (double-penalty effect and/or representativeness), as well as the relative skill between different model configurations for equivalent neighbourhoods, the skill relative to a reference forecast, e.g. persistence. This framework represents just one way in which the representativeness and double penalty effect can potentially be estimated. In the paper the framework is applied to consider whether the 1.5 km MetUM configuration is more skilful than the 4 km configuration. Currently the other scenarios of importance for the operational utilisation of such a framework are being assessed. Crown Copyright

5 These are: a. Can the framework assist in the decision-making for model-upgrades at the same horizontal resolution? The classic test vs. control scenario; and b. Can the framework successfully compare deterministic forecasts with ensembles? It is hoped that in proving and then adopting this strategic approach comparisons between deterministic and ensemble prediction systems (EPS) are easy to do, and enable the optimization of post-processing to ensure optimal skill of forecast products, through identifying the neighbourhood sizes that optimise skill. Incorporated in this process is statistical significance testing of the score differences to add credibility to the results and assist in decision making. The benefits of this strategy include: Providing a different perspective of deterministic forecast skill through using probabilistic metrics, and treating the near-convection-resolving deterministic model forecasts probabilistically through the use of spatial fractions (probabilities). Providing a readiness for comparison to convective-scale ensembles, potentially providing useful guidance on the pre- and post-processing of individual ensemble members to spatial probabilities to maximize ensemble skill. Counteracting the spatial and temporal representativeness errors of both the single site observations and forecasts on verification metrics. To illustrate one part of the new framework Fig. 3 shows the cumulative skill benefit (deficit) of the UKV over the UK4 for December Here DMO and equivalent neighbourhoods from the UKV and UK4 are compared. These graphs summarise all hourly output from the 36h forecast. The S denotes that the score differences are statistically significant at the 5% level. An interesting result is that for 10 m wind the UKV DMO is more skilful than the UK4 DMO when RPS is used. This is in contrast to the skill deficit seen in Fig. 1. This result suggests that the UKV DMO places the wind speed in the correct category more frequently. When comparing neighbourhoods of the same size, UKV is still more skilful but with a reduced margin as the neighbourhood size increases. From the results we conclude that: o Any neighbourhood has, for the most part, a positive impact. o The general behaviour of the scores as a function of neighbourhood size can provide clues as to whether the lack of skill seen at the grid scale is to do with the double penalty or representativeness mismatches, or whether it is simply due to a model's inability to forecast certain event thresholds, e.g. low cloud bases, visibility, cloud amounts. Using neighbourhoods is generally beneficial for more extreme thresholds. o Using a neighbourhood can mitigate against the double penalty effect and representativeness mismatches, by introducing or enhancing forecast skill, but a basic level of skill must be present to do so. o A poor, relatively unskilful forecast can not be made to appear more skilful very easily. o It can be shown that a near-convection-resolving model with a neighbourhood is more skilful than a convection-permitting model single grid point or neighbourhood. o The benefit is parameter dependent, e.g. for this forecast sample temperature and wind results suggest ~15% can be ascribed to these effects. For the non-continuous parameters such as TCA % or more is possible. o Failure to show skill relative to another (coarser) model using a neighbourhood would seem to suggest a more fundamental model deficiency. Clearly UKV TCA and CBH forecasts shown here are inherently less skilful, and the difference is not simply due to the doublepenalty effect. Crown Copyright

6 Figure 3: Cumulative skill benefit (deficit) for the UKV over theuk4 for 03Z forecast initialisations in December Results show comparison of DMO vs DMO and equivalent sized neighbourhoods. Some important development areas remain, for example, an assessment of the bias. We also want to know whether all parameters need a neighbourhood as default, and whether it is sufficient to use the same neighbourhood size for all parameters. Perhaps the neighbourhood sizes should increase with forecast lead time to represent the increase in forecast error. All these issues will be actively addressed in the next year. Crown Copyright

7 3. Demonstrating model improvement across timescales At the Met Office we are fortunate that the Unified Model is a true seamless nested modelling system which is run from the (sub-) km short-range Numerical Weather Prediction (NWP) scale to climate scale where the resolution is O(100 km). In maintaining and developing a common model code across all these space and time scales, a strategy is needed to ensure, for example, that improvements which improve climate simulations do not detrimentally affect forecast skill at the short-range NWP scale. How can one assess the relative skill whilst still factoring in that shortrange NWP and a decadal forecasts answer fundamentally different questions? It is as yet unclear whether we can assess the same parameters (in terms of their definition) at increasing temporal scales, or the same parameters in the same way (i.e. using the same method for all time scales). One attribute we want to assess is whether our model can reliably predict events of interest, bearing in mind that the nature of the forecast changes as we move from shortto long-range, and from deterministic to probabilistic. The design of a seamless verification approach must incorporate this change. The method should always be aligned to the question that one wishes to answer. Therefore the method must be able to adapt as the forecast definition changes, yet remain capable of assessing forecasts in a relative sense. One of the key attributes that forecasts of all types require, is the ability to discriminate. Mason and Weigel (2009) and Weigel and Mason (2011) put forward a Generalized Discrimination Score (GDS) which at least might provide a relatively simple candidate score to fulfil this requirement. The score can be applied to binary dichotomous (yes-no), polychotomous (multi-category), continuous, discrete and continuous probabilistic forecasts. This is a score we hope to explore more fully in the coming year or two. 4. Other research highlights In addressing the question of comparing forecasts at different time scales, one of the key questions is how skilful longer-lead forecasts (which are potentially much coarser resolution) are with respect to surface weather elements (not just circulation patterns). As a first step to bridging this gap an experimental global surface index has been formulated (J. Maksymczuk, pers. comm.) using the traditional UK index framework, measuring raw model skill. Forecasts are verified at 00 and 12Z using the following lead times: t+6, 12, 18, 24, 30, 36, 42, 48, 60, 72, 96, 120, 144h. For the UK version the UK index site list is used (which is QC assured). Globally we are more vulnerable to poor observations. Parameters in this index include temperature, 10 m vector wind, cloud cover, cloud base height, 24hr precipitation (same thresholds as in current UK Index for ETS scores). Figure 4 tracks the monthly and 12-month mean indices for the UK and globally. Tracking the changes in surface parameters for core model development will be important as NWP and longer time scales (including coupling atmosphere and ocean) converge to ensure that forecasts of surface parameters are not detrimentally impacted at weather time scales. Other key highlights are: Novel application of spatial methods (MODE Davies et al. 2007, FSS Roberts and Lean 2008, SAL Wernli et al. 2009) to satellite cloud masks/analyses and cloud cover forecasts to understand spatial biases in cloud cover (Crocker and Mittermaier 2012, Mittermaier and Bullock 2012). Implementing and testing the Symmetric Equitable Error in Probablity Space (SEEPS, Rodwell et al. 2009, ECMWF headline measure) score for 6-h precipitation in regional models as part of the EUMETNET SRNWP-V programme (North et al. 2012). Crown Copyright

8 Figure 4: Global surface indices for UK and global site lists. Both monthly and 12-month running means are shown. Time series starts April Crown Copyright

9 Appendix: Definition of scores The Brier Score (BS, Brier 1950), is by far the most common scalar accuracy measure for twocategory (event/no-event) probabilistic forecasts. By definition the BS is analogous to the meansquared error (MSE), but where the forecast is a probability and the observation is either a 0 or 1, corresponding to not observed, or observed. It represents an error in probability space and is a measure of accuracy. It follows that the smaller the error, the smaller the BS (Eq. 1), the more accurate the forecast. 1 2 (1) BS (pi oi ) N i 1 The BS can also be calculated for a reference forecast, often the long-term climatological frequency of an event, which represents a constant reference forecast. Other reference forecasts can be used. This enables the calculation of a Brier Skill Score like that described in Eq. 2. BS BSref (2) BSS BSref For parameters like wind and temperature there is a clear desire to verify the distribution instead of focusing on a few thresholds, also in part due the large seasonal variations for these parameters. The Ranked Probability Score (RPS, see e.g. Epstein, 1969) is defined as: (3) 1 K 2 RPS ( CDF fcst, k CDFobs, k ) K 1 k 1 The RPS is a multi-category extension of the BS over J categories. It is a squared-error score with respect to the bin into which the observed event falls. This makes it sensitive to distance, and Murphy (1970) points out that it is particularly suited to ordered variables where being more than one category off would be significant from a forecaster s perspective, and this should be reflected in the score. The key difference is that cumulative probabilities CDF fcst,k and CDF obs,k are used to compute the score. The empirical distribution of the forecast values is determined through the application of a range of thresholds that should encapsulate all forecast values. Like the BS the perfect score is 0 with the range being between 0 and 1. It measures the sum of the squared differences in cumulative probability space for a multi-category probability forecast and tends to penalize forecasts more severely when probabilities are further from the actual outcome. The Continuous Ranked Probability Score (CRPS, e.g. Matheson and Winkler, 1976) is written as: (4) CRPS N Pfcst ( x) Pobs( x) This determines the differences between the forecast (P fcst ) and observed (P obs, Heaviside) cumulative probability distributions. A perfect forecast has a CRPS of 0. Skill scores can be calculated for both the RPS and CRPS by defining a reference forecast, using the general skill score formula, analogous to Eq dx Crown Copyright

10 References Brier G. 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78, 1 3. Crocker R. and M. Mittermaier 2012: Exploratory use of a satellite cloud mask to verify NWP models. Submitted to Meteorol. Apps. Davis, C. A., B. G. Brown, and R. G. Bullock (2006): Object-based verification of precipitation forecasts. Part I: Methodology and application to mesoscale rain areas. Mon. Wea. Rev., 134, Ebert E. 2008: Fuzzy verification of high-resolution gridded forecasts: A review and proposed framework. Meteorol. Appl., 15, Epstein E. (1969): A scoring system for probability forecasts of ranked probabilities. J. Appl. Meteor., 8, Lorenz E. 1969: Atmospheric predictability as revealed by naturally occurring analogues. J.Atmos. Sci., 26, Mason S.J. and A. P. Weigel 2009: A generic forecast verification framework for administrative purposes. Mon. Wea. Rev., 137: Mass C., D. Ovens, K. Westrick and B. Colle 2002: Does increasing horizontal resolution produce more skillful forecasts? the results of two years of real-time numerical weather prediction over the pacific northwest. Bull. Amer. Meteorol. Soc., 83(3), Matheson J. and R. Winkler 1976: Scoring rules for continuous probability distributions. Manage. Sci., 22, Mittermaier M.P., N. Roberts and S.A. Thompson 2011: A long-term assessment of precipitation forecast skill using the Fractions Skill Score. Meteorol. Apps, DOI= /met.296. Mittermaier M.P. 2012: A strategy for verifying near-convection-resolving forecasts at observing sites. Submitted to Wea. Forecasting. Mittermaier M. 2012: A critical assessment of surface cloud observations and their use for verifying cloud forecasts. In press, QJRMS, March Mittermaier M.P., and R. Bullock 2012: Using MODE to explore the spatial and temporal characteristics of cloud cover forecasts from high-resolution NWP models. Submitted to Meteorol. Apps. Murphy A. (1970). The ranked probability score and the probability score: a comparison. Mon. Wea. Rev., 98, North R., M. Trueman, M. Mittermaier, M. Rodwell 2012:An assessment of the SEEPS and SEDI metrics for the verification of 6h forecast precipitation accumulations. Submitted to Meteorol. Apps. Roberts N.M., Lean H.W. 2008: Scale-selective verification of rainfall accumulations from highresolution forecasts of convective events. Mon. Wea. Rev., 136: Rodwell, M.J., D.S Richardson, T.D. Hewson and T. Haiden A new equitable score suitable for verifying precipitation in numerical weather prediction. Quart. J. Roy. Meteor. Soc. 136: Wernli H., M. Paulat, M. Hagen, and C. Frei SAL - a novel quality measure for the verification of quantitative precipitation forecasts. Mon. Wea. Rev., 136, Weigel A.P. and S.J. Mason 2011: The Generalized Discrimination Score for ensemble forecasts. Mon. Wea. Rev., 139: Crown Copyright