Use of medium-range ensembles at the Met Office I: PREVIN a system for the production of probabilistic forecast information from the ECMWF EPS

Transcription

1 Meteorol. Appl. 9, (2002) DOI: /S Use of medium-range ensembles at the Met Office I: PREVIN a system for the production of probabilistic forecast information from the ECMWF EPS T P Legg, K R Mylne and C Woolcock, Met Office, London Road, Bracknell, Berkshire RG12 2SZ, United Kingdom This is the first of a pair of papers covering the production and use of probability forecasts at ranges of 3 to 10 days at the Met Office using the ECMWF Ensemble Prediction System (EPS). The use of ensembles is intended to provide a set of forecasts which cover the range of possible uncertainty, recognising that it is impossible to obtain a single deterministic forecast which is always correct. We present a brief review of ensemble forecasting techniques and their use for the generation of probability forecasts. A wide range of probability forecasting products and tools are now available to forecasters at the Met Office, generated from the EPS. These will be described, and their use and interpretation discussed, both for site-specific forecast data and for fields of data covering a wider area. An important part of any forecasting system is verification: this is covered in some detail using several different methods, for the site-specific forecasts of surface weather parameters. Comparisons are made between the probabilistic forecasts and equivalent deterministic forecasts generated from the high-resolution ECMWF model, and it is evident that the former are more skilful according to most assessment methods for forecasts more than three days ahead. The companion paper shows how forecasters use the vast amounts of information available in forecast production (Young and Carroll 2002). 1. Introduction The present work is the first of a pair of papers covering the production, use and application of probability forecasts at the Met Office. Here we are concerned with forecasts at time-ranges of 3 to10 days ahead, and the main tool used is the Ensemble Prediction System (EPS) of the European Centre for Medium-range Weather Forecasts (ECMWF). The EPS provides a set of 51 different numerical global forecasts, designed to cover more or less the range of uncertainty of the atmospheric evolution (Persson 2001). There is uncertainty in any analysis that might be used as the initial conditions for a numerical forecast, and the forecast atmospheric evolution is highly sensitive to such errors, as described in a little more detail in the next section. Errors in the analysis derive from a number of sources, notably observational errors, regions where few observations are available, and limitations of the data assimilation system. Limitations in the model, including its physics and the parameterisation of physical processes, add further to the forecast uncertainty. Historically, forecasts have usually been made based primarily upon the output of a single model, and the degree of uncertainty estimated subjectively by the level of agreement between a few model runs from different NWP centres. An ensemble is better suited to the task of creating a range of possible outcomes and gives some indication of the most likely outcome and the range of uncertainty. These uncertainties vary with time, as the sensitivity of the model to initial differences is flow-dependent and can vary due to the non-linear nature of atmospheric evolution. The EPS will produce, for any chosen forecast time and location, a set of 51 forecast values of any parameter. The resulting set of forecasts can be used to estimate probabilities of certain defined events occurring, or of different atmospheric flow patterns. For a point variable, a probability density function (pdf) can hence be derived from which the range of possible outcomes can be directly seen. From this, probabilities of certain events (e.g. rainfall of at least 10mm within a 12-hour period) can be estimated at that location. Maps can also be generated showing the spatial variation of many variables such as ensemble-mean temperature, probability of temperature exceeding 25 C, probability of severe gales, etc. Examples of these are presented in section 3. Clustering can be performed, to average together groups of ensemble members with similar 255

2 T P Legg, K R Mylne and C Woolcock solutions. Summaries of these clusters can be plotted on maps so as to convey all possible outcomes without the user having to look at 51 individual charts for each field of interest. In section 4 we present verification information for probabilistic forecasts of surface weather elements derived from the EPS, and make comparisons with equivalent results for forecasts generated from the ECMWF deterministic model which runs at a higher horizontal resolution than the ensemble. Four methods of verification are employed here: (i) Brier Scores (and Brier Skill Scores) (Brier 1950), which are a measure akin to mean square error for probabilities; (ii) Reliability (Brier 1950; Murphy 1973), which measures the correspondence between forecast probability and frequency of occurrence; (iii) the Relative Operating Characteristic, which explores the relationship between Hit Rate and False-Alarm Rate for a forecast system and thereby assesses the usefulness of the forecasts for decision-making; and (iv) Rank Histograms (outlier statistics) which can be used to show how frequently the range of the ensemble fails to encompass the verifying value and hence whether the spread of the ensemble is realistic (Hamill & Colucci 1997, 1998). Of these, the first three are described in more detail by Stanski et al. (1989), together with illustrations of their calculation. A review of many probabilistic verification measures, including those described in the present paper, is given by Wilks (1995). The present paper describes and illustrates the basic methods of producing probabilistic forecast information from the EPS, and provides some details on verification. The companion paper considers practical applications of the ensemble products in more detail. Young & Carroll (2001) describe the methods by which operational forecasters at the Met Office use probabilistic information as they provide services to customers. In a related paper, Mylne (2001) examines from the user s viewpoint the use of probabilistic forecasts as a decision-making tool by estimation of the economic value of the forecasts. 2. Background to probability forecasts using ensembles Increases in computer power in recent years have made the production of many forecast runs per day a feasible exercise, and hence we are no longer limited to a single set of forecast output. The main reason for using an ensemble of forecasts, as already mentioned, is to try to cover the range of possible uncertainty when making a forecast. The approach used is to attempt to estimate the forecast pdf by sampling the initial-condition pdf. In section 4 below we shall examine the extent to which this aim is met. A single deterministic forecast carries no indication of its likely accuracy. Uncertainties in weather forecasts derive mainly from 256 uncertainties in the initial atmospheric state (due to errors in observations, areas of the globe where there are few in-situ observations, and shortcomings in the data assimilation process) and approximations in the atmospheric model (especially in sub-grid-scale physical processes). Small errors in the initial state of the atmosphere, and model approximations, are amplified in time due to the non-linear nature of atmospheric evolution; hence the forecast errors can become very large. Thanks to the success of developments in Numerical Weather Prediction (NWP), large errors in synopticscale evolution are now rare in the first 2 3 days of the forecast, but significant uncertainty will always be commonplace beyond about 3 days. Uncertainty varies with the synoptic situation, and on some occasions the atmosphere can be quite predictable well beyond 3 days, but equally there are occasions when the atmosphere is highly unpredictable even at short range, around 24 hours. There is evidence, for example from the severe storms of December 1999 in France and Germany, that poor predictability in the shortrange can often occur when there is potential for severe weather. Operational ensemble systems such as the EPS have, so far, mostly been designed for mediumrange forecasting. Ensemble techniques are also now commonly used for long-range forecasting (see, for example, Graham et al. 2000), and are starting to be developed also for shorter-range forecasting (e.g. Stensrud et al. 1999; Du & Tracton 1999; Hamill & Colucci 1997, 1998). Future developments are likely to see increasing use of ensembles for shorter-range forecasting, both in severe weather situations and for probabilistic prediction of fine-scale details such as precipitation. The ECMWF EPS is designed for use in the mediumrange, between 3 and 10 days ahead. The method employed by the EPS is to perturb the initial conditions, altering them by small amounts within the likely analysis error. These perturbations are determined by using so-called singular vectors (SVs) to identify the fastest growing modes of uncertainty at the time the forecast is initialised, as described more fully by Molteni et al. (1996), Rabier et al. (1996), and Buizza et al. (1999, 2000). SVs arise when searching for the perturbation that, when added to a given basic state, will achieve maximum error growth over a specified time interval (Ehrendorfer et al. 1999); they are calculated at low resolution using a linearised version of the ECMWF model. For the ECMWF EPS this time interval is set at 48 hours, and the ensemble is intended to work best only at forecast times greater than 48 hours. Perturbations are scaled such that at the 48-hour optimisation time the spread of the ensemble is, on average, equal to the mean forecast error. Since the perturbations are chosen for maximum rate of growth over the first 48 hours, this means that the initial perturbations are considerably smaller than typical analysis errors. In

3 Use of medium-range ensembles at the Met Office, Part 1 more recent developments of the EPS, described by Buizza et al. (2000), additional initial-condition perturbations have been added; these are evolved from SVs calculated 48 hours previously, and have been shown to improve ensemble skill, particularly at shorter range (Barkmeijer et al. 1998). Stochastic perturbations have also been added to the model physics (Buizza et al. 1999), in order to attempt to estimate uncertainty due to model errors (as distinct from uncertainty in the initial conditions). In running the ensemble operationally, 25 perturbations are calculated daily from linear combinations of new and evolved singular vectors. Following this, 51 separate sets of initial conditions for the model are generated: one from the unperturbed ECMWF analysis (the control) plus 25 pairs created by adding each perturbation to the control analysis and by subtracting each from it. A full non-linear model integration is then run from each set of initial conditions. If two requirements are satisfied the set of initial states must provide a realistic estimation of the probability distribution of analysis errors, and the computed phase-space trajectories must present good approximations of atmospheric trajectories then the resulting ensemble statistics should approximate the correct probability density function. The current operational implementation of the EPS runs at a horizontal resolution of TL255, which equates to around 80 km in mid-latitudes, although many of the results and examples shown in this paper are based on output from the previous version at TL159 (120 km). Alongside the EPS, ECMWF also runs a high-resolution deterministic forecast at resolution TL511 (40 km), which was recently increased from TL319 (60 km). Various aspects of the EPS are described in more detail by Persson (2001). Other centres have explored different ways of generating ensembles. At the National Centers for Environmental Prediction (NCEP) in the United States, initial-condition perturbations are calculated using a different technique called error breeding (Toth & Kalnay 1993). As well as varying the initial atmospheric analysis as described above, one can vary the physics used in the model (Houtekamer et al. 1996). Houtekamer et al. (1996) developed a procedure in which each ensemble member differs in initial conditions (random perturbations are applied to the observations in different analysis cycles), in sub-grid-scale parameters, and even in orography. A combination of these two approaches can be used, by taking several members from each of two or more models to produce a multi-model ensemble (e.g. Evans et al. 1999; Mylne et al. 2002). Also, the outputs from forecasts from several different operational models can be taken together, to form what is popularly known as a Poor Man s ensemble (since it is computationally cheaper yet can still provide many of the benefits of an ensemble approach; see Ziehmann 2000). Once the ensemble has been run, output from it may be used to estimate the pdf of any model parameter, or equivalently to estimate probabilities of any forecast event derivable from the model fields. Probabilities are simply estimated by the proportion of ensemble members predicting an event to occur: for example, if 10 out of 51 members predict rainfall exceeding 10 mm at a particular location and time period, then the probability of exceeding 10 mm is estimated to be 20%. 3. Examples of forecast products The Met Office has developed a comprehensive range of tools and displays designed to aid forecasters in assimilation of information from the EPS, and in the production of probability forecasts. The principal elements of this system, known as PREVIN (Predictability Visualisation), will be described and illustrated in this section. The products fall into two broad categories: chart products covering the area around the UK, and site-specific information. All products cover forecasts out to 10 days (240 hours) ahead, mostly at 12-hour intervals, and are presented to forecasters as images on the Met Office s internal web system. 3.1 Chart products A useful idea of predictability and the range of uncertainty on the broad scale can be gained from spaghetti charts. These have been widely used since the introduction of ensemble forecasting. A spaghetti chart has one given contour plotted for each ensemble member, and gets its name from the fact that in the later stages of a forecast, when predictability is low, the contours often look like a pile of spaghetti. The example in Figure 1 shows the 546 dm contour for 500 hpa height fields; in this case a strong signal for a marked trough over the UK is seen but there is more uncertainty elsewhere within the chart area, especially over the eastern seaboard of Canada. Although useful for giving a broad picture of the predictability of the atmosphere in different regions, care is needed in interpreting spaghetti charts. Especially where the gradient of a field is shallow, small differences between ensemble members can lead to large differences in the horizontal position of a contour, falsely suggesting that local predictability is poor ( this may be the case in the example in Figure 1 to the north of the cut-off low, around 55ºN 65ºW. This effect can be particularly misleading in areas of high pressure where gradients are shallow but predictability is usually high. Nevertheless, with a little caution, spaghetti charts can often give a good indica- 257

4 T P Legg, K R Mylne and C Woolcock Figure 1. Spaghetti chart showing the position of the 546 dm contour of the 500 hpa height field produced by each ensemble member, data time 12Z on 1 November 2000, forecast time T+120. Figure 2. Ensemble mean forecasts of PMSL, data time 12Z on 21 August 2000, forecast times T+24 to T+216 over an area surrounding the UK. (The operational system displays these every 12 hours reduced here to save space.) 258

5 Use of medium-range ensembles at the Met Office, Part 1 tion of whether a region and time-frame of interest is broadly predictable or has become completely unpredictable. Spaghetti charts are available in Previn both for 500 hpa heights (at 528, 546 and 564 dm) and for 850 hpa wet-bulb potential temperature. A clear idea of broad developments in synoptic pattern can be seen on the Ensemble Mean charts, which are generated for 12-hour intervals (a reduced example, showing frames every 24 hours, is shown in Figure 2). These are available for fields such as mean sea level pressure, 500 hpa height, hpa thickness, and 850 hpa temperature. The ensemble mean is useful as it provides a clue to the most probable evolution, and it acts as a flow-dependent predictability filter, retaining the more predictable flow while smoothing out unpredictable features. Normally it will tend to be the smallscale features that are smoothed out, but on occasions these may be quite predictable and will be retained. Any broad change in the expected nature of the flow can readily be seen; the set of ensemble means in Figure 2 shows a ridge building across the UK around 48 hours ahead (T+48) which gets pushed away and replaced by a trough towards T+120, but by T+216 there are so many different patterns among the 51 members that the overall mean pattern becomes weak. Indeed, as ensemble members gradually diverge at later forecast times, the ensemble mean often does tend towards climatology. Ensemble means have been shown to have better accuracy than either the control member or the mean accuracy of individual members in terms of r.m.s. errors (Legg 1998; Molteni et al. 1996), but this is largely because they rarely produce intense features. It is also important to note that an ensemble mean forecast is not meteorologically consistent or realistic to the extent that a single model forecast is. This might be most apparent in a situation where two different but equally likely scenarios are produced by the ensemble; the ensemble mean will fail to show either and will give a misleading forecast. The ensemble mean also wastes much useful information in the ensemble, which can be better accessed by use of probability forecasts. If the forecaster needs to see the synoptic detail within the individual ensemble members then postage stamp charts are available. Figure 3 shows a reduced example at T+168 (7 days) a selection of 30 members is shown here, but all 51 members are presented to users. Experienced forecasters can readily assimilate a lot of information by glancing over a set of postage stamps, and in particular can quickly spot extreme members or significantly different scenarios. The examples presented in Figure 3 show that, in an overall cyclonic and mobile situation, significant differences may be found between members, giving substantial uncertainty in expected UK weather. Differences between members in timing of developments mean that it is impossible to give an exact forecast for a week ahead. To help summarise the information within the ensemble for the user, clusters of similar members can Figure 3. Postage stamp charts of PMSL, data time 12Z on 24 October 2000, forecast time T+168 over an area surrounding the UK. (Some members have been omitted for clarity there are 51 members altogether.) 259

6 T P Legg, K R Mylne and C Woolcock also be calculated and displayed (Figure 4) to show the main sets of possible solutions for a selection of fields. The clustering is performed on the mean sea level pressure field at T+168, in terms of shape (curvature) and position of contour lines. The principal method of clustering which we use is complete linkage, but other methods also used include single linkage, median sort and Ward (see Wilks 1995 for details of these methods). Figure 4 shows the mean and median (the median is defined here as the single member closest to the mean) mean sea level pressure charts for the two clusters from a particular ensemble run. Forecasters are presented with calculated values of medians, means and standard deviations for other fields also. On occasions the system can produce as many as ten clusters; three or four is more typical. Another way of visualising the main possible synoptic types at a glance is by the use of tubing (Atger 1999b). Whereas clustering makes an assumption that groups of significantly different forecast solutions are likely (multi-modality), tubing assumes that the forecast pdf is usually mono-modal. It therefore first identifies a central cluster of members closest to the ensemble mean, and then identifies those members which are most different from the ensemble mean. The latter are referred to as tube extremes. The central cluster mean, sometimes referred to as the centroid, may then be used as a guide to the most probable solution, while the tube extremes identify the patterns within the ensemble range which are most different from the central cluster mean. The PREVIN system displays charts of the central cluster mean and tube extremes based on tubing calculated at ECMWF. Colour-shaded charts of probabilities of certain defined events are displayed for a selection of fields, covering an area around the UK (Figure 5). Again, these provide useful summaries, including regional variations in expected weather conditions. Shades of blue indicate probabilities of cold or wet conditions; red is used for warm, dry or hazardous events. Temperature anomaly fields here are defined with respect to ECMWF Re-Analysis climatology. A rather different display from those described so far shows forecast cyclone tracks. Several different sets of charts are available. First, the positions of low pressure centres in individual ensemble members can be displayed as a scatter chart (see Young and Carroll 2001: Fig. 20). Further information can be added by showing the tracks over a 48-hour period centred on a chosen time (Figure 6). All 51 ensemble members produce forecast low centres whose movement can be seen on Figure 4. Median and mean of PMSL, for two example clusters, data time 12Z on 1 June 2000, forecast time T+144 over an area surrounding the UK. 260

7 Use of medium-range ensembles at the Met Office, Part 1 Figure 5. Fields of variables, data time 12Z on 26 October 2000, forecast day 4 over an area surrounding the UK. (Note: Arrangement of charts has been altered slightly from the operational display, to fit better on the page.) these charts; individual lows are colour-coded according to their depth. In the example shown, most ensemble members have produced three depressions, with a spread in their tracks and positions. The tracking software (Terry & Atlas 1996) works by identifying a low centre and following it according to the direction and strength of the overlying 500h Pa-level flow to track its positions at 12-hour intervals. The use of these products by Met Office forecasters is described in Vol 9, issue 3 by Young and Carroll (2002). 3.2 Site-specific products The data from which the products described in this sub-section are derived are extracted from the EPS output to give values representative of specific sites. Fortyone stations are available for display and can be chosen 261

8 T P Legg, K R Mylne and C Woolcock Figure 6. Cyclone tracks, data time 12Z on 4 December 2000, covering the forecast period T+72 to T+120 over the North Atlantic area. from a menu; data are available at 12-hour intervals up to 10 days ahead for 2 m temperature, precipitation (accumulated during the 12 hours ending at the stated time), and 10 m wind speed. Values for each site are interpolated from the four nearest model grid-points; at the resolution currently used in the EPS, grid-points are roughly 80 km apart (120 km apart up until 20 November 2000). Interpolation is currently done at ECMWF, and a set of 51 values from the 51 ensemble members is supplied to the Met Office in BUFR code. Direct use of interpolated model grid-point values of weather parameters for local sites is subject to considerable errors due to local site biases, geographical effects, and approximations in model parametrisations. For example, sites some distance inland may be excessively influenced by model grid-points over the sea. This simple interpolation is used here on the assumption that for medium-range forecasting the range of solutions will be dominated by synoptic variability between ensemble members which will largely dominate over the local site-specific biases. Nevertheless, work is in progress to apply site-specific bias corrections in the future. The probability density function (pdf) graph, an example of which is presented in Figure 7, shows the range covered by the ensemble forecast values and, within that range, where the most likely value lies. The probabilities shown are of the variable s value 262 lying within unit range; for example, for 16 C the probability on the graph is that of the temperature being between 15.5 and 16.5 C. A red dot denotes the corresponding value from the high-resolution deterministic ECMWF model. Solid vertical lines are set threshold values (here 15, 20, 25 C). Vertical dashed lines indicate anomalies (departures from long-term seasonal averages for that station) of (2, 4, 6 C; the thick line is the seasonal normal. (These anomalies are relative to observed station data, over a 30-year period where possible, and hence errors may arise due to model biases.) The graphs are accompanied by tabulated probabilities of certain events, defined in terms of values being above or below certain limits e.g. 15 C, 20 C, 25 C, 30 C for temperatures, 0.1 mm, 1.0 mm, 5.0 mm, 10.0 mm for 12-hour precipitation accumulation. Note that this probability is equal to the area under the pdf over the part of the x-axis defining the event: for example, in Figure 6 the probability of T>20 C. Included also are some joint and conditional events; for example, the probability of both temperature below 2 C and precipitation occurring at the same time, as a rough guide to snow probability. Note that joint and conditional probabilities can only be calculated by analysis of each ensemble member for the combined event they cannot be obtained from the probabilities of the two separate events since it is important to determine the probability of them occurring simultaneously.

9 Use of medium-range ensembles at the Met Office, Part 1 Figure 7. Probability density function, for temperature at Heathrow Airport, data time 12Z on 3 September 2000, forecast time T+120. (For full explanation of lines, see text.) Figure 8. Probability time-series, for Aviemore, data time 12Z on 21 August 2000, forecast times T+12 to T+240. (Small squares at top or bottom of graph indicate whether the operational model predicted the event.) In the PDF example given here (Figure 7), at 120 hours ahead (T+120) for Heathrow the temperature is most likely to be about 20 C, although there is some spread and the coldest members suggest temperatures of C. Similar graphs are available for precipitation accumulation and for wind speed (not shown). Sometimes a distinctly bi-modal distribution may be seen, for example when there are two broadly different 263

10 T P Legg, K R Mylne and C Woolcock flow types produced by the ensemble. On occasions, the operational value is found to lie outside the range of the ensemble members. This might simply indicate another possible forecast solution, or may also be due to different site-specific biases in the higher-resolution deterministic model. The event probabilities tabulated alongside the PDFs can be collected together and themselves plotted as a function of forecast time out to ten days ahead. Several such curves are plotted on one pair of axes as shown in Figure 8. This provides an instant way of seeing how event probabilities change through the forecast period; a heavy-rainfall event or the onset of a cold spell is clear to see. In this example, high temperatures are most likely around T+96, after which rainfall event probabilities become higher. Solid lines are used for time-series of probabilities verifying at 12Z and dashed lines are for probabilities verifying at 00Z. Another way of presenting the probability time-series information is to consider a fixed verifying time and look at how the event probabilities vary as the data time is brought closer. This idea is illustrated in Figure 9, with forecast times of between 1 and 10 days ago giving event probabilities for a fixed date. Here, the more recent runs of the ensemble favoured relatively warm, dry conditions with increasing confidence. 3.3 Early warnings of severe weather One other use of ensembles, which is currently being developed in the Met Office, is to provide first guess early warnings of severe weather events. This is achieved by determining how many ensemble members have forecast values of certain parameters exceeding defined extreme thresholds. With a large ensemble size, it is hoped that the possibility of severe events can reli- Figure 9. Time sequence of probabilities, for Stornoway, verifying at 12Z on 25 August 2000, from successive runs of the ensemble preceding this time. 264

11 Use of medium-range ensembles at the Met Office, Part 1 ably be identified up to a few days ahead. Issued early warnings are given in terms of probabilities of an event in each of 12 areas covering the UK. The direct use of model output to determine whether a fixed threshold is likely to be reached is not adequate in the case of extreme or severe weather events. Model output has a tendency to underestimate the highest wind speeds and rainfall amounts, mainly because of the resolution of the model and sub-grid-scale processes. It is necessary to calibrate the forecast data to determine what model threshold must be used to correspond to observed events of the defined severity. A first guess at the calibration was obtained by comparing past occasions when severe weather was observed with the corresponding model analysis values on those occasions. ECMWF EPS output is automatically scanned daily to evaluate whether any severe events are likely during the following five days. These events include severe gales, heavy and/or prolonged rainfall, and heavy snowfall. Adjustments are made which include an allowance for differences between ensemble members in timing of events. If any such events are expected within any of a set of defined UK regions with a probability above a defined critical level, then forecasters are alerted and, in conjunction with all other available forecast information (including some of the ensemble products referred to above), can prepare an appropriate Early Warning message to be transmitted as part of the National Severe Weather Warning Service. 4. Verification Verification is important, both to assess how skilful forecasts are, and to inform users of the potential benefits to be obtained from the forecasts; the idea of using verification to aid users in the application of probability forecasts is developed further in Mylne (2001). There are important distinctions between verification of probabilistic and deterministic forecast information. Deterministic forecasts might be assessed, for example, in terms of the error in the (best guess) value of the forecast variable; for probability forecasts it is the event probability which is assessed. Many probabilistic skill measures are based on the assessment of two-state categorical events, such as rain/no rain, frost/no frost, temperature above/below 25ºC, etc. A review of many probabilistic verification measures is given by Wilks (1995); details and illustrations of their calculation are shown by Stanski et al. (1989). A brief example of assessment results from an operational ensemble system using a selection of verification methods is presented by Toth et al. (1998). Much of the work in the literature has focused on verification of fields such as 500 hpa height (e.g. Molteni et al. 1996), but the present work concentrates mainly on the assessment of site-specific surface weather parameters, which are more difficult to predict due to the influence of local site-specific effects. Note that when assessing probabilistic forecasts a very large number of cases must be considered to produce meaningful and representative results. We shall concentrate on four particular methods of verification, as set out below. Verification is based on forecast values obtained each day for the 41 UK sites, and the observed values of these parameters at each station. Data have been gathered since August 1997; the results presented below cover a period of three years beginning at that date, although results for individual threemonth seasons are also produced. 4.1 Brier Score The Brier Score is effectively a root-mean-square error for a set of probability forecasts. It examines the differences between the forecast probability, p f, and the occurrence of the event, p o, which is defined to take the value 1 when the event is observed to occur and 0 for a non-event. The Brier Score from a large number N of forecasts can be calculated as 1 N 2 BS = ( pf po) N n= 1 Note that the Brier Score is negatively oriented, i.e. better forecasts achieve lower values; values fall in the range 0.0 to 1.0. Comparisons can also be made with a reference set of forecasts, typically climatology, persistence, or some other forecast model which we are hoping to improve upon. The results of such a comparison can be expressed as a Brier Skill Score (BSS), which is zero if our forecast system performs no better than the reference, and 1.0 if our forecast system is perfect. A negative value of BSS indicates forecasts that are poorer than the reference system. The BSS can be calculated as BSS = 1 BS BS where BS fc is the Brier Score of our forecasting system, and BS ref is that of the reference forecasts. The reference set of forecasts used in our work is obtained by always using the climatological probability. The Brier Score is strongly influenced by the climatological frequency of occurrence of an event. Rare events will often be forecast with probability zero so the Brier Score calculated over many cases will be small (apparently good) due to a large number of correct very low fc ref 265

12 T P Legg, K R Mylne and C Woolcock probabilities. For this reason one cannot directly compare Brier Scores for different events, or even for the same event at different locations or seasons. Brier Skill Scores, however, can be meaningfully compared, since they remove this effect by comparing the skill of one forecast system for an event with that of another system. A decomposition of the Brier Score into three separate components was proposed by Murphy (1973). These three components are reliability, resolution, and uncertainty. Reliability is a measure of our ability to assign event probabilities realistically, and is discussed separately in section 4.2. The resolution measures how much our forecasts deviate from the climatological probability of the event, indicating how much forecast information is present; numerically this term is negative. Uncertainty is affected only by what actually happens (the observations), and indicates the intrinsic difficulty in forecasting the event. Brier Score Decomposition is also discussed by Atger (1999a) and by Wilks (1995). Note that reliability can easily be improved at the cost of worse resolution and vice versa one needs to achieve an improvement in both in order to get a real gain in skill. Brier Score Decompositions are produced by our verification system, but the results are not shown here. Examples of Brier Scores and Skill Scores for forecasts of various high temperature values and 12-hour rainfall amounts are presented in Figure 10 for all forecast times up to 10 days ahead. Note that the dashed lines (a) Top (b) Top (a) Bottom Figure 10. (a) Brier Scores. (b) Brier Skill Scores. For warm events (spot temperature exceeding 15, 20, 25ºC) and wet events (12-hour precipitation accumulations exceeding 0.1, 1.0, 5.0, 10.0 mm), forecast times T+24 to T+240, from all data August 1997 to August (b) Bottom

13 Use of medium-range ensembles at the Met Office, Part 1 give the scores for deterministic forecasts from the ECMWF high-resolution deterministic model for comparison. For rainfall, the Brier Scores are lower for larger amounts, as expected. Brier Skill Scores are rather low for a threshold amount of 0.1mm, due to the model s limited skill in forecasting small amounts of rainfall (in particular a tendency to produce small amounts of rainfall when in fact none occurs). Skill for a threshold of 5 mm is better, but drops off again when the less common event of 10 mm in 12 hours is assessed. This graph illustrates that, as might be intuitively expected, the more extreme the event (and the events used in Figure 10 are not extreme climatologically), the more difficult it is to predict, even probabilistically. Nevertheless, the ensemble still gives a clear advantage over the high-resolution deterministic forecast at day 3 and beyond. 4.2 Reliability diagrams For a perfect probability forecast system, out of all occasions when an event s probability is forecast to be p, the event should actually occur on a fraction p of occasions (Brier 1950). By grouping forecasts together according to the forecast probability, a graph can be plotted to show how closely this ideal is followed as a function of p, and for a perfectly reliable forecast system the graph would follow a diagonal line from (0.0, 0.0) to (1.0, 1.0). If the curve showing the fraction of event occurrences as a function of forecast probability p lies below this line, then forecast probabilities are mostly too high; if above, then forecast probabilities are too low. If the slope of the curve is too shallow this is termed over-confidence since high forecast probabilities tend to be too high and low probabilities too low. This is a characteristic of ensembles which have insufficient spread to cover the full forecast uncertainty. The reliability curve also gives an indication of resolution: for example, if the line is almost horizontal then the forecast system lacks resolution, i.e. it cannot distinguish between occasions when the probability of an event is high or low. Examples of reliability curves for temperature and rainfall event definitions are given in Figure 11. These tend to show an over-confidence in prediction of such events (forecast probabilities too high), and also some lack of resolution. For both sets of temperature reliability curves (night-time low temperatures and daytime high temperatures) it is notable that the forecasts become less reliable and lack resolution for the more extreme events (e.g. T < (2ºC, T > 25ºC); conversely the reliability of forecasts of T > 20ºC is remarkably good. Often included with reliability graphs, as here, are subsidiary plots showing the relative frequency with which each probability level has been forecast ( sharpness ). These show, for example, that high probabilities of rare events (e.g. temperature > 25ºC, rainfall > 10 mm) are rarely forecast. Probabilities of rainfall > 0.1 mm are frequently over-estimated for point observations. This is a common characteristic of models, which represent local showers as small amounts of rain spread over the whole grid-box. Provided it is based on a sufficiently large and representative set of data, the reliability diagram can be used as a means of calibrating probability forecasts. For example, if it is found that, when the forecast system produces a probability of p=50% for a given event, that event is only observed to occur on 30% of occasions, then the forecast probability should be altered to 30% before the forecast is issued. However, it must be noted that although such calibration results in reliable probability forecasts, it does so only by limiting the range of forecast probabilities used. This may mean that certain Figure 11. Reliability diagrams, (a) for spot temperature below 2, 0, 2, 5ºC at forecast time T+84, (b) for spot temperature above 15, 20, 25ºC at forecast time T+96, (c) for 12-hour precipitation accumulation exceeding 0.1, 1.0, 5.0, 10.0 mm at forecast time T+96; all using data August 1997 to August

14 T P Legg, K R Mylne and C Woolcock (a) (b) Figure 12. (a) Relative Operating Characteristic curves. For warm events (spot temperature exceeding 15, 20ºC) and 12-hour precipitation (accumulation exceeding 1.0, 5.0 mm), for forecast time T+96, from all data August 1997 to August The dot on these graphs refers to results for the high-resolution deterministic model. (b) Areas under Relative Operating Characteristic curves. For warm events (spot temperature exceeding 15, 20, 25ºC) and 12-hour precipitation (accumulation exceeding 0.1, 1.0, 5.0, 10.0 mm), for forecast times T+24 to T+240, from all data August 1997 to August

15 Use of medium-range ensembles at the Met Office, Part 1 calibrated probabilities would never be issued, for example for T < 2ºC the calibrated probability would never be more than 70%. 4.3 Relative Operating Characteristic The Relative Operating Characteristic (ROC) is a measure which has gained favour in recent years, and originated in signal detection theory (Stanski et al. 1989). It shows the relationship between Hit Rate (HR) and False-Alarm Rate (FAR). The quantities HR and FAR are calculated with respect to the observations, as follows: HR = (No. of hits) / (Total no. of observed events) FAR = (No. of false alarms) / (Total no. of non-events) A fuller description of ROC is given in the Appendix to Mylne (2001). By calculating HR and FAR at a set of probability thresholds, usually at intervals of 0.1, a graph can be plotted showing the relationship between the two. As the probability threshold decreases, the number of times the event is forecast to occur increases monotonically, and hence both the HR and FAR increase. The ROC graph for a probability forecast therefore forms a monotonically decreasing curve from (1.0, 1.0) to (0.0, 0.0). Ideally, we would like to have high hit rates and low false-alarm rates, and hence a curve tending towards the upper-left portion of the graph; if all yes cases were forecast with probability 1.0 and all no events were predicted definitely not to occur, then the ROC curve would run up the left-hand abscissa and then along the top of the graph. A skill-less (useless) forecasting system, unable to differentiate between when an event is more or less likely to occur, will produce hits and false-alarms with equal frequency, so the ROC curve will lie along the diagonal. Figure 12a shows curves for certain defined events, for four-day forecasts. All tend towards the top-left of the graphs to varying degrees, indicating that the probability forecasts have skill. Because a deterministic forecast can only give probabilities of 1 or 0, the ROC for deterministic forecasts is given by a single point. In the examples shown, this single point usually lies on or below the curve obtained from the probabilistic forecasts. For rarer events, both hit rate and false-alarm rate tend to be lower, so the points tend to lie nearer the bottom-left of the graph. Note that the ROC is a measure only of the forecast system s ability to discriminate events from non-events and is totally independent of reliability, since it is based on a stratification by observations. A further measure that can be determined is the area under the ROC curve, which provides a useful summary of the information. This area has a maximum possible value of 1.0, whereas a skill-less forecasting system would produce a ROC area of 0.5. Different sets of forecasts can usefully be compared by examining the ROC areas. Results for our system are shown in Figure 12b; ROC areas for temperature are greatest for relatively common events. For rainfall the area is greatest for R > 1 mm. Skill is reduced for more extreme events, but also for R > 0.1 mm due to the model bias already Figure 13. Rank histogram for 12-hour precipitation at T+96, from all ensemble data August 1997 to August

16 T P Legg, K R Mylne and C Woolcock noted. Note the intuitively expected result, especially for rainfall, that ROC areas decrease with increasing forecast time. Also, the ROC areas from ensemble forecasts exceed those from deterministic forecasts, though it is difficult to draw conclusions here because the deterministic ROC is only a single point whereas the probabilistic ROC is a smooth curve so this is not a truly fair comparison. Mylne (2001) provides a better illustration of the advantages of probability forecasts by comparing the economic value of a deterministic forecast with the optimal value that can be obtained from a probability forecast, using exactly the same verification information. 4.4 Rank histograms and numbers of outliers Given an ensemble forecast generated from m members, the m values can be ranked in order and used to divide the forecast distribution into (m+1) bins which for a perfect system would all be equally probable. It is then of interest, over many cases, to examine the distribution of verifying observations within these (m+1) bins. This can be illustrated by plotting a rank histogram (Hamill & Colucci 1997, 1998). In an example such as Figure 13, we would ideally hope to see an equal population across each of the bins. Note that for an ideal forecast system this implies a proportion equal to 2/(m+1) of all observations occurring in the outermost bins which correspond to verifying (observed) values lying outside the range of the ensemble. Note that this is closely related to reliability ( a perfectly reliable forecasting system would give a flat rank histogram. By studying rank histograms constructed in this way we can determine whether our forecast system comes close to achieving this ideal. If the spread of the ensemble is realistic then the ideal rank histogram should approximately be attained. It has long been noted from our results that this is not the case: it is found that the outermost bins are over-populated, often considerably so, indicating that the verification too often lies outside the range covered by the ensemble, particularly at shorter time-ranges. This may imply that the spread of the ensemble is insufficient, on average. Indeed, the over-population of the outermost bins is more noticeable here, for surface weather assessments, than in previous work that concentrated on broad-scale model parameters such as 500 hpa height (Molteni et al. 1996); this could however be due to certain stations frequently populating one of the outermost bins, due to stationspecific biases. Ways of determining the probability of a value lying beyond the extremes of an ensemble s distribution, and how far beyond, are currently being addressed. Post-hoc corrections could be made to achieve calibration of probability forecasts, to assign realistic probabilities to the tails of the ensemble distribution (Hamill & Colucci 1998); this is easier if the shape of the rank histogram is not too extreme. Bias 270 corrections will in future be applied on a station-bystation basis, to overcome individual station characteristics. Figure 13 is a rank histogram for four-day forecasts of rainfall. Many of the cases contributing to the extreme left-hand bar (all ensemble members too wet) arose when no rain fell but the model did produce small amounts of rain, a known bias problem. For longer forecast times the outlier proportions are lower (not shown). 5. Conclusions This paper has described the Met Office s operational system for production of probability forecasts from the ECMWF Ensemble Prediction System. Numerous tools are available for displaying both chart products based on model fields and site-specific products based on interpolated point forecasts. Extensive verification of the site-specific probability forecasts is also available. This shows the clear advantage of probabilistic forecasts over traditional deterministic methods, even though the deterministic forecast is run at higher horizontal resolution. Absolute skill of site-specific forecasts based on direct interpolation from model fields is relatively poor, and work is currently in progress to implement site-specific bias corrections to help address this problem. Practical applications of the system by forecasters are considered in the companion paper by Young & Carroll (2001). Further verification from a user perspective, including a method for the application of probability forecasts for decision-making, is described by Mylne (2001). Acknowledgements Thanks are due to Mike Harrison for originally setting up this project and providing direction and encouragement in its early stages. The help of Kelvyn Robertson in the development of the cyclone-tracking software is gratefully acknowledged. References Atger, F. (1999a) The skill of ensemble prediction systems. Mon. Wea. Rev. 127: Atger, F. (1999b) Tubing: an alternative to clustering for the classification of ensemble forecasts. Wea. and Forecasting 14: Barkmeijer, J., Buizza, R. & Palmer, T. N. (1999) 3D-Var Hessian singular vectors and their potential use in the ECMWF Ensemble Prediction System. Q. J. R. Meteorol. Soc. 125: Brier, G. W. (1950) Verification of forecasts expressed in terms of probability. Mon. Wea. Rev. 78: 1 3. Buizza, R., Miller, M. & Palmer, T. N. (1999) Stochastic representation of model uncertainties in the ECMWF EPS. Q. J. R. Meteorol. Soc. 125:

17 Use of medium-range ensembles at the Met Office, Part 1 Buizza, R., Barkmeijer, J., Palmer, T. N. & Richardson, D. S. (2000) Current status and future developments of the ECMWF Ensemble Prediction System. Meteorol. Appl. 7: Du, J. & Tracton, M. S. (1999) Impact of lateral boundary conditions on regional model ensemble prediction. In: Research Activities in Atmospheric and Oceanic Modelling, ed. H. Ritchie. Report 28, CAS/JSC Working Group Numerical Experimentation (WGNE), WMO/TD-No. 942, Ehrendorfer, M., Errico, R. M. & Raeder, K. D. (1999) Singular-vector perturbation growth in a primitive equation model with moist physics. J. Atmos. Sci. 56: Evans, R. E., Harrison, M. S. J. & Graham, R. J. (1999) Joint medium-range ensembles from the UKMO and ECMWF models. Proc. Workshop on Predictability, October 1997, ECMWF, Shinfield Park, Reading. Graham, R. J., Evans, A. D. L., Mylne, K. R., Harrison, M. S. J. & Robertson, K. B. (1999) An assessment of seasonal predictability using atmospheric general circulation models. Q. J. R. Meteorol. Soc. 126: Hamill, T. M. & Colucci, S. J. (1997) Verification of Eta-RSM short-range ensemble forecasts. Mon. Wea. Rev. 125: Hamill, T. M. & Colucci, S. J. (1998) Evaluation of Eta-RSM ensemble probabilistic precipitation forecasts. Mon. Wea. Rev. 126: Houtekamer, P. L., Lefaivre, L., Derome, J., Ritchie, H. & Mitchell, H. L. (1996) A system simulation approach to ensemble precision. Mon. Wea. Rev. 124: Legg, T. P. (1998) Ensembles in the medium-range: The use and assessment of ensembles for forecasting surface temperature and precipitation 1 to 10 days ahead. Proc. WMO International Workshop on Dynamical Extended-Range Forecasting, Toulouse, France, November 1997, WMO TD no. 881, pp (PWPR Report Series no. 11). Molteni, F., Buizza, R., Palmer, T. N. & Petroliagis, T. (1996) The ECMWF Ensemble Prediction System: methodology and validation. Q. J. R. Meteorol. Soc. 122: Murphy, A. H. (1973) A new vector partition of the probability score. J. Appl. Meteorol. 12: Mylne, K. R. (2001) Decision-making from probability forecasts using calculations of forecast value. Meteorol. Appl. 9: Mylne, K. R., Evans, R. E. & Clark, R. T. (2002) Multi-model multi-analysis ensembles in quasi-operational mediumrange forecasting. To appear in Q. J. R. Meteorol. Soc. Persson, A. (2001) User Guide to ECMWF Forecast Products. ECMWF, Reading, Meteorological Bulletin M3.2, 112pp. Rabier, F., Klinker, E., Courtier, P. & Hollingsworth, A. (1996) Sensitivity of forecast errors to initial conditions. Q. J. R. Meteorol. Soc. 122: Stanski, H. R., Wilson, L. J. & Burrows, W. R. (1989) A Survey of Common Verification Methods in Meteorology. WMO WWW Tech. Report no. 8, WMO TD no. 358, 114pp. Stensrud, D. J., Brooks, H. E., Du, J., Tracton, M. S. & Rogers, E. (1997) Using ensembles for short-range forecasting. Mon. Wea. Rev. 127: Terry, J. & Atlas, R. (1996) Objective cyclone-tracking and its application to EPS-1 Scatterometer Forecast Impact Studies. Paper delivered at 15th Conference on Weather Analysis and Forecasting, Norfolk, Virginia, 1996, American Meteorol. Soc. Toth, Z. & Kalnay, E. (1993) Ensemble forecasting at the NMC: the generation of perturbations. Bull. Amer. Meteorol. Soc. 74: Toth, Z., Kalnay, E., Tracton, M. S., Wobus, R. & Irwin, J. (1997) A synoptic evaluation of the NCEP ensemble. Wea. and Forecasting, 12: Toth, Z., Zhu, Y., Marchok, T., Tracton, M. S. & Kalnay, E. (1998) Verification of the NCEP global ensemble forecasts. Proc. 12th Conf. on NWP, Amer. Meteorol. Soc., pp Wilks, D. S. (1995) Statistical Methods in the Atmospheric Sciences An Introduction. International Geophysics Series vol. 59, Academic Press. Young, M. V. & Carroll, E. B. (2002). Use of medium-range ensembles at the Met Office II: Applications for mediumrange forecasting. Meteorol. Appl. 9: Ziehmann, C. (2000) Comparison of a single-model EPS with a multi-model ensemble consisting of a few operational models. Tellus 52A: