Point verification and improved communication of the low-to-medium cloud cover forecasts

Transcription

1 METEOROLOGICAL APPLICATIONS Meteorol. Appl. 24: (2017) Published online in Wiley Online Library (wileyonlinelibrary.com) DOI: /met.1645 Point verification and improved communication of the low-to-medium cloud cover forecasts Junella Tam* and Wai-kin Wong Hong Kong Observatory, Hong Kong ABSTRACT: The amount of sunshine weighs heavily in our perception of the weather. It is largely determined by cloud cover, especially that at the low-to-medium level. Therefore, when reviewing the Hong Kong Observatory s weather symbol forecasting product, verification on the low-to-medium cloud field is carried out against the synoptic observations at the Hong Kong International Airport. Several metrics are used to examine the different aspects of the forecasts, and consideration is given to the non-gaussian nature of the reported cloud amount for a fairer assessment. Based on the data from January to mid-august 2015, the median of the forecasts from the European Centre for Medium-Range Weather Forecasts Ensemble Prediction System is found to outperform the other model forecasts, particularly when the performance is examined by forecast day. This paper presents these results and also discusses the potential of using the field in deriving site-specific weather symbol forecasts. KEY WORDS cloud verification; weather symbols; forecasting systems Received 8 May 2016; Revised 4 October 2016; Accepted 6 October Introduction Weather symbols are used to describe the current or forecast weather conditions in a concise manner. The selection of weather symbols is largely based on the amount of sunshine and precipitation. The Hong Kong Observatory (HKO) collects in situ observations for rainfall from over 160 rain gauges, and the overall spatial distribution can be estimated from radar data. However, measurements of sunshine are rather limited and this makes both the forecast and the verification of weather symbols fairly challenging, but this study offers a way of tackling the issue. Currently, the observations available for analysis include the sunshine duration at King s Park and the cloud amounts reported at the Hong Kong International Airport (HKIA) as part of the surface synoptic observations (SYNOP). Although the two sites are some 26 km apart, certain relationships between these quantities can be observed. Figure 1 shows that the daily averaged total cloud amount can stand at around 7 oktas regardless of the number of hours of sunshine recorded, suggesting that it would be hard to distinguish a sunny day from a cloudy day using solely the total cloud amount. In addition to the total cloud amount, however, a SYNOP report also contains the low cloud amount, and if unavailable, the medium cloud amount. Displaying a stronger correspondence, the low-to-medium cloud amount seems to be a more appropriate proxy for the amount of sunshine. Verification of the low-to-medium cloud amount is carried out for different numerical weather prediction (NWP) models in Section 2. The percentages of low clouds and of medium clouds from the deterministic models of the European Centre for Medium-Range Weather Forecasts (ECMWF), Japan * Correspondence: J. Tam, Hong Kong Observatory, 134A Nathan Road, Tsim Sha Tsui, Hong Kong. yttam@hko.gov.hk; jtam. hko@gmail.com Meteorological Agency (JMA), the US National Centers for Environmental Prediction (NCEP) as well as the Ensemble Prediction System (EPS) of the ECMWF are considered. Since the forecast ranges of these models all cover a 10-day period after the time of initialization, the T + 1h to T+ 240 h forecasts are taken to assess their performance in the short to medium term. The data are interpolated to the HKIA and compared against the SYNOP observations according to the guidelines issued by the World Meteorological Organization (WMO). The various metrics examined indicate that the median of the ECMWF EPS outperforms the rest of the models throughout the 240 h forecast period. Built on that basis, the potential of using the ECMWF EPS median low-to-medium cloud cover to derive the forecast weather symbols is then assessed. The symbols currently in use at the HKO forecasting office to complement the official local forecasts over the whole of Hong Kong come in four categories in terms of the amount of sunshine expected: sunny, sunny periods, sunny intervals and cloudy. Though they do not exactly represent the weather conditions at the HKIA, these symbols issued by the duty forecasters in the HKO forecasting office nevertheless serve as a useful reference for formulating an algorithm to assign forecast weather symbols automatically. Figure 2 shows the distribution of the SYNOP cloud amounts for each category between 1 January and 19 August The median of the total cloud amount corresponding to sunny intervals and that corresponding to cloudy are observed to coincide. To compare the mean values of these two distributions, Welch s two-sample t-test is run. The results shown in Table 1 suggest that the difference in mean of these two categories is less than an okta. In contrast, the means of the low-to-medium cloud amounts corresponding to sunny intervals and cloudy are more easily distinguishable, as roughly 3 and 4 oktas, respectively. Using 0, 1, 3 and 4 oktas of forecast low-to-medium cloud amounts as thresholds for sunny, sunny periods, sunny intervals and cloudy, respectively (such that a forecast cloud 2017 Royal Meteorological Society

2 Verification and communication of the low-to-medium cloud forecasts 467 Figure 1. The total number of hours of sunshine recorded at King s Park for each day is plotted against the corresponding daily averaged SYNOP cloud amount reported at the HKIA between 1 January and 19 August The correlation between the sunshine duration and (a) the total cloud amount is 0.60, but that between the sunshine duration and (b) the low-to-medium cloud amount is stronger, at amount that lies between 0 and 1 okta gives sunny, while a forecast cloud amount greater than 4 oktas gives cloudy ), the forecasts for 15 sites around Hong Kong are compared against the weather symbols issued by weather forecasters at the time for the whole territory in Section 3. The results are found to be comparable with those of similar studies, and a particular case is presented to illustrate how such a visual forecast product would perform. 2. Low-to-medium cloud verification The WMO s guidelines on evaluating cloud amount (WMO, 2012) discuss in depth the various aspects of handling observation data. Considering that the ultimate goal of this study is to develop a methodology that takes human perception of the weather into account, the manual SYNOP observations at the Figure 2. The distribution of the SYNOP cloud amounts corresponding to the four terms used by the HKO forecasting office to describe the expected amount of sunshine between 1 January and 19 August The boxes span 50% of the data with the tops and bottoms marking the 75th and the 25th percentiles respectively. Each of the whiskers covers 1.5 times the interquartile range (unless the most extreme point is closer), and the circles represent the outliers beyond that. It can be observed that the median total cloud amount corresponding to sunny intervals coincides with that corresponding to cloudy, suggesting that the total cloud amount may not be very useful in determining the amount of sunshine received. HKIA are taken as ground truths for the purpose of this paper. The aim of this section is to identify the model forecasts that best match with the reported low-to-medium cloud amounts before discussing the assignment of weather symbols in Section Marginal and joint distributions The joint distributions of the observed and forecast cloud amounts provide a clear depiction of the association between the two. The multi-category contingency tables corresponding to the various NWP model outputs are plotted as heat

3 468 J. Tam and W.-K. Wong Table 1. Key statistics from Welch s two-sample t-tests conducted on the SYNOP cloud amounts (in oktas) between 1 January and 19 August Sunny Sunny periods Sunny intervals Cloudy Total cloud amount Sample mean Low-to-medium cloud amount Sample mean , 2.33] [ 2.17, 1.89] [ 0.64, 0.40] , 1.32] [ 1.24, 0.94] [ 1.21, 0.86] The observations are split into four groups according to the terms used by the HKO forecasting office to describe the expected amount of sunshine at the time. The mean cloud amount is calculated for each of these groups, and the difference between the means of each pair of neighbouring groups is also calculated (not shown) along with the 95% confidence interval over which this difference is expected to lie. These statistics quantify how distinct the samples are. maps in Figure 3. Ideally, the maps should be darkest along the diagonal to represent a perfect correspondence between observations and forecasts. Instead, NCEP s map in Figure 3(c) is darkest along the bottom, suggesting that it grossly underestimates the low-to-medium cloud amount. The marginal distribution on the side shows that over 30% of the forecasts give 0 okta, and such a bias is reflected throughout the rest of this section. Note that the marginal distribution (top horizontal panel) of the observed low-to-medium cloud amount is bimodal, peaking at 1 and 4 oktas, and tailing off towards 8 oktas. This feature seems to be replicated best by the JMA model (vertical panel in Figure 3(b)), except that one of the peaks has shifted from 1 to 0 okta. This shift may not seem significant at first glance, but as Mittermaier (2012) explains, cloud amounts are rarely considered in isolation but are usually taken as being greater than a defined amount, thus this will have an impact on the verification scores examined below. The following verification metrics are based on the two-by-two contingency tables constructed using an increasing number of oktas as the exceedence threshold. The multi-category tables in Figures 3(a) (e) can be sliced into four blocks like Figure 3(f) (e.g. with the top-right representing hits, i.e. forecasts that correctly predict the cloud amount to exceed certain oktas). Details on the calculation of the metrics involving the numbers of hits, false alarms, misses and correct negatives are documented in the Appendix Frequency bias The frequency bias provides a quantitative way to compare the distributions of the observations and forecasts. The biases of the NWP models for the full range of exceedence thresholds are plotted in Figure 4, and NCEP s tendency to underforecast is reflected by most of its values being below 1 in Figure 4(c). JMA s shift from 1 to 0 okta can also be observed through this metric. Figure 4(b) shows the value dipping below 1 for cloud amounts 1 okta but overshooting when amounts 2 oktas are considered, representing an underforecast of exactly 1 okta of clouds. Beyond that, the frequency bias keeps increasing. This is in part due to the other peak in the marginal distribution at 4 oktas. Referring to the heat map in Figure 3(b), it is observed that the darker shades are mostly above the diagonal, which indicates that the model generally gives cloudy forecasts in less cloudy situations and results in a large number of false alarms. The large frequency biases towards the cloudy end for the ECMWF models (Figures 4(a), (d) and (e)) can be explained in a similar way. The corresponding joint distribution heat maps (Figures 3(a), (d), and (e)) show that the dark shades spread upwards at 4 oktas. Once the exceedence threshold for constructing the two-by-two contingency tables slides past 4 oktas, these boxes are counted as false alarms rather than hits. However, overforecasting over this range should not be too much of a concern, considering that the weather symbol classification only involves the values between 0 and 4 oktas (Figure 2) forecasts of 4 oktas will all simply map to the cloudy symbol. More rigorous verification regarding the symbol forecasting is described in Section 3, but in terms of the frequency bias, it can be seen from Figure 4 that the ECMWF models perform relatively well over the range of interest for this study Log-odds ratio The odds ratio compares the negative and positive associations between forecasts and observations. As it does not depend on the marginal distributions, Stephenson (2000) considers it an equitable score that cannot be hedged by predicting events with higher frequencies of occurrence. For unskilled forecasts, the value simply gives zero. The set of U-shaped curves in Figure 5 therefore indicates a decline in the significance of forecast skill as the lowest cloud amounts are progressively removed before recovering towards the cloudy end. It can be seen from Figure 3(e) that the distribution of the ECMWF EPS median forecasts is centred on 2 oktas, as supposed to the corresponding peaks at 1 and 4 oktas. Moving the exceedence threshold from 1 to 2 oktas, the darkest box corresponding to forecasts of 2 oktas for 1 okta of clouds in the joint distribution is no longer counted as hits but rather as false alarms (which agrees with the frequency bias of over 1 shown in Figure 4(e) as an indication for a tendency to overforecast), thus reducing the odds ratio. Further increasing the exceedence threshold to 3 oktas, the box corresponding to forecasts of 2 oktas for 4 oktas of clouds contributes towards misses rather than hits such that the odds of a hit, and hence the odds ratio, are reduced. It is not until the exceedence threshold reaches 4 oktas that they (the darkest boxes corresponding to instances where 4 oktas of clouds are reported but lower cloud amounts, mostly 2 oktas, are forecast) become correct negatives and contribute positively towards the odds ratio. Nevertheless, both the mean and median of the ECMWF EPS still perform better than the other forecasting systems examined. The odds of the yes predictions being correct are greater than the odds of them being incorrect by the highest margin across all the exceedence thresholds. While this might seem consistent with the frequency bias results, it is perhaps surprising to see that NCEP does not score too poorly according to this metric. The log-odds ratio considers the correct negatives while the frequency bias does not. NCEP s tendency to underforecast, in fact, reduces the odds of a false alarm and counteracts the drop in the odds of a hit arising from the large number of misses. This illustrates that, in order to evaluate the characteristics of a model forecast thoroughly, multiple verification metrics should be considered.

4 Verification and communication of the low-to-medium cloud forecasts 469 Figure 3. Heat maps showing the joint distributions of the observed (along the x-axis) and forecast (along the y-axis) cloud amounts, together with bar plots showing the marginal distributions of the datasets. The verification period is between 1 January and 19 August The observations are the SYNOP low-to-medium cloud amounts reported at the HKIA, and the forecasts are the interpolated values of (a) the ECMWF deterministic model, (b) JMA, (c) NCEP, (d) the ECMWF EPS mean and (e) the ECMWF EPS median. Given an exceedence threshold, each of these heat maps can be transformed into a two-by-two contingency table (f), with the top-right representing hits, i.e. forecasts that correctly predict the cloud amount to exceed the threshold, the top-left representing false alarms, the bottom-right representing the misses, and the bottom-left representing the correct negatives Royal Meteorological Society Meteorol. Appl. 24: (2017)

5 470 J. Tam and W.-K. Wong Figure 4. The frequency biases (Equation (A.1)) between the observed and the forecast low-to-medium cloud amounts reaching the thresholds specified along the x-axis. The verification period is between 1 January and 19 August The models under consideration are (a) the ECMWF deterministic model, (b) JMA, (c) NCEP, (d) the ECMWF EPS mean and (e) the ECMWF EPS median. The ideal value is 1 which means that the observed event (of the cloud amount exceeding a certain number of oktas) occurs as often as it is forecast. The model that stands out in Figure 5 is JMA, of which the corresponding curve drops off the scale at 8 oktas instead of rising further. This is attributed to the relatively high numbers of misses and false alarms at this threshold which contribute negatively towards the score, and the relatively low number of hits which is shown as a light-grey box in the top right of the heat map in Figure 3(b). Although NCEP in Figure 3(c) and the ECMWF EPS mean in Figure 3(d) also seem to suffer from a low number of hits, their false alarms are not very high as indicated by the light shades of the boxes along the top row to the left of

6 Verification and communication of the low-to-medium cloud forecasts 471 Figure 5. The log-odds ratios (Equation (A.4)) calculated based on the model forecast low-to-medium cloud amounts reaching the thresholds specified along the x-axis, with the envelope around each line corresponding to the 95% confidence interval. A score of zero represents that skill is due to pure chance, and the verification period is between 1 January and 19 August Figure 6. SEDS (Equation (A.5)) calculated based on the model forecast low-to-medium cloud amounts reaching the thresholds specified along the x-axis, and the envelope around each line corresponds to the 95% confidence interval. The perfect score is 1 which is achieved with zero misses and false alarms, and the verification period is between 1 January and 19 August the hits box. The remaining model outputs, i.e. the ECMWF deterministic model in Figure 3(a) and the ECMWF EPS median in Figure 3(e), may have quite a number of false alarms, but the hits are rather high and there are not as many misses as JMA. For JMA, the box that corresponds to a forecast amount of 7 oktas but an observed amount of 8 oktas counts towards hits when cloud amounts 7 oktas are considered. Once the threshold is moved up to 8 oktas, the box counts towards misses instead, resulting in a drastic drop in the log-odds ratio. A similar drop at 8 oktas can also be observed with the next metric Symmetric Extreme Dependency Score (SEDS) Hogan et al. (2009) introduces the SEDS for the verification of biased variables, to address the non-gaussian nature of the cloud field in particular. Though Ferro and Stephenson (2010) demonstrate that the score is still dependent on the frequency of occurrence, Mittermaier (2012) considers it to be a suitable metric for cloud verification because it is equitable for large samples and less susceptible to hedging. Taking it as a measure of forecast skill, the ECMWF models (both the deterministic and the EPS) are again observed in Figure 6 to outperform the others, but when the forecasts are stratified by forecast day as in Figure 7, their performance begins to diverge and that of the ECMWF EPS median appears to be the most consistent throughout. The mean s sharp drop in score for the overcast scenario for the 144 h forecasts and beyond could be attributed to an underforecast of the cloud cover, as the averaging tends to smooth out the forecasts and results in a narrower distribution with thinner tails at both ends. Given that the verification period includes a model change of the ECMWF systems, the results are split into before and after 15 April 2015 in an attempt to examine the impact on the cloud forecasts in the region. Figure 8 shows that SEDS is lower after the change, but this is true for all the models, so the poorer performance is likely to be due to seasonal effects. Viewing these verification results, it appears that the ECMWF EPS median has the best overall performance in forecasting the low-to-medium cloud amount. Its forecasts are therefore used to generate weather symbols and the performance is evaluated in the following section. 3. Weather symbols As mentioned in Section 1, the symbols describe the expected amount of sunshine, or conversely, the cloud cover. There are four categories for the state of sky: sunny, sunny periods, sunny intervals and cloudy. The corresponding distributions of the observed cloud amount are described in Table 1. For sunny, the mean low-to-medium cloud amount is shown in Table 1 to be 0.77 okta so the corresponding range of cloud amount is set to be [0, 1]. Similarly, the ranges for sunny periods, sunny intervals and cloudy are set around the means and are (1, 3], (3, 4] and (4, 8], respectively. Although these interval values are obtained from SYNOP observations, they are assumed to be applicable to the model forecasts without any fine-tuning because the field verifies reasonably well, especially in terms of the frequency bias which is around 1 over this range as shown in Figure 4(e). The ECMWF EPS median low-to-medium cloud field is thus interpolated to the 15 sites around Hong Kong shown in Figure 12 and converted into weather symbols using these thresholds to describe the expected state of the sky Statistical evaluation To assess the potential of such forecasts, the weather symbols issued by the forecasters are taken as proxies for the human perception of the state of the sky over the whole of Hong Kong for comparison. Although there is a forecast element in the symbols issued by the HKO forecasting office to complement the local weather forecasts, they are also supposed to cater for the weather conditions at the time, so they are assumed not to be

7 472 J. Tam and W.-K. Wong Figure 7. SEDS calculated based on the model forecast low-to-medium cloud amounts reaching the thresholds specified along the x-axis for different forecast days: (a) T + 1 h to T + 24 h; (b) T + 49 h to T + 72 h; (c) T + 97 h to T h; (d) T h to T h and (e) T h to T h, with the envelope around each line representing the 95% confidence interval. The perfect score is 1, and the verification period is between 1 January and 19 August It is shown that the performance of the ECMWF EPS mean is on par with that of the median until the final forecast day. too far from the state of the sky that the human forecasters could observe at the time. As in Section 2.1, heat maps are compiled to compare the distributions of these observed and forecasted weather symbols, and it is encouraging to see in Figure 9 that the 2017 Royal Meteorological Society marginal distributions share a similar shape, in which the counts for cloudy and sunny periods are the highest. A more objective measure is perhaps the percent hits, which denotes the percentage of the samples lying on the diagonal. In Meteorol. Appl. 24: (2017)

8 Verification and communication of the low-to-medium cloud forecasts 473 Though intuitive and straight forward to communicate to a general audience, both the percent hits and proportion correct by category are criticized by Jolliffe et al. (2003) for weighing all the correct forecasts the same regardless of the sample distribution, thus encouraging the forecasting of the most frequently occurring categories. Other commonly used metrics, namely the Heidke skill score and the Peirce skill score, also fail to give sufficient credit to the correct forecasts and near hits of the rarer events. In contrast, the score introduced by Gerrity (1992) considers the underlying distribution as well as the off-diagonal information, such that less penalty is assigned to an incorrect forecast of a rare event than a similar-sized error of a common event. Given that the observed frequencies of the state of sky range from 8.37% for sunny to 41.0% for cloudy as shown in Figure 9, it seems fairer to use the Gerrity skill score (GSS) when assessing the forecast performance. Using the data from Figure 9, the GSS is calculated using Equation (A.10) to be 0.40 and the break-down by forecast day is plotted in Figure 10. It shows that the GSS ranges from 0.27 to Since the sky-cover forecast verification of the NOAA National Weather Service based on the Global Forecasting System model output statistics (Kluepfel, 2009) and that of the FMI based on the HIgh Resolution Limited Area Model (HIRLAM) give scores of 0.38 and 0.46 respectively, it seems that the ECMWF EPS median of the low-to-medium cloud amount forecasts performs reasonably well in forecasting the state-of-sky categories Regional variation Figure 8. SEDS calculated based on the model forecast low-to-medium cloud amounts reaching the thresholds specified along the x-axis over different parts of the verification period: (a) between the 0000 UTC run on 1 January and the 0000 UTC run on 15 April 2015, i.e. before a model change of the ECMWF systems, and (b) between the 0012 UTC run on 15 April 2015 and 0012 UTC run on 19 August 2015, i.e. after the change. It is shown that the scores are lower in (b) which is possibly due to seasonal effects. The 95% confidence interval around each line gives a quantitative estimate of the uncertainty of the verification result. this case, it is calculated to be 48% which seems comparable to the results from the cloud verification studies done by the National Oceanic and Atmospheric Administration (NOAA) (Kluepfel, 2009) and the Finnish Meteorological Institute (FMI) (WMO, 2012) which give 45% and 61% respectively. Another verification metric computed based on contingency tables is the proportion correct by category, which counts not only hits but also the correct negatives. Table 2 shows the values for the sunny, sunny periods, sunny intervals and cloudy categories by each of the forecast days, and they average to 0.92, 0.67, 0.70 and 0.68, respectively. In the previous section, the forecasts over 15 stations are compared against the weather symbols issued for the whole of Hong Kong over the verification period, because collectively they should give a fair representation of the overall weather condition in the territory. In this section, it is hoped to see if the regional variation in cloud cover can be represented by the forecasts. Take the 3 h interval of HKT ( UTC) on 11 August 2015 as an example. The duty forecaster expected isolated showers but sunny periods in general for that day. The rainfall distribution map in Figure 11 shows that rainfall in the hour preceding 3 pm was confined in the northern part of the territory, which is reflected in the cloudier pictures captured by the web cameras at locations such as the Wetland Park, and the blue sky observed down south at Cape D Aguilar. The derived weather symbols for that time period from the latest available run, the 1200 UTC run from 10 August 2015, are shown in Figure 12. Map (a) corresponds to the output from the current operational system, which is based primarily on the total cloud cover forecasts and adjusted by the low-to-medium cloud amounts as described in Leung and Tam (2016). It gives an overall cloudy picture for Hong Kong and does not quite match with what is observed in Figure 11 especially over the south. In contrast, Map (b), which is derived from the ECMWF EPS median low-to-medium cloud cover, seems to correspond with the observations better. The symbols for sunny periods (with dotted borders) dominate, but a few symbols for sunny intervals (with thick solid borders) are also given for the northern part close to the Wetland Park. This particular case demonstrates how such a product can complement the official forecasts, by communicating certain subtleties in the regional differences in the state of sky that would otherwise make the worded forecasts fairly clumsy. The regional information that the product offers also supports the HKO s

9 474 J. Tam and W.-K. Wong Figure 9. Heat map showing the joint distributions of the observed (along the x-axis) state of sky based on the weather symbols issued by the forecasting office and the forecast (along the y-axis) state of sky derived from the ECMWF EPS median low-to-medium cloud field, together with bar plots showing the marginal distributions of the datasets. The verification period is between 1 January and 19 August Table 2. The proportion correct by category calculated for the forecast state of sky derived from the ECMWF EPS median low-to-medium cloud amount over 15 sites around Hong Kong against the weather symbols issued by the HKO forecasting office for the whole territory between 1 January and 19 August ECMWF forecast lead time T + 1 h to T + 24 h T + 25 h to T + 48 h T + 49 h to T + 72 h T + 73 h to T + 96 h T + 97 h to T h T h to T h T h to T h T h to T h T h to T h T h to T h 2017 Royal Meteorological Society Sunny Sunny periods Sunny intervals Cloudy Meteorol. Appl. 24: (2017)

10 Verification and communication of the low-to-medium cloud forecasts 475 Figure 10. The Gerrity skill score (Equation (A.10)) by the forecast days along the x-axis for the weather symbols derived from the ECMWF EPS median low-to-medium cloud forecasts between 1 January and 19 August The perfect score is 1, and the dotted line indicates the average score over the entire forecast period of 10 days. increasing focus on location-specific forecasts. As the channel through which weather forecasts are communicated gradually shifts from mass media to smartphones, forecasts are required to be at an increasingly high resolution to provide information that is most relevant to each individual user. 4. Conclusion A major emphasis of this study is on the verification metrics. Because these metrics all have different properties and each of them scrutinizes certain aspects of the forecasts, it is always advisable to consider a number of them. Those covered in this paper are recommended by the World Meteorological Organization (WMO). As the literature suggests that cloud amounts are rarely considered okta by okta, but are usually taken as the amounts exceeding a specific threshold, the metrics used in Section 2 for evaluating the model cloud cover are all designed for two-by-two contingency tables. The Gerrity skill score (GSS), on the other hand, is applicable to multi-category contingency tables and is therefore used to assess the weather symbol forecasts in Section 3. Limited by the availability of observations, the low-to-medium cloud verification is done at a single point and the European Centre for Medium-Range Weather Forecasts Ensemble Prediction System (ECMWF EPS) median is shown in Section 2 to match the surface synoptic observations (SYNOP) at the Hong Kong International Airport (HKIA) best over the range 0 to 4 oktas. A frequency bias of around 1 indicates that the forecast for clouds exceeding a threshold is made almost exactly as often as it is observed, whereas the relatively high log-odds ratios suggest that the odds of forecasting a hit are greater than the odds of a false alarm by a significant amount. The Symmetric Extreme Dependency Score (SEDS), which takes the marginal distribution of the observations into account and is difficult to hedge, shows that the EPS median has the most consistent performance across the forecast lead times. In Section 3, the forecast field is interpolated onto 15 sites around Hong Kong to derive weather symbols, and the results are compared against the actual weather symbols issued by the duty forecasters for the whole of Hong Kong. The verification results are on par with those from similar studies on sky cover, and at the same time, the output seems to be able to portray a regional variation in the amount of sunshine received over the city. Mapping the ECMWF EPS median low-to-medium cloud amounts to the four state-of-sky categories using the exceedence thresholds of 0, 1, 3 and 4 oktas therefore appears to match with the human perception of the weather conditions to a Wetland Park Cape D Aguilar Figure 11. The rainfall distribution map shows that rain fell in the northern part of Hong Kong in the hour preceding 1500 HKT on 11 August 2015, suggesting that it was cloudier in the north of Hong Kong. These observations are used to assess whether regional variation in cloud cover can be represented by the forecasts.

11 476 J. Tam and W.-K. Wong (a) the bottom-right representing misses, i.e. forecasts failing to predict that the cloud amounts would exceed the threshold; the top-left representing false alarms, i.e. forecasts incorrectly predicting that the cloud amounts would exceed the threshold; and the bottom-left representing correct negatives, i.e. forecasts correctly predicting the cloud amounts to stay under the threshold. A.1. Frequency bias To quantitatively measure the overall relative frequencies of how often an event is forecast against how often it is observed, there is: Hits + False Alarms Frequency Bias = (A.1) Hits + Misses (b) Ranging from zero to infinity, the perfect score is exactly 1, below (above) which indicates a tendency to underforecast (overforecast). A.2. Log-odds ratio Given that an event has occurred, the ratio of the probability of forecasting it to the probability of not forecasting it is given by: Odds of a hit = Hits Misses (A.2) Conversely, given that it has not occurred, the ratio of the probability of forecasting it to happen to the probability of forecasting it to not happen is given by: Odds of a false alarm = Figure 12. Weather symbols derived for HKT on 11 August 2015 using model cloud cover forecasts from the 1200 UTC run on 10 August The symbols with thin solid borders correspond to cloudy, thick solid borders correspond to sunny intervals, and those with dotted borders correspond to sunny periods. Map (a) is primarily based on the total cloud cover forecasts which shows cloudy conditions over the whole of Hong Kong, whereas Map (b) is derived using the ECMWF EPS median low-to-medium cloud cover which more accurately depicts the regional difference in the cloud cover between the north and the south observed in Figure 10. False Alarms Correct Negatives (A.3) These can be put in a ratio for comparing negative with positive associations between the categorical forecasts and observations according to Stephenson (2000). Taking the natural logarithm: Log-odds ratio = ln (Odds ratio) ) ( Odds of a hit = ln Odds of a false alarm (A.4) which runs from minus infinity to plus infinity, with zero representing that skill is due to pure chance. A.3. Symmetric Extreme Dependency Score (SEDS) certain extent. The results from this study is hoped to contribute to the future development of the HKO s automated forecast products. Hogan et al. (2009) introduce the SEDS specifically for the verification of biased variables: Acknowledgements where H = The authors would like to thank the two anonymous reviewers for their helpful comments leading to the improvement of this paper. base rate for forecast events, p = for observed events. The perfect score is 1, with zero misses or false alarms. Though Ferro and Stephenson (2010) demonstrate that SEDS is still dependent on the frequency of occurrence, Mittermaier (2012) considers it to be a suitable metric for cloud verification because it is equitable for large samples and less susceptible to hedging. Nevertheless, her paper finds that overforecasting could improve the score for total cloud cover (possibly because its observed distribution has one of the peaks at the overcast end). However, Appendix: Contingency table verification metrics Given a multi-category table of the observed and forecast cloud amounts, a two-by-two contingency table can easily be constructed by slicing it into four blocks with: the top-right representing hits, i.e. forecasts correctly predicting the cloud amounts to exceed certain oktas; 2017 Royal Meteorological Society SEDS = Hits Hits+Misses ln q + ln p 1 ln H + ln p (A.5) Hits+False Alarms is Sample Size Hits+Misses is the base rate Sample Size is the hit rate, q = the Meteorol. Appl. 24: (2017)

12 Verification and communication of the low-to-medium cloud forecasts 477 the same conclusion cannot be drawn here when the bimodal distribution is of a different shape. For low-to-medium cloud cover, the base rate for observed events p generally decreases as the exceedence threshold increases, and it does so drastically towards the overcast end. As can be seen from Figure 3, the integral of the observation distribution to the right of the exceedence threshold becomes small very quickly. Similarly, the base rate of forecast q shows a declining trend, but the rate of decrease depends on the marginal distribution of the forecasts. Given that: Frequency Bias = q (A.6) p Figure 4 implies that q ECMWF p for the ECMWF models (Figures 4(a), (d) and (e)) but q JMA > p for JMA (Figure 4(b)) over the lower thresholds as it tends to overforecast. As p is the same across the models, the base rate for forecast events is higher for JMA than for the ECMWF such that q JMA > q ECMWF. Note that all these rates (H, q and p) are less than 1, so their logarithms are negative and the minus signs simply cancel each other out in the fraction in Equation (A.5). Comparing only the absolute values of these logarithms (which increase as the arguments decrease such that ln q JMA < lnq ECMWF ), the numerator of the first term of Equation (A.5) should be smaller for JMA than ECMWF, which seems to be in line with the resulting scores shown in Figure 6, suggesting that overforecasting does not necessarily work as a hedging strategy. Furthermore, the fact that SEDS is transpose symmetric, as demonstrated by Stephenson (2000) and Hogan et al. (2009), should mean that underforecasting does not work either, which is illustrated by NCEP. In actual fact, however, all these also depend on the hit rate H, as the score essentially compares the absolute values of ln q in the numerator and of ln H in the denominator. This only implies that the difference in lnh between JMA and ECMWF is at least as large as that of lnq. To score well, a model must have a decent hit rate but not by simply raising the base rate for forecast events. A.4. Gerrity skill score For a K-category contingency table constructed with N samples, the frequencies of the observations O i (i = 1,,K)are given by: p i = N ( ) O i (A.7) N where N(O i ) denotes the number of observations in category i. The scoring matrix is defined, for the diagonal, as: i ( s ii = 1 i 1 ) 1 K 1 1 r=1 + a K 1 a r where a i = r i r=1 r=i and for the off-diagonal as: s ij = 1 K 1 The Gerrity skill score: GSS = 1 N ( j 1 r=1 K i=1 ) K 1 1 (j i) + a a r r r=j K N ( ) F i, O j sij j=1 p r r=1 p r (A.8) (A.9) (A.10) where N(F i,o j ) denotes the number of forecasts in category i that correspond to observations in category j, thus ranges from 1to 1 with 1 being the perfect score and 0 indicating no skill. References Ferro CAT, Stephenson DB Extremal dependence indices: improved verification measures for deterministic forecasts of rare binary events. Weather Forecasting 26: Gerrity JP Jr A note on Gandin and Murphy s equitable skill score. Mon. Weather Rev. 120: Hogan RJ, O Connor EJ, Illingworth AJ Verification of cloudfraction forecasts. Q. J. R. Meteorol. Soc. 135: Jolliffe IT, Stephenson DB (eds) Forecast Verification: A Practitioner s Guide in Atmospheric Science, John Wiley and Sons. Kluepfel C Sky cover forecast verification in the national weather service. Developmental Testbed Center Verification Workshop, 26 August 2009, Boulder, CO. Leung CYY, Tam JYT Development of an automatic integrated weather forecasting portal for the public. The 30th Guangdong-Hong Kong-Macao Seminar on Meteorological Science and Technology, April 2016, Guangzhou, China. Mittermaier MP A critical assessment of surface cloud observations and their use for verifying cloud forecasts. Q. J. R. Meteorol. Soc. 138: Stephenson DB Use of the odds ratio for diagnosing forecast skill. Weather Forecasting 15: World Meteorological Organization Recommended methods for evaluating cloud and related parameters (WWRP ) _1_web.pdf (accessed 1 August 2015).