Post-processing rainfall forecasts from a numerical weather prediction model in Australia

Transcription

1 WATER FOR A HEALTHY COUNTRY FLAGSHIP Post-processing rainfall forecasts from a numerical weather prediction model in Australia Durga Lal Shrestha, David Robertson, James C. Bennett, QJ Wang November 23

2 Water for a Healthy Country Flagship Report series ISSN: X Australia is founding its future on science and innovation. Its national science agency, CSIRO, is a powerhouse of ideas, technologies and skills. CSIRO initiated the National Research Flagships to address Australia s major research challenges and opportunities. They apply large scale, long term, multidisciplinary science and aim for widespread adoption of solutions. The Flagship Collaboration Fund supports the best and brightest researchers to address these complex challenges through partnerships between CSIRO, universities, research agencies and industry. The Water for a Healthy Country Flagship aims to provide Australia with solutions for water resource management, creating economic gains of $3 billion per annum by 23, while protecting or restoring our major water ecosystems. The work contained in this report is part of the water information research and development alliance (WIRADA), a collaboration between CSIRO and the Bureau of Meteorology. For more information about Water for a Healthy Country Flagship or the National Research Flagship Initiative visit Citation Shrestha DL, Robertson DE, Bennett JC, Wang QJ (23) Post-processing rainfall forecasts from a numerical weather prediction model in Australia. CSIRO Water for a Healthy Country Flagship, Australia. Copyright and disclaimer 23 CSIRO To the extent permitted by law, all rights are reserved and no part of this publication covered by copyright may be reproduced or copied in any form or by any means except with the written permission of CSIRO. Important disclaimer CSIRO advises that the information contained in this publication comprises general statements based on scientific research. The reader is advised and needs to be aware that such information may be incomplete or unable to be used in any specific situation. No reliance or actions must therefore be made on that information without seeking prior expert professional, scientific and technical advice. To the extent permitted by law, CSIRO (including its employees and consultants) excludes all liability to any person for any consequences, including but not limited to all losses, damages, costs, expenses and any other compensation, arising directly or indirectly from using this publication (in part or in whole) and any information or material contained in it.

3 Contents Acknowledgments... v Executive summary... vii Introduction... 2 Method Bayesian joint probability (BJP) modelling approach Schaake shuffle method Study Area and Data Study area Observation data Numerical weather prediction model output Ensemble Rainfall Forecast Verification Ensemble rainfall forecasts Verification method... 5 Results and Discussion Spatial variation of forecast performance Variation of forecast performance with lead time Variation of forecast performance with post-processing time step Summary and Recommendations References Appendix A... 3 Post processing NWP rainfall forecasts i

4 Figures Figure. Framework for ensemble flood and short-term streamflow forecasting system (source: Wang and Robertson, 2a)... Figure 2. Bayesian joint probability modelling approach used in this study... 4 Figure 2.2 Illustration of the Schaake shuffle for a single site and multiple lead times... 6 Figure 3. Location of catchments used in this study. Descriptions of catchments are given in Table Figure 3.2 Variability of observed rainfalls and raw NWP rainfall forecasts in the South Esk catchment. The rainfall is average 3 hour totals during the period from August 2 to April 22 for the period 2: UTC to : UTC (3 hour lead time). The gray-dashed lines in the right show the ACCESS-G model grid... 9 Figure 4. Rainfall forecasts over the South Esk catchment on two forecast days. (a) 2//28 and (b) 2//7... Figure 4.2 Illustration of leave-one-out-month cross-validation method. Here n is the number of forecast months... Figure 4.3 Computation of continuous ranked probability score... 2 Figure 5. Diurnal cycle of the observed, raw NWP and post-processed rainfall forecasts (catchment average) in catchments. The time step of the rainfall is 3 hour and period is from August 2 to April Figure 5.2 Biases of post-processed rainfall forecasts across the catchments. The skill is measured by percentage bias of catchment average rainfall. Blue shows skill of post-processed NWP model forecasts; red shows skill of raw NWP model forecasts... 5 Figure 5.3 Bias in the raw and post-processed rainfall forecasts across all subareas. (a) Bias as a function of lead time for subareas having 5 th and 95 th percentile bias, (b) range of absolute bias as a function of fraction of subareas... 5 Figure 5.4 Spatial variation of 24 hour average rainfall values in the South Esk catchment. The dashedgrey lines in the centre (vertical) panels show the NWP model grid. Top row is for -day lead time, middle row 3-day lead time and bottom row 9-day lead time... 6 Figure 5.5 Bias and CRPS skill of the raw NWP forecasts and post-processed forecasts for the South Esk catchment... 6 Figure 5.6 Delineation for Cotter catchment and ACCESS-G model grids (dashed lines)... 7 Figure 5.7 Bias (%) for individual forecast periods as a function of lead time for 3 hour rainfall total for all 9 subareas in the Cotter. Each panel is for each subarea location given in Figure Figure 5.8 Bias for catchment average rainfall for the Cotter. (a) Individual period, and (b) cumulative totals... 8 Figure 5.9 Variation in CRPS of the post-processed rainfall forecasts for individual periods as a function of lead time in subareas in the Cotter catchment. Subarea locations are shown in Figure Figure 5. CRPS for catchment average rainfall for the Cotter catchment. (a) Individual periods, and (b) cumulative totals... 9 Figure 5. Reliability diagrams for the post-processed ensemble rainfall forecasts for individual forecast periods as a function of lead time for one subarea in the Cotter catchment. (a) probability of rainfall, (b) 25 th percentile of rainfall amounts greater than.2 mm, (c) 5 th percentile of rainfall amounts greater than.2 mm, and (d) 75 th percentile of rainfall amounts greater than.2 mm... 2 Figure 5.2 Same as Figure 5. but for catchment average 24 hour rainfall totals... 2 ii Post processing NWP rainfall forecasts

5 Figure 5.3 ROC curves for individual forecast periods as a function of lead time for one of subarea in the Cotter catchment. (a) Probability of rainfall, (b) 25 th percentile of rainfall amounts greater than.2 mm, (c) 5 th percentile of rainfall amounts greater than.2 mm, and (d) 75 th percentile of rainfall amounts greater than.2 mm Figure 5.4 Same as Figure 5.3, but for cumulative totals for catchment average rainfall Figure 5.5 Area under ROC curves of Figure 5.3 and Figure 5.4 as a function of lead time. (a) Individual forecast periods for one of the subarea in the Cotter catchment, and (b) cumulative rainfall totals for catchment average Figure 5.6 Catchment average bias (%) as a function of lead time for different post-processing time steps in the Cotter catchment. (a) Individual forecast period, and (b) cumulative totals Figure 5.7 Same as Figure 5.6 but for the CRPS Figure 5.8 Reliability diagram for catchment average rainfall for day (4 panels in top row), day 2 (second row), day 4 (third row), and day 6 (bottom row). 4 Panels in the first column are for probability of rainfall, second column 25 th percentile of rainfall amounts greater than.2 mm, third column 5 th percentile of rainfall amounts greater than.2mm, and last column 75 th percentile of rainfall amounts greater than.2 mm Figure 5.9 Same as Figure 5.8 but for ROC curves for cumulative totals Figure 5.2 Area under ROC curves of Figure Post processing NWP rainfall forecasts iii

6 Tables Table 3. Characteristics of catchments used in this study... 7 Table 4. Time steps and number of BJP models... iv Post processing NWP rainfall forecasts

7 Acknowledgments This work is part of the water information research and development alliance between CSIRO s Water for a Healthy Country Flagship and the Bureau of Meteorology. Post processing NWP rainfall forecasts v

8 vi Post processing NWP rainfall forecasts

9 Executive summary Bias free and reliable ensemble rainfall forecasts are required to produce reliable and skilful ensemble streamflow forecasts. The rainfall forecasts that are publicly available from Australian NWP models are deterministic and often contain systematic errors. Therefore, it is necessary to remove the systematic biases and reliably quantify uncertainty in rainfall forecasts before they can be used for streamflow forecasting. This report presents the further development and application of a rainfall post-processing method to remove forecast bias and reliably quantify forecast uncertainty. We post-process rainfall forecasts from a global Australian NWP model by combining a simplified version of the Bayesian joint probability (BJP) modelling approach and the Schaake shuffle. The BJP modelling approach constructs statistical relationships between NWP forecasts and observed rainfalls. It corrects biases in the NWP model forecasts and generates an ensemble of possible forecasts that reflects the uncertainty in the NWP forecast. The BJP modelling approach is applied to individual locations and individual forecast periods to produce probabilistic rainfall forecasts. Ensemble forecasts with appropriate spatial and temporal correlations are then produced by linking samples from the forecast probability distributions using the Schaake shuffle. The post-processing method is evaluated for a range of small-medium sized Australian catchments that cover a wide range of climatic conditions and hydrological characteristics. We show that the postprocessing method significantly reduces the forecast bias and error, produces forecasts that successfully discriminate between events and non-events for both small and large precipitation occurrences, and reliably quantifies the forecast uncertainty. Post processing NWP rainfall forecasts vii

10

11 . Introduction Previous WIRADA research (Wang and Robertson, 2a) has developed a framework for an ensemble flood and short-term streamflow forecasting system (see Figure.) for Australia. Within this framework (Robertson et al., 23) have developed a post-processing method for rainfall forecasts from the ACCESS numerical weather prediction (NWP) models. The method uses a simplified version of the Bayesian joint probability modelling (BJP) approach (Wang et al., 29; Wang and Robertson, 2b) to produce forecast probability distributions for individual locations and individual forecast periods. Ensemble forecasts with appropriate temporal correlations are then produced by linking samples from the forecast probability distributions using the Schaake Shuffle method (Clark et al., 24). The BJP modelling approach provides a highly flexible probability model with relatively few parameters, through its use of a parametric transformation for data normalisation and variance stabilisation, and Bayesian parameter inference methods. Robertson et al. (23) applied this post-processing method to rainfall forecasts from APS (Australian Parallel Suite version ) version of the Australian Community Climate and Earth System Simulated (ACCESS) NWP model at rain gauge locations in the Ovens catchment in southeast Australia. They demonstrated that ensemble forecasts produced using this method are more skilful and reliable than the raw NWP forecasts, both for individual forecast periods and for cumulative totals for multiple lead times. Figure. Framework for ensemble flood and short-term streamflow forecasting system (source: Wang and Robertson, 2a) The results from the Ovens catchment provided a proof-of-concept for a single catchment in a single climate zone. Further evaluation of the proposed method across a wide range of climate zones is required to provide confidence in the robustness of the method. In addition, work in the Ovens catchment applied the rainfall post-processing method at individual locations (rain gauge stations). The problem with applying the post-processing method separately for individual stations is that it does not preserve the spatial covaribility between neighbouring stations. Preserving the space-time variability in rainfall is critical for hydrological applications (Clark et al., 24). Furthermore, hydrological applications require postprocessing of mean areal rainfall over the catchments rather than point rainfall at rain gauge locations. Recently, the Bureau of Meteorology upgraded the ACCESS suite of NWP models from APS to APS (Australian Parallel Suite version ). The upgrade of the ACCESSS suite of NWP models has resulted in increases in the skill of rainfall forecasts (Bureau of Meteorology, 22). However, the rainfall forecasts Post processing NWP rainfall forecasts

12 from ACCESS models still require post-processing to produce reliable and bias free ensemble rainfall forecasts as raw output that is publicly available from Australian NWP models is deterministic and often contains systematic errors (Shrestha et al., 23). This report presents the further development and application of the rainfall post-processing method (Robertson et al., 23) to remove forecast bias and reliably quantify forecast uncertainty. This study assesses the performance of the post-processing method in ten catchments in which the Bureau of Meteorology is seeking to demonstrate an experimental short-term forecasting service. These ten catchments cover a wide range of sizes, climatic conditions and hydrological characteristics. In a comparison with the previous study of Robertson et al. (23), the main contributions of this study are to (i) apply the method to more catchments that cover a wide range of climatic conditions and hydrological characteristics, (ii) demonstrate the method to post-process mean areal rainfall over the catchment, (iii) extend the method to multiple sites (locations) and multiple lead times, (iv) evaluate the consequence of applying the method at different time steps, and (v) apply the method to rainfall forecasts from an upgraded version of NWP model extending to days. 2 Post processing NWP rainfall forecasts

13 2. Method Post-processing of a deterministic rainfall forecast from NWP model consists of two parts. First, we apply a simplified version of the Bayesian joint probability (BJP) modelling approach (Robertson et al., 23) to generate bias free and reliable probabilistic forecasts from a deterministic NWP rainfall forecast. Next, the Schaake shuffle (Clark et al., 24) is used to generate ensemble forecasts by linking samples from probability distributions at each lead time and each location. This method to generate ensemble rainfall forecasts from a deterministic rainfall forecast is hereafter referred to as post-processing. 2.. Bayesian joint probability (BJP) modelling approach The BJP modelling approach (Wang et al., 29; Wang and Robertson, 2b) has been successfully applied in seasonal streamflow forecasting in Australia. Recently, Robertson et al. (23) simplified the BJP modelling approach to post-process raw NWP rainfall forecasts for individual lead times and individual rainfall gauging locations. Detailed descriptions of the BJP modelling approach are provided in Wang et al. (29) and Wang and Robertson (2b). Here we present a brief overview of the simplified BJP modelling approach adapted from Robertson et al. (23) to highlight the difference from the original implementation. The BJP modelling approach assumes that the model predictor x (in this case NWP rainfall forecast) and predict and y (in this case observed rainfall) follow a joint bivariate normal distribution in a transformed space (see Figure 2.). For this study we apply the log-sinh transformation (Wang et al., 22) to normalise the variables and stabilise their variances rather than the Yeo-Johnson transformation (Yeo and Johnson, 2) used in the original formulation of the BJP modelling approach. The log-sinh transformation is mathematically defined as xˆ = ln(sinh( α x + β xx)) β x yˆ = ln(sinh( α y + β y y)) β y (2.) where xˆ and ŷ are transformed variables of x and y, respectively, α x and β x are parameters of the transformation for x, α y and β y are parameters of the transformation for y. The transformed variables xˆ and ŷ are assumed to follow a bivariate normal distribution p ( xˆ, yˆ) ~ N( μ, Σ) (2.2) where µ xˆ μ = µ yˆ 2 σ xˆ Σ = ρ xy ˆˆσ xˆ σ yˆ ρ xy ˆˆσ xˆ σ yˆ 2 σ yˆ (2.3) Post processing NWP rainfall forecasts 3

14 The model parameters θ = [ α x β x µ xˆ σ xˆ α y β y µ yˆ σ yˆ ρ xˆ yˆ ] describe two transformation parameters α x and β x, mean µ xˆ and standard deviation σ xˆ for xˆ and transformation parameters and standard deviation σ ŷ for ŷ, and a correlation coefficient reparameterised to ease parameter inference. ρ xˆ ˆy α y and. All model parameters are β y, mean The original formulation of the BJP modelling approach for seasonal forecasting infers model parameters and their uncertainties using Markov chain Monte Carlo methods to sample from the posterior parameter distribution p( θ D), where D = { x t, y t }, x t and y t are the observed predictor and predictand data for event t, t =,2,..., n. Formulation of the posterior parameter distribution is detailed in Robertson et al. (23) An important feature of the posterior parameter distribution is the treatment of (near) zero rainfall values in the likelihood function. Where either forecast or observed rainfall can be considered to be zero, they are treated as censored data (i.e. having values less than or equal to (near) zero) in the likelihood function, which enables the use of the continuous bivariate normal distribution for a problem which is otherwise solved using a mixed discrete-continuous probability distribution. Different censoring thresholds are used for the predictor and predictand to reflect the differing precisions of available data. The censoring threshold for observed rainfall is.2 mm which is the minimum measurable rainfall amount for the majority of operational tipping bucket rain gauges. The censoring threshold for NWP rainfall forecasts is set to. mm. A lower threshold is used for NWP rainfall forecasts because they represent average rainfall over a large spatial extent. Therefore rainfall forecasts lower than the minimum measurable amount are likely to result in measurable rainfall at some specific locations. A non-zero threshold was imposed in the NWP rainfall forecasts because the data contained some very small values that were found to be artefacts of numerical processing methods. µ ŷ Predictor NWP rainfall forecast pp(xx yy )~NN(μμ, Σ) Predictand Observed rainfall Conditional dist. Posterior parameter dist. Figure 2. Bayesian joint probability modelling approach used in this study For operational streamflow forecasting applications considerably more data are available to infer model parameters than for seasonal forecasting applications. This will reduce parameter uncertainty, and, therefore, in this study we obtain a single set of model parameters that gives the maximum a posteriori (MAP) solution. We obtain the MAP solution for model parameters θ of bivariate normal distribution by maximising the posterior density of the model parameters p( θ D) using a stepwise approach. First we infer transformation, mean and standard deviation parameters for the marginal distribution of each predictor and predictand separately using the shuffled complex evolution algorithm ((Duan et al.,, 994). We then use these parameter values to infer the remaining correlation coefficient parameter from the bivariate normal distribution. Once the parameters θ are inferred from historical data, a forecasting problem is stated as finding the predictand ŷ given the predictor xˆ, i.e. the conditional bivariate normal distribution f ( yˆ xˆ = xˆ t+ ) which is given by 4 Post processing NWP rainfall forecasts

15 where 2 f ( yˆ xˆ = xˆ t ) ~ N( µ, σ ) (2.4) + yˆ xˆ yˆ xˆ µ σ yˆ xˆ 2 yˆ xˆ ( xˆ µ xˆ ) = µ yˆ + ρ xy ˆˆσ yˆ σ 2 yˆ = σ ( ρ 2 xy ˆˆ ) xˆ (2.5) An ensemble of forecasts is generated using the conditional distribution of equation 2.4 to represent the forecast probability distribution. An inverse of equation 2. is then applied to give the forecast values in untransformed space. In forecast mode, a data augmentation procedure is used for censored predictor data (Wang and Robertson, 2b). Where a predictor value is equal to the censoring threshold, data augmentation is used to generate a value less than the censoring threshold and the joint distribution is conditioned on the augmented predictor value Schaake shuffle method Probability forecasts produced by applying a BJP modelling approach separately to each forecast lead time and each location will not contain the appropriate spatial and temporal correlation structures necessary for streamflow forecasting (Clark et al., 24; Schaake et al., 27; Wu et al., 2). We apply the Schaake shuffle (Clark et al., 24) to generate ensembles with appropriate spatial and temporal correlations by linking samples from the probabilistic forecasts according to the ranks of historical observations. The Schaake shuffle method uses historically observed rainfall time series of the same duration as the rainfall forecasts as the empirical basis for the spatial and temporal correlation structures. The number of observed time series sampled from the historical record is equal to the number of forecast ensemble members. These time series represent the set of possible sequences for an ensemble forecast. Observation Ensemble forecasts Lead time Randomly select historical observations and rank them for i=:nleads % Sort the forecasts fcast(:,i) = sort(fcast(:,i)) % Sort the observations [~, rank] = sort(obs(:,i)) % Shuffle the forecasts according % to rank of the observations shuffled(rank,i) = fcast(:,i) end Shuffle forecasts according to rank of the observations Lead time Post processing NWP rainfall forecasts 5

16 Figure 2.2 Illustration of the Schaake shuffle for a single site and multiple lead times Figure 2.2 shows the Schaake shuffle method for single site. In this example, the ensemble size is 4 and the forecast duration is 3 time steps. For each lead time, the forecast ensemble members are ranked by magnitude. This ranking is repeated for the set of historical time series. To create a temporally coherent forecast, each ranked forecast is serially matched to one historical time series. For example, the green observed time series in Figure 2.2 has a rainfall that is ranked 2nd highest at lead time one, highest at lead time two, and 3rd highest at lead time 3. A forecast time series is assembled from this sequence: the 2nd highest forecast ensemble member at lead one is matched to the highest forecast ensemble member at lead time 2 and the 3rd highest forecast ensemble member at lead time 3 to generate the green forecast time series in Figure 2.2. This process is repeated for the entire ensemble. The post-processing applies a slightly more complex Schaake shuffle that samples historical time series from time and space to impart historical spatial and temporal patterns to post-processed rainfall forecasts. 6 Post processing NWP rainfall forecasts

17 3. Study Area and Data 3.. Study area 2 E 3 E 4 E 5 E S ORD 2 S 2 STN 3 S 3 MCL GBR WLL CTT MRR WTT 5, 2, km 4 S 4 ONK STE 2 E 3 E 4 E 5 E Figure 3. Location of catchments used in this study. Descriptions of catchments are given in Table 3. Post-processing has been applied to NWP forecasts for ten catchments where the Bureau of Meteorology is seeking to demonstrate an experimental real time short-term streamflow forecasting service. These ten catchments cover a wide range of sizes, climatic conditions and hydrological characteristics. Figure 3. shows the location of the catchments and Table 3. shows characteristics of the catchments. Table 3. Characteristics of catchments used in this study CATCHMENT ID CATCHMENT NAME DESCRIPTION NO. OF SUBAREAS CATCHMENT AREA (KM 2 ) NO. OF NWP GRID CELL STN Stanley Subtropical MCL Macleay Subtropical WLL Wollondilly Temperate CTT Cotter Alpine GBR Goobarragandra Alpine MRR Upper Murray Alpine STE South Esk Alpine-temperate WTT Watts Temperate ONK Onkaparinga Dry-temperate ORD ORD Subtropical-monsoon ANNUAL RAINFALL (MM) Post processing NWP rainfall forecasts 7

18 3.2. Observation data Each catchment is divided into a number of subareas (see Table 3.) which forms the basis of the semidistributed hydrological model used to produce streamflow forecasts Bennett et al. (23). In each subarea, the hydrological model uses the mean areal rainfall amounts estimated to fall within the subarea (hereafter referred to as subarea average rainfall). Subarea average rainfall is derived by inverse distance weighting of rainfall from the nearby rain gauge stations. Hourly rainfall data of subarea average rainfall were quality controlled and supplied by the Bureau of Meteorology. No additional verification of these data has been performed. A previous study (Robertson et al., 23) argued that the post-processing method on rain gauge data from the Ovens catchment limits the influence of artefacts resulting from missing data that are introduced by the interpolation techniques (inverse distance weighting) currently in operational use. Since subarea average rainfall is input to the hydrological model for real time streamflow forecasting, this study uses subarea rainfall rather than rain gauge observation. This allows us to directly evaluate the benefit of the post-processing method for streamflow forecasting. Furthermore, for hydrological applications the localisation of rainfall is important at the catchment scale, so it is useful to evaluate rainfall forecasts on catchment averages (see, e.g., Oberto et al., 26; Rossa et al., 28) Numerical weather prediction model output The rainfall forecasts used here are generated with the ACCESS NWP model. The ACCESS model (Bureau of Meteorology, 2) has been in operational use by the Bureau of Meteorology since August 2. The ACCESS NWP model system is based on the UK Met Office s Unified Model/Variational Assimilation (UM/VAR) system. Several variants of ACCESS are run operationally, extending from a course resolution global model down to the high resolution city-based models. Collectively, these ACCESS variants form the Australian Parallel Suite (APS). This study uses the most recent version of the global model (ACCESS-G) from the ACCESS APS system (Bureau of Meteorology, 22). The most significant recent changes for this global model include: an increase in model horizontal resolution from approximately 8 km to 4 km, an increase in the number of atmospheric levels from 5 to 7, assimilation of additional new satellite observation types and use of more recent versions of the UM/VAR software which includes improved physical parameterisation. Key features of various components and physical parameterisations are given in above two references. ACCESS-G data assimilation is performed 4 times per day at the nominal assimilation base times of :, 6:, 2: and 8: UTC. However, full model forecasts of 24 hour duration are only run at : and 2: UTC. Because of the increased resolution of APS ACCESS-G the model integration takes significantly longer (about 7 hours) to complete compared to the previous APS ACCESS-G. Although the internal model time step of ACCESS-G is 2 minutes, the rainfall forecasts available for this study are at 3 hourly intervals. In operational conditions, streamflow forecasts are assumed to be issued once a day at 23: UTC (9 am local time). For this study we use the most recently issued NWP forecast (2: UTC) that is available (around 9: UTC) when the streamflow forecasts are made. Thus, the first nine hours of NWP rainfall forecasts are neglected and post-processing is applied to forecasts for lead times from (2: UTC) to 24 hours. NWP forecasts for the first few hours are generally regarded as not reliable because of the socalled spin-up time (Kasahara et al., 992; Shrestha et. al, 23). Given that we do not use the first nine hours of forecast our results are considered to be free from model spinup effects. In the rest of this report, lead time 3 hour means for a forecast period from 2: UTC to : UTC. 8 Post processing NWP rainfall forecasts

19 -4.3 Observation Raw NWP Rainfall (mm/3h) Figure 3.2 Variability of observed rainfalls and raw NWP rainfall forecasts in the South Esk catchment. The rainfall is average 3 hour totals during the period from August 2 to April 22 for the period 2: UTC to : UTC (3 hour lead time). The gray-dashed lines in the right show the ACCESS-G model grid Hindcasts for the ACCESS suite of models are not available. An archive of real time forecasts for a 2 month period (approximately 64 forecasts) extending from August 2 to April 22 is used in this study. While a longer record is desirable to calibrate a post-processing method, it is unlikely (due to the high cost of generating hindcasts) to be available for operational forecasting applications in Australia. Forecast rainfall for each subarea was generated by taking the area weighted average of NWP forecast rainfall for all grid cells intersecting the subarea (Shrestha et al., 23). Since the number of grid cells (see Table 3.) covering the whole catchment is typically less than the number of subareas, some of the subareas have the same rainfall forecast value, and thus the raw NWP forecasts are unlikely to capture gradients of rainfall across the catchment. As an example of how NWP forecasts inadequately describe the spatial gradients in rainfall, we show the delineation of the South Esk catchment and ACCESS-G model grids (Figure 3.2). The catchment is delineated into 42 subareas for hydrological modelling. The average 3 hour observed rainfall at a 3 hour lead time (2 UTC to UTC) is shown in left figure and NWP rainfall forecast in right figure. The catchment has elevation changes of >5 m to <2 m in the space of 3 km, and this is reflected in an extremely steep rainfall gradient (>2 mm annual rainfall to <6 mm). Figure 3.2 clearly shows it is not possible to simulate such high spatial variability with the 4 km ACCESS-G grid cells. Generally, atmospheric numerical models under-represent the small-scale spatial variability of rainfall because their finite-difference computational fluid dynamical schemes contain smoothing in the form of both implicit and explicit numerical diffusion, as well as physically based subgrid-scale turbulent mixing. A convenient rule of thumb is that NWP models do not accurately resolve features on a scale of less than 5 times the grid length of the model (Harris et al., 2). Post processing NWP rainfall forecasts 9

20 4. Ensemble Rainfall Forecast Verification 4.. Ensemble rainfall forecasts The post-processing method was applied for three different time steps 3 hour, 6 hour and 24 hour. The 6 hour and 24 hour rainfall totals are obtained by accumulating 3 hour ACCESS-G rainfall forecasts over 6 hour and 24 hour periods, respectively. For each time step, separate models were established to postprocess NWP rainfall forecasts for each lead time and each location. Table 4. presents the number of the BJP models developed for each time step and each location. The number of the BJP models developed for each catchment is the number of the subareas multiplied by the number of forecast lead times. Thus, for the South Esk catchment 3277 BJP models were developed for 3 hour time step to produce forecasts of 9.5 days. It takes about two minutes to build each BJP model, but it takes only a fraction of seconds to generate the forecasts for the whole catchment. Thus, computational time will not be an issue to produce postprocessed forecasts in a real time. Table 4. Time steps and number of BJP models TIME STEP (HOURS) FORECAST PERIOD (HOURS) NO. LEAD TIME NO. BJP MODEL 3 * * Forecast period corresponds to a period 2:-22: UTC Figure 4. shows an example of ensemble rainfall forecasts produced by post-processing NWP model forecasts on 28 October 2 and 7 November 2 for lead times out to about 9.5 days in the South Esk catchment. The observed rainfall and raw NWP rainfall forecasts are also shown as a comparison to the ensemble rainfall forecasts. For these two selected events, the NWP model forecasts are higher than observed rainfall, while the mean of the ensemble forecasts is very close to the observation. In Section 4.2 we present the verification methods we use to rigorously assess the quality of the rainfall forecasts after post-processing. 3 (a) 25 Observation Raw numerical weather prediction Ensemble mean Ensemble member 3 (b) Rainfall (mm/ 3h) 5 Rainfall (mm/ 3h) Lead time (hours) Lead time (hours) Figure 4. Rainfall forecasts over the South Esk catchment on two forecast days. (a) 2//28 and (b) 2//7 Post processing NWP rainfall forecasts

21 4.2. Verification method The performance of the post-processing method is evaluated using a leave-one-month-out cross-validation procedure. The procedure is implemented by inferring parameters of the simplified BJP model using all available data with the exception of one month. Rainfall for all the days in the left-out month are then forecast and compared to corresponding observations. This procedure is repeated for all months in NWP evaluation period so that the forecasts are produced for all available NWP model forecasts. This procedure is used to ensure that the forecasts are verified independent of model fitting and a similar number of data are used to fit the model as will be available operationally. The results shown in this study are all from cross-validation data. Figure 4.2 presents the leave-one-month-out cross-validation method. Training data set st cycle A i i+ n. Verification sample.. i th cycle i i+ n Training data set. Verification sample. n th cycle i i+ n Training data set Verification J F M A M J J A S Figure 4.2 Illustration of leave-one-out-month cross-validation method. Here n is the number of forecast months Many aspects of the performance of the post-processed ensemble rainfall forecasts need to be assessed. The performance of forecasts is assessed for individual forecast periods and for cumulative forecast totals. Furthermore, the forecasts are evaluated for individual subareas and for catchment averages. This enables the performance of the post-processing probability model and the efficacy of the Schaake shuffle ensemble generation method to be assessed separately. A detailed assessment of the strengths and weaknesses of a set of forecasts usually requires more than one or two summary scores ((Jolliffe and Stephenson,, 22). In this study, the forecast performance is evaluated using bias, continuous ranked probability score (CRPS), reliability diagrams and relative operating characteristics (ROC) plots. Bias assesses the difference between the mean of rainfall forecasts and the mean of the corresponding observations. Biases in rainfall forecasts will potentially be amplified in streamflow forecasts and therefore it is important that rainfall forecasts have minimal bias. Forecast bias, as a percentage of the observed value, is assessed for the raw NWP forecasts and post-processed forecasts for individual lead times and cumulative totals throughout the forecast period. For post-processed rainfall forecasts, the bias is computed from mean of the forecast ensemble. The continuous ranked probability score (CRPS) is a summary statistic comparing the closeness of the forecast and observed cumulative distribution functions (see also Figure 4.3) and is expressed as CRPS ( F, y ) = ( F ( y) H ( y y )) dy (4.) t t t t 2 Model fitting and generating forecasts where t is time, F is the cumulative distribution function of ensemble forecasts, y t is the observation and H is the Heaviside function (which equals for values greater than the observed and otherwise). (Hersbach, 2) discusses the application of CRPS to ensemble forecasts. We compare the CRPS score of the raw NWP rainfall forecasts and post-processed rainfall forecasts. For the raw deterministic NWP rainfall forecasts, the CRPS reduces to the mean absolute error. Post processing NWP rainfall forecasts

22 .9.8 Forecast probability Observation probability CRPS Probability.7.6 Z CRPS(F t; y t) = (F t (y)! H (y! y t)) 2 dy.5! Rainfall (mm/hour) Figure 4.3 Computation of continuous ranked probability score The reliability diagram is a commonly used visual tool to verify probabilistic forecasts. Statistical reliability is commonly demonstrated with reliability diagrams (or reliability curves). A reliability diagram plots the observed frequency of a set of events against the forecast probability for those events and shows how well the forecast probability corresponds to the observed frequency (Wilks, 26). The reliability of the forecast probability of an event of rainfall greater than.2 mm (probability of rainfall, PoP) and 25 th, 5 th and 75th percentiles of rainfall amounts greater than.2 mm are assessed using reliability diagrams. We produce reliability diagrams using forecasts for individual forecast periods and for cumulative forecasts, and for individual subareas and catchment averages. The reliability diagram for individual forecast periods and subareas assesses the reliability of forecasts made using individual BJP models. For an individual forecast a period reliability diagram is drawn from the data pooled from a 24-hour period (i.e. for day, lead times of -24 hours; for day 2, lead times of hours and so on). The reliability diagrams for the cumulative totals and catchment averages assess the ability of the Schaake shuffle method to restore the appropriate space-time correlation structure of the forecast ensembles. We assess the reliability of forecast total rainfall for day (lead times of 24 hours) to day 9 (lead times of hours). Significant streamflow events primarily result from significant rainfall events. Therefore, it is important for rainfall forecasts to be able to identify significant rainfall events when they occur (Robertson et al., 23). The ROC curve measures the ability of the forecast to discriminate between the events and non events, such as rainfall exceeding (or not exceeding) certain thresholds. The ROC curve plots the hit rate against the false alarm rate for a range of decision probability thresholds. Hit rate and false alarm rate are computed by constructing a 2 by 2 contingency table for each decision threshold. Thus, probability forecasts for a continuous variable can be first converted into probabilistic forecasts of a binary event by specifying exceedance above or below a threshold (say 24 hour rainfall totals greater than 5 mm) and then these can be converted into a continuous set of deterministic forecasts of a binary event by using a sequence of probability decision thresholds (i.e. the event is yes if the probability of 24 hour rainfall exceeding 5 mm is greater than decision threshold.8) (see also Jolliffe and Stephenson, 22). For unskilled forecasts a ROC plot will follow a diagonal line, as hit rates will be equal to false alarm rates. Perfect forecasts on a ROC plot travel vertically from the origin to the top left of the diagram and then horizontally to the top right. For a ROC curve, we used the same forecast events (i.e..2 mm and 25 th, 5 th and 75 th percentiles of rainfall amounts greater than.2 mm) used in the reliability diagram. For an ensemble of m forecasts, there are m distinct probability thresholds corresponding to at least, 2, 3,, m of the forecasts predicting the chosen events (Toth et al., 23).Thus, instead of using uniform probability decision threshold values, we have selected probabilities of forecasts of the events (i.e..2 mm and 25 th, 5 th and 75 th percentiles of rainfall amounts greater than.2 mm) as decision threshold values to compute hit rate and false alarm rate for the ROC curve. Like other scores mentioned before, forecast discrimination is assessed for individual lead times and for cumulative totals throughout the forecast period for individual subarea and catchment averages. 2 Post processing NWP rainfall forecasts

23 5. Results and Discussion 5.. Spatial variation of forecast performance We present the performance of 3 hour total rainfall forecasts from raw and post-processed NWP model outputs in this section; comparisons of 3 hour, 6 hour and 24 hour rainfall totals are given in next section. Figure 5. presents catchment average rainfall observations, raw NWP and post-processed rainfall forecasts throughout the forecast periods in ten catchments. The results show that there is a significant variation of rainfall throughout the catchments. Most importantly, the evidence of the diurnal cycle is seen in both observed and raw NWP forecasts but it is not observed in the rainfall forecasts from earlier version (APS) of the ACCESS-G model. Several catchments in the tropics and subtropics (e.g. Stanley, Macleay and Ord) have very high diurnal variation, while the South Esk and Onkaparinga are less affected by diurnal variation. In all catchments, there is a displacement of 3 hours to 2 hours between diurnal cycles of observations and raw forecasts. The post-processing method performs very well in not only reproducing the peak of the diurnal cycle but also reproducing the timing of the peak of the cycle. Accurately estimating the timing of the peak of the rainfall forecasts is crucial for flood forecasting. Figure 5.2 shows a comparison of the biases in the raw NWP rainfall forecasts and post-processed rainfall forecasts across the ten catchments. The bias of catchment average rainfall forecasts is computed for raw NWP rainfall forecasts and the mean of post-processed ensemble rainfall forecasts. The bias in the raw NWP forecasts varies significantly across the catchments and displays a diurnal cycle. Diurnal cycle in the bias score was also observed in ACCESS-G APS model in the Ovens catchment (Shrestha et. al., 23). The NWP model performs very badly in the Ord catchment with the bias over 7%, which is likely the product of the limited ability of NWP models to simulate the tropical cyclones in the wet season. In the South Esk (STE) and Upper Murray (MRR) catchments, the performance of the NWP model is better than other catchments. In most of the catchments the NWP model tends to underestimate rainfall (negative percentage bias). In the Macleay (MCL) and Wollondilly (WLL) catchments, the NWP model is overestimating the rainfall for some forecast periods. The rainfall forecasts from the post-processing method display no bias or very little bias in all catchments throughout the forecast periods. The postprocessing method is also able to remove the diurnal signal present in the bias score of the raw NWP forecasts. Figure 5.3 shows the bias in the raw and post-processed rainfall forecasts across all subareas. The results show that about % of subareas have significant bias up to 8% (Figure 5.3a). The largest biases occur in subareas in the Ord catchment where the skill of the raw NWP forecasts is poor. There is a very strong diurnal cycle in the bias score. It can be seen from Figure 5.3b that about % of subareas have absolute bias in the raw NWP forecasts larger than 6%. More than 3% of subareas have bias in the order of the observed rainfall values. The post-processing method significantly reduces the forecast bias. The biases are consistently smaller for all subareas. Figure 5.4 presents a spatial variation of average daily rainfalls in the South Esk catchment for observations, raw NWP model forecasts and post-processed NWP forecasts for day (top panel), day 3 (middle panel) and day 9 (bottom panel) lead times. Catchment rainfall is strongly influenced by topography, with lowest average rainfall occurring at sites lower in the catchment, to the west, and highest totals occurring high in the catchment, to the east. The subareas in the valley receive average daily rainfall value of about.5-2 mm. The subareas in the headwaters of the catchment (northern part) have the highest rainfall of about mm/day. The NWP model displays a little variation of the rainfall forecasts across the subarea. The NWP model overestimates rainfall in south west low land areas in day and day 3 lead times. The model is reasonably good in the central part of the catchment for day lead time. In the north-east part, the model significantly underestimates the rainfall for all lead times. The post-processed rainfall forecasts show a similar pattern of observed rainfall and there is a significant improvement of the skill over the raw NWP Post processing NWP rainfall forecasts 3

24 forecasts. The post-processing method is able to reproduce the subarea variability which was not seen in the raw NWP forecasts. Reproducing rainfall variability of similar statistical structure as the observed is important for hydrological modelling. For example, studies of the effect of rainfall variability on basin response (see, e.g. Ogden and Julien, 993) suggest that runoff volume is sensitive to the small-scale spatial and temporal variability of the rainfall field, which is provided as input to a rainfall runoff model. 8 STN MCL WLL CTT Avg Rainfall (mm/day) GBR MRR 8 STE WTT ONK ORD Lead time (hours) Observation Raw NWP Post-processed Figure 5. Diurnal cycle of the observed, raw NWP and post-processed rainfall forecasts (catchment average) in catchments. The time step of the rainfall is 3 hour and period is from August 2 to April 22 Figure 5.5 presents the bias and CRPS skill of the raw NWP and post-processed rainfall forecasts for the South Esk catchment. One can see that there is a significant bias in the raw NWP forecasts. There is a positive bias up to 5% in the low rainfall lowland areas, whereas the bias is negative in high rainfall highland areas. This is a typical characteristic of many NWP models (see, e.g., Shrestha et al., 23). The NWP model performs reasonably well in the central part of the catchment. The post-processing method reduces the forecast bias to below % for most of the subareas. The CRPS value (lower panel) of the raw NWP forecasts varies from about.5 to.5 mm/3 hour. Applying the post-processing method reduces the CRPS value across all subareas in the South Esk. The maximum value of CRPS is about.4 mm/3hour in highland areas and in lowland areas it reduces to below.5 mm/3 hour. 4 Post processing NWP rainfall forecasts

25 Figure 5.2 Biases of post-processed rainfall forecasts across the catchments. The skill is measured by percentage bias of catchment average rainfall. Blue shows skill of post-processed NWP model forecasts; red shows skill of raw NWP model forecasts 8 7 (a) Raw NWP for 9% subareas Post-processed for 9% subareas (b) Raw NWP range Post-processed range 6 8 Bias (%) Absolute bias (%) Lead time (days) Fraction of subareas Figure 5.3 Bias in the raw and post-processed rainfall forecasts across all subareas. (a) Bias as a function of lead time for subareas having 5 th and 95 th percentile bias, (b) range of absolute bias as a function of fraction of subareas Post processing NWP rainfall forecasts 5

26 -4.3 Observation Raw NWP Post-processed Day Day Day Rainfall (mm/day) Figure 5.4 Spatial variation of 24 hour average rainfall values in the South Esk catchment. The dashed-grey lines in the centre (vertical) panels show the NWP model grid. Top row is for -day lead time, middle row 3-day lead time and bottom row 9-day lead time Raw NWP model Post-processed NWP model Bias (%) < N I L E R I V E R N I L E R I V E R >5 SO U T H ESK R I V ERS T P A U L S R I V ER S O U T H ES K R I V ER I S T P A U L S R V ER CRPS (mm) < N I L E R I V E R N I L E R I V E R >.5 SO U T H ESK R I V ERS T P A UL S R I V ER SO U T H ESK R I V ERS T P A U L S R I V ER Kilometres Figure 5.5 Bias and CRPS skill of the raw NWP forecasts and post-processed forecasts for the South Esk catchment 6 Post processing NWP rainfall forecasts

27 5.2. Variation of forecast performance with lead time We have presented biases of the raw and post-processed rainfall forecasts across ten catchments in the previous section. In this section, we focus on the Cotter catchment for 3 hour rainfall forecasts. The results for other time steps and catchments are given in Appendix A. Figure 5.6 shows subarea delineation in the Cotter and ACCESS-G model grid cells. The catchment is delineated into 9 subareas for hydrological modelling. The streamflow direction is from subareas a and b towards i. The total catchment area is about 47 km 2 and the area of the subarea ranges from 2 km 2 to 6 km 2 which is much less than the area of an ACCESS-G model grid cell (~2 km 2 ). All of these subareas are mainly contained within two NWP model grid cells. Thus, the ACCESS-G model is unlikely to capture gradients of rainfall across the catchment km km i h g f e d c a b Figure 5.6 Delineation for Cotter catchment and ACCESS-G model grids (dashed lines) (a) Raw NWP forecast (b) (c) 75 Post-processed forecast (d) (e) (f) 75 5 Bias (%) (g) (h) (i) Lead time (hours) Figure 5.7 Bias (%) for individual forecast periods as a function of lead time for 3 hour rainfall total for all 9 subareas in the Cotter. Each panel is for each subarea location given in Figure 5.6 Post processing NWP rainfall forecasts 7

28 Figure 5.7 presents the bias in the raw NWP rainfall forecasts and post-processed forecasts in all 9 subareas in the Cotter as a function of lead time for 3 hour rainfall totals. The bias in the raw NWP rainfall forecasts is up to -75%. The NWP model significantly underestimates 3 hour rainfall totals. The NWP model forecasts are too high in two subareas near the outlet (subareas h and i) for some forecast periods. The postprocessed forecasts display a little forecast bias throughout all subareas and throughout the forecast periods. As mentioned before, the bias of the post-processed rainfall forecasts is the bias of the mean of ensemble rainfall forecasts. It can be seen from the figure that post-processing corrects forecasts both when they are too high and too low. Bias correction using the BJP modelling approach is more sophisticated than just correcting the mean bias. Using different marginal distributions, and particularly transformations, for the raw NWP rainfall forecasts and observed data allows for a non-linear bias correction(robertson et al., 23). One can see from the figure that the bias in the raw NWP rainfall forecasts tends to be cyclic which is likely the product of the limited ability of NWP models to describe the diurnal cycle. The diurnal cycle was also observed in the bias of rainfall forecasts from the earlier version (APS) of the ACCESS-G model (Shrestha et al., 23). Poorly representing the timing and magnitude of the diurnal cycles, particularly in rainfall, is a known problem with many NWP models and is commonly related to the representation and parameterisation of convective processes (Kaufmann et al., 23; Dai and Trenberth, 24; Evans and Westra, 22). Figure 5.8 shows the bias for catchment average rainfall for the Cotter catchment. As expected, catchment average bias is reduced compared to biases in individual subareas. Catchment average biases in the postprocessed rainfall ensembles are minimal throughout the individual forecast period (Figure 5.8a). Figure 5.8b shows catchment average bias in the post-processed cumulative rainfall ensembles and in the raw NWP. The result shows that bias for the raw NWP forecasts fluctuates from about -5% to -3% at shorter lead times, and then are relatively stable near -4%. However, the bias in the post-processed ensemble forecasts is very little and remains stable after 2 hours. These results of catchment average and cumulative total rainfall forecasts demonstrate the efficacy of the Schaake shuffle in restoring both the spatial and temporal correlations in the forecast ensemble. (a) (b) Bias (%) Raw NWP forecast Post-processed forecast Lead time (hours) Figure 5.8 Bias for catchment average rainfall for the Cotter. (a) Individual period, and (b) cumulative totals Improvement in absolute bias is calculated by subtracting absolute bias for raw NWP forecasts from absolute bias of the post-processed rainfall forecasts. The results for 3 hour, 6 hour and 24 hour rainfall totals for all ten catchments are given in Appendix A (Figure A.). In general, the post-processing reduces the forecast bias across the range of catchments. Figure 5.9 presents the CRPS scores of the raw NWP and post-processed rainfall forecasts for individual periods for all 9 subareas in the Cotter. The CRPS score of the raw NWP forecasts shows a diurnal cycle and in general, it increases with increasing lead time. The CRPS score of the post-processed rainfall forecasts also shows the diurnal cycle and it increases with increasing lead time. The CRPS of the raw NWP forecasts is higher than that of the post-processed rainfall forecasts for all lead times and for all subareas. The CRPS score of the post-processed forecasts of the catchment average rainfall for individual period and cumulative rainfall totals is shown in Figure 5.. The CRPS score of the catchment average rainfall 8 Post processing NWP rainfall forecasts

29 forecasts (Figure 5.a) for individual forecast period displays similar behaviour to that of individual subareas, but the magnitude of the score is a little lower Raw NWP forecast Post-processed forecast (a) (b) (c).35 (d) (e) (f).3 CRPS (mm/h) (g) (h) (i) Lead time (hours) Figure 5.9 Variation in CRPS of the post-processed rainfall forecasts for individual periods as a function of lead time in subareas in the Cotter catchment. Subarea locations are shown in Figure (a) (b) Raw NWP forecast Post-processed forecast CRPS (mm/h) Lead time (hours) Figure 5. CRPS for catchment average rainfall for the Cotter catchment. (a) Individual periods, and (b) cumulative totals The CRPS score for cumulative rainfall totals for both raw and post-processed rainfall forecasts are smaller than the forecasts for individual periods because errors in individual periods will tend to compensate for each other. The CRPS score first increases to 2 hour lead time, then decreases until 48 hour lead time, and finally becomes stable around value of. and.7 mm/hour for raw and post-processed rainfall forecasts, respectively. The consistently lower value of CRPS score of the catchment average rainfall forecast for both individual periods and cumulative totals indicates that the Schaake shuffle maintains realistic spatial and Post processing NWP rainfall forecasts 9

30 temporal correlations in the forecast ensemble. CRPS skill score of the post-processed rainfall forecasts is computed with reference to CRPS score of the raw NWP forecasts. The results for 3 hour, 6 hour and 24 hour rainfall totals for all ten catchments are given in Appendix A (Figure A.2). CRPS skill score is positive across the catchments throughout the forecast periods for all time steps. This means that the postprocessed rainfall forecasts are always better than the raw NWP forecasts for the test catchments. (a) Precip >.2 mm (PoP) (b) Precip >.52 mm (25 th %) (c) Precip >.3 mm (5 th %) (d) Precip > 3.62 mm (75 th %) Observed relative frequency Forecast probability Day Day 2 Day 3 Day 4 Day 6 Day Figure 5. Reliability diagrams for the post-processed ensemble rainfall forecasts for individual forecast periods as a function of lead time for one subarea in the Cotter catchment. (a) probability of rainfall, (b) 25 th percentile of rainfall amounts greater than.2 mm, (c) 5 th percentile of rainfall amounts greater than.2 mm, and (d) 75 th percentile of rainfall amounts greater than.2 mm (a) 24 hour precip >.2 mm (PoP) (b) 24 hour precip >.53 mm (25 th %) (c) 24 hour precip >.33 mm (5 th %) (d) 24 hour precip > 3.47 mm (75 th %) Observed relative frequency Forecast probability Day Day 2 Day 3 Day 4 Day 6 Day Figure 5.2 Same as Figure 5. but for catchment average 24 hour rainfall totals Figure 5. presents reliability diagrams for the probability of rainfall exceeding four thresholds:.2 mm and 25 th, 5 th and 75 th percentile of rainfall amounts greater than.2 mm in 3 hour period for one of the subareas in the Cotter. The reliability is computed for forecasts pooled for day (lead time 3-24 hour), day 2 (27-48 hour), and so on. One can see that the forecast probability of a rainfall event of.2 mm (probability of precipitation, PoP) and 25 th percentile of rainfall amounts greater than.2 mm (.52 mm in this subarea) for individual forecast periods (Figure 5.a and b) appears to be reliable, with the observed relative frequencies closely following the : line up to 4 day lead times. For higher threshold values (Figure 5.c and d), the reliability diagrams further deviate from the : line, indicating they are less reliable. We note that the rainfalls exceeding the 75 th percentile are by definition rare, and this reduces the pool of forecasts available to assess reliability. Assessing reliability on few forecast events can lead to unstable estimates of forecast reliability. For longer lead times (day 6 and day 9), forecasts are not reliable for all threshold values. In fact the post-processing method over-estimates the probability of exceeding the thresholds. One can see in the inset diagram that the sample size for higher probability bins is small, so forecast probability and observed relative frequency in the higher probability bins are subject to considerable sampling variability. Note that deviations from the diagonal may be caused by both sampling effects and serial correlation in the forecast-observation pairs (Pinson et al., 2). Further analysis is required to investigate the stability of the reliability diagrams for small sample sizes (see, e.g., Bröcker and Smith, 27). 2 Post processing NWP rainfall forecasts

31 Figure 5.2 shows that 24 hour rainfall totals for catchment average are reliable for all threshold values and for all lead times except day 9 for 75 th percentile threshold value. As expected, the reliability decreases for longer threshold values and longer lead times (e.g., day 6 and day 9). Interestingly, the probabilistic forecasts for cumulative rainfall totals seem to slightly under-estimate the probability of exceeding thresholds. The probabilistic forecasts of 24 hour rainfall totals for catchment average are produced by summing ensemble members of individual forecast periods. These forecasts are most likely to be reliable if the forecasts for individual periods are reliable and the ensemble members have the appropriate spacetime correlation structures. Here we have demonstrated that the probability distributions of forecasts for both individual periods and cumulative totals at both individual subareas and for catchment averages are reliable and therefore the space-time correlations have been successfully instilled by the Schaake shuffle. Average absolute deviation from the : reliability line (perfect reliability) is computed for the postprocessed rainfall forecasts. The results for 3 hour, 6 hour and 24 hour rainfall totals for all ten catchments are given in Appendix A (Figure A.3). In general, the post-processed rainfall forecasts are reliable for shorter lead times and for lower threshold values. Figure 5.3 presents the ROC curves as a function of lead time and rainfall threshold value for one of the subareas in the Cotter catchment. The ROC curve measures the ability of the forecast to discriminate between the events and non-events. Here, an event is defined for 3 hour rainfall totals exceeding.2,.52,.3 and 3.62 mm. The area under the ROC curve (AUC) is also shown in Figure 5.5. Note that AUC is. for a perfect forecast (i.e. high hit rate) and.5 for a climatology forecast (diagonal line in ROC curve). At shorter lead times, the ROC curves for forecasts of individual periods tend to approach the top left corner of the plot (AUC=.), while at longer lead times they are closer to the diagonal line. This suggests that as would be expected forecasts for shorter lead times have a greater ability to discriminate between events and non-events than forecasts for longer lead times. The contrast in forecast discrimination with lead time is stronger for the high rainfall events (rainfall > 5 th and 75 th percentile) than for the low rainfall events (rainfall > PoP and 25 th percentile). This is also evidenced in the AUC plot (Figure 5.5a). After a lead time of 7 days (Figure 5.5a), the post-processed rainfall forecasts are less skilful than a climatology forecast. The ROC curves for cumulative rainfall totals for catchment average are shown in Figure 5.4. The results show that cumulative rainfall forecasts for catchment average better discriminate the events and nonevents than for individual forecast periods, particularly for longer lead times. Furthermore, one can see that the spread of the ROC curve is significantly less than the curves for individual forecast periods (see Figure 5.5b also). The forecast discrimination is stronger for shorter lead times than for longer lead times for higher rainfall threshold values (i.e. cumulative rainfall >5 th and 75 th percentile). For the events of lower threshold values (i.e. cumulative rainfall >.2 mm and 25 th percentile) there are no clear differences in forecast discrimination with lead time. The improved forecast discrimination of cumulative rainfall forecasts for catchment average supports the earlier finding that the space-time correlations restored by the Schaake shuffle method are appropriate. The area under the ROC curves for 3 hour, 6 hour and 24 hour rainfall totals across ten catchments is given in Appendix A (Figure A.4). In general, the post-processed rainfall forecast better discriminates between event and non-events for shorter lead time and for lower thresholds. Discrimination is better for 24 hour rainfall totals compared to 6 hour and 3 hour rainfall totals. Post processing NWP rainfall forecasts 2

32 (a) Precip >.2 mm (PoP) (b) Precip >.52 mm (25 th %) (c) Precip >.3 mm (5 th %) (d) Precip > 3.62 mm (75 th %) Hit rate False alarm rate Lead time (hours) Figure 5.3 ROC curves for individual forecast periods as a function of lead time for one of subarea in the Cotter catchment. (a) Probability of rainfall, (b) 25 th percentile of rainfall amounts greater than.2 mm, (c) 5 th percentile of rainfall amounts greater than.2 mm, and (d) 75 th percentile of rainfall amounts greater than.2 mm (a) Cumulative Precip > PoP (b) Cumulative Precip > 25 th % (c) Cumulative Precip > 5 th % (d) Cumulative Precip > 75 th % Hit rate False alarm rate Lead time (hours) Figure 5.4 Same as Figure 5.3, but for cumulative totals for catchment average rainfall Figure 5.5 Area under ROC curves of Figure 5.3 and Figure 5.4 as a function of lead time. (a) Individual forecast periods for one of the subarea in the Cotter catchment, and (b) cumulative rainfall totals for catchment average 22 Post processing NWP rainfall forecasts

33 5.3. Variation of forecast performance with post-processing time step The previous section presents the performance of post-processing for 3 hour rainfall totals. (Shrestha et al., 23) investigated the skill of the raw NWP models with rainfall accumulation periods. As expected they found that the skill of the NWP model rainfall forecasts increases with increasing accumulation periods because of cancelling timing errors within the accumulation periods. Here we investigate the effect of accumulation periods on the performance of the post-processing method. Forecasts and observations are first accumulated over 3 hour, 6 hour and 24 hour periods. Post-processing is then applied for these periods separately and rainfall forecast ensembles are produced. In order to compare the skill of these periods, 3 hour and 6 hour post-processed rainfall forecasts and observations are accumulated to a 24 hour period. Figure 5.6 presents catchment average bias in the Cotter River for individual forecast periods and cumulative rainfall totals. In comparison to raw NWP rainfall forecasts, all post-processed forecasts exhibit much less bias. There were no substantial differences in the biases for the three different time steps. It may not have significant effect in streamflow forecasting due to the low-pass filtering function of the rainfallrunoff model. Bennett et al. (23) describe the performance of streamflow forecasts with different rainfall accumulation periods. (a) (b) - Bias (%) hour 6 hour 24 hour raw 24 hour Lead time (hours) Figure 5.6 Catchment average bias (%) as a function of lead time for different post-processing time steps in the Cotter catchment. (a) Individual forecast period, and (b) cumulative totals (a) (b).5 CRPS (mm/h)..5 3 hour 6 hour 24 hour raw 24 hour Lead time (hours) Figure 5.7 Same as Figure 5.6 but for the CRPS Post processing NWP rainfall forecasts 23

34 (a) Precip. >.2 mm (PoP) (b) Precip. >.97 mm (25 th %) (c) Precip. > 3.24 mm (5 th %) (d) Precip. > 9.48 mm (75 th %).8.6 Day (e) (f) (g) (h).8.6 Day 2 Observed relative frequency.4.2 (i).8 2 (j) 3 2 (k) 4 2 (l) Day (m) (n) (o) (p).8.6 Day Forecast probability 3 hour 6 hour 24 hour Figure 5.8 Reliability diagram for catchment average rainfall for day (4 panels in top row), day 2 (second row), day 4 (third row), and day 6 (bottom row). 4 Panels in the first column are for probability of rainfall, second column 25 th percentile of rainfall amounts greater than.2 mm, third column 5 th percentile of rainfall amounts greater than.2mm, and last column 75 th percentile of rainfall amounts greater than.2 mm Figure 5.7 compares CRPS for post-processed rainfall forecasts for 3 hour, 6 hour and 24 hour periods. Like bias, the CRPS for the three accumulation periods are comparable for instantaneous rainfall forecasts and cumulative rainfall forecasts. Post-processed rainfall forecasts for 24 hour totals are marginally better than 3-hour and 6-hour post-processed rainfall forecasts for longer lead times, but these differences in performance are negligible. 24 Post processing NWP rainfall forecasts

35 Cumulative precip. >.2 mm (PoP) (a) Cumulative precip. >.97 mm (25 th %) (b) Cumulative precip. > 3.27 mm (5 th %) (c) Cumulative precip. > 9.44 mm (75 th %) (d).8.6 Day.4.2 Cumulative precip. >.2 mm (PoP) (e) Cumulative precip. >.57 mm (25 th %) (f) Cumulative precip. > 4.68 mm (5 th %) (g) Cumulative precip. > 5.57 mm (75 th %) (h) Day to Day 2 Hit rate Cumulative precip. >.2 mm (PoP) (i) Cumulative precip. > 2.22 mm (25 th %) (j) Cumulative precip. > 7.8 mm (5 th %) (k) Cumulative precip. > mm (75 th %) (l) Day to Day 4 Cumulative precip. >.2 mm (PoP) (m) Cumulative precip. > 6.76 mm (25 th %) (n) Cumulative precip. > mm (5 th %) (o) Cumulative precip. > mm (75 th %) (p) Day to Day False alarm rate 3 hour 6 hour 24 hour Figure 5.9 Same as Figure 5.8 but for ROC curves for cumulative totals AUC (a) Cumulative precip. > PoP (b) Cumulative precip. > 25 th % (c) Cumulative precip. > 5 th % Lead time (days) 3 hour 6 hour 24 hour (d) Cumulative precip. > 75 th % Figure 5.2 Area under ROC curves of Figure 5.9 Post processing NWP rainfall forecasts 25

36 Figure 5.8 presents a comparison of reliability diagrams for 3 hour, 6 hour and 24 hour post-processed catchment average rainfall forecasts in the Cotter catchment. The threshold values selected are the same as previously i.e..2 mm (PoP), and 25 th, 5 th and 75 th percentile of rainfall amounts greater than.2 mm (note that the magnitude is different than in Figure 5. as the accumulation period is different). For shorter lead times, the reliability diagrams of the 24 hour rainfall forecasts are closer to the diagonal line than for 3 hour and 6 hour post-processed rainfall forecasts except for higher threshold values in day 2 and day 4. For day 6, 24 hour post-processed rainfall forecasts are not reliable compared to 3 hour and 6 hour. Note that for rainfall greater than the 5 th and 75 th percentile, there are fewer data in the higher probability bins (see inset diagram). Thus the results are subject to sampling variability. Since the deviation from the diagonal line may be caused by sampling effects and serial correlation in the forecast-observation pairs (Bröcker and Smith, 27; Pinson et al., 2), any conclusions cannot be drawn where the sample size is too small. Figure 5.9 shows ROC curves for cumulative rainfall for four different threshold values:.2 mm (PoP), and 25 th, 5 th and 75 th percentiles of rainfall amounts greater than.2 mm. For shorter lead times, 24 hour postprocessed rainfall forecasts are better than 3 hour and 6 hour rainfall forecasts. The difference between 24 hour and the other time steps decreases when the rainfall threshold value increases. For day 9, all forecasts are comparable. Sometime it is difficult to compare two ROC curves as they cross each other (see, e.g., Figure 5.9o and p). So we have also shown AUC plots to better assess the forecast skill. Figure 5.2 presents the AUC for the ROC curves shown in Figure 5.9. Since it is common practice to consider AUC of more than.8 to be indicative of a good forecast and AUC of.7 as the limit for the useful forecast ((Buizza et al.,, 999), these values and no skill value of.5 are also shown in the figure. The results show that 24 hour post-processed forecasts are better than 3 hour and 6 hour forecasts for all lead times except day 8 and day 9. For all threshold values, AUC values of 24 hour forecasts are above.8, which indicates that the post-processed forecasts are good at discriminating the events. For 3 hour and 6 hour forecasts, the AUC value is about.7 which indicates that the forecasts although not as good, are still useful. 26 Post processing NWP rainfall forecasts

37 6. Summary and Recommendations Bias free and reliable ensemble rainfall forecasts are required to produce reliable and skilful ensemble streamflow forecasts. The rainfall forecasts that are publicly available from Australian NWP models are deterministic and often contain systematic errors. Therefore, it is necessary to remove the systematic biases and reliably quantify uncertainty in rainfall forecasts before they can be used for streamflow forecasting. This report presents the further development and application of a rainfall post-processing method to remove forecast bias and reliably quantify forecast uncertainty. The post-processing method consists of first applying a simplified Bayesian joint probability model for individual locations and individual forecast periods to produce probabilistic rainfall forecasts. Ensemble forecasts with appropriate spatial and temporal correlations are then produced by linking samples from the forecast probability distributions using the Schaake shuffle (Clark et al., 24). The post-processing method is evaluated for a range of small to medium sized catchments that cover a wide range of climatic conditions and hydrological characteristics. The main conclusions and recommendations are as follows: The skill of the raw NWP forecasts varies significantly across the catchments. The NWP model is very poor in subtropical/monsoonal catchments. The bias of the raw NWP forecasts in these regions can be over 7% (over-forecasting) for some forecast periods. In other regions, the NWP model is not producing enough rainfall for most of the forecast periods. Post-processing significantly improves forecast bias across the catchments. The raw NWP model is unable to resolve small scale catchment rainfall variability. The skill of the raw NWP forecasts varies considerably across hydrological model subareas. Post-processing restores small scale topographical variation and consequently improves skill at each of the forecast locations. The skill of both raw NWP and post-processed rainfall forecasts varies with forecast lead time. The diurnal variability that can be seen in both observations and NWP model forecasts is also present in the bias of the raw NWP forecasts. Post-processing consistently and significantly improves forecast bias for all forecast lead times. The bias of the post-processed rainfall forecasts does not show any evidence of diurnal variability. The post-processing method performs very well in not only reproducing the magnitude of the peak of the diurnal cycle but also reproducing the timing of the peak. Accurately estimating the timing of the peak of the rainfall forecasts is crucial for flood forecasting. Post processing produces ensemble forecasts with minimal bias for; individual forecast periods, cumulative totals for multiple lead times, individual locations (subareas) and for catchment average rainfalls. The post-processed forecast ensembles are demonstrated to successfully discriminate between events and non-events for both small and large rainfall occurrences, and reliably quantify the forecast uncertainty for shorter lead times. The experiments with time steps of rainfall accumulation periods show that the skill of postprocessed rainfall forecasts for 3 hour, 6 hour and 24 hour totals is comparable. As NWP models are regularly updated, forecast data sets from the latest NWP models are generally short. This poses a major challenge in establishing robust statistical models for calibrating the post BJP processing model. Further analysis is required to investigate the extent of data required to infer the BJP model parameters and reliably quantify forecast uncertainty. In this study, the BJP modelling approach uses only NWP rainfall forecasts as a single predictor. However there are other outputs available from NWP models which can potentially be used as predictors in the BJP model (see, e.g., Wang et al., 29). Future work is envisaged to assess the Post processing NWP rainfall forecasts 27

38 skill of the post-processing method with a combination of NWP rainfall forecasts and other climate forecasts. This study applies the BJP modelling approach to a deterministic forecast from an NWP model. The authors are extending the method to post-process ensemble rainfall forecasts from different sources (e.g., seamless rainfall forecasts, Bowler et al., 26). The Schaake shuffle assumes stationarity in the spatial and temporal correlation structure. In this study the stationarity problem is assumed to be reduced by using a short period of historical observations (same period of forecasts data, although much longer period of observation is available,). The seasonal variability of the correlation was not considered in this study because of using a short period of data. This study post-processes rainfall forecasts from the NWP model for individual lead times and individual locations separately. In future it is planned to extend the method that jointly postprocesses rainfall forecasts from multiple lead times and locations. 28 Post processing NWP rainfall forecasts

39 7. References Bennett, J.C., Robertson, D.E., Shrestha, D.L., Wang, Q., 23. Performance of a new ensemble streamflow forecasting system for Australia, CSIRO Water for a Healthy Country Flagship, Australia. Bowler, N.E., Pierce, C.E., Seed, A.W., 26. STEPS: A probabilistic precipitation forecasting scheme which merges an extrapolation nowcast with downscaled NWP. Q. J. R. Meteorolog. Soc., 32(62): Bröcker, J., Smith, L.A., 27. Increasing the reliability of reliability diagrams. Weather and Forecasting, 22(3): Buizza, R., Hollingsworth, A., Lalaurette, F., Ghelli, A., 999. Probabilistic predictions of precipitation using the ECMWF ensemble prediction system. Weather and Forecasting, 4(2): Bureau of Meteorology, 2. Operational implementation of the ACCESS numerical weather prediction systems. NMOC Operations Bulletin No. 83, available at: (last access: June 23), Bureau of Meteorology, Melbourne, Australia. Bureau of Meteorology, 22. APS upgrade of the ACCESS-G numerical weather prediction system. NMOC Operations Bulletin No. 93, available at: (last access: June 23), Bureau of Meteorology, Melbourne, Australia. Clark, M., Gangopadhyay, S., Hay, L., Rajagopalan, B., Wilby, R., 24. The Schaake shuffle: A method for reconstructing space-time variability in forecasted precipitation and temperature fields. Journal of Hydrometeorology, 5(): Dai, A., Trenberth, K.E., 24. The diurnal cycle and its depiction in the community climate system model. Journal of Climate, 7(5): Duan, Q., Sorooshian, S., Gupta, V.K., 994. Optimal use of the SCE-UA global optimization method for calibrating watershed models. J. Hydrol., 58(3-4): Evans, J., Westra, S., 22. Investigating the mechanisms of diurnal rainfall variability using a regional climate model. Journal of Climate, 25(2): Harris, D., Foufoula-Georgiou, E., Droegemeier, K.K., Levit, J.J., 2. Multiscale Statistical Properties of a High-Resolution Precipitation Forecast. Journal of Hydrometeorology, 2(4): Hersbach, H., 2. Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems. Weather and Forecasting, 5(5): Jolliffe, I.T., Stephenson, D.B. (Eds.), 22. Forecast verification: a practitioner's guide in atmospheric science. Wiley, West Sussex, England, 247 pp. Kaufmann, P., Schubiger, F., Binder, P., 23. Precipitation forecasting by a mesoscale numerical weather prediction (NWP) model: eight years of experience. Hydrol. Earth Syst. Sci., 7(6): Oberto, E., Turco, M., Bertolotto, P., 26. Latest results in the precipitation verification over Northern Italy, COSMO Newsletter, pp Ogden, F.L., Julien, P.Y., 993. Runoff sensitivity to temporal and spatial rainfall variability at runoff plane and small basin scales. Water Resour. Res., 29(8): Pinson, P., McSharry, P., Madsen, H., 2. Reliability diagrams for non parametric density forecasts of continuous variables: Accounting for serial correlation. Q. J. R. Meteorolog. Soc., 36(646): Robertson, D.E., Shrestha, D.L., Wang, Q.J., 23. Post-processing rainfall forecasts from numerical weather prediction models for short-term streamflow forecasting. Hydrol. Earth Syst. Sci., 7(9): Rossa, A., Nurmi, P., Ebert, E., 28. Overview of methods for the verification of quantitative precipitation forecasts. In: Michaelides, S. (Ed.), Precipitation: advances in measurement, estimation, and prediction. Springer-Verlag, Berlin, pp Schaake, J. et al., 27. Precipitation and temperature ensemble forecasts from single-value forecasts. Hydrol. Earth Syst. Sci. Discuss., 4: Post processing NWP rainfall forecasts 29

40 Shrestha, D.L., Robertson, D.E., Wang, Q.J., Pagano, T.C., Hapuarachchi, H.A.P., 23. Evaluation of numerical weather prediction model precipitation forecasts for short-term streamflow forecasting purpose. Hydrol. Earth Syst. Sci., 7(5): Toth, Z., Talagrand, O., Candille, G., Zhu, Y., 23. Probability and Ensemble Forecasts. In: Jolliffe, I.T., Stephenson, D.B. (Eds.), Forecast Verification: A Practitioner's Guide in Atmospheric Science. J. Wiley, Chichester, pp Wang, Q.J., Robertson, D.E., 2a. A high level framework for ensemble flood and shortterm river flow forecasting system in Australia A discussion paper, CSIRO Water for a Healthy Country Flagship, Australia. Wang, Q.J., Robertson, D.E., 2b. Multisite probabilistic forecasting of seasonal flows for streams with zero value occurrences. Water Resour. Res., 47: W2546. Wang, Q.J., Robertson, D.E., Chiew, F.H.S., 29. A Bayesian joint probability modeling approach for seasonal forecasting of streamflows at multiple sites. Water Resour. Res., 45: W547. Wang, Q.J., Shrestha, D.L., Robertson, D.E., Pokhrel, P., 22. A log-sinh transformation for data normalization and variance stabilization. Water Resour. Res., 48: W554. Wilks, D.S., 26. Statistical methods in the atmospheric sciences. Academic Press, San Diego, California. Wu, L. et al., 2. Generation of ensemble precipitation forecast from single-valued quantitative precipitation forecast for hydrologic ensemble prediction. J. Hydrol., 399(3-4): Yeo, I.K., Johnson, R.A., 2. A new family of power transformations to improve normality or symmetry. Biometrika, 87(4): Post processing NWP rainfall forecasts

41 Appendix A (a) 3-hour STN MCL WLL CTT GBR MRR STE WTT ONK ORD (b) 6-hour (c) 24-hour Lead time (days) Improvement in % Bias Figure A.. Improvement in absolute bias from post-processed rainfall forecasts over raw NWP forecasts for 3 hour, 6 hour and 24 hour periods. Improvement in absolute bias is calculated by subtracting absolute bias for raw NWP forecasts from absolute bias of post-processed rainfall forecasts. Blue indicates that post-processed forecasts outperform raw NWP rainfall forecasts, while red indicates that raw NWP rainfall forecasts outperform the postprocessed rainfall forecasts (a) 3-hour (b) 6-hour (c) 24-hour STN MCL WLL CTT GBR MRR STE WTT ONK ORD Lead time (days) CRPS skill score (%) Figure A.2. CRPS skill score of post-processed rainfall forecasts over raw NWP forecasts for 3 hour, 6 hour and 24 hour periods. CRPS skill score of post-processed rainfall forecasts is computed with reference of CRPS score of the raw NWP forecasts. The dark blue indicates that post-processed forecasts significantly outperform raw NWP rainfall forecasts, while white indicates that post-processed forecasts marginally outperform raw NWP rainfall forecasts. The post-processed rainfall forecast is always better than the raw NWP forecast with respect to CRPS score Post processing NWP rainfall forecasts 3

42 STN MCL WLL CTT GBR MRR STE WTT ONK ORD STN MCL WLL CTT GBR MRR STE WTT ONK ORD STN MCL WLL CTT GBR MRR STE WTT ONK ORD (a) 3-hour (b) 6-hour (c) 24-hour 24-hour Precip > PoP 24-hour Precip > 25 th % 24-hour Precip > 5 th % STN MCL WLL CTT GBR MRR STE WTT ONK ORD Lead time (days) NaN Mean absolute deviation of reliability line 24-hour Precip > 75 th % Figure A.3. Average absolute deviation of reliability line from perfect reliability for 3 hour, 6 hour and 24 hour periods. The dark blue indicates that post-processed rainfall forecasts are not less reliable, while white indicates that post-processed forecasts are perfectly reliable. The black indicates that there are not enough data in some of probability bins to compute reliability diagrams. Note that these results are subjected to sampling variability 32 Post processing NWP rainfall forecasts

43 STN MCL WLL CTT GBR MRR STE WTT ONK ORD STN MCL WLL CTT GBR MRR STE WTT ONK ORD STN MCL WLL CTT GBR MRR STE WTT ONK ORD (a) 3-hour (b) 6-hour (c) 24-hour 24-hour Precip > PoP 24-hour Precip > 25 th % 24-hour Precip > 5 th % STN MCL WLL CTT GBR MRR STE WTT ONK ORD Lead time (days) Area under relative operating characteristics curve 24-hour Precip > 75 th % Figure A.4. Area under ROC curve for 3 hour, 6 hour and 24 hour periods. The dark blue indicates that postprocessed rainfall forecasts are better than climatology forecasts, while red indicates that post-processed forecasts are worse than climatology forecasts Post processing NWP rainfall forecasts 33