VERIFICATION OF PROBABILISTIC STREAMFLOW FORECASTS

Transcription

1 VERIFICATION OF PROBABILISTIC STREAMFLOW FORECASTS by Tempei Hashino, A. Allen Bradley, and Stuart. S. Schwartz Sponsored by National Oceanic and Atmospheric Administration (NOAA) No. NA86GP365 and No. NA6GP569 IIHR Report No. 427 IIHR-Hydroscience & Engineering and Department of Civil and Environmental Engineering The University of Iowa Iowa City IA August 22

2 ACKNOWLEDGMENTS This report basically constitutes the master thesis of Tempei Hashino. Funding for the research was provided by National Oceanic and Atmospheric Administration (NOAA) under the following grants: #NA86GP365 and #NA6GP 569. This support is gratefully acknowledged. i

3 EXECTIVE SUMMARY Long-range streamflow forecasts, such as the ensemble streamflow predictions (ESP) produced by the National Weather Service (NWS) Advanced Hydrologic Prediction Services (AHPS), are usually probabilistic forecasts. The format of the forecast is essentially a continuous probability distribution function, which predicts the likelihood of occurence of a streamflow variable, conditioned on the current hydroclimatic state. Although significant advances in forecast verification methodologies have been made in recent years, many of these approaches are not directly applicable to probabilistic streamflow forecasts. The main purposes of this research are () to extend the distributions-oriented (DO) approach to the verification of probability distribution forecasts of streamflow, and (2) to demonstrate the usefulness of the DO approach in assessing the quality of streamflow forecasts. Techniques for forecast verification using the DO approach are proposed and studied using probability distribution forecasts for an experimental forecasting system for the Upper Des Moines River basin. One significant obstacle in the verification of probabilistic streamflow forecasts is the small data sample available for verification. Verification sample sizes for long-range hydrologic forecasts are typically much smaller than those available for weather forecasts. Since verification with the DO approach is equivalent to estimation of the joint distribution of forecasts and observations, application of the DO approach to streamflow forecasts with small samples results in large estimation uncertainties. Three continuous statistical modeling approaches are considered that deal with estimation uncertainties by reducing the dimensionality D of the verification problem. Based on Monte Carlo experiments, the continuous approach with a logistic regression or kernel density estimation produces better estimates of forecast quality, especially with small sample sizes (say 5 or ), than the traditional discrete approach with a contingency table. Moreover, the continuous approaches work better whether the forecasts were issued in discrete or continuous numbers. A significant concern when using the ESP technique for streamflow forecasting is hydrologic model biases. The simulation biases of the hydrologic model propagate to the probability distribution forecasts through the ensemble traces produced ii

4 by the hydrological model, and could degrade the quality of the forecasts. Bias correction methods are often applied to try to reduce the effects of model biases. The impacts of three bias correction methods on streamflow forecast quality are examined using the DO techniques developed for streamflow forecast verification. The three bias correction methods examined are the Event-Bias Correction method (), the reression-type method, and the Quantile-mapping method (). The results showed that all bias correction methods improve skill scores, mostly by reducing the conditional bias (Reliability) and unconditional bias (Mean Error). It is remarkable that in some cases the bias correction methods also improve the association (potential skill) between forecasts and observations. The forecasts modified by tend to have the lowest sharpness and discrimination over all flow quantiles, whereas tends to give the highest sharpness and discrimination. The regressiontype methods seem to be in between of these. This application shows a strength of the proposed DO approach for probabilistic streamflow verification. Specifically, the approach produces detailed information on many aspects of forecast quality, which helps in determining the differences between alternate forecasting systems. iii

5 TABLE OF CONTENTS Page LIST OF TABLES vi LIST OF FIGURES viii CHAPTER INTRODUCTION FORECASTING SYSTEM Study Area and Data Resources Probabilistic Forecasting System Proposed Verification Approach Forecasts for a Discrete Event Verification Dataset Discussion Summary and Conclusions VERIFICATION APPROACH Introduction Distributions-Oriented Measures Bias Accuracy Calibration-Refinement Measures Likelihood-Base Rate Measures Estimation of Measures Basic Statistics Other Derivative Estimators Estimation of CR Decompositions Example of Verification Absolute and Relative Measures Marginal and Conditional Distributions Discussion Summary and Conclusions DISTRIBUTIONS-ORIENTED METHODS FOR SMALL VERIFICATION DATASET Introduction Monte Carlo Simulation with Analytical Model for Joint Distribution Assumptions and Procedure iv

6 4.2.2 Result and Discussion Monte Carlo Simulation with Stochastic Model of Streamflow Forecasting System Assumptions and Procedure Result and Discussion Monte Carlo Simulation with Discrete Joint Distribution Model Assumptions and Procedure Result and Discussion Summary and Conclusions ASSESSMENT OF BIAS CORRECTION METHODS FOR ENSEMBLE FORECASTS Introduction Biases in Historical Simulations Bias Correction Methods Event-Bias Correction Method Regression-Type Method Quantile-Mapping Method Result and Discussion Performance Measures CR Factorization and Decompositions LBR Factorization and Decompositions Results for All Months Summary and Conclusions SUMMARY AND CONCLUSIONS APPENDIX 6. Distributions-Oriented Methods for Small Verification Dataset Assessment of Bias Correction Methods for Ensemble Forecasts Future Study and Remarks A STATISTICAL METHODS A. Logistic Regression Method A.2 Kernel Density Estimation Method A.3 Combination Method A.4 Contingency Table Approach B SELECTED FIGURES AND TABLES REFERENCES v

7 LIST OF TABLES Table Page 2. Example of verification dataset for June-September volume forecasts Parameters of beta distributions for the analytical model and true forecast quality measures Root Mean Squared Error (RMSE) in MSE/σ 2 x, ME/σ x, TY2/σ 2 x, and DIS/σ 2 x for the forecasts generated for nonexceedance probability p =.25 by the analytical model Root Mean Squared Error (RMSE) in MSE/σ 2 x, ME/σ x, TY2/σ 2 x, and DIS/σ 2 x for the forecasts generated for nonexceedance probability p =.5 by the analytical model Root Mean Squared Error (RMSE) in REL/σ 2 x for the forecasts generated for nonexceedance probability p =.25 by the analytical model Root Mean Squared Error (RMSE) in RES/σ 2 x for the forecasts generated for nonexceedance probability p =.25 by the analytical model Root Mean Squared Error (RMSE) in REL/σ 2 x for the forecasts generated for nonexceedance probability p =.5 by the analytical model Root Mean Squared Error (RMSE) in RES/σ 2 x for the forecasts generated for nonexceedance probability p =.5 by the analytical model Parameters used in fitting the distribution to observed monthly volume (U) and the first three L-moments of the ensemble volumes (X l, X l2, and X l3 ) Summary statistics of the standardized random variables Root Mean Squared Error (RMSE) in REL/σ 2 x for the forecasts generated for nonexceedance probability p =.25 by the stochastic model Root Mean Squared Error (RMSE) in RES/σ 2 x for the forecasts generated for nonexceedance probability p =.25 by the stochastic model Root Mean Squared Error (RMSE) in REL/σ 2 x for the forecasts generated for nonexceedance probability p =.5 by the stochastic model Root Mean Squared Error (RMSE) in RES/σ 2 x for the forecasts generated for nonexceedance probability p =.5 by the stochastic model. 62 vi

8 4.4 Basic information and true forecast quality measures of Subjective 2-24-h Projection Probability-of-Precipitation Forecasts for United States during October 98-March 98 from Wilks (995) Root Mean Squared Error (RMSE) in REL/σ 2 x for the forecasts generated by the discrete model Root Mean Squared Error (RMSE) in RES/σ 2 x for the forecasts generated by the discrete model Mean, Standard Deviation (SD), and Coefficient of Variation (CV) of the observed monthly volume (cfsd) for the Des Moines River at Stratford Mean Error (ME), Root Mean Square Error (RMSE), correlation coefficient (CC), and Mean Square Error (MSE) Skill Score (SS MSE ) between the observed monthly volume and historical simulations.. 73 B. BIAS in REL/σ 2 x for the forecasts generated for nonexceedance probability p =.25 by the analytical model of joint distribution B.2 Standard Deviation in REL/σ 2 x for the forecasts generated for nonexceedance probability p =.25 by the analytical model of joint distribution B.3 BIAS in RES/σ 2 x for the forecasts generated for nonexceedance probability p =.25 by the analytical model of joint distribution B.4 Standard Deviation in RES/σ 2 x for the forecasts generated for nonexceedance probability p =.25 by the analytical model of joint distribution B.5 BIAS in REL/σ 2 x for the forecasts generated for nonexceedance probability p =.5 by the analytical model of joint distribution B.6 Standard Deviation in REL/σ 2 x for the forecasts generated for nonexceedance probability p =.5 by the analytical model of joint distribution B.7 BIAS in RES/σ 2 x for the forecasts generated for nonexceedance probability p =.5 by the analytical model of joint distribution B.8 Standard Deviation in RES/σ 2 x for the forecasts generated for nonexceedance probability p =.5 by the analytical model of joint distribution vii

9 LIST OF FIGURES Figure Page 2. Map of Des Moines River Basin Ensemble traces simulated for forecast on June Probability distribution forecast for June-September volume. The ensemble traces are simulated with the current conditions as of June Schematic of the current Extended Streamflow Prediction System. 3. Mean Error (ME) and Mean Square Error (MSE) for June-September seasonal volume forecasts CR (on left) and LBR (on right) decompositions of MSE for June- September seasonal volume forecasts Various decompositions of MSE Skill Score for June-September seasonal volume forecasts. The upper left indicates CR decompositions, Relative Resolution (RRES) and Relative Reliability (RREL). The upper right indicates LBR decompositions, Relative Discrimination (RDIS), Relative Sharpness (RS), and Relative Type 2 Conditional Bias (RTY2). The lower left shows Potential Skill, Reliability Measure, and Unconditional Bias Measure Reliability diagram for June-September seasonal volume forecasts issued for.25 quantile Discrimination diagram for June-September seasonal volume forecasts issued for.25 quantile MSE/σ 2 x, ME/σ x, TY2/σ 2 x, and DIS/σ 2 x estimated by two approaches for nonexceedance probability p =.25; D is discretized (-binned) approach (DSC), C represents a continuous approach such as LRM, KDM, and CM. The maximum, upper quartile, median, lower quartile, and minimum are indicated from top to bottom. The forecasts are produced by the analytical model MSE/σ 2 x, ME/σ x, TY2/σ 2 x, and DIS/σ 2 x estimated by two approaches for nonexceedance probability p =.5; D is discretized (-binned) approach (DSC), C represents a continuous approach such as LRM, KDM, and CM. The maximum, upper quartile, median, lower quartile, and minimum are indicated from top to bottom. The forecasts are produced by the analytical model viii

10 4.3 Conditional mean of the observations given the forecasts µ x f and marginal distribution of the forecasts s(f) estimated by three methods, DSC, LRM, and KDM, for nonexceedance probability p =.25 with a sample size 5. The forecasts are produced by the analytical model Conditional distribution of the forecasts given the observations r(f x) estimated by three methods, DSC, LRM, and KDM, for nonexceedance probability p =.25 with a sample size 5. The forecasts are produced by the analytical model Conditional mean of the observations given the forecasts µ x f and marginal distribution of forecasts s(f) estimated by three methods, DSC, LRM, and KDM, for nonexceedance probability p =.5 with a sample size 5. The forecasts are produced by the analytical model Conditional distribution of the forecasts given the observations r(f x) estimated by three methods, DSC, LRM, and KDM, for nonexceedance probability p =.5 with a sample size 5. The forecasts are produced by the analytical model CR decompositions estimated by four approaches for nonexceedance probability p =.25; D is discretized (-binned) approach (DSC), L is logistic regression (LRM), K is kernel density estimation directly applied to r(f x) (KDM), and C is combination of logistic regression and kernel density estimation (CM). The maximum, upper quartile, median, lower quartile, and minimum are indicated from top to bottom. The forecasts are produced by the analytical model CR decompositions estimated by 4 approaches for nonexceedance probability p =.5; D is discretized (-binned) approach (DSC), L is logistic regression (LRM), K is kernel density estimation directly applied to r(f x) (KDM), and C is combination of logistic regression and kernel density estimation (CM). The maximum, upper quartile, median, lower quartile, and minimum are indicated from top to bottom. The forecasts are produced by the analytical model Relations between observations and L-moments for September monthly volume Scatterplot of transformed observed monthly volume and transformed L-moments of monthly volume ensembles Conditional mean of the observations given the forecasts µ x f estimated by three methods, DSC, LRM, and KDM, for nonexceedance probability p =.25 with sample sizes 5 and. The forecasts are produced by the stochastic model Conditional mean of the observations given the forecasts µ x f estimated by three methods, DSC, LRM, and KDM, for nonexceedance probability p =.5 with sample sizes 5 and. The forecasts are produced by the stochastic model ix

11 4.3 CR decompositions estimated by four approaches for nonexceedance probability p =.25; D is discretized (-binned) approach (DSC), L is logistic regression (LRM), K is kernel density estimation directly applied to r(f x) (KDM), and C is combination of logistic regression and kernel density estimation (CM). The maximum, upper quartile, median, lower quartile, and minimum are indicated from top to bottom. The forecasts are produced by the stochastic model CR decompositions estimated by four approaches for nonexceedance probability p =.5; D is discretized (-binned) approach (DSC), L is logistic regression (LRM), K is kernel density estimation directly applied to r(f x) (KDM), and C is combination of logistic regression and kernel density estimation (CM). The maximum, upper quartile, median, lower quartile, and minimum are indicated from top to bottom. The forecasts are produced by the stochastic model True marginal and conditional distributions of the discrete forecasts Conditional mean of the observations given the forecasts µ x f estimated by three methods, DSC, LRM, and KDM, with sample sizes 5 and. The forecasts are produced by the discrete model CR decompositions estimated by four approaches for Discrete Forecast; D is discretized (2-binned) approach (DSC), L is logistic regression (LRM), K is kernel density estimation directly applied to r(f x) (KDM), and C is combination of logistic regression and kernel density estimation (CM). The maximum, upper quartile, median, lower quartile, and minimum are indicated from top to bottom. The forecasts are produced by the discrete model Example of Bias Correction Method applied to ensemble traces Comparison of observed monthly volume and historical simulation from January 988 to December Example of the bias-correction for -month lead time forecast with initial condition of January in 949; (Event-Bias Correction Method) is left and (Linear Interpolation) right Observed monthly volume versus simulated monthly volume with power function for May and September Observed monthly volume versus simulated monthly volume with LOWESS regression for May and September Example of the Quantile Mapping method () for -month lead time forecast with initial condition of January in MSE Skill Score (left) and Skill Score for Bias Correction (right) versus forecasted month for, 2, and 3-month lead times, averaged over the quantiles x

12 5.8 Skill Score for Bias Correction for May and September monthly volumes, averaged over the quantiles Comparison of Mean Error (left) and Mean Square Error (right) by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for, 2, and 3-month lead time September monthly volume forecasts Comparison of MSE Skill Score (left) and measure of association (right) by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for, 2, and 3-month lead time September monthly volume forecasts Comparison of Decompositions of Skill Score by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for, 2, and 3-month lead time September monthly volume forecasts. The measure of reliability is left, and the measure of unconditional bias is right Performance measures and decompositions of MSE Skill Score by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for, 2, and 3-month lead time May monthly volume forecasts Marginal distribution of the forecasts s(f) and the conditional mean of the forecasts µ x f by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for -month lead time September monthly volume forecasts CR decompositions by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for, 2, and 3-month lead time September monthly volume forecasts Conditional distributions of the forecasts r(f x = ) (left) and r(f x = ) (right) by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for -month lead time September monthly volume forecasts Conditional mean of the forecasts given the observations µ f x for five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for -month lead time September monthly volume forecasts xi

13 5.7 Conditional mean of the forecasts given the observations µ f x for and bias correction methods with. The forecasts were issued for September monthly volume with -month lead time. The three curves for each colour in the bottom two figures show µ f x=, µ f, and µ f x= from top to bottom Relative sharpness by five Bias Correction methods, actual (non biascorrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for, 2, and 3-month lead time September monthly volume forecasts LBR decompositions by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for, 2, 3-month lead time September monthly volume forecasts Mean Error and unconditional bias from decomposition of Skill Score by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (PS- S), for all the months with, 3, and 6-month lead times CR decompositions by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for all the months with, 3, and 6-month lead times LBR decompositions by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for all the months with, 3, and 6-month lead times Relative sharpness by five Bias Correction methods, actual (non biascorrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for all the months with, 3, and 6-month lead times MSE Skill Score and potential skill by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for all the months with, 3, and 6-month lead times A. Example of logistic regression applied to the pairs of forecasts and observations A.2 Unbounded estimation with biweight kernel A.3 Bounded estimation with floating boundary kernel A.4 Bounded estimation with biweight kernel and reflection boundary technique xii

14 A.5 Example of kernel density estimation method applied to forecasts to estimate the marginal distribution s(f) B. CR decompositions by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for -month lead time May monthly volume forecasts B.2 LBR decompositions by five Bias Correction methods, actual (non bias-corrected) streamflow simulation (), and pseudoperfect streamflow simulation (), for -month lead time May monthly volume forecasts xiii

15 CHAPTER INTRODUCTION After the devastating floods in 993 in the Midwest, the National Weather Service (NWS) proposed development of Advanced Hydrologic Prediction Services (AHPS) for streamflow forecasting. The first demonstration of AHPS system was carried out for the Des Moines River basin. AHPS produces short-range forecasts of the flood levels and the timing of flood crests. AHPS also produces long-range probabilistic streamflow forecasts. The forecast includes the chance (or probability) of exceeding minor, moderate, or major flooding, and the chance of exceeding certain water levels, volumes, and flows on the river over the next 9 days. These probabilistic forecasts are issued as probability distributions for streamflow, where streamflow is treated as a continuous random variable. Hence, they are called probability distribution forecasts, as opposed to more traditional probabilistic forecasts for discrete events. The probability distribution forecast AHPS produces has the advantage that users can obtain probabilistic forecasts for the events they are interested in. On the other hand, intuitively it is more difficult to evaluate probability distribution forecasts than categorized forecasts. This research defines forecast verification as the procedure to assess the degree of agreement between forecasts and observations, following Murphy and Daan (985). Forecast verification has traditionally been implemented using one or more verification measures (Murphy, 993). This approach fails to give a complete picture of the forecast quality for many kinds of forecasts, not to mention for the probability distribution forecasts. In the late 98s, Murphy and Winkler (987) proposed a general framework of forecast verification called the Distributions-Oriented (DO) approach. Based on the joint distribution of forecasts and observations, this approach unifies and imposes a structure on the verification methodology, provides insight into the relationships among verification measures, and creates a sound scientific basis to develop and/or choose particular verification measures in specific contexts (Murphy and Winkler, 987). The original DO approach assumes the forecasts and observations are expressed as discrete variables. Thus, the DO approach is not directly applicable to

16 2 probability distribution forecasts of continuous variables. The objectives of this research are () to extend the DO approach to the verification problem of probability distribution forecasts (or ensemble forecasts) of streamflow, and (2) to demonstrate its usefulness in assessing the quality of streamflow forecasts. In the application of the DO approach to streamflow forecasts, the major problem stems from the small sample size. For instance, in the case of meteorological forecasts, say, maximum daily temperature, 365 pairs of forecasts and observations would be available per year. After 5 years, 825 pairs could be utilized for verification. But if the forecast of interest is summer season flow volume, after 5 years, just 5 pairs would be available for verification. The DO approach outlined by Murphy (997) requires the construction of the joint distribution of forecasts and observations, where forecasts and observations are discrete random variables. With such a small sample, categorizing continuous probabilistic forecasts into discrete bins may not be appropriate to estimate the joint distribution, and the verification may lead to a distorted impression of forecasting system. In this research, an alternative approach which does not categorize the probabilistic forecasts is investigated. In order to demonstrate that the DO approach provides useful information to assess forecast quality, this research addresses the assessment of bias correction methods applied to ensemble forecasts. The forecasting system in this research is based on Extended Streamflow Prediction (ESP), which produces probabilistic forecasts by doing statistical analysis of ensemble traces. The ensemble traces are simulated by a hydrological model. However, in most cases the hydrological model may have some bias associated with its assumptions or input data. In practice, bias correction methods are utilized to correct the bias in simulations. However, it is not clear how the bias in ensemble traces propagates to the probabilistic forecasts, and how these methods improve the forecasts. The probabilistic forecasts modified with the bias correction methods are investigated by using the DO approach.

17 3 CHAPTER 2 FORECASTING SYSTEM An experimental forecasting system for the Des Moines River basin (Bradley and Schwartz, 2) is used to develop and test approaches for verification of probabilistic streamflow forecasts. Like the Advanced Hydrologic Prediction Services (AHPS) forecasts from the National Weather Service (NWS), the experimental forecasts are made using an ensemble forecasting technique. This chapter explains the study area and input datasets first, and then discusses how forecasts are made. An overview of approach used for verification is given along with the development of a verification dataset from the forecasts. 2. Study Area and Data Resources The study area is the Upper Des Moines River basin stretching from the southern part of Minnesota to central Iowa (Figure 2.). This research uses the discharge data obtained at Stratford, Iowa. The Des Moines River basin contains two major reservoirs, and the Upper Des Moines River is a main source of inflow into Saylorville Reservoir. This is why Stratford was chosen as the station for this research, since long-term forecasts of inflow into reservoir are important in reservoir operations. The drainage area of the Upper Des Moines River basin is about 4,2 km 2, and the elevation ranges from 29 to 58 m above mean sea level. The gently rolling terrain, formed by continental glaciation and subsequent erosion, supports extensive cultivated corn fields. The Upper Des Moines River has two main tributaries: the West Fork and the East Fork Des Moines River. The West Fork River has its origins in the glacial moraine area of Pipestone, Lyon, and Murray Counties, Minnesota at the elevation of about 58 m. The southeastward flow meets the East Fork, which flows southeasterly from Jackson County, Minnesota. The subbasins of the West and East Forks have many lakes, especially in Minnesota. The Upper Des Moines River passes through Fort Dodge, Iowa and joins the Boone River before Stratford. According to USGS NWISWeb Data for the Nation ( the daily mean streamflow, obtained from 82 years of records, varies from 5 to

18 4 6, cfs. The maximum peak streamflow of 42,3 cfs was recorded on 2 April 993. For more on the hydrological characteristics, see Bae and Georgakakos (992). The Hydrological Simulation Program-Fortran (HSPF) (Donigian et al., 984, and Bicknell et al., 997) was applied to the Upper Des Moines River basin, and the basin was modeled as a single lumped catchment. HSPF is a lumped hydrologic model that can simulate both watershed hydrology and water quality continuously. The time series of simulated streamflow is obtained by inputting a set of mean areal meteorological time series for the land segment. The input time series data consists of daily data of precipitation and potential evapotranspiration, and hourly data of air temperature, dew point temperature, wind movement, cloud cover, and solar radiation. For calibration, daily streamflow records were obtained at Stratford, Iowa, from U.S. Geological Survey (USGS). Precipitation and air temperature data obtained from the National Climatic Data Center were interpolated over the basin. The dew point temperature, wind movement, and cloud cover were obtained for three surface airways stations from National Center for Atmospheric Research (NCAR). The solar radiation and potential evapotranspiration time series were estimated based on the air temperature, dew point temperature, wind movement, and cloud cover data (see Shuttleworth, 993). HSPF model was calibrated with two objective functions at Stratford; the first one is the root mean squared error of the simulated and observed flows, and the second one is the root mean squared error of the logarithms of these flows. The two objective functions were evaluated, using weekly time step flows. To automate the calibration of HSPF model parameters, Shuffled Complex Evolution global optimization method (SCE-UA) was applied. 2.2 Probabilistic Forecasting System The experimental forecasting system implemented in this research is based on Extended Streamflow Prediction (ESP) (Day 985). ESP produces probabilistic forecasts by statistical analysis of different realizations in the future. This is the same concept as NWS uses in AHPS. To explain the basic idea of ESP, an example of streamflow forecasts is shown. Assume the present time is June st 965, and a forecast will be made of June- September flow volume. ESP assumes that historical meteorological time series represent possible realizations in the future. One streamflow trace is simulated by inputting each historical meteorological time series into HSPF, using the current

19 5 Figure 2.: Map of Des Moines River Basin. watershed conditions as the initial conditions. Since 48 years of historical record are available (from 948 to 996, excluding the current year), 48 streamflow traces are obtained (Figure 2.2). As June-September volume is of interest in this example, flow volumes are computed from the streamflow traces. Then, the cumulative distribution function of the ensemble traces is estimated by weighting each trace, using the method proposed by Smith et al. (992). Finally, the probability distribution forecasts are produced for June-September volume in terms of nonexceedance probability (Figure 2.3); for any value of the volume (threshold), the likelihood of the event whose volume is less than or equal to the threshold is obtained. 2.3 Proposed Verification Approach 2.3. Forecasts for a Discrete Event From the framework of ESP explained above, probability distribution forecasts are obtained in terms of nonexceedance probability. The mathematical definition

20 6 5 Des Moines at Stratford 45 4 Streamflow (cfs-days) Days (after June ) Figure 2.2: Ensemble traces simulated for forecast on June Des Moines River near Stratford, Iowa Volume (cfs-days) Nonexceedance Probability (%) Figure 2.3: Probability distribution forecast for June-September volume. The ensemble traces are simulated with the current conditions as of June 965.

21 7 of the forecasts is given as: G t (y) = P {Y y α t }, (2.) where P {Y y α t } is the probability that the forecast variable Y, for example monthly streamflow volume, is less than or equal to some threshold y, conditioned on the state of the hydroclimatic system α t at a certain forecast date t. Obviously, it is not straightforward to verify the forecasts in the form of G t (y). The following discusses the approach taken for the verification of the probability distribution forecasts. First, consider a discrete event Y y p where y p has the climatological nonexceedance probability p. The probabilistic forecast for the event Y y p is simply given as f(y p ) = G t (y p ). (2.2) Then, the corresponding observation can be discretized as: x(y p ) =, Y y p =, Y > y p. (2.3) Therefore, pairs of probabilistic forecasts f(y p ) and discrete observations x(y p ) are obtained for the discrete event Y y p. The pairs that are used to estimate the joint distribution of f(y p ) and x(y p ) are called a verification dataset. From one verification dataset, one set of measures of forecast quality for a threshold y p is computed. To evaluate the quality of probability distribution forecast over the range of possible outcomes, this research uses nine thresholds y p with p =.5,.,.25,.33,.5,.66,.75,.9,.95. Hence, nine verification datasets are obtained Verification Dataset One verification dataset for threshold y p is made up of the N pairs of forecasts f(y p ) and observations x(y p ). It is important to note that this verification dataset contains a portion of information needed to obtain a complete picture of the forecast quality. Table 2. shows the example of verification dataset for June-September volume forecasts. The forecasts are issued on June st every year for the event Y y p with the threshold y p = (p =.66). According to the probability distribution forecast shown in Figure 2.3, the probabilistic forecast on June st in 965 for the event is f(y p ) = The volume observed from June to September in is , which is slightly greater than y p. Therefore, the corresponding discrete observation x(y p ) is equal to, which indicates that the event did not occur.

22 8 Table 2.: Example of verification dataset for June- September volume forecasts. Pairs Obs. c (cfs-d) Date of Forecast a f(y p ) x(y p ) b Y 949/6/ /6/ /6/ /6/ /6/ : : : : 964/6/ /6/ /6/ /6/ : : : : a The forecasts were issued on June st every year. b The threshold for forecasts, y p = is.66 quantile. c Obs. is observed June-September volume. 2.4 Discussion More research on forecasts using ensembles has been done in the meteorological field. In the meteorological field, ensemble forecasts are often called an Ensemble Prediction System (EPS), whereas they are called Extended Streamflow Prediction (ESP) in the hydrological field. The main difference between current EPS and ESP is that the ensemble traces in meteorological and hydrological fields are produced in the different ways. Figure 2.4 shows the current version of ESP used in NWS. NWS has extended the original idea to facilitate incorporation of climate outlooks into the ESP (Perica, 998). The NWS ESP program produces ensemble traces

23 9 by inputting historical meteorological events adjusted with meteorological and climatological forecasts, and deterministic precipitation forecasts. Another way to incorporate climate outlooks and meteorology probabilistic forecasts is to adjust weights for ensemble traces simulated with historical meteorological events (Croley, 2). More investigations on incorporation of climate and meteorology forecasts into ESP are needed. On the other hand, in meteorological research, the ensembles of geopotential heights, temperatures, or moisture are created from slightly different initial conditions. The main methods to generate the initial conditions of the ensemble members are () Monte Carlo methods, (2) methods which generate perturbations dynamically constrained by the flow of the day, including breeding and singular vectors, (3) the perturbed observations method which uses data assimilation cycles with random errors, and (4) methods which make perturbations by varying the model parameterizations of subgrid-scale physical processes (Hou et al. 998). These ensembles of meteorological variables could be directly input into a hydrological model to produce an ensemble of streamflow. As mentioned in Chapter, AHPS provides probabilistic forecasts which indicate the exceedance probability of certain levels over the next 9 days. In the meteorological field, ensemble traces have been utilized mainly in the following four ways (Anderson 996): () use the ensemble mean forecast as a substitute for a single discrete forecast; (2) produce a small, easily understood set of forecast states by clustering algorithms; (3) make a priori predictions of forecast skill, that is, figure out the relation between ensemble spread and skill of the control forecast; and (4) examine the entire ensemble to extract as much information as possible. For example, the quantitative precipitation forecasts are given by exceedance probability for some continuous thresholds. In fact, this is the same method as AHPS utilizes. Nowadays, more efforts have been put into item (4). As we see, many aspects of ensemble forecasting are common in meteorological and hydrological fields. The cooperation between the researchers in these fields is necessary to improve ensemble forecasts of streamflow. 2.5 Summary and Conclusions This research utilizes an experimental forecasting system that has been developed for the Upper Des Moines River basin. The discharge at Stratford, Iowa, was

24 EXTENDED (ENSEMBLE) STREAMFLOW PREDICTION PROCEDURE Historical series of precipitation and temperature 948 P Meteorological forecasts/climate outlooks: - - to 5-day - 6- to -day - monthly outlook month outlooks Adjusted series of precipitation and temperature 948 P Current conditions: - snow pack - soil moisture - streamflow - reservoir levels Q Streamflow traces Streamflow traces T NWS adjustment procedure T NWS hydrologic Model P T time 995 P T time Precipitation deterministic Forecast 24(48) hours time % time % % Figure 2.4: Schematic of the current Extended Streamflow Prediction System. Source: Perica, Sanja, Integration of Meteorological Forecasts/Climate Outlooks Into Extended Streamflow Prediction (ESP) System, rl/papers/ams/ams98-6.htm, (accessed March, 998).

25 chosen since long-term forecasts of inflow into the downstream reservoir are important for operations. The Upper Des Moines River basin drains about 4,2 km 2, and the gently rolling terrain was formed by continental glaciation and subsequent erosion. In 993, the record-breaking peak streamflow of 42,3 cfs was observed. The Hydrological Simulation Program-Fortran (HSPF), which is a lumped hydrologic model, was applied to the Upper Des Moines River basin. Sets of mean areal time series data, such as daily precipitation, potential evapotranspiration, hourly data of air temperature, and so on, were produced from various sources. HSPF model was calibrated with two objective functions at Stratford, and the optimum parameters were automatically obtained by Shuffled Complex Evolution global optimization method (SCE-UA). The experimental forecasting system is based on the idea of Extended Streamflow Prediction (ESP). The time series of historical meteorological information obtained were inputted into the HSPF model, and then streamflow was simulated with the current hydroclimatological conditions on a forecast date. The outputs of streamflow are assumed to be different realizations in the future, which are called ensemble traces. Finally, the probabilistic distribution forecast, for the forecast date expressed in nonexceedance probability, was produced by statistical analysis of the ensemble traces. The problem is to verify the forecast for continuous range of streamflows, since the probability distribution forecast gives a probabilistic forecast for any possible outcome. One solution is to consider a discrete event that a forecast variable is less than or equal to a threshold. For the event, one probabilistic forecast is derived from the probability distribution forecast in terms of nonexceedance probability. On the other hand, the corresponding continuous observation is converted into or ; indicates the event did not occur, and means the event occurred. This pair of probabilistic forecast and discrete observation is obtained every forecast date. Thus, one verification dataset for a threshold is made up of as many pairs of forecasts and observations as forecast date. Since nine quantiles of observations are used as the thresholds covering the possible outcomes, nine verification datasets are computed. Investigation of these nine verification datasets can be considered equivalent to examination of forecast quality of the probability distribution forecast.

26 2 CHAPTER 3 VERIFICATION APPROACH The proposed approach for verification of ensemble streamflow predictions involves selecting discrete events. The probabilistic forecast for an event a forecast variable is less than or equal to a threshold is obtained from the probability distribution forecast. The corresponding continuous observation is also converted into a discrete number: indicates that the event occurred, means that the event did not occur. The verification dataset for the event consists of the pairs of probabilistic forecasts and discrete observations. Using the verification datasets derived for discrete events, forecast quality of the probability distribution forecast can be assessed over the range of possible outcomes. In this chapter, a distributions-oriented (DO) approach for forecast verification is described. The DO approach is extended to case of continuous probabilistic forecasts with parametric and nonparametric techniques to estimate the joint distribution of forecasts and observations. Secondly, the technical methods are described in detail. Then, DO measures and other common measures for forecast verification are discussed. The technical methods in the extended DO approach will be assessed in the next chapter. 3. Introduction Verification procedures can be classified into two categories (Murphy, 997): a measures-oriented (MO) approach and a distributions-oriented (DO) approach. The MO approach is traditionally used in the verification process. Literally, this approach emphasizes calculating quantitative measures of only one or two aspects of forecasting quality such as bias, accuracy, or skill, and then makes conclusions based on these measures. In most cases, the mean squared error (hereafter referred to as MSE), and the correlation coefficient (CC) are used as the accuracy measure. However, CC is shown to be a measure of potential skill by Murphy et al. (989). Although many verification measures had been developed, until the 98 s the investigation of the relationships between measures, examination of their relative strengths and weaknesses, or general concepts about verification itself had

27 3 not been studied extensively (Murphy and Winkler, 987). For example, Barnston (992) showed the nonlinear, one-to-one relationship between CC and RMSE for standardized forecasts and observations, and the significant variation of the mean correspondence between CC and Heidke score with the number of equally likely Heidke categories. Murphy (995) concluded that the coefficient of determination is superior to the CC as the measure of linear association, and both of them are not proper as the measure of skill. The DO approach was developed in the 98 s. Since then, the DO approach has played an important role, especially in the verification of meteorological forecasts. For instance, the diagnostic verification of Climate Prediction Center Long- Lead Outlooks has been done with this approach (Wilks 2). The forecasts made by human forecasters and guidance products from numerical weather prediction models were investigated (Brooks and Doswell, 996). Also, the verification of the forecasts produced based on the Ensemble Prediction System (EPS) has been done (e.g., Hamill and colucci 997, and Hou et al. 998). The DO approach involves the use of the joint distribution of forecasts and observations, from which all the measures of forecast quality are derived systematically. The reason why the DO approach is preferable is that it gives insights on forecast quality from various aspects and allows the user to identify the situations in which forecast performance may be weak or strong, something the MO approach fails to do (Brooks and Doswell III, 996). The major difficulty in applying this approach to verification stems from the estimation of the joint distribution. Two fundamental characteristics of the verification problem are complexity and dimensionality, which are quantitatively defined by Murphy (99). Complexity is defined by the number of factorizations (C F ), number of basic factors in each factorization (C BF ), or total number of basic factors (C T BF ). For example, in Absolute Verification (AV), where one kind of observation and one forecasting system are examined, the joint distribution can be factorized into one conditional and one marginal distribution. Thus, C F = 2, C BF = 2, and C T BF = 4. On the other hand, the general definition of dimensionality D is that D is the number of degrees of freedom in order to estimate the joint distribution of forecasts and observations. In the case where a forecasting system uses n x categories for observations and n f for forecasts, the dimensionality D is defined as D = n f n x. (3.)

28 4 For example, when a forecast is issued in categories from to with. interval for a dichotomous (two-category) observation, the verification problem has the dimension D = 2 = 2. Then, given 5 pairs of forecasts and observations, which is not unusual with hydrological variables, it is no wonder that some bins may not have enough (or any) subsamples to estimate the joint distribution. Hence, most verification problems suffer from the curse of dimensionality (Murphy, 997). In Chapter 4, techniques to reduce the dimensionality (but not the complexity) are investigated. This chapter describes the measures of the DO approach tailored to probabilistic forecasts and dichotomous observations, and their estimators, in such a way that the dimensionality of the joint distribution is reduced. As a result, all of the DO measures are derived from six basic variables and one integral. 3.2 Distributions-Oriented Measures One can derive the distributions-oriented (DO) measures from the joint distribution of forecasts and observations p(f, x),where f is the probabilistic forecast issued for an event that forecast variable Y is equal to or less than a threshold y p (f = f(y p )), and x is the corresponding discrete observation (x = x(y p )), which takes on for occurrence of the event and for no occurrence. The measures derived from the joint distribution can be examined over the range of thresholds for which the forecasts are issued. To cast light on understanding of the joint distribution from various aspects, it can be factorized into one conditional and one marginal distribution in two ways (Murphy and Winkler, 987): CR factorization: p(f, x) = q(x f)s(f) (3.2) LBR factorization: p(f, x) = r(f x)t(x). (3.3) The calibration refinement (CR) factorization is used more often, and easy to understand partly because a forecast is issued first, then the observation is compared (Murphy and Winkler 987, Brooks and Doswell III 996). On the other hand, it is easier to reconstruct the marginal and conditional distribution for the likelihoodbased rate (LBR) factorization, since the observation random variable x takes on or only. This research mainly utilizes the following measures described in Murphy (997).