No part of this presentation may be copied, reproduced, or otherwise utilized without permission. WRF Webcast Improving the Accuracy of Short-Term Water Demand Forecasts August 29, 2017
Presenters Maureen Hodgins, Research Manager Water Research Foundation Thomas M. Fullerton, Jr., Ph.D. Professor & Trade in the Americas Chair University of Texas at El Paso (UTEP) David Torres, M.S. Economist El Paso Water Adam Walke, M.S. Economist UTEP Border Region Modeling Project
Webcast Agenda Introductions Maureen Hodgins 5 min Overview Tom Fullerton 5 min Key Research Findings Adam Walke 30 min El Paso Water Modeling Efforts David Torres 5 min Question & Answers 15 min
WRF Research Focus Area - Demand 2012 to 2018/19 10 projects 8 funded Forecasting
Demand Forecasting Summary
Water use estimates -Residential end uses, 4309 (2016) -Multi-family, 4554 (est 2018) -CII, 4375 (2015) & 4619 (est 2019) Forecasting methods -Customer data, 4527 (2016) -Short term, 4501, (2017) -Long term, 4667 (est 2020) Factors impacting demand -Recession, 4458 (2016) -United Kingdom - Behavioral changes, 4649 (2016) -Passive efficiency, 4495 (est 2018) Planning with uncertainty -Uncertainty & long term forecasts, 4558 (2016) Urban Landscapes -Irrigation controllers, 4227 (2016) -Urban landscape research needs (est 2017) Sizing Infrastructure -Demand patterns for sizing meters & service lines, 4689 (est 2018)
Finding the 4501 Products
4501 Products
Please Type Your Question Here! Slides and recording will be available to WRF subscribers WITHIN 24 hours after the webcast!
Agenda Overview of project 4501 Key findings: Survey of water utility planners & forecasters Short-term water demand forecasting manual Final Report: Accuracy of utility-generated forecasts Forecasting case study: El Paso Water Brief overview of add-on case studies El Paso Water modeling efforts Questions & answers
Acknowledgements Project Advisory Committee: Jack Kiefer (Hazen and Sawyer) Chris Meenan (Las Vegas Valley Water District) Paul Merchant (South West Water) Graduate Research Assistants: Juan P. Cardenas Alejandro Ceballos Omar Solis
Acknowledgements Water Research Foundation El Paso Water City of Phoenix Tampa Bay Water Three anonymous utilities Hunt Communities
Survey Targeted to utility staff involved in shortterm planning and forecasting 198 unique survey responses 42 states represented 70% of respondents use water demand forecasts - only those respondents answered the subsequent questions
Approximately how often are forecasts generated or updated? Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation.
If forecasts are generated for different water demand scenarios, what types of scenarios are considered? Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation.
What types of forecasting methodologies are used to generate your utility's forecasts? Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation.
Forecasting Manual Organized around:
Factors to Consider in Choosing a Forecasting Methodology Accuracy track record of the methodology Cost of implementing the methodology Data and computational requirements Organizational goals in forecasting Prediction only versus analyzing scenarios & alternative policies Forecast horizon (short- medium- or longterm forecasts)
Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation.
Factors to Consider in Developing a Forecast Sample size & handling of missing data Time between forecasts & vintage of data Which demand variable to forecast: Demand in each customer category? Demand in each part of the service area? Separate forecasts for the customer base and water demand per customer? Which (if any) predictor variables to use
Predictor variables used in 53 published studies for short-term water demand forecasting Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation.
Issues to Consider in Evaluating a Forecast Evaluation serves to: Assess whether previous methods are working Choose models with good chances of success Good compared to what? Use benchmarks Evaluation criteria Forecast error summary statistics Tests of forecast error differentials Tests of directional forecast accuracy
Final Report Analysis of utility-generated forecasts: Tampa Bay Water City of Phoenix Three anonymous utilities Analysis of forecasts developed by the research team with utility data: El Paso Water City of Phoenix One anonymous utility
Analysis of Utility-Generated Forecasts Accuracy analyses were conducted for 5 sets of utility-generated forecasts; in one case there were not enough observations for statistical tests of forecast accuracy. Utility forecasts were compared to random walk benchmark forecasts. Forecasts were grouped in different ways (e.g. by step-length, by geography, by customer class).
Utility Forecast Accuracy Summary: Root Mean Squared Error (RMSE) # Horizon (Frequency) Methodology Utility % Better 1 2 Weeks (Daily) 2 1 Week (Weekly) 3 1 Year (Monthly) 4 2 Years (Monthly) 5 4 Years (Annual) Regression with weather and demographic explanatory variables and lagged demand Regression with weather explanatory variables and lagged demand Econometric model including price, weather, employment, unemployment rate, and lagged demand Expert judgment taking into account climatic & economic conditions End-use model based on survey data, and data on demographic trends, prices, and conservation policies 71% 100% 75% Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation. 0% 0%
Utility Forecast Accuracy Summary: Error Differential Regression Test # Horizon (Frequency) Methodology Utility % Significant 1 2 Weeks (Daily) 2 1 Week (Weekly) 3 1 Year (Monthly) 4 2 Years (Monthly) 5 4 Years (Annual) Regression with weather and demographic explanatory variables and lagged demand Regression with weather explanatory variables and lagged demand Econometric model including price, weather, employment, unemployment rate, and lagged demand Expert judgment taking into account climatic & economic conditions End-use model based on survey data, and data on demographic trends, prices, and conservation policies 29% 83% 50% Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation. 0% NA
Utility Forecast Accuracy Summary: Chi-Square Test of Independence # Horizon (Frequency) Methodology Utility % Significant 1 2 Weeks (Daily) 2 1 Week (Weekly) 3 1 Year (Monthly) 4 2 Years (Monthly) 5 4 Years (Annual) Regression with weather and demographic explanatory variables and lagged demand Regression with weather explanatory variables and lagged demand Econometric model including price, weather, employment, unemployment rate, and lagged demand Expert judgment taking into account climatic & economic conditions End-use model based on survey data, and data on demographic trends, prices, and conservation policies 57% 78% 75% Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation. 0% NA
Characteristics of Successful Utility-Generated Forecasts Combine statistical methods like regression with expert judgment Use time series data on key predictor variables like weather, prices, and economic indicators In one case, forecasts are generated without forecasting predictor variables All successful models harness the predictive power of lagged demand
El Paso Case Study El Paso Water serves El Paso, Texas Located in the Chihuahuan Desert Historically faced water supply constraints Adopted a comprehensive water conservation strategy in 1991
Total Water Consumption in El Paso: 1959-2015 Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation.
Methodology I: Linear Transfer Function ARIMA In a Linear Transfer Function (LTF) approach, demand is first modeled as a function of explanatory variable lags. Any remaining unexplained systematic variation in demand is then modeled using autoregressive (AR) and moving average (MA) parameters.
Methodology II: Vector Autoregression In the most basic version of the Vector Autoregression (VAR) approach, each variable is modeled as a function of lags of itself and lags of all of the other variables. Instead of choosing specific lags of each variable to include in the model, the analyst only has to select one lag order for all of the variables.
Methodology III: Random Walk A simple random walk (RW) forecast posits no change in demand (forecast equals actual demand one period earlier): F t = A t-1 A random walk with drift (RWD) simply adds the average annual change in demand to the random walk: F t = A t-1 + d In the case of highly seasonal variables measured at a monthly frequency, a random walk can be defined as: F t = A t-12
Data (January 1994 December 2013) From El Paso Water: Total Demand = Per-Customer Demand Customer Base Real Average Price = Total Revenues/(Total Demand CPI) From the Bureau of Labor Statistics: Nonfarm Employment in El Paso County From the National Oceanic and Atmospheric Administration: Days per Month with Temperatures above 90 F Total Monthly Precipitation in Inches The customer base is first-differenced; the other variables are first- and twelfth-differenced.
Estimation Results LTF and VAR methods can provide insights into the relationships between variables: Slope coefficients indicate how demand changes when explanatory variables change. In the case of the VAR model, there is a very large number of coefficients, so an alternative means of deciphering relationships is desirable. An impulse response function is a convenient way of depicting the reactions of endogenous variables to shocks in other endogenous variables.
LTF Estimation Results Per-Customer Usage Customers Constant 0.002 345.198** Average Price t-3-2.899** Days over 90 F t 0.089** Days over 90 F t-1 0.131** Rainfall t-1-0.396** Nonfarm Employment t 0.121** Nonfarm Employment t-32 42.440** AR t-12-0.295** 0.369** AR t-18-0.175** MA t-1-0.706** -0.299** MA t-3 0.157* MA t-12-0.265** R-Squared 0.702 0.237 F-Statistic 59.808** 10.602** * Probability value <.05; ** Probability value <.01 Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation.
Elasticity Estimates The estimated elasticity of demand with respect to price is -0.32. This suggests that a 10% price increase would result in a 3.2% decline in demand.
Change in Deseasonalized Per Capita Demand (Thousands of Gallons per Month) Per-Customer Demand: VAR Impulse Response Function 1.5 1.0 0.5 0.0-0.5-1.0 1 2 3 4 5 6 7 8 9 10 The figure shows the response of demand to a one standard deviation shock in the residual of price Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation.
Forecast Evaluation Criteria Criteria based on the size of the forecast errors: Root Mean Squared Error Forecast error differential regression test H 0 : Both sets of forecasts are equally accurate Criteria for evaluating directional accuracy: Chi-square test H 0 : Forecasted and actual events are independent (i.e. forecasts don t provide useful information for predicting the direction of change)
Most Accurate Forecast for El Paso based on Root Mean Squared Error, 2011-2013 Step-length Per-customer demand Customer base 1 month LTF VAR 2 months LTF LTF 3 months LTF LTF 4 months LTF RWD 5 months LTF RWD 6 months LTF RWD 7 months LTF RWD 8 months LTF RWD 9 months LTF VAR 10 months LTF VAR 11 months LTF VAR 12 months LTF VAR Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation.
Are the LTF forecasts significantly better than the VAR forecasts? Step-length Per-customer demand Customer base 1 month no no 2 months no no 3 months yes no 4 months yes no 5 months yes no 6 months yes no 7 months yes no 8 months yes no 9 months yes no 10 months yes no 11 months yes no 12 months yes no Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation.
Are the LTF forecasts significantly better than the Random Walk with Drift forecasts? Step-length Per-customer demand Customer base 1 month yes no 2 months yes no 3 months yes no 4 months yes no 5 months yes no 6 months yes no 7 months yes no 8 months yes no 9 months yes no 10 months yes yes 11 months yes yes 12 months yes yes Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation.
Do the forecasts provide useful information on directional changes in per-customer demand? Step-length LTF VAR 1 month yes yes 2 months yes yes 3 months yes yes 4 months yes no 5 months yes yes 6 months yes yes 7 months yes yes 8 months yes yes 9 months yes yes 10 months yes no 11 months yes no 12 months yes no Source: Fullerton, T.F., Jr. and Walke, A.G. (2017) Improving the Accuracy of Short-Term Water Demand Forecasts. Denver, Colo.: Water Research Foundation.
Combining Forecasts If separate forecasts each contribute complementary information that is useful for prediction, it may make sense to combine the forecasts as a strategy for improving accuracy. Forecasts can be combined by assigning a separate weight to each one based on it s relative accuracy. Demand = b 0 + b 1 LTF + b 2 VAR + b 3 RWD + e
Per-Customer Demand: Combined Results 2008-2013 Variable Coefficient t-statistic Probability Constant 1.5737 3.0507 0.0033 LTF 0.5893 6.4515 0.0000 VAR 0.2957 3.0559 0.0033 RWD 0.0259 0.3905 0.6974 AR t-1 0.3035 2.4850 0.0155 MA t-3 0.4215 3.6689 0.0005 R-squared 0.9732 Durbin-Watson 2.0305 F-statistic 472.8418 Probability (F-stat) 0.0000
Customer Base: Combined Results 2008-2013 Variable Coefficient t-statistic Probability Constant 1,446.399 1.4732 0.1454 LTF 0.0749 0.5023 0.6171 VAR 0.0360 0.2256 0.8222 RWD 0.8817 20.2273 0.0000 MA t-5-0.3338-2.6728 0.0094 R-squared 0.9985 Durbin-Watson 2.0701 F-statistic 11,119.20 Probability (F-stat) 0.0000
Conclusions of El Paso Case Study The LTF approach offers an improvement over the alternatives considered in forecasting percustomer water demand for El Paso. Random walks are competitive in forecasting the number of customers. Experimentation with alternative forecasting methods and comparisons of forecast accuracy can help inform decisions about what is the best forecasting approach to use in a given context.
Results of Add-On Case Study I Are there benefits to separately modeling and forecasting each sub-component of total water demand? Data from the City of Phoenix were analyzed. Disaggregation by customer category is the best alternative for a majority of the steplengths considered, but it is not significantly better than directly modeling aggregate demand for this sample.
Results of Add-On Case Study II Can proxies serve as predictor variables in place of price data when the latter are unavailable? Data from an anonymous utility were analyzed to address this question. During a period when water rates were changing significantly, proxy variables were poor substitutes for actual price data in forecasting water demand.
Overarching Conclusion Forecast evaluation is not the final word in the forecasting process but a tool for continual reassessment and improvement of prediction strategies.
Questions?
Thank You Comments or questions, please contact: mhodgins@waterrf.org tomf@utep.edu agwalke@utep.edu david.torres@epwu.org For more information visit: www.waterrf.org