TRANSIT FORECASTING UNCERTAINTY & ACCURACY DAVID SCHMITT, AICP WITH VERY SPECIAL THANKS TO HONGBO CHI

Transcription

1 TRANSIT FORECASTING UNCERTAINTY & ACCURACY DAVID SCHMITT, AICP WITH VERY SPECIAL THANKS TO HONGBO CHI November 18, 2016

2 TOPICS Motivation Database Historical Transit Demand Forecasting Accuracy Impact of Forecasting Assumptions Freely-Available Materials Reference Class Forecasting Some Other Accuracy-Related Projects

3 MOTIVATION Empirical observations (by others): Large demand inaccuracies from large transit projects (Flyvbjerg, FTA, and others) Forecasting accuracy for large-scale transportation projects is not improving over time (Flyvbjerg [worldwide] and TRB [USA toll roads]) Empirical observations (by the author): Assessing uncertainty is not standard practice Absence of documenting uncertainty & risk in practice Lack of knowledge about forecast accuracy Given historical inaccuracy, need exists to improve & promote better assessment of forecast uncertainty and risk

4 TRANSIT FORECASTING ACCURACY DATABASE Developed to: Quantify and track industrywide accuracy trends; Quantify and track the accuracy of upstream assumptions and exogenous forecasts; and Provide empirical data to support the implementation of reference class forecasting, quality control and due diligence practices in the United States 82 large-scale transit projects Project description and characteristics (city, length, # stations, CBD/non-CBD, mode) Tracks differences in forecasted/actual values of 10 project assumptions and exogenous forecasts Forecasted ridership (year of forecast, forecast year, value) Observed ridership (year of observation, value) Allows for multiple records of forecasted and observed ridership

5 TRANSIT FORECASTING ACCURACY DATABASE: PROJECTS BY MODE & DECADE OF OPENING Mode Total Bus Bus Rapid Transit (BRT) Commuter Rail Streetcar/Trolley Urban Heavy Rail Urban Light Rail Downtown People Mover (DPM) Total total records of forecasted ridership (mean= 1.7 per project) 307 total records of observed ridership (mean= 3.7 per project)

6 COMPUTING ACCURACY Accuracy i = AAAAAAAAAAAAAAAAAAAAAAAAAAAAA ii FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF ii Accuracy: actual / forecasted ridership forecasted > actual ridership (strongly over-forecast) forecasted > actual ridership (over-forecast) = 1.00 forecasted matches actual ridership forecasted < actual ridership (under-forecast) forecasted < actual ridership (strongly under-forecast) For all projects in database: N= accuracy (avg) For Florida projects in database: N= accuracy (avg) 6

7 7

8 Enjoying the Outside View David Schmitt May 19, 2015 Pag e 8

9 Reference Class Groups 1 Time Period Hypothesis More recent projects are more accurate and more appropriate reference class Result More recent projects (2007-present) are, on average, more accurate than less recent projects ( )

10 Reference Class Groups 1 Time Period 2 Mode Hypothesis More recent projects are more accurate and more appropriate reference class Tested Downtown People Movers (DPM), Bus/BRT, Light Rail, Heavy Rail, & Commuter Rail Result More recent projects (2007-present) are, on average, more accurate than less recent projects ( ) Light rail (better) and DPMs (worse) projects have statistically significant differences in average accuracy

11 Reference Class Groups 1 Time Period 2 Mode 3 Project Development Phase Hypothesis More recent projects are more accurate and more appropriate reference class Tested Downtown People Movers (DPM), Bus/BRT, Light Rail, Heavy Rail, & Commuter Rail Forecasts are more accurate in later stages of project development: Planning Engineering Full Funding Grant Agreement Result More recent projects (2007-present) are, on average, more accurate than less recent projects ( ) Light rail (better) and DPMs (worse) projects have statistically significant differences in average accuracy No statistically significant difference in forecast accuracy between any two project phases

12 Reference Class Groups 1 Time Period 2 Mode 3 Project Development Phase Hypothesis More recent projects are more accurate and more appropriate reference class Tested Downtown People Movers (DPM), Bus/BRT, Light Rail, Heavy Rail, & Commuter Rail Forecasts are more accurate in later stages of project development: Planning Engineering Full Funding Grant Agreement Result More recent projects (2007-present) are, on average, more accurate than less recent projects ( ) Light rail (better) and DPMs (worse) projects have statistically significant differences in average accuracy No statistically significant difference in forecast accuracy between any two project phases 4 Impact to Transit System Smaller changes to transit system are easier to predict (more accurate) than larger changes: 1 st rail mode (largest) new line extension (smallest) Transit Forecasting Accuracy Database: Enjoying the Outside View No statistically significant difference between projects with small or large changes David Schmitt May 19, 2015 Pag e 12

13 PROJECT ASSUMPTIONS & EXOGENOUS FORECASTS Examples: Project characteristics (level of service, travel time, fare) Transit system (supporting and competing networks) Roadway system (level of congestion) Demographics (population, employment estimates) External conditions (economic, auto fuel prices) Provided to transit forecasters, and typically accepted without review Are these assumptions and forecasts biased? If they are biased, how much impact do they have on the demand forecast inaccuracy?

14 HISTORICAL (IN)ACCURACY OF PROJECT ASSUMPTIONS & EXOGENOUS FORECASTS Filled cells represent highest proportion of each row Significant optimism bias in assumptions & forecasts, which increases risk of ridership forecasting inaccuracy 14

15 HOW MUCH DO INACCURATE ASSUMPTIONS CONTRIBUTE TO DEMAND INACCURACY? Process Quantify level of inaccuracy for each assumption Compute change in forecasted demand by apply elasticity to corrected assumption Re-estimate transit demand forecast Re-compute forecast accuracy

16 PROPOSED ELASTICITIES Project Assumption Elasticity Unit Source Project Service Levels (desired unit: veh/hr) Employment Estimates (desired unit: percentage error in jobs) Project Travel Time (desired: travel time) Population Estimates (desired unit: percentage error in population) Project Fare (desired: transit fare in cents) Supporting transit network (desired: veh/hr) Economic Conditions (desired: unknown) Competing transit network (desired: veh/hr) Auto Fuel Price (desired: price/gallon) Roadway Congestion (desired: index or qualitative level) Average transit ridership growth in USA (desired: hisotircal ridership by city/region) -0.2 Rail Headway Patronage Impacts of Changes in Transit Fares and Services (Pickrell Report) 0.5 Service Quantity TCRP, Chapter 9, Page Percentage error in Patronage Impacts of Changes in Transit Fares and Services employment (Pickrell Report) -0.3 Rail Operating Speed Patronage Impacts of Changes in Transit Fares and Services (Pickrell Report) 1.0 Percentage error in population -0.3 Rail Fare Rail Fare 0.4 Feeder Bus Headway 0.1 Relative change in unemployment rate between YOF and YOO -0.4 Bus Headway 0.1 Auto Operating Cost (cents) 0.34 Fuel Price 0.25 Fuel Price Patronage Impacts of Changes in Transit Fares and Services (Pickrell Report) Patronage Impacts of Changes in Transit Fares and Services (Pickrell Report) Transit Elasticities and Price Elasticities (Victoria Transport Policy Institute), Short-term, pg 8, 2004 Patronage Impacts of Changes in Transit Fares and Services (Pickrell Report) Unable to find elasticity in research Patronage Impacts of Changes in Transit Fares and Services (Pickrell Report) Patronage Impacts of Changes in Transit Fares and Services (Pickrell Report) Transit Elasticities and Price Elasticities (Victoria Transport Policy Institute), pg 8, 2004 Transit Elasticities and Price Elasticities (Victoria Transport Policy Institute), pg 16, National change in VMT Unable to find roadway congestion directly in research % Per year APTA Historical Ridership, ; Annual ridership growth reflects past 30 recorded years ( ); actual values

17 Transit Project Demand Forecast Accuracy: Original vs. Adjusted (with 45-degree line, dashed lines represent ±20% accuracy) 2.50 Project Demand Accuracy due to Input Inaccuracy (Actual Forecasted Demand) y = x R² = Original Project Demand Accuracy (Actual Forecasted Demand) 17

18 OTHER RESULTS Original Accurate ( ) Adjusted Overestimated (<0.8) Underestimated (1.2+) Total % Over-estimated (<0.8) % Accurate ( ) % Under-estimated (1.2+) % Total % % 59% 31% 10% 100% 70% Rank Project Characteristic/Assumption Average Impact on Demand Accuracy (Abs Value) 1 Employment Estimates 20.8% 2 Population Estimates 14.7% 3 Project Service Levels 13.1% 4 Supporting Transit Network 9.2% 5 Competing Transit Network 9.0% 6 Economic Conditions 7.4% 7 Auto Fuel Price 6.2% 8 Duration between Forecast Year/Opening % 9 Project Travel Time 4.2% 10 Project Fare 4.1% 11 Roadway Congestion 2.1%

19 OTHER RESULTS (2) # Projects that did not significantly alter accuracy 44 72% # Projects that became accurate 12 20% # Projects that became inaccurate 5 8% Overall Results: Historical Accuracy (original) 0.64 Historical Accuracy (revised) 0.73 Inaccurate inputs & assumptions accounts for ~25% of the demand forecast 19

20 SO SOMETHING ELSE IS GOING ON (SOME IDEAS) Data supporting travel model is not unbiased and/or error-free Travel model mis-specification(s) Mis-estimation of travel model Data used to validate model is not unbiased and/or error-free Insufficient calibration / validation of travel models Insufficient calibration / validation within key corridor(s) WHAT ELSE IS RESPONSIBLE FOR ~75% OF DEMAND INACCURACY?

21 FREELY AVAILABLE MATERIALS FOR APPLICATION Item Background information on Reference Class Forecasting and application methods Reference Class Reports & Project Assumption Accuracy Reports Location See Appendix to presentations/143_ %20Transit%20Forecasting%20Acc uracy%20database%20summary%20 v5%20-%20with%20script.pptx Methodology used to identify reference classes See TRB paper # Text and figures summarizing the accuracy of USA transit projects constructed within the past 5- and 10-years (for immediate use in forecasting reports) 21

22 REFERENCE CLASS FORECASTING The Outside View : the use of base-rate and distributional results derived from similar past situations and their outcomes to de-bias forecasts made using traditional methods The American Planning Association recommended Reference Class Forecasting in 2005, over 10 years ago Empirical observations: Absence of reference class forecasting in USA practice Absence of reference classes focused on USA transit The Inside View : focused on the project itself, its objective and characteristics, and extrapolating travel patterns into the future Objective: Determine appropriate reference classes for USA transit ridership forecasting 22

23 REFERENCE CLASS RECOMMENDATIONS Reference Class Conditions for Application Major transit projects constructed since 2007 Travel model properties have been thoroughly reviewed LRT projects only Project mode is LRT All projects If the conditions for other two classes cannot be met Reference Class Reports and corresponding Project Assumption Accuracy Reports can be found in the Appendix to this presentation 23

24 Mean = 0.85 Median = 0.83 Std. Dev = 0.22 Variance = 0.05 Example of Reference Class Report 24

25 APPLICATION EXAMPLE 1 Objective: address optimism bias in the project demand forecast Important: identify appropriate reference class Application Examples Original BRT forecast: 10,000 boardings/day Adjust forecast to reflect average median of empirical forecast error Adjusted forecast to reflect average error of transit forecasts: 8,500 Adjust forecast to reflect median error of transit forecasts: 8,300 Express error range in terms of risk acceptance 80% risk acceptance = 9,800 50% risk acceptance = 8,000 30% risk acceptance = 7,000 25

26 70% of projects have ratio 0.70; Funding agency accepts 30% of the historical risk if they assume ridership is 10,000 x 0.70 = 7,000 26

27 50% of projects have ratio 0.80; Funding agency accepts 50% of the historical risk if they assume ridership is 10,000 x 0.80 = 8,000 27

28 20% of projects have ratio 0.98; Funding agency accepts 80% of the historical risk if they assume ridership is 10,000 x 0.98 = 9,800 28

29 APPLICATION EXAMPLE 1 Objective: address optimism bias in the project demand forecast Important: identify appropriate reference class Application Examples Original BRT forecast: 10,000 boardings/day Adjust forecast to reflect average median of empirical forecast error Adjusted forecast to reflect average error of transit forecasts: 8,500 Adjust forecast to reflect median error of transit forecasts: 8,300 Express error range in terms of risk acceptance 80% risk acceptance = 9,800 50% risk acceptance = 8,000 30% risk acceptance = 7,000 29

30 Traffic Forecasting Accuracy Assessment Research SOME OTHER ACCURACY-RELATED PROJECTS FORECAST (Traffic Forecasting Accuracy Database) Objectives: Analyze the accuracy of design traffic forecasts developed as part of Equivalent Single Axle Load (ESAL) determinations Identify issues with traffic forecasts produced using the Central Florida Regional Planning Model (CFRPM) Identify reasons why CFRPM may not have been used to produce traffic forecasts in lieu of other methods Assist greatly the Department in prioritizing CFRPM improvements and future data collection efforts Currently incorporates the 31 ESAL reports produced in 2001; plans for NCHRP (Traffic Forecasting Accuracy Assessment Research) Objective: to develop a process to analyze and improve the accuracy, reliability, and utility of project-level traffic forecasts The product of this research will provide guidance to MPOs, state DOTs, and others to improve the accuracy, reliability, and utility of traffic forecasting methods as applied to transportation planning, design, and operation efforts both short and long term.

31 31

32 FINAL COMMENTS To contribute/assist with projects not currently in the database, please contact David Schmitt 32

33 THANK YOU! 33