Beginning to Enjoy the Outside View A Glance at Transit Forecasting Uncertainty & Accuracy Using the Transit Forecasting Accuracy Database

Beginning to Enjoy the Outside View A Glance at Transit Forecasting Uncertainty & Accuracy Using the Transit Forecasting Accuracy Database presented by David Schmitt, AICP with very special thanks to Hongbo Chi December 9, 2015

Agenda 1. Database 2. Accuracy 3. Reference Class Forecasting 4. Applications 5. Materials

Motivation Accurate transit demand forecasts matter To decision makers sound investment of public dollars To transit operators fare revenue covers meaningful percentage of operating costs To designers/engineers to right-size the system To planners compute reliable estimates of project s costs and benefits Empirical evidence suggests real gap between need and practice Large inaccuracies in demand from large transit projects (Flyvbjerg, FTA, and others) Few details on empirical transit forecasting accuracy Absence of documenting uncertainty, bias & risk in practice Given importance of accurate forecasts and historical inaccuracy, need exists to improve & promote better assessment of forecast uncertainty, bias and risk Enjoying the Outside View 3

Transit Forecasting Accuracy Database Developed to: Quantify and track industrywide accuracy trends; Quantify and track the accuracy of upstream assumptions and exogenous forecasts; and Provide empirical data to support the implementation of reference class forecasting, quality control and due diligence practices in the United States 65 large-scale transit projects Project description and characteristics (city, length, # stations, CBD/non-CBD, mode) Tracks differences in forecasted/actual values of 10 project assumptions and exogenous forecasts Forecasted ridership (year of forecast, forecast year, value) Observed ridership (year of observation, value) Allows for multiple records of forecasted and observed ridership Enjoying the Outside View 4

Projects by Mode & Decade of Opening Mode 1980s 1990s 2000s 2010s Total Bus 1 2 1-4 6% Bus Rapid Transit (BRT) - - 2 1 3 5% Commuter Rail - - 7-7 11% Streetcar/Trolley - - 1-1 2% Urban Heavy Rail 4 3 6-13 20% Urban Light Rail 5 7 20 1 33 51% Downtown People Mover (DPM) 2 1 1-4 6% Total 12 13 38 2 65 100% 18% 20% 58% 3% 100% 120 total records of forecasted ridership (mean= 1.8 per project) 218 total records of observed ridership (mean= 3.4 per project) Enjoying the Outside View 5

Historical (In)Accuracy of Project Assumptions & Exogenous Forecasts Filled cells represent highest proportion of each row Significant optimism bias in assumptions & forecasts, which increases risk of ridership forecasting inaccuracy 6

Computing Accuracy = Accuracy: actual / forecasted ridership 0.00-0.79 forecasted > actual ridership (strongly over-forecast) 0.80-0.99 forecasted > actual ridership (over-forecast) = 1.00 forecasted matches actual ridership 1.01-1.20 forecasted < actual ridership (under-forecast) 1.21+ forecasted < actual ridership (strongly under-forecast) For all projects in database: N=61 0.63 accuracy (avg) For Florida projects in database: N=5 0.26 accuracy (avg) Enjoying the Outside View 7

Enjoying the Outside View 8

Reference Class Forecasting The Outside View : the use of base-rate and distributional results derived from similar past situations and their outcomes to de-bias forecasts made using traditional methods The American Planning Association recommended Reference Class Forecasting in 2005, over 10 years ago Empirical observations: Absence of reference class forecasting in USA practice Absence of reference classes focused on USA transit The Inside View : focused on the project itself, its objective and characteristics, and extrapolating travel patterns into the future Objective: Determine appropriate reference classes for USA transit ridership forecasting Enjoying the Outside View 9

Reference Class Recommendations Reference Class Conditions for Application Major transit projects constructed since 2007 Travel model properties have been thoroughly reviewed LRT projects only Project mode is LRT All projects If the conditions for other two classes cannot be met Reference Class Reports and corresponding Project Assumption Accuracy Reports can be found in the Appendix to this presentation Enjoying the Outside View 10

Mean = 0.85 Median = 0.83 Std. Dev = 0.22 Variance = 0.05 Example of Reference Class Report 11

Application Example 1 Objective: address optimism bias in the project demand forecast Important: identify appropriate reference class Application Examples Original BRT forecast: 10,000 boardings/day Adjust forecast to reflect average median of empirical forecast error Adjusted forecast to reflect average error of transit forecasts: 8,500 Adjust forecast to reflect median error of transit forecasts: 8,300 Express error range in terms of risk acceptance 80% risk acceptance = 9,800 50% risk acceptance = 8,000 30% risk acceptance = 7,000 12

70% of projects have ratio 0.70; Funding agency accepts 30% of the historical risk if they assume ridership is 10,000 x 0.70 = 7,000 13

50% of projects have ratio 0.80; Funding agency accepts 50% of the historical risk if they assume ridership is 10,000 x 0.80 = 8,000 14

20% of projects have ratio 0.98; Funding agency accepts 80% of the historical risk if they assume ridership is 10,000 x 0.98 = 9,800 15

Application Example 1 Objective: address optimism bias in the project demand forecast Important: identify appropriate reference class Application Examples Original BRT forecast: 10,000 boardings/day Adjust forecast to reflect average median of empirical forecast error Adjusted forecast to reflect average error of transit forecasts: 8,500 Adjust forecast to reflect median error of transit forecasts: 8,300 Express error range in terms of risk acceptance 80% risk acceptance = 9,800 50% risk acceptance = 8,000 30% risk acceptance = 7,000 16

Project Assumption Accuracy Report [2007-today] Characteristic N Well Below Assumed Levels Actual Levels Are Below Assumed Levels At Assumed Levels Above Assumed Levels Well Above Assumed Levels Supporting transit network 11 9% 36% 45% 0% 9% Project Service Levels 11 18% 36% 36% 9% 0% Economic Conditions 9 33% 56% 0% 11% 0% Competing transit network 8 0% 13% 38% 38% 13% Employment Estimates 6 0% 67% 0% 17% 17% Project Travel Time 5 0% 40% 60% 0% 0% Project Fare 4 0% 0% 50% 25% 25% Population Estimates 4 0% 75% 0% 25% 0% Auto Fuel Price 3 0% 0% 0% 67% 33% Roadway congestion 2 50% 50% 0% 0% 0% 17

Application Example 2 Objective: develop forecasting range based on empirical uncertainty of project inputs and assumptions Original BRT forecast: 10,000 boardings/day Additional Forecast # Supporting Transit Network Levels Project Service Levels Empirical Frequency Ridership Forecast Original Original value Original value 16% 10,000 1 Original value Lower by 10% 16% 9,500 2 Original value Lower by 25% 8% 8,500 3 Lower by 10% Original value 13% 9,000 4 Lower by 10% Lower by 10% 13% 8,200 5 Lower by 10% Lower by 25% 6% 6,900 6 Lower by 25% Original value 3% 7,600 7 Lower by 25% Lower by 10% 3% 6,900 8 Lower by 25% Lower by 25% 2% 5,700 X Higher than original values Higher than original values 20% Exceeds 10,000 Funding agency now has stronger sense of forecast reliance on accurate inputs given empirical results: ~1 in 2 Chances inaccurate inputs will cause forecast to be 7,600-10,000 ~1 in 3 Chances inaccurate inputs will cause forecast to meet or exceed 10,000 ~1 in 7 Chances inaccurate inputs produce forecast of 7,600 18

Freely Available Materials Item Background information on Reference Class Forecasting and application methods Reference Class Reports & Project Assumption Accuracy Reports Location See Appendix to http://www.trbappcon.org/2015conf/ presentations/143_2015-05- 19%20Transit%20Forecasting%20Acc uracy%20database%20summary%20 v5%20-%20with%20script.pptx Methodology used to identify reference classes See TRB paper #16-1603 Text and figures summarizing the accuracy of USA transit projects constructed within the past 5- and 10-years (for immediate use in forecasting reports) See Appendix to this presentation 19

Final Comments Subsequent updates to the information provided here will be made publicly-available on a regular basis (through TMIP listserv or similar service) To contribute/assist with projects not currently in the database, please contact David Schmitt (daves1997@gmail.com) Enjoying the Outside View 20

THANK YOU! David Schmitt, AICP daves1997@gmail.com 21

References 1. Flyvbjerg, Bent. From Nobel Prize to Project Management: Getting Risks Right. Project Management Journal. August 2006. 2. Flyvbjerg, Bent. How (In)accurate Are Demand Forecasts in Public Works Projects?: The Case of Transportation. Journal of American Planning Association. Vol. 71, No. 2. Spring 2005. 3. Flyvbjerg, Bent. Quality Control and Due Diligence in Project Management: Getting Decisions Right By Taking the Outside View. International Journal of Project Management. 2012. 4. Kahneman, Daniel and Amos Tversky. Intuitive Prediction: Biases and Corrective Procedures. Decision Research. June 1977. 5. Lovallo, Dan and Daniel Kahneman. Delusions of Success: How Optimism Undermines Executives Decisions. Harvard Business Review. July 2003. 6. Nicolaisen, Morten Skou and Patrick Arthur Driscoll. Ex-Post Evaluations of Demand Forecast Accuracy: A Literature Review. Transport Reviews. Vol. 34, No. 4, pp. 540-557. 2014. 7. Taleb, Nassim Nicholas. Antifragile: Things That Gain from Disorder. Random House, 2012-11-27. ibooks. 8. Taleb, Nassim Nicholas. The Black Swan: Second Edition. Random House Trade Paperbacks, 2010-05-11. ibooks. 9. Transportation Research Board. National Cooperative Highway Research Program Synthesis 364: Estimating Toll Road Demand and Revenue A Synthesis of Highway Practice. 2006. 10. U.K. Department of Transport. Procedures for Dealing with Optimism Bias in Transport Planning: Guidance Document. June 2004. 11. U.S. Department of Transportation: Federal Transit Administration. Before-and-After Studies of New Starts Projects [annual reports to Congress]. 2007-2013. 12. U.S. Department of Transportation: Federal Transit Administration. Predicted and Actual Impacts of New Starts Projects: Capital Cost, Operating Cost and Ridership Data. September 2003. 13. U.S. Department of Transportation: Federal Transit Administration. The Predicted and Actual Impacts of New Starts Projects - 2007: Capital Cost and Ridership. April 2008. 14. U.S. Department of Transportation: Transportation Systems Center. Urban Rail Transit Projects: Forecast Versus Actual Ridership and Costs. October 1989. 15. U.S. Department of Transportation: Federal Transit Administration. Travel Forecasting for New Starts: A Workshop Sponsored by the Federal Transit Administration. Phoenix and Tampa, 2009. 16. U.S. Department of Transportation: Travel Model Improvement Program Webinar: Shining a Light Inside the Black Box (Webinar I). February 14, 2008. 17. Wachs, Martin. Ethics and Advocacy in Forecasting for Public Policy. Business & Professional Ethics Journal, Vol. 9, Nos. 1 & 2. 18. Web site: https://www.planning.org/newsreleases/2005/apr07.htm. Accessed December 2014. 19. Web site: http://www.homereserve.com/images/classic_room.jpg. Accessed January 2015. 20. Web site: http://static3.businessinsider.com/image/4e020c7cccd1d5c239010000-1200/23-back-bay-in-boston-ma.jpg.. Accessed January 2015. 21. Wikipedia. http://en.wikipedia.org/wiki/reference_class_forecasting. Accessed February 2015. Enjoying the Outside View 22

Appendix Accuracy Summary for USA Transit Projects Constructed within Past 10 Years 23

Overview N=32 67% strongly overforecasted 31% accurately forecasted Mean accuracy ratio = 0.68 Enjoying the Outside View 24

Project Assumption Accuracy Table 25

Text Over the past 10 years in which data is available, the transit forecasting industry has produced an accuracy ratio of 0.68. This means over the past 10 years, actual ridership is, on average, 32% lower than forecasted ridership for the 32 projects in that sample. Two-thirds of the sampled projects have been over-forecasted, while 31% have been forecasted accurately. Fortunately, the accuracy of the project assumptions and exogenous forecasts used to develop the demand forecast were available for this sample. Project assumptions include the project's service levels, travel time, and fare. The assumed supporting and competing transit networks is also included. Also recorded is the general accuracy from the population and employment estimates, whose forecasts are generally done by agencies external to the project team. A general assessment of macro-economic conditions is also recorded. Each project assumption and exogenous forecast was assigned one of five categories that related its actual level to the level assumed at the time the forecast was made. The five categories are collapsed into three groups: Optimistically biased, defined as a ~10+% variation that would artificially increase demand, Conservatively biased, defined as a ~10+% variation that would artificially decrease demand, and Accurately assumed, defined as neither optimistically nor conservatively biased. While many items were described quantitatively, in many cases the actual values were qualitatively described. In these cases, a professional transit demand forecaster provided his best assessment of the Before/After Study text and assigned the categories accordingly. Items not mentioned were assigned an 'unknown' category and excluded from analysis. It should be noted that the details and analyses provided are unevenly reported among the various reports. No project reports the details on all 10 project assumptions and exogenous forecasts. Table X describes the accuracy of the project assumptions and exogenous forecasts for projects constructed over the past 10 years. All but two assumptions/forecasts have been optimistically biased, indicating that the information being provided to transit forecasters is not accurate. Given the limitations of sample size, this analysis will focus on the two project assumptions and one exogenous forecast that reflect at least 10 projects: employment estimates, the supporting transit network and project service levels. Over the past 10 years, the employment estimates provided to transit forecasters has been optimistically biased in 9 of 11 projects that provided some level of employment information. The service levels of the supporting transit network, routes that feed into the project, have been optimistically biased in 8 of 14 projects reporting this characteristic. Project service levels have been optimistically biased in 12 of 27 projects. Artificially high assumptions of these characteristics will generally produce artificially high demand forecasts, all other things being equal. 26

Appendix Accuracy Summary for USA Transit Projects Constructed within Past 5 Years 27

Overview N=12 60% accurately forecasted 33% strongly overforecasted Mean accuracy ratio = 0.85 Enjoying the Outside View 28

Project Assumption Accuracy Table 29

Text Over the past five years in which data is available, the transit forecasting industry has produced an accuracy ratio of 0.85. This means over the past five years, actual ridership is, on average, 15% lower than forecasted ridership for the 12 projects in that sample. The breakdown of the 12 projects is shown in Figure X. Nearly 60% of the projects have been accurate; that is, the actual ridership has been within 20% of the forecasted demand. Table X describes the accuracy of the project assumptions and exogenous forecasts for projects constructed over the past five years. All but three assumptions/forecasts have been optimistically biased, indicating that the information being provided to transit forecasters is not accurate. Given the limitations of sample size, this analysis will focus on the two project assumptions that reflect at least 10 projects: the project and supporting transit network service levels. Over the past five years, the project service levels have been optimistically biased in 6 of 11 projects that provided the actual service levels. The service levels of the supporting transit network, routes that feed into the project, have been optimistically biased in 5 of 11 projects reporting this characteristic. Artificially high assumptions of these characteristics will generally produce artificially high demand forecasts, all other things being equal. 30