Review of the NLTF Revenue Forecasting Model

Transcription

1 Final Report 1 May 2014 Review of the NLTF Revenue Forecasting Model Prepared for Ministry of Transport

2 Authorship Aaron Schiff and John Small (09) Covec Ltd, All rights reserved. Disclaimer Although every effort has been made to ensure the accuracy of the material and the integrity of the analysis presented herein, Covec Ltd accepts no liability for any actions taken on the basis of its contents.

3 Contents Executive Summary Summary of forecasting requirements Context Issues with the existing model Summary of our methodology PED volume forecasting Light RUC volume forecasting Heavy RUC volume forecasting Discussion of forecasting issues Suggested improvements to the Excel model i i i iii iv v xi xv xix xx 1 Background and scope 1 2 Needs assessment Purpose of the forecasts Consequences of forecast errors The forecasting process Forecast outputs and characteristics Scenario analysis 3 3 Context New Zealand transport trends Other New Zealand trends International transport trends 17 4 Issues with the existing NLTF model Forecast accuracy and reliability Design and implementation 22 5 Literature review Private transport activity Commercial transport activity 31 6 Data review Transport activity data Potential explanatory variables 38

4 7 Petrol excise duty forecasting Data Modelling strategy Pure time series models [PED model 1] Regression models [PED models 2a-2e] Hybrid models [PED models 3a & 3b] Additional PED volume models PED volume model evaluation and comparison PED volume confidence intervals and sensitivity testing Recommendations for PED modelling 95 8 Road user charges forecasting Data Modelling strategy Light RUC models Heavy RUC models Discussion Commentary on various forecasting issues Suggested improvements to the spreadsheet model References 158

5 Executive Summary This report reviews forecasts of National Land Transport Fund (NLTF) revenue. We focus on the main components of NLTF revenues: petrol excise duty (PED) and heavy and light road user charges (RUCs). We consider options for modelling and forecasting the volumes of PED and RUCs (litres and km respectively) to which duties and charges are applied. We also suggest ways that the design and implementation of the current Excel forecasting model could be improved. This work was conducted in close consultation with members of the NLTF revenue forecasting group. Summary of forecasting requirements The primary requirement is for forecasts of annual NLTF revenues with a high degree of accuracy over the next three years. Forecasts over a ten year period are also required, but greater uncertainty beyond the three year horizon is acceptable. In addition: The model and forecasts need to be updated in a timely fashion every quarter when new data becomes available, with a minimum amount of manual work; A set of scenarios reflecting alternative forecast assumptions and confidence intervals (eg low, medium, high) are required, but the accuracy of the medium scenario is of greatest importance; The model should be capable of easily producing forecasts under various assumptions about key factors affecting NLTF revenues; and The forecasts should be readily explainable and understandable, and the reasons for any significant departure from current trends should be apparent. Context Recent events have made forecasting future transport activity challenging. Figure 1 shows the correlation between total annual vehicle kilometres travelled (VKT) and total real GDP in New Zealand over time. Between 2001 and 2006 there was a strong positive correlation, but in 2006 this correlation was disrupted. Growth in GDP and VKT resumed in 2007, and both declined during the 2008/09 recession (also indicating a positive correlation), but since 2010, real GDP growth has resumed while total VKT has essentially remained constant. At the same time, there have been significant changes within the transport fleet (Figure 2). On a per-capita basis, VKT of light petrol and medium diesel vehicles has declined over time, with the decline in light petrol VKT per capita apparent since In contrast, VKT per capita of light diesel vehicles has increased significantly since 2001, with most of this increase occurring between 2001 and VKT per capita of heavy diesel vehicles has fluctuated, but at the end of 2012 was at a similar level as in Similar trends have been observed in other developed countries, and a key question is whether the recent stagnation of some types of road transport activity is a temporary i

6 Annual VKT per capita (index) effect of the global financial crisis and recession, or a permanent shift reflecting changes in factors such as demographics, travel preferences, and urban design. Figure 1 Total annual VKT and annual real GDP in New Zealand. Source: Ministry of Transport and Statistics New Zealand Figure 2 Annual VKT per capita indexes (2001Q4 = 100) for New Zealand. 140 Heavy diesel Medium diesel 130 Light diesel Light petrol Total Year Ended Quarter Source: Covec analysis of Ministry of Transport and Statistics New Zealand data ii

7 PED litres (millions) Issues with the existing model There are concerns about the accuracy and reliability of the PED and RUC forecasts produced by the model. While an initial review (Deloitte, 2012) of the model s accuracy found no evidence of structural breaks and that the models were performing well, in practice the model has over-predicted PED and heavy RUC volumes by up to 5%, and under-estimated light RUC by up to 2%. The PED forecasts are of particular concern, with the model predicting strong growth in PED volume over the next three years, in contrast to the general decline in PED volumes since 2007/08 (Figure 3). These forecasts imply that within three years, annual PED volumes will exceed the highest level observed over the past 13 years. Figure 3 PED volume forecasts produced by the existing model. 3,500 3,400 3,300 3,200 Actual Forecast 3,100 3,000 2,900 2,800 2,700 2,600 2,500 Source: Ministry of Transport and Covec analysis. In addition: The model uses complex error-correction models (ECMs) to generate the forecasts, which depend on a relatively large number of explanatory variables and that generate forecasts that have been difficult to interpret and explain, particularly in the short term. A relatively complex process of seasonal adjustment is used for all variables, even in the absence of clear seasonal patterns (eg for PED volumes). Ad hoc changes to the ECM coefficients are allowed for, without any robust basis for making these changes. iii

8 The workflows for updating the model each quarter when new data arrives and for specifying forecasting scenarios are complex and require a number of manual steps that are time consuming and may be error-prone. To ensure that the model has been updated correctly, several MoT staff members must update it independently, and compare the results. Summary of our methodology We applied the following methodology to PED and heavy and light RUC volumes: Preliminary analysis of quarterly volumes to understand long-term trends and short-term fluctuations, including testing for predictable seasonal patterns. Construction of a dataset of potential explanatory variables for PED and RUC volumes, based on results from a literature review and suggestions from the NLTF revenue forecasting group and subgroup. Testing a variety of econometric models for PED and RUC volume, including: o o o Simple pure time series models based only on past values of PED and RUC, and deterministic trends and seasonal dummy variables. The time series models were selected from a relatively large class of such models using process of statistical model selection that chooses the model most likely to have generated the data that has been observed. Regression models, including various ways of modelling short-run dynamics (autoregressive error models, lagged dependent variables, and simple ECMs). Hybrid models, including: Modelling PED volume as the product of light petrol VKT and fuel efficiency Modelling heavy and light RUC volume as functions of economic activity in transport-intensive sectors Testing a set of additional models following presentation of our initial results to the NLTF revenue forecasting group. Evaluation and testing of all models, including: o o o Within-sample goodness of fit and residual diagnostic testing Truncated-sample forecasting performance The plausibility of out-of-sample 10-year ahead forecasts iv

9 PED Volume (million litres) Choosing a short-list of models on the basis of the above evaluation in consultation with a subgroup of the NLTF revenue forecasting group. Developing confidence intervals for a baseline forecast generated from the shortlisted models and performing sensitivity testing on these models. Making recommendations for modelling PED, light RUC, and heavy RUC volumes on the basis of all of the above analysis. PED volume forecasting PED volume volatility and seasonality A significant challenge in PED volume forecasting is the high volatility of the quarterly data (Figure 4). We found no evidence of this volatility being due to a predictable seasonal pattern. Instead the volatility appears to be driven by the random timing of large fuel import shipments into New Zealand. To partially overcome this problem, we recommend that forecasting be done using the 4-quarter moving average of PED volume. The volatility of volumes (and revenues) should be reflected in the range of forecasts produced for PED. Figure 4 Quarterly PED volume and polynomial trend. 1, Quarter Source: Covec analysis of Ministry of Transport data. PED volume analysis Three general categories of models of PED volume were tested and compared: Pure time series models. v

10 Simple regression models relating PED volume to various explanatory variables including real petrol prices, seasonally adjusted real GDP, and the seasonally adjusted unemployment rate. Hybrid models where PED volume is calculated from a combination of a light petrol VKT model and a fuel efficiency assumption. Two versions were tested, using total light petrol VKT and per-capita light petrol VKT. The latter incorporates total population directly into the PED volume forecast. Table 1 summarises the eight models for PED volume that were estimated and tested, including some additional models requested by the NLTF revenue forecasting group. Table 1 Summary of variables included in the selected PED models. Model Type Real GDP Real petrol price Uempl rate Fuel eff Total pop Trend Young pop propn Urban pop propn AKL pop propn 1 Time series 2a 2b 2c 2d 2e 3a 3b Regression (AR errors) Regression (ADL) Regression (ECM) Regression (AR errors) Regression (ADL) Hybrid (total VKT) Hybrid (per-cap VKT) Table 2 summarises the goodness of fit of these models and their performance when estimated with a truncated sample of the data up to the second quarter of 2011 and using the model to produce forecasts of PED volume compared with actuals up to the third quarter of The regression models explain around 80% of the variation in the 4-quarter moving average of PED volume, although only 10 15% of the variation in actual quarterly PED volume. The hybrid models have a lower goodness of fit but perform significantly better than the other models on the truncated sample forecasting test as indicated by the RMSE and the average percentage errors. vi

11 PED volume (million litres) Table 2 Summary of goodness of fit and truncated-sample forecast RMSE of the PED volume models. 1 2a 2b 2c 2d 2e 3a 3b Full sample goodness of fit R 2 vs PED volume R 2 vs PED volume (MA) Truncated sample forecasting performance RMSE (million litres) Average quarterly error (%) error (%) error (%) Figure 5 illustrates the truncated sample forecasting performance of the PED volume models, where the tendency of all models except the hybrid models to over-forecast PED volumes can be clearly seen. Figure 5 Comparison of the truncated-sample forecasting performance of the PED models Truncated sample PED volume forecasts Actual Time series Regression (a) Regression (b) Regression (c) Regression (d) Regression (e) Hybrid (a) Hybrid (b) Quarter Source: Covec analysis. Figure 6 compares the annual PED volume forecasts produced by these models under a baseline scenario and in our view the forecasts produced by the two hybrid models and regression model (b) are the most plausible. The PED models were also reviewed in terms of the practicality of using them to generate forecasts and their relative advantages and disadvantages (Table 3). Taking all of the above into consideration, we recommend the use of the per-capita hybrid model (model 3b) to forecast PED volumes. vii

12 PED volume (million litres) Figure 6 Annual PED volume forecast comparison. 3,600 Annual PED volumes 3,400 3,200 3,000 2,800 2,600 2,400 Actual Current model Time series Regression (a) Regression (b) Regression (c) Regression (d) Regression (e) Hybrid (a) Source: Covec analysis. Year ended June Table 3 Advantages and disadvantages of the PED volume models. Model(s) Advantages Disadvantages Time-series [1] Regression [2a & 2d] (AR errors) Regression [2b & 2e] (lagged dependent variable) Regression [2c] (ECM) Hybrid [3a & 3b] Very simple implementation No forecast drivers required Model can evolve over time Simple implementation Clear link between explanatory variables and forecasts Simple implementation Clear link between explanatory variables and forecasts Includes demographic variables Sophisticated short-run dynamics Uses potentially more reliable VKT data (compared to PED volumes) Clear link between explanatory variables and forecasts Allows analysis of changing fuel efficiency Cannot test alternative scenarios Provides no explanation for trends Unsophisticated short-run dynamics Extrapolates past relationship with GDP and unemployment Very sensitive to demographic assumptions No Stats NZ forecast of urban population for model 2b Population data is only observed in Census years Difficult to interpret and explain trends Extrapolates past relationship with GDP and unemployment Basis for forecasting efficiency is not clear Model 3a extrapolates past relationship with GDP and unemployment Model 3b includes an unexplained deterministic trend viii

13 PED volume (m litres) Recommended PED volume model The recommended PED volume model (model 3b) is a hybrid model that forecasts PED volume as a function of per-capita light petrol VKT, total population, and fuel efficiency. Per-capita light petrol VKT is modelled as a function of real per-capita GDP, the real petrol price, and a negative time trend. Fuel efficiency is also modelled as a function of a positive time trend, although this could be easily replaced by a more sophisticated efficiency model if such a model were to be developed. Figure 7 and Table 4 show an indicative PED volume forecast produced by model 3b. In the short term, PED volumes are forecast to increase slightly as economic activity increases, unemployment falls, and real petrol prices remain constant. In the longer term, the downwards trend in light petrol VKT per capita, higher real petrol prices, and increasing fuel efficiency are forecast to lead to a decline in PED volumes, with total volume in 2023 at a similar level as it was in 2013 (see Figure 8). Figure 7 Indicative forecast and 67% confidence interval from the recommended PED volume model. 3,400 3,300 Actual Forecast 3,200 3,100 3,000 2,900 2,800 2,700 2,600 Source: Covec analysis. Year ended June ix

14 Average contribution to annual forecast change Table 4 Indicative forecasts and confidence intervals produced by PED model 3b. YE June 90% lower PED volume (million litres) 67% lower Base ,013 67% upper 90% upper 90% lower PED volume (annual % change) 67% lower Base 67% upper 90% upper ,884 2,924 2,989 3,058 3, ,894 2,949 3,037 3,130 3, ,913 2,968 3,057 3,150 3, ,916 2,971 3,060 3,153 3, ,915 2,970 3,059 3,152 3, ,908 2,963 3,051 3,144 3, ,896 2,951 3,038 3,130 3, ,885 2,939 3,026 3,117 3, ,874 2,928 3,015 3,105 3, ,865 2,918 3,004 3,095 3, Figure 8 Approximate decomposition of the forecasts produced by PED model 3b. 0.8% 0.7% 0.6% 0.5% 0.4% 0.3% 0.2% 0.1% 0.0% -0.1% -0.2% to to to to to to to to to Real petrol price Real GDP per capita VKT time trend to Efficiency Population Dynamic & interaction Source: Covec analysis. x

15 Light RUC volume forecasting Unlike PED volumes, light RUC volumes (net km) were found to have a predictable seasonal pattern. Rather than performing seasonal adjustment on the data, it is more straightforward and transparent to include seasonal factors (eg quarterly dummy variables) in the regression models. We evaluated a large number of light RUC models, and compared the results from time series, regression, and hybrid models. In this case the hybrid models were based on GDP levels in various goods-producing sectors, with sub-models to forecast GDP in these sectors. Table 5 summarises the models that were estimated for light RUC km, including some additional models requested by the NLTF forecasting group. Table 5 Summary of explanatory variables in the light RUC models. Model Type Real GDP Real diesel price Real light RUC price TPW sector GDP Const. sector GDP Trend 1 Time series Goods imports 2a Regression 2b Regression 3 Hybrid Table 6 summarises the performance of these four models. The regression and hybrid models have better goodness of fit, but the time series model performs significantly better on the truncated sample forecasting test (Figure 9). Table 6 Summary of goodness of fit and truncated-sample forecast RMSE of the light RUC models. 1 2a 2b 3 Full sample goodness of fit R 2 vs light RUC km Truncated sample forecasting performance RMSE (million km) Average quarterly error (%) 0.4% 1.4% 2.2% 1.6% 2012 error (%) 0.6% 0.9% 1.8% 1.8% 2013 error (%) 0.3% 3.7% 4.1% 3.6% The models predict similar growth in the short term but the regression and hybrid models predict relatively strong growth in the longer term, at a faster rate than the current model (Figure 10). In our view the short-term forecasts are plausible but the long-term forecasts may need to be moderated. xi

16 Light RUC volume (million km) Light RUC net km (millions) Figure 9 Comparison of truncated sample forecasts produced by the light RUC models. 2,400 Truncated sample light RUC volume forecasts 2,300 2,200 2,100 2,000 1,900 1,800 1,700 Actual Time series Regression (a) Regression (b) Hybrid Quarter Source: Covec analysis. Figure 10 Annual light RUC forecast comparison. 12,000 Annual light RUC volumes 11,000 10,000 9,000 8,000 7,000 6,000 Actual Current model Time series 5,000 Regression (a) Regression (b) Hybrid Source: Covec analysis. Year ended June xii

17 Light RUC volume (m km) Recommended light RUC volume model On the basis of the above analysis, in consultation with a subgroup of the NLTF revenue forecasting group, light RUC model 2b was selected for further analysis and we recommend this model for light RUC forecasting. This is a simple regression model that forecasts light RUC volumes as a function of total real GDP, the real diesel price, the real light RUC price, real imports of goods, and a positive time trend. Figure 11 and Table 7 show an indicative forecast produced by this model. Under this scenario and in this model, growth is driven by higher real GDP, higher real imports of goods, and lower real diesel prices, offset by higher real light RUC prices in the short term (Figure 12). In the longer term the real GDP effect is important while there is also a time trend that drives up light RUC km in all periods. Imports do not feature in the long term in this forecast, however this is due to an assumption about future imports growth and alternative assumptions will lead to a different forecast. Figure 11 Indicative forecast and 67% confidence interval for the recommended light RUC model. 14,000 12,000 Actual Forecast 10,000 8,000 6,000 4,000 2,000 0 Source: Covec analysis. Year ended June xiii

18 Average contribution to annual forecast change Table 7 Indicative forecasts and confidence intervals for light RUC model 2b. YE June 90% lower Light RUC volume (million km) 67% lower Base ,150 67% upper 90% upper Light RUC volume (annual % change) 90% lower 67% lower Base 67% upper 90% upper ,084 8,210 8,405 8,600 8, % 0.7% 3.1% 5.5% 7.1% ,252 8,420 8,680 8,940 9, % 2.6% 3.3% 4.0% 4.4% ,543 8,711 8,971 9,231 9, % 3.5% 3.3% 3.3% 3.2% ,966 9,134 9,394 9,654 9, % 4.9% 4.7% 4.6% 4.5% ,342 9,510 9,770 10,030 10, % 4.1% 4.0% 3.9% 3.8% ,674 9,841 10,101 10,361 10, % 3.5% 3.4% 3.3% 3.3% ,001 10,169 10,429 10,689 10, % 3.3% 3.2% 3.2% 3.1% ,340 10,508 10,768 11,028 11, % 3.3% 3.2% 3.2% 3.1% ,684 10,852 11,112 11,372 11, % 3.3% 3.2% 3.1% 3.1% ,034 11,201 11,461 11,721 11, % 3.2% 3.1% 3.1% 3.0% Figure 12 Approximate decomposition of forecasts produced by light RUC model 2b. 1.0% 0.5% 0.0% -0.5% -1.0% -1.5% -2.0% to to to to to to to to to to Real GDP Real diesel price Real light RUC price Time trend Real imports Dynamic correction Source: Covec analysis. xiv

19 Heavy RUC volume forecasting Heavy RUC volumes (net km) were also found to have a predictable seasonal pattern, and quarterly dummy variables were tested in the models. We again evaluated a large number of heavy RUC volume models, falling into the same three classes as for light RUC: time series, simple regression, and hybrid models involving sectoral GDP and the proportions of heavy vehicles with 2-4 and 7+ axles. Table 8 summarises the heavy RUC models that were evaluated including additional models requested by the NLTF revenue forecasting group. Table 8 Summary of variables in the heavy RUC models. Model Type Real GDP Real heavy RUC price Forest GDP TPW GDP Real export of goods Real import of goods Trend 2-4 axles propn 7+ axles propn 1 Time series 2a Regression 2b Regression 3a Hybrid 3b Hybrid Table 9 summarises the forecasting performance of the heavy RUC models. The two regression models and one of the hybrid models have the highest goodness of fit, while the time series model performs best on the truncated sample forecasting test (see also Figure 13). Table 9 Summary of goodness of fit and truncated-sample forecast RMSE of the heavy RUC models. 1 2a 2b 3a 3b Full sample goodness of fit R 2 vs heavy RUC km Truncated sample forecasting performance RMSE (million km) Average quarterly error (%) error (%) error (%) Figure 14 compares indicative forecasts produced by the heavy RUC models. On the basis of recent trends in heavy RUC volumes, in our view it is unclear which of these models produces more plausible forecasts. However, all models produce lower heavy RUC volume forecasts than the existing NLTF forecasting model. xv

20 Heavy RUC volume (million km) Heavy RUC net km (millions) Figure 13 Comparison of truncated sample forecasting performance of the heavy RUC models Truncated sample heavy RUC volume forecasts Actual Time series Regression (a) Regression (b) Hybrid (a) Hybrid (b) Quarter Source: Covec analysis. Figure 14 Annual heavy RUC forecast comparison. 5,000 Annual heavy RUC volumes 4,500 4,000 3,500 3,000 2,500 Actual Current model Time series Regression (a) Regression (b) Hybrid (a) Hybrid (b) Source: Covec analysis. Year ended June xvi

21 Heavy RUC volume (m km) Recommended heavy RUC volume model On the basis of our analysis, we recommend heavy RUC model 2b for forecasting. This model forecasts heavy RUC volumes based on real exports and imports of goods, and the real heavy RUC price. Figure 15 and Table 10 show an indicative forecast produced by this model. The model forecasts relatively weak growth in heavy RUC volumes in the first three years, and steady growth thereafter. Figure 15 Indicative forecast and 67% confidence interval from the recommended heavy RUC model. 4,500 4,000 Actual Forecast 3,500 3,000 2,500 2,000 1,500 1, Year ended June The relatively weak growth in the short term is largely caused by the initial dynamic correction of the model to the estimated trend, given that actual heavy RUC volumes in 2013 were relatively high (Figure 16). In the longer term, growth is largely driven by growth in exports, although this depends on the particular long-term assumption of export growth (and relatively low import growth). xvii

22 Average contribution to annual forecast change Table 10 Indicative forecasts and confidence intervals produced by heavy RUC model 2b. YE June 90% lower Heavy RUC volume (million km) 67% lower Base ,552 67% upper 90% upper Heavy RUC volume (annual % change) 90% lower 67% lower Base 67% upper 90% upper ,491 3,536 3,606 3,678 3, % -0.5% 1.5% 3.5% 4.9% ,441 3,500 3,593 3,688 3, % -1.0% -0.4% 0.3% 0.7% ,471 3,530 3,623 3,720 3, % 0.9% 0.9% 0.9% 0.9% ,520 3,580 3,675 3,772 3, % 1.4% 1.4% 1.4% 1.4% ,546 3,606 3,702 3,800 3, % 0.7% 0.7% 0.7% 0.7% ,566 3,626 3,723 3,821 3, % 0.6% 0.6% 0.6% 0.6% ,586 3,647 3,744 3,844 3, % 0.6% 0.6% 0.6% 0.6% ,608 3,670 3,767 3,867 3, % 0.6% 0.6% 0.6% 0.6% ,630 3,692 3,790 3,890 3, % 0.6% 0.6% 0.6% 0.6% ,652 3,714 3,813 3,914 3, % 0.6% 0.6% 0.6% 0.6% Figure 16 Approximate decomposition of forecast changes in heavy RUC model 2b. 0.6% 0.4% 0.2% 0.0% -0.2% -0.4% -0.6% -0.8% -1.0% -1.2% -1.4% to Source: Covec analysis to to to Real exports of goods Real heavy RUC price to to to to Real imports of goods Dynamic & interaction to to xviii

23 Discussion of forecasting issues During our review, the NLTF forecasting group and subgroup raised a number of general questions about the forecasting approach and models: Plausibility and risks of the forecasts: Forecasts produced by econometric models necessarily assume that the relationships embodied in the models continue to hold in the future. In our view this is not problematic as long as the models have been thoroughly tested and continue to be reviewed regularly. It is also not clear that an alternative (non-econometric) approach based on ad hoc models or simple extrapolation would produce more accurate forecasts, and such an approach may be criticised because of its arbitrary nature. Overall, in our view the models recommended in this report produce plausible forecasts of PED and RUC volumes. However there is always some risk that the relationships embodied in these models fundamentally changes. This risk can be mitigated by reviewing the models on a regular basis. Speed of modelled changes: The requirement that the forecasts can be updated each quarter led us to estimate quarterly models for PED and RUC volumes. This implies that changes in the explanatory variables in the models affect PED and RUC volumes in the current quarter (and in future). In our view this is reasonable given that the explanatory variables tend to be correlated over time and that many of the models incorporate dynamic variables (eg lags of the dependent variable) that imply that the dependent variable takes time to adjust to shocks. Scope for the use of multiple models: We have recommended a single model for each volume forecast. A possible alternative approach involves running multiple models in parallel and either using these multiple models to produce a range of forecasts, or combining their forecasts into a single meta-forecast. We have some concerns with the use of multiple models: o o o Given that a single forecast of PED and RUC volumes is ultimately required, there is a risk that the process for choosing a single forecast from multiple models will become arbitrary, which will reduce accuracy and transparency. The use of multiple models will make it more difficult to explain how the forecasts have been derived, as it will be necessary to explain the forecast produced by each model as well as the process used for combining them. It is not clear that an approach based on multiple models will perform better than the use of a single model that is regularly re-estimated and tested. Re-testing and re-estimation effectively uses multiple models over time, but since only one model is in use at any given point in time, the issues associated with combining multiple models are avoided. xix

24 For these reasons we prefer the use of a single model for each volume forecast. However, if the NLTF forecasting group wishes to use multiple models to generate forecasts, in section we discuss how this can be done in a reasonably robust way. Potential for remediation of the existing spreadsheet model: In our view it is technically possible to remediate the existing Excel model by replacing the econometric models in it with new models and making some other changes to the design and structure. However our advice is that it is likely to be no more costly (and possibly less costly) to build a new model for PED and RUC volumes. This is because the complexity of the existing model s structure means that modifications would need to be made very carefully and tested thoroughly to ensure that there are no unwanted side-effects, which will increase costs. Recommendations for future review of the econometric models: We recommend that the coefficients of the models for PED and RUC volumes be reestimated using the latest available data on an annual basis. This will allow the coefficients of the model to be updated as new information becomes available. We expect this annual update would be a straightforward task that the Ministry could undertake internally or could contract out at relatively low cost. We also recommend that the econometric models be fully re-tested and their structure changed if necessary every three years. Suggested improvements to the Excel model Following our econometric analysis and review of the existing Excel spreadsheet model, we suggest the following improvements could be made: Replace ECMs with simpler regression models The ECMs in the existing model are relatively complex, particularly the short-run components of the models. This means that a relatively large number of inputs are required to generate forecasts, and it can be difficult to explain the short-run predictions generated by the models. Our econometric analysis found that relatively simple models can perform well in forecasting PED and RUC volumes, including modelling short-run dynamics through the use of lagged dependent variables or autoregressive error terms, which are easier to implement and interpret than ECMs. In our view, for PED and RUC volumes, any additional benefits of using ECMs to more accurately capture short-run dynamics are outweighed by the practical disadvantages of this approach. This is particularly true given that highly accurate quarterly forecasts are not required. Improve scenario analysis The ability to analyse scenarios in the model could be improved by clearly separating actual data inputs from scenarios, and simplifying the way that forecasting scenarios are specified in the Excel model. As a general principle, anything that needs to be updated by the spreadsheet user to produce a new forecast should be easily accessible and in a centralised location rather than dispersed throughout the spreadsheet tabs. A single information tab could contain and summarise all of the relevant inputs to the model when a new forecast is generated. xx

25 Improve outputs of the model In section below we suggest a number of simple outputs that can be generated automatically from the model each time a new forecast is required. These include tables and charts of the forecast levels and growth rates (and their confidence intervals), as well as an approximate breakdown of the drivers of the forecasts. It is possible to build the Excel model in such a way that these outputs update automatically each time a new forecast is generated. Remove parameter shocks The ability to analyse parameter shocks in the model introduces considerable complexity while potentially undermining the credibility of the econometric models as there is no simple, non-arbitrary way to make such adjustments. In our view it would be better for the coefficients of the econometric models to be re-estimated on a regular basis outside the spreadsheet model, including diagnostic testing. Remove coefficient re-estimation but re-test econometric models regularly The coefficients of the econometric models need to be updated regularly, particularly given the recent disruption to past transport correlations and the open question of whether these are temporary or permanent changes. However in our view this should be done outside the Excel model so that a proper set of diagnostic tests can be performed, and the structure of the models can be updated if necessary. Remove seasonal adjustment for PED and use quarterly dummies instead of seasonal adjustment for other variables There is no predictable seasonal pattern in PED volumes, with the quarterly volatility largely driven by random factors that essentially cannot be forecasted. Therefore in our view it is preferable for some simple form of smoothing (eg the 4-quarter moving average) to be applied to PED volumes for use in the analysis. Other variables such as RUC volumes do have predictable seasonal patterns. Our recommendation is to include quarterly dummy variables in the models where necessary to capture seasonal effects, rather than seasonally adjusting the variables prior to analysis. Include forecast uncertainties (confidence intervals) As well as generating forecasts under different input assumptions, it would be helpful if the model could reflect the uncertainty associated in the econometric models through the calculation of confidence intervals for the forecasts. The implementation of this will be greatly simplified by using simple regressions models to generate the forecasts, rather than ECMs. Simplify the updating process The model could be built in such a way that additional observations can be added to a data table and this flows through the model automatically, including updating the date ranges applied to output tables and charts. This would reduce the manual work required to produce updates and eliminate errors that may be created during updating. xxi

26 Simplify models for other components of NLTF revenues While not part of our review, we noted that the existing model includes relatively complex models for the other minor components of NLTF revenues (eg CNG and LPG excise, driver licensing, etc). In our view it would be preferable to greatly simplify these models, for example to use simple time-series models. xxii

27 1 Background and scope We have been asked by the Ministry of Transport to review the model it uses to generate forecasts of National Land Transport Fund (NLTF) revenues. The current model was developed during 2010 and 2011 (Deloitte, 2011a & 2011b) and was most recently reviewed in 2012 (Deloitte, 2012). The Ministry has commissioned our review out of concerns that the current model does not fully meet its needs, and most importantly there are concerns about the accuracy and reliability of the forecasts it produces. Due to the constrained timeframe for this review, we focus on the main components of NLTF revenues: petrol excise duty (PED), and road user charges (RUCs). For modelling purposes, RUC revenues are split into two categories: light RUC, applying to vehicles up to six tonnes, and heavy RUC for heavier vehicles. Light RUC applies mostly to private cars and vans, and some buses and small trucks, while heavy RUC mostly corresponds to large transport trucks and buses. Together, PED and RUC revenues comprise around 91 percent of current NLTF gross revenue. The key task in forecasting PED and RUC revenues is forecasting the volumes (litres and kilometres, respectively) to which the duties and charges will be applied. Accordingly, our analysis focuses on the volume forecasts for PED and RUC. Other revenue sources that we do not review include fuel excise duty on LPG and CNG, motor vehicle re-licensing and registrations, and charges for motor vehicle change of ownership and administration activities. The current NLTF forecasting model uses a class of econometric models known as errorcorrection models (ECMs) to generate quarterly PED and RUC forecasts. These models forecast long-term trends in volumes as functions of a relatively small number of key drivers. The models also incorporate more complex auxiliary models of short-term variation around these long-term trends. Additional methodologies are used to handle seasonal variation in quarterly data, and to permit some types of sensitivity testing. The Ministry has asked us to review these forecasting models and consider whether other models may be able to produce better forecasts of NLTF revenues. We have also been asked to review the design and implementation of the current model, including the ease with which the model can be used and updated. This report summarises our findings and is organised as follows. Section 2 reviews the Ministry s needs and requirements with regard to NLTF revenue forecasting. Section 3 describes recent trends in New Zealand transport activity, and section 4 gives an overview of the most significant issues with the current model. Section 5 briefly reviews recent literature on transport activity modelling and forecasting, and section 6 reviews the data that is available for NLTF revenue forecasting. Sections 7 and 8 review forecasting of PED and RUC revenues respectively, and section 9 concludes with a discussion of some forecasting issues and suggests improvements in the design and implementation of the model. 1

28 2 Needs assessment Through discussions with MoT officials and members of the NLTF revenue forecasting group, we have assessed the requirements of the forecasting model in terms of the forecasts that it produces and the ways that the model can be used. 2.1 Purpose of the forecasts Forecasts of NLTF revenue are required for a variety of purposes. The forecasts are provided to Treasury, for inclusion in Crown financial budgets and plans. These cover a four-year timeframe and are updated each March and October, as well as for inclusion in the government s annual Budget publication. The government also uses the forecasts to inform its government policy statement (GPS) on transport, which is updated every three years, and to understand the future revenues available for transport investments and initiatives. NZTA uses the forecasts for its planning, and in particular uses the forecasts of NLTF revenues over the next three years to plan and sequence transport projects. 2.2 Consequences of forecast errors NZTA is directly affected by errors in forecasting NLTF revenues, particularly if actual revenue is lower than forecast. NZTA develops a project plan on the basis of short-term (1-3 year ahead) revenue forecasts. Once a plan is committed and projects are commenced, there is limited scope to delay or reorganise projects in response to a revenue shortfall. Short-term revenue forecasts that are higher than actual revenues therefore cause NZTA to borrow to meet its commitments, with corresponding financing costs. Forecasting errors also make it difficult for the government to plan transport policies and investments. Again, the greatest difficulty arises if actual revenue is lower than forecast, meaning that policies and investments may not be able to be implemented in the time expected. 2.3 The forecasting process There is a requirement to update the forecasts in a timely fashion each quarter when new data becomes available, although revenue forecasts are only required and published for June years. The model update process is undertaken by MoT, with forecasts being reviewed by the NLTF revenue forecasting group, consisting of officials from MoT, NZTA, and Treasury. It is desirable that the amount of work required each quarter to update the forecasts is minimised. This includes minimising the work necessary to enter new data into the model and produce forecasts under different scenarios. To ensure that the model has been updated correctly, several MoT staff members update it independently, and the results are compared. Sense checks are also done by 2

29 comparing the forecasts to previous periods, and trying to identify the causes of any significant changes. 2.4 Forecast outputs and characteristics Forecasts of all components of NLTF revenue are required but the majority of revenues arise from PED and RUC. It is necessary for the model to handle some technical complexities including payments to licensing agents, and refunds. Forecasts of NLTF revenues for each component and in total are required on an annual basis (June years) over at least a ten year period. The ability to generate at least a simple set of scenarios (eg low, medium, high) is required, to reflect the uncertainty associated with the forecasts. However, while scenarios are generated, the baseline (medium) scenario is generally adopted by users of the forecasts, eg by Treasury in its financial reports. Thus while representation of uncertainty is useful, accuracy of the baseline forecast is very important. There is also a requirement that the forecasts be readily explainable to users of the forecasts. This includes changes in trends and reversions to past trends. This means that it is desirable for the forecasting model and process to be relatively simple and transparent, although accuracy is still of primary importance. 2.5 Scenario analysis Ideally, the model should have the ability to generate forecasts under a range of scenarios, in order to test the effect of various shocks and trends on the forecasts. MoT staff told us that the types of scenario analysis that would be useful include the effects of changes in (among other things): Vehicle fuel efficiency and the mix of vehicles in the fleet including the shift from light petrol vehicles to light diesel; The propensity of drivers in different age groups or in different regions to own vehicles and to drive or use other forms of transport; The efficiency of freight supply chains, eg truck sizes and operational improvements that reduce empty running; Urban density and land uses on people s need to drive or use other forms of transport; and Transport fuel prices, taxes and user charges. In general, analysing the effect of any one of these changes requires that a suitable variable be included in the model in some form, and that a suitably robust relationship is established between that variable and revenues. Thus while the ability to test scenarios on all of the above may be desirable, it may not be practical within the context of a forecasting model. 3

30 For example, forecasting the effects of changes in fuel efficiency will require, at a minimum: An average fuel efficiency parameter in the model; Sensible estimates of the size of this parameter; An understanding of how this parameter affects NLTF revenues; and Guidance as to how this parameter may change over time. Each additional type of analysis therefore increases the complexity of the model, and requires sufficient data in order to estimate the necessary relationships. While this would allow a greater range of scenarios to be tested, it is not clear whether this will improve the accuracy of the forecasts. In general, forecasting favours simple models that fit the data well, while policy and scenario analysis tends towards more complex models that permit richer analysis but may be less suitable for forecasting, particularly in the short term. We return to this issue in sections 7 and 8 when we analyse options for re-designing the current forecasting model. 4

31 3 Context In this section we briefly analyse recent trends in transport activity in New Zealand and internationally, and discuss the implications for NLTF revenue forecasting. 3.1 New Zealand transport trends As a preliminary step, we have analysed overall trends in transport activity in New Zealand, using vehicle-kilometres travelled (VKT) as the measure of activity. The quarterly VKT data exhibits some seasonal fluctuations; to smooth this out we have calculated rolling annual totals as the basis for analysis Relationship between transport activity and economic activity Figure 17 illustrates the overall context for NLTF forecasting by showing the relationship between total annual VKT (for all types of vehicles) and total annual real GDP. The available data spans the time period from 2001 to Between 2001 and 2005, there was a very strong positive correlation between total VKT and total real GDP. After 2005, this correlation breaks down. During 2006, real GDP increased but total VKT declined slightly. In 2007 the positive correlation between VKT and GDP resumed, and the global financial crisis during 2008 was associated with a decline in both real GDP and VTK (also indicating a positive correlation). However from 2010 the economy has recovered and real GDP growth has resumed, while VKT has been volatile but essentially has not increased during the past three years. Figure 17 Total annual VKT and annual real GDP in New Zealand. Source: Ministry of Transport and Statistics New Zealand 5

32 This suggests that the apparently strong historical positive correlation between GDP and transport activity may no longer be reliable. At least, forecasting models estimated on the basis of this historic relationship are likely to forecast a reversion to that historic trend, while the most recent data indicates that the relationship between VKT and GDP is no longer so simple, and/or is being over-ridden by other factors that may require further investigation. This could include factors such as: Changes in the unemployment rate Changes in fuel prices or other transport-related prices Changes in population demographics, eg the age distribution or the rate of urbanisation Changes in the propensity to use of public transport, for example caused by improvements in the quality of public transport services. Our subsequent analysis considers all of these variables (and others) as potential drivers of transport activity and as potential explanations for the deviation from the historic correlation between transport activity and GDP. Further insights are provided by breaking down VKT into broad classes by vehicle type. Figure 18 shows the correlation between annual VKT of light (under 3,500 kg) petrolpowered vehicles and annual real GDP in New Zealand. In this case, similar features as Figure 17 are apparent, but the decline in VKT in recent years is even stronger. Between the third quarter of 2005 and the first quarter of 2013, annual real GDP increased by 12.3%, while annual VKT of light petrol vehicles decreased by 3.4%. Figure 18 Annual VKT for light petrol vehicles and annual real GDP in New Zealand. Source: Ministry of Transport and Statistics New Zealand 6

33 For diesel-powered vehicles, different correlations between VKT and GDP can be observed for different weight classes. There has been a strong positive correlation between VKT of light (< 3,500 kg) diesel vehicles and real GDP, and this correlation appears to be largely undisturbed by the global financial crisis and corresponding recession (Figure 19). Figure 19 Annual VKT for light diesel vehicles and annual real GDP in New Zealand. Source: Ministry of Transport and Statistics New Zealand. Among medium (3,500 kg 6,000 kg) diesel vehicles, the positive correlation between VKT and real GDP observed until late 2009 has essentially been reversed in subsequent years, with VKT of this type of vehicle declining sharply while real GDP has increased (Figure 20). For heavy vehicles (> 6,000 kg), the positive correlation between real GDP and VKT has essentially remained over time, however there is some suggestion of a weakening of this correlation in the most recent data (Figure 21). This analysis is summarised in Figure 22, showing the correlation between the VKT measures and real GDP, calculated on a rolling basis over two years. All measures of VKT are essentially perfectly correlated with real GDP up to the end of 2005, and the diesel VKT measures remain so until late The volatility during the global financial crisis and recession is apparent, but it is also apparent that, with the exceptions of heavy and light diesel VKT, correlations with GDP have not returned to their former levels. 7

34 Figure 20 Annual VKT for medium diesel vehicles and annual real GDP in New Zealand. Source: Ministry of Transport and Statistics New Zealand. Figure 21 Annual VKT for heavy diesel vehicles and annual real GDP in New Zealand. Source: Ministry of Transport and Statistics New Zealand. 8

35 Annual VKT per capita (km) Figure 22 Rolling (8-quarter) correlation between annual VKT and annual real GDP. Source: Covec analysis of Ministry of Transport and Statistics New Zealand data Per-capita transport activity Similar trends are also observed if transport activity is measured on a per-capita basis. Figure 23 shows total annual VKT per capita for all types of vehicle in New Zealand, and it is apparent that per-capita transport activity started to decline around Figure 23 Total annual VKT per capita in New Zealand. 9,800 9,600 9,400 9,200 9,000 8,800 8,600 8,400 Year Ended Quarter Source: Ministry of Transport and Statistics New Zealand. 9

36 Consistent with the above analysis, different patterns of VKT per capita are observed for different vehicle types (Figure 24). Most notable is the increasing use of light diesel vehicles, while use of light petrol and medium diesel vehicles has fallen. Figure 24 Annual VKT per capita indexes (2001Q4 = 100) for New Zealand. Annual VKT per capita (index) 140 Heavy diesel Medium diesel 130 Light diesel Light petrol Total Year Ended Quarter Source: Covec analysis of Ministry of Transport and Statistics New Zealand data Household travel behaviour The New Zealand Household Travel Survey, conducted by the Ministry of Transport, gives some insight into travel behaviour. The frequency of data releases from the survey (every three years) makes it unsuitable for direct use in generating NLTF forecasts, but it is a useful source of information about overall transport trends. Figure 25 shows the annual average distance driven per capita by drivers in different age groups in small vehicles. One apparently clear trend is the declining volume of percapita travel by people in the age group, while growth in per-capita travel among other age groups has been relatively modest or static. The total volume of travel across all age groups reported in the travel survey grew strongly from 18.3 billion km to 29.1 billion km in but has since remained essentially constant. 10

37 km per capita per annum Figure 25 Distance driven per capita in cars, vans, utes, and SUVs, by age group. 16, / ,000 12,000 10,000 8,000 6,000 4,000 2, Age group Source: Household Travel Survey and Statistics New Zealand NLTF revenues and volumes The above changes in transport activity have translated into changes in the size and composition of NLTF revenues over time. Light RUC volumes have grown relatively strongly over time, from 5.66 billion km in 2000/01 to 8.15 billion km in 2012/13, an average annual growth rate of 3.1% (Figure 26). Heavy RUC volumes have increased at a slower rate, from 2.70 billion km in 2000/01 to 3.55 billion km in 2012/13, an average annual growth rate of 2.3%. In contrast with RUC km, PED volumes (Figure 27) do not exhibit a general upwards trend, and display considerable volatility over time. We return to the issue of PED volatility in section 7; for now we note that PED volumes over the past 13 years have displayed a general pattern of increase until around 2007/08, followed by a general decline. The total PED volume in 2012/13 was almost identical to the level in 2000/01. 11

38 PED litres (millions) Net RUC km (millions) Figure 26 Annual net RUC km. 9,000 Light RUC Heavy RUC 8,000 7,000 6,000 5,000 4,000 3,000 2,000 1,000 0 Source: Ministry of Transport. Figure 27 Annual PED volumes (litres). 3,300 3,200 3,100 3,000 2,900 2,800 2,700 2,600 Source: Calculated from Ministry of Transport data. 12

39 Other New Zealand trends In addition to the above, other recent trends in New Zealand may have an impact on transport activity and NLTF revenues, including changing demographics, changes in the structure of the New Zealand economy, and changes in the use of public transport in cities. Our analysis in subsequent sections includes all of these variables, as well as prices and others that may have an effect on transport activity Demographics The key demographic changes occurring in New Zealand are changes in the age distribution of the population and increasing urbanisation. Figure 28 shows the distribution of age in the New Zealand population across broad generational groups. Overall the population is ageing, with a significant increase in the proportion of the population aged between 55 and 74, and a smaller increase in the proportion of the population aged over 75. The proportions of the population in the and age groups in recent years have slightly declined or remained constant. Figure 28 Population age distribution in New Zealand. 35% Under % 25% 20% 15% 10% 5% 0% Source: Statistics New Zealand Figure 29 shows the proportion of the total New Zealand population living in urban areas, as defined by Statistics New Zealand. This proportion has generally increased over time, although the urbanisation rate was relatively constant for much of the 2000s. Also notable is the significant increase in urbanisation in 2013, although it is not clear whether all of this increase occurred in one year (as the data suggests) or whether this is a feature of the 2013 Census that has not been applied to the urban population estimates in the years since the 2006 Census. This means that any modelling based on the urban population data should be undertaken with caution. 13

40 Proportion of real GDP Figure 29 Proportion of the New Zealand population living in urban areas. 85.4% 85.2% 85.0% 84.8% 84.6% 84.4% 84.2% Source: Statistics New Zealand Economic structure The nature of New Zealand s economy has gradually changed over time (Figure 30). Figure 30 Broad breakdown of New Zealand s real GDP. 70% Primary Manufacturing, construction & wholesale Retail & services 60% 50% 40% 30% 20% 10% 0% Source: Calculated from Statistics New Zealand data. 14

41 Annual trips per capita The primary sector has generally declined in relative importance, while the tertiary (retail and services) sector has increased. Secondary industries have also generally declined over time. One notable feature is that after around 2009 the growth in the share of activity attributable to retail and services has been curtailed, while the decline in secondary industries has stopped Public transport patronage Use of public transport in New Zealand has steadily increased over time (Figure 31) from 22 trips per capita in 2001 to just under 30 trips per capita in 2013, while total patronage has increased from 86 million boardings in 2001 to 133 million boardings in 2013, an average annual growth rate of 3.7%. Bus is the predominant public transport mode, with metropolitan rail networks only available in Auckland and Wellington, and ferry services only available in Auckland, Wellington, and Christchurch. Figure 31 Annual public transport patronage per capita. 35 Bus Rail Ferry All Source: Ministry of Transport and Statistics New Zealand. Year ended June Transport prices Figure 32 shows annual average real price indexes for selected transport-related goods and services in New Zealand. Retail petrol and diesel prices have generally increased over time, although considerable volatility has been observed in recent years. Real prices for the purchase of vehicles and for passenger transport services have fallen steadily over time, while real domestic air transport prices have increased but at a slower rate than fuel prices. 15

42 Cars per capita Index Figure 32 Real transport price indexes. 2,000 1,800 1,600 1,400 1,200 1, Petrol Purchase of vehicles Domestic air transport Diesel Passenger transport services Calendar year Source: Calculated from Statistics New Zealand and Ministry of Transport data Private vehicle ownership The rate of vehicle ownership appears to have changed in recent years (Figure 33). Figure 33 Registered cars per capita in New Zealand Source: Calculated from Statistics New Zealand data. 16

43 After increasing relatively strongly between 1991 and 2007, the number of registered cars per capita in New Zealand has generally declined, although there was a small increase between 2012 and While we do not model vehicle ownership directly in our analysis, this change in propensity to own private vehicles will show up in petrol volumes and road user charges, and is therefore implicit in our modelling. 3.3 International transport trends While we have not been able to undertake a comprehensive analysis of transport activity in other countries, similar trends as in New Zealand have been observed elsewhere. Figure 34 shows an index of annual VMT in the United States and United Kingdom. Aggregate transport activity grew strongly for most of the 1980s and 1990s, but experienced weaker growth in the early 2000s, declined during the financial crisis and recession in the mid-2000s, and has subsequently remained constant. Figure 34 Index of annual VMT in the US and UK. Source: US Department of Transportation & UK Department for Transport. Internationally, there has been considerable interest in whether the reduction in transport activity observed since the mid-2000s, in contrast to three decades of steady growth prior, is a temporary or permanent change. There appear to be two schools of thought regarding this question. One school suggests that this is a temporary shock, caused mainly by the global financial crisis, higher unemployment, and volatile fuel prices. For example, the International Transport Forum (2012) stated: The 2008 financial crisis triggered a severe, sudden and synchronised drop in demand leading to strong reductions in global output, trade and transport volumes. 17

44 The ITF generally considers these shocks to be temporary, and concludes: Transport flows are expected to grow strongly driven by higher GDP and larger populations. In the OECD, passenger transport volumes in 2050 are expected to be 10% to 50% higher than in Freight transport is expected to grow by 50% to 130%. The other school of thought, typified by authors such as Litman (2013) is that lower or declining transport volumes is the new normal, due to permanent rather than temporary changes in demographics, urbanisation, and the structure of the economy. Litman argues: Aging population, rising fuel prices, increasing urbanization, improving travel options, increasing health and environmental concerns, and changing consumer preferences are reducing demand for automobile travel and increasing demand for alternatives. Automobile travel will not disappear, but at the margin (compared with current travel patterns) many people would prefer to drive less and rely more on walking, cycling, public transport and telework, provided they are convenient, comfortable and affordable. This controversy makes forecasting future transport volumes challenging. At the very least, uncertainty over future transport projects is greater than it has been in the past, as previously steady growth has failed to occur for much of the past decade. We return to this issue in the New Zealand context in our analysis of private transport activity in section 7 below. 18

45 4 Issues with the existing NLTF model The primary concern with the existing model relates to the accuracy and reliability of PED and RUC revenue forecasts. Other issues arise from the way that the model has been designed and implemented. This review is based on our review of the model provided to us by the Ministry of Transport (updated to October 2013), and discussions with Ministry staff. 4.1 Forecast accuracy and reliability The structure of the existing model is explained in detail by Deloitte (2011a). Time-series econometric techniques were used to estimate error-correction models (ECMs) for PED and RUC volumes (litres and km respectively), as a function of a number of explanatory variables. These models are used to generate forecasts of PED and RUC volumes, from which NLTF revenues are calculated by applying appropriate dollar values. ECMs contain two components a long-run trend and a short-run error correction component that models the way the dependent variable deviates from the long-run trend in the short term. Over time, the long-run trend dominates, but the short-run component is also important for short-term forecasting accuracy. The models were estimated using quarterly data. This is because although only annual forecasts are required, the models need to be updated and new forecasts generated every quarter. Some of the quarterly data series, particularly PED volumes, exhibit significant volatility. Some of this may be seasonal, but some simply arises from random factors affecting the timing of fuel imports (see section below). This creates significant technical challenges for the short-run models, and makes it difficult to generate highly accurate short-run forecasts. The PED, heavy RUC, and light RUC volume forecasts generated by the model provided to us are shown in Figure 35 to Figure 37. PED volume is forecast to grow strongly in the first four years, and more slowly thereafter. Questions have been raised about the reliability of this forecast, given that PED volumes have generally been falling over the past five years. In contrast, the model predicts that by , PED volume will exceed the peak that it reached in More generally, the Ministry has expressed concern that the only negative long-run driver of PED volume in the model is real petrol prices, and given that real petrol prices are expected to be constant in the short term, the model predicts strong growth in PED volume in the coming years as a result of increased economic activity. The RUC volume forecasts are more consistent with recent trends, although the growth rate of heavy RUC is forecasted to accelerate over time. In comparison with PED, heavy RUC volumes appear to have recovered from the recession, with increases in the three years since The growth rate of light RUC has also been relatively stable over time, and the forecast is generally in line with this. The RUC volumes are also significantly more stable than PED volumes, which makes forecasting easier. 19

46 RUC km (millions) PED litres (millions) Figure 35 PED volume forecasts produced by the existing model. 3,500 3,400 3,300 3,200 Actual Forecast 3,100 3,000 2,900 2,800 2,700 2,600 2,500 Source: Ministry of Transport and Covec analysis. Figure 36 Heavy RUC volume forecasts produced by the existing model. 5,000 4,500 4,000 3,500 Actual Forecast 3,000 2,500 2,000 1,500 1, Source: Ministry of Transport and Covec analysis. 20

47 RUC km (millions) Figure 37 Light RUC volume forecasts produced by the existing model. 11,000 10,000 9,000 8,000 7,000 6,000 5,000 4,000 3,000 2,000 1,000 Actual Forecast 0 Source: Ministry of Transport and Covec analysis. In an initial review of the model s forecasting accuracy, Deloitte (2012) assessed the model s forecasts for the and years. Figure 38 summarises Deloitte s analysis of the model s forecasting performance in those years for heavy and light RUC, and PED. The total forecast error was decomposed into model error (due to the imperfect fit of the model) and input error (due to incorrect assumptions about forecast drivers). In the input error dominated and model errors were relatively small, but in the model errors became more significant. This is to be expected with forecasting over longer time horizons, but also raises the question of whether the model continues to be valid. To address this question, Deloitte conducted some simple structural break tests by examining regression residuals and testing the significance of dummy variables at different points. No evidence of a structural break was found in the PED model. In respect of RUC, Deloitte found: Some evidence of structural breaks for heavy RUC, but negligible overall improvement from incorporating these in the model. No strong evidence of structural breaks for light RUC. More recently, the Ministry reviewed actual NLTF revenue compared to the forecasts and found that the forecasting model had over-predicted PED and heavy RUC by up to 5%, and under-estimated light RUC revenue by up to 2%. 21

48 Figure 38 Forecasting errors of the NLTF revenue model in and % 5% Model error Input error 4% 3% 2% 1% 0% -1% -2% Light RUC Heavy RUC PED Light RUC Heavy RUC PED Source: Adapted from Deloitte (2012), table Design and implementation We have briefly reviewed the Excel implementation of the existing NLTF forecasting model. Overall, the model is complex, and this partly reflects the complexity of the forecasting task, but the design seems unnecessarily complex in some ways. In this section we comment on the model s structure and its ease of use, then in section 9.2 below we suggest ways that the model could be improved, taking into account our modelling of PED and RUC volumes. The model is structured around a series of sub-models for the various components of the NLTF. Each sub-model takes various data and assumptions, and generates a quarterly forecast of one component of NLTF revenue, which are then aggregated to June years for presentation. Separation between actual data and forecast assumptions The model does not maintain a clear separation between actual (observed) data for the various drivers of the NLTF revenue forecasts and future assumptions about these variables. For example, Figure 39 shows the table where actual and future forecast economic variables are entered. It is not clear from this table where the actual data ends and the forecasts begin. This may make it difficult to update the actual data, and may lead to errors when specifying forecast scenarios. 22

49 Figure 39 Table for entering actual and forecast economic variables. Source: NLTF forecasting model Re-estimation of model coefficients The model allows for re-estimation of the coefficients of the underlying econometric models when new data becomes available, via Excel s built in linear regression functions. The ability to re-estimate the coefficients is useful, but additional econometric diagnostic tests should be performed at the same time, to ensure that the models remain valid. Such testing is not possible within Excel, and so by allowing the coefficients to be updated there is a danger that the models will become invalid over time but this may not be noticed. In our view it would be better for re-estimation of the models to be undertaken as a separate process, every one or two years, including a full set of statistical diagnostic testing (see section below). The steps required to update the coefficients in the model are also relatively complex and may be error-prone (Figure 40). In part this is because the existing model is unable to detect the presence of new actual data that has been entered, and consequently a number of manual steps are required to update the data (see also below). 23

50 Figure 40 Instructions for re-estimating coefficients in the existing model. Source: NLTF forecasting model ECM structure is complex The heart of the forecasting model is a number of ECMs, which allow for relatively sophisticated time-series dynamics. However, the short-run components of the models appear to be complex, and include a number of variables that are not included in the long-run components of the models. This may make it difficult to explain the short-run forecasts that the model produces. For example, the long-run model for PED contains three variables: real GDP, real household consumption, and the real petrol price. The short-run model for PED contains real GDP as well as five other variables that do not appear in the long-run model: real investment, the stock of passenger vehicles, the working age population, the retirement age population, and the difference between retail mortgage rates and the 90- day interest rate (the interest wedge ). While the long-run model for PED is relatively easy to understand, the short-run model depends on six different forecasts, and may produce dynamics that are not easy to explain. Given the focus on short-term forecasts, this is of concern. While we have not undertaken a full econometric review of the ECMs in the existing model, a brief analysis suggests problems, including the use of a number of statistically insignificant variables. For example, in the long-run PED model, both real GDP and household consumption are insignificant; this is likely due to the very high correlation between these variables and it may not be useful to include both in the model. A complex seasonal adjustment model is used All of the NLTF volumes (and other variables) in the model appear to have a process of seasonal adjustment applied to them, regardless of whether reliable seasonal effects are present. This is particularly problematic for PED volume, which does not appear to have a regular seasonal pattern (see section below). The result is seasonal adjustment factors for PED volume that are very unstable (Figure 41). 24

51 Seasonal adjustment factor Figure 41 Seasonal adjustment factors for PED volume in the existing model Q1 Q2 Q3 Q4 Source: Covec analysis of the NLTF forecasting model. Ad hoc parameter shocks are allowed for The model allows for manual adjustment to all of the regression coefficients in the models, via either a level or percentage adjustment (Figure 42). This adds complexity to the model and increases the possibility of errors, for limited benefit in our view. Adjusting the parameters of the models defeats the purpose of using econometric models for forecasting, and the basis for making any adjustments to the parameters is unclear. Such adjustments reduce the transparency and predictability of the forecasts produced by the model, and also require a relatively complex infrastructure to allow these shocks to flow through to the forecast calculations. Figure 42 Parameter shock control panel in the existing model. Source: NLTF forecasting model 25

52 Workflow for specifying the forecast scenario is complex Ideally, the model would allow the user to quickly and easily specify a forecast scenario encompassing all of the exogenous variables in the model. Instead, the forecasts of exogenous variables are mixed with actual data for these variables (see above), and a separate set of shocks to the variables is allowed for (Figure 43). This structure makes it difficult to understand exactly what the forecast scenario is, and the process for testing different scenarios appears to be complex and error-prone, particularly given the fact that a large number of variables need to be specified. Figure 43 Specification of variable shocks in the existing model. Source: NLTF forecasting model Model updating process is complex Finally, the process for updating the model each quarter when new data becomes available appears to be complex and time-consuming. This is partly because the model does not automatically detect the presence of new data and manual changes are required to move the forecasting period forwards (Figure 44). Figure 44 Instructions for updating the existing model with new actual data. Source: NLTF forecasting model 26