GDP forecast errors Satish Ranchhod

Transcription

1 GDP forecast errors Satish Ranchhod Editor s note This paper looks more closely at our forecasts of growth in Gross Domestic Product (GDP). It considers two different measures of GDP, production and expenditure, and also examines the impact of revisions to this series on assessment of our forecast errors. Executive Summary This note examines the Reserve Bank s forecasting performance with respect to quarterly growth in Gross Domestic Product (GDP) between December 1994 and March Frequent revisions to GDP data by Statistics New Zealand complicate analysis of our forecasting performance. Using GDP data as reported in the middle of the following year (which is generally accepted in the literature as the most suitable outturn for assessing forecasting performance) we find a positive bias in the Reserve Bank s forecasts of midterm GDP growth we have tended to over-predict quarterly growth 4 to 8-quarters ahead. On average our forecasts for these horizons have been higher than actual quarterly growth by approximately 5 percentage points. This holds for both the production and expenditure measures of GDP. However, the observed bias is not significantly different from zero at most horizons. This finding is likely to be influenced by the volatility of the sample and the limited number of available observations. 1 Introduction This note examines the Reserve Bank s statistical forecasting performance for quarterly growth in Gross Domestic Product (GDP) growth between December 1994 and March ly forecasts of growth in both Expenditure (GDPE) and Production (GDPP) based real GDP are examined. Greater weight is placed on the findings for GDPP. GDPP is thought to have been more reliable than GDPE over the sample period and is less volatile. The size of the calculated forecast errors, and hence our evaluation of forecast accuracy, is influenced by the particular vintage of data that is used as the outturns for calculating errors. Initial estimates of GDP are frequently subject to revisions and re-basing over time; the effect of such changes can be large. However, revisions to estimates that occur after a number of years are of limited relevance to forecasters or forecast users (Zarnowitz and Braun, 1992). We examine the Reserve Bank s forecasting performance using three vintages of the GDP data: First release, un-revised data Revised GDP data, defined as the data as at the middle of the following year Latest release GDP data (from the March 2002 quarter) Following the work of Zarnowitz and Braun (1992), Batchelor (2001) and St Clair and Yates (2001), we place greater weight on those forecast errors calculated using revised data. 1 The use of quarterly growth rates aims in part to avoid the difficulty of overlapping observations. However, serial correlation may still exist in quarterly forecast errors. Consequentially, we examine the forecast errors at all horizons for serial correlation. Also, when conducting t-tests on our mean forecast errors we use a correction for serial correlation.

2 Revised data tends to be more reliable than first release data, but is still timely relative to latest release data, making it more relevant for policy decisions. Forecast errors are defined as forecast less actual and are calculated using deseasonalised data. Hence, a positive mean forecast error reflects a tendency to over-predict the rate of GDP growth while a negative error reflects a tendency to under-predict. We assess forecasting performance in terms of forecast bias (as measured by the mean forecast error) and by the size of forecast errors (as measured by the mean absolute forecast errors and the root mean forecast error). ly forecasts up to 9 quarters ahead are considered. Because of the delay in publishing GDP data, the quarter prior to the publication date of our forecasts and the current quarter are both in fact forecasting quarters. We note that quarterly GDP growth tends to be highly volatile. This makes it more difficult to prove bias statistically. Further, we have, at most, 32 observations for each forecast horizon. The tests used to determine the significance of any observed bias in our forecast errors have limited power when the number of observations is small. The remainder of the note is structured as follows: Sections 2 and 3 present our findings for production and expenditure based GDP. Section 4 considered the effects of using different vintages of outturns when calculating forecast errors. Section 5 concludes. 2 Forecast errors for production-based GDP On average, the Reserve Bank has not tended to either under- or over-predict quarterly GDPP growth for short-term horizons, using any of the vintages of data. However, 4 to 8 steps ahead, forecast errors calculated using first release and revised data suggest that, on average, we have over-predicted GDPP growth. On the other hand, using latest release data as actual GDP indicates that we have not significantly under- or over-predicted real GDPP growth over any of the forecast horizons considered. This reflects the fact that while we have over-predicted GDPP beyond the shortterm (using first release or revised data), data revisions have tended to increase measured GDPP, making our errors appear smaller and the bias statistically insignificant when using latest release data. Figure 2.1 plots the mean forecast errors RMSE for all outturns. Figure 2.2 plots the actual forecast errors for the 4-quarters ahead horizon. We note that at this horizon forecast errors calculated using all three outturns display similar patterns. However, the forecast errors calculated using latest release data appear slightly more volatile.

3 Figure 2.1 Mean forecast error and summary forecast statistics for GDPP: All outturns 0.3 % % First release Revised Latest release Previous Current s ahead - RMSE First release Revised Latest release RMSE Previous Current s ahead

4 Figure 2.2 GDPP forecast errors for the 4-quarters ahead horizon % Initial Revised Latest % Forecast date We now turn to a closer examination of each data vintage. 2.1 Forecast errors calculated using first release data Our findings using first release data are summarised in table 2.1. The Reserve Bank s forecasts of GDPP growth for short-term forecasting horizons have been unbiased. However, we have tended to over-predict GDPP growth in the medium term. Mean forecast errors 2 to 9 quarters ahead range between 2 and 0.36 percentage points. They are significantly different from zero at the 2 quarters ahead horizon and the 4 to 7 quarters ahead horizons. As one would expect, the size of our forecast errors (as measured by MAE and RMSE) increases as the forecasting horizon lengthens. To examine whether there has been a change in our forecast performance over time, we use a rolling three-year sample of our forecast errors. We find that for short-term horizons, mean errors have remained relatively constant over the sample period but the size of our forecast errors has grown. At longer horizons our mean errors have fallen over the sample period (less over-prediction), but the average size of our errors has grown. Note that this and all similar results are likely to be a feature of the volatility of the actual data series, rather than a change in the nature of our forecasts. The volatility of both GDPP and GDPE has increased substantially since 1997, using both initial release and revised data vintages (the pattern is less clear using latest release data).

5 Table 2.1 ly GDPP growth forecast errors statistics calculated using first release data s Mean MAE RMSE ahead Previous Current * * ** ** * Notes: Asterisks indicate the significance with which the null hypothesis: Mean Error = 0 can be rejected: ** = Significant at the 5 per cent level * = Significant at the 10 per cent level 2.2 Forecast errors calculated using revised data Table 2.2 summarises our findings using revised data (ie GDPP as it stands in the middle of the following year). The results are qualitatively similar to the findings using first release data. Using this measure of actual GDPP we also find that, on average, our forecasts of GDPP growth for short-term horizons are unbiased. Forecasts for mid- to long-term horizons, however, have consistently over-predicted GDPP growth. On average, our forecast of quarterly GDPP growth 4 to 8 quarters ahead have been at least 4 per cent higher than actual GDPP growth. However, only mean forecast errors for the 5 and 6-quarters ahead forecasting horizons are significantly different from zero. The average size of our forecast errors grows as the forecasting horizon lengthens. We find that for all horizons, the mean forecast errors have remained fairly constant over time. However, we again find that the size of our forecast errors has been growing over the sample period at most horizons. Table 2.2 ly GDPP growth forecast errors statistics: calculated using revised data s Mean MAE RMSE ahead Previous Current * ** Notes: Asterisks indicate the significance with which the null hypothesis: Mean Error = 0 can be rejected: ** = Significant at the 5 per cent level * = Significant at the 10 per cent level

6 2.3 Forecast errors calculated using latest release data Table 2.3 summaries our findings for forecast errors calculated using latest release data. Mean forecast errors calculated using this vintage of data tend to be lower than those calculated using first release or revised data. As will be discussed in section 4, this reflects that GDP has been revised over time upwards closer to our forecasts. Using latest release data indicates that at all forecast horizons, we have not consistently under- or over-predicted quarterly GDPP growth; the mean forecast errors for all horizons are not statistically significant from zero at conventional significance levels. We note that, as with other outturns, the mean forecast errors are largest for the 5 to 7-quarters ahead forecast horizons. Forecasts for these horizons have, on average, over-predicted GDPP growth by at least 0.15 percentage points. Forecasting accuracy is similar at all horizons. The mean forecast errors for short- to medium-term horizons (up to 3 quarters ahead) have increased over the sample period. Not surprisingly, the mean forecast errors for these horizons also appear to be slightly higher (greater over-prediction) during the Asian crisis. The size of the forecast errors for forecasts up to 3 quarters ahead has also grown slightly over time. The size of forecast errors for longer horizons (7 to 9 quarters ahead) has fallen slightly over the sample period. Table 2.3 ly GDPP growth forecast errors statistics calculated using latest release data s Mean MAE RMSE ahead Previous Current Forecast errors for expenditure-based GDP The findings of our tests are influenced by not only the chosen vintage of data (initial release, revised, or latest release), but also by the chosen measure of GDP (production or expenditure). For near-term forecast horizons, mean forecast errors calculated using GDPE demonstrate slightly greater over-prediction than those calculated using GDPP. They are similar for longer horizons. GDPE forecast errors tend to be more volatile than GDPP errors when using revised or latest release data. This difference is especially pronounced when we consider revised data. Note that there is no statistical difference between the mean forecast errors for the two GDP measures (this holds for all forecast horizons). The observed differences in statistical tests are likely to reflect the differences in the sample variances and the weak power of our statistical tests given their limited sample size. However, using both measures of GDP, there appears to be a positive bias in our medium-term forecasts (though evidence of this bias is stronger in our forecasts of GDPP). As with GDPP we find that, on average, the Reserve Bank has not tended to under- or overpredict quarterly GDPE growth for short-term horizons. However, first release and revised data indicate that, on average, we have over-predicted GDPE growth 4 to 7 quarters ahead.

7 Successive revisions to GDPE data have reduced evidence of bias in our forecasts. The mean forecast errors and RMSE for all outturns are presented in figure 3.1. Forecast errors for the 4 quarters ahead horizon are plotted in figure 3.2. At this horizon we observe a similar pattern in the forecast errors calculated using all three vintages of outturns. However, the forecast errors calculated using latest release data appear to be slightly more variable. Figure 3.1 Mean forecast error and summary forecast statistics for GDPE: all outturns % % First release Revised Latest release Previous Current s ahead RMSE First release Revised Latest release RMSE Previous Current s ahead

8 Figure 3.2 GDPE forecast errors for the 6-step ahead horizon % Initial Revised Latest % Forecast date Forecast errors calculated using first release data Table 3.1 summarises our findings for forecast errors calculated using first release GDPE data. Forecasts for the previous, current and 1 quarter ahead forecast horizons are statistically unbiased. 2 Forecasts 2 to 8 quarters ahead have tended to over-estimate quarterly GDPE growth. Mean forecast errors for these horizons have been at least 3. However, only the mean forecast errors for the 2 quarters ahead and 4 to 6 quarters ahead horizons are significantly different from zero at the 10 per cent level or better. The accuracy of our forecasts is similar at all forecast horizons. We again use a rolling 3 year sample to examine whether our forecast performance has changed over time. At most horizons our mean forecast errors have fallen over the sample period. The size of our errors for near term horizons has tended to increase over the sample period, while for longer horizons the size of our errors has fallen. Table 3.1 ly GDPE growth forecast errors statistics calculated using first release data s Mean MAE RMSE ahead Previous Current * * * * Notes: Asterisks indicate the significance with which the null hypothesis: Mean Error = 0 can be rejected: ** = Significant at the 5 per cent level * = Significant at the 10 per cent level 2 We note that the mean forecast error for the previous quarters is quite large but not statistically different from zero.

9 3.2 Forecast errors calculated using revised data When revised data is used to calculate the GDPE forecast errors we find that, on average, forecasts for the current quarter and 1 quarter ahead have tended to be relatively unbiased. Forecast for the 2 to 8 quarters ahead horizons have, on average, over-predicted quarterly GDPE growth by at least 5 percentage points. However, our mean forecast errors are not significantly different from zero at any of the horizons examined despite being very similar to mean errors calculated using initial release data. The size of the forecast errors is similar at all horizons. Our findings using revised data are summarised in table 3.2. Over the sample period our mean forecast errors at most forecasting horizons have remained relatively constant. The size of our errors has tended to increase slightly. Table 3.2 ly GDPE growth forecast errors statistics calculated using revised data s Mean MAE RMSE ahead Previous Current Forecast errors calculated using latest release data Table 3.3 summarises our findings using latest release data. These indicate that we have not consistently under- or over-predicted GDPE at any of the horizons considered. As with GDPP, the mean forecast errors calculated using latest release data are noticeably lower than those calculated using either first release or revised data (at most horizons). However, they are also more variable than either of the other outturns at most horizons. This makes it more it more difficult to demonstrate bias statistically. For near-term forecasting horizons, mean forecast errors calculated using latest release data have tended to increase over the sample period. They have fallen slightly for longer-term horizons. The size of our errors has remained relatively constant or fallen over time. Table 3.3 ly GDPE growth forecast errors statistics calculated using latest release data s Mean MAE RMSE ahead Previous Current

10 4 Comparison of different vintages of outturns As noted earlier, the size and bias of the calculated forecast errors are influenced by the particular vintage of outturns used. Our examination indicates that there are stronger signs of bias in those forecast errors that are calculated using first release and revised data than in those calculated using latest release data. We now briefly examine the properties of the different outturns and the effect this may have had on calculation of our forecast errors. Figures 4.1 and 4.2 present revisions to GDP that have occurred over the sample period. For both the production and expenditure measures of GDP we see that subsequent revisions have tended to revise the level of GDP upwards. For GDPP, this is particularly pronounced for the revision that occurred in For GDPE we see that there was a substantial upward revision in Figure 4.1 Revisions to GDPP 1350 Historical GDPP figures GDPP index (Base: 1990=1000) May May-98 May-99 May-00 May-01 Jan-89 Jul-89 Jan-90 Jul-90 Jan-91 Jul-91 Jan-92 Jul-92 Jan-93 Jul-93 Jan-94 Jul-94 Jan-95 Jul-95 Jan-96 Jul-96 Jan-97 Jul-97 Jan-98 Jul-98 Jan-99 Jul-99 Jan-00 Jul-00 Jan-01 Date 3 4 This revision was particularly large because it also included a change from fixed weight to chain-linked data when calculating GDP. From September 1997 Statistics NZ included new information on a number of aggregates such as household consumption. Revisions also occurred in the measurement of the gross fixed capital formation and imports series.

11 Figure 4.2 Revisions to GDPE 1350 Historical GDPE figures GDPP index (Base: 1990=1000) May May-98 May-99 May-00 May-01 Jul-89 Jan-90 Jul-90 Jan-91 Jul-91 Jan-92 Jul-92 Jan-93 Jul-93 Jan-94 Jul-94 Jan-95 Jul-95 Jan-96 Jul-96 Jan-97 Jul-97 Jan-98 Jul-98 Jan-99 Jul-99 Jan-00 Jul-00 Jan-01 Date When using latest release data, forecast errors for both GDPP and GDPE are more variable, and mean forecast errors tend to be lower, than when using other vintages of outturns. However, the size of the forecast errors calculated using latest release data tends to be larger than those calculated using either of the other outturns. This reflects that there are more large negative forecast errors when using latest release data. These pull down the mean forecast errors, but increase the average size of the errors. While we do not find statistically significant differences between the three vintages of outturns (in terms of means and variances), the differences that exist are large enough to influence our findings of bias for each series. 5 In particular, we note that the effect of the successive upward revisions to GDP figures has been to reduce the mean of our forecast errors but also increase their volatility. This reduces the likelihood of detecting a statistically significant bias in our forecasts. However, revisions to GDP that occur after a number of years are of limited use to forecasters and decision-makers. The more timely nature of revised data suggests that it is more useful for policy-makers. It is also likely to be more reliable than first release GDP figures. Hence it is generally argued that forecast errors calculated using this revised data should be given more weight than those calculated using either first or latest release data. 5 Conclusion We have tended to over-predict GDP growth, and successive data revisions have tended to revise up GDP. As a result, mean forecast errors calculated using latest release data are lower than those calculated using first release and revised data. These revisions have also increased the size of our forecast errors (as measured by RMSE and MAE). However, as data revisions that occur after a number of years are of little use to forecasters and decision-makers, we focus on forecast errors calculated using revised (middle of the following year) data. Using revised data, our findings indicate that the Reserve Bank has not tended to over- or under-predict GDP growth in the near term. However, on average we have over-predicted 5 This finding may also be influenced by the small sample size.

12 quarterly GDP growth 4 to 8-quarters ahead. Our forecasts of GDP for these horizons have tended to be 5 percentage points higher than actual quarterly growth. This holds for both production and expenditure-based measures of GDP. The size of our forecast errors tends to be larger for longer forecast horizons. We obtain similar findings using first release data. However, when latest release data is used, we find that the Reserve Bank has not tended to over- or under-predict GDP growth at any of the horizons considered. In 1997 the Reserve Bank changed its forecasting methodology. Prior to June 1997 our forecasts were framed in terms of how we believed the economy would evolve assuming the Reserve Bank took no policy action. Since this time our forecasts have allowed for adjustments in monetary conditions that are consistent the Reserve Bank s objectives in relation to inflation. There is some weak evidence to suggest that our forecasting performance for near-term horizons was more accurate during the period in which our forecasts were based on no policy response. However, any difference in forecasting performance during the two periods cannot necessarily be ascribed to the differing forecasting methodologies. Events specific to each sample period (such as the Asian crisis and the droughts that were experienced) are likely to have large influences on our findings. GDP data itself has been considerably more volatile since 1997 than in the few years prior to that. This naturally leads to greater forecast errors. This paper aims only to describe our GDP forecast errors and not relate them to our CPI forecast errors. However, it is worth noting that we tend to over-predict quarterly GDP growth but under-predict quarterly CPI inflation. This is the opposite of what might be expected. The likely reconciliation of this seeming contradiction is the fact that we do not only consider total output when making our inflation forecasts, but rather on the balance between demand and supply. Since 1997 this notion has been captured formally using an output gap. The role of GDP errors and GDP revisions in explaining our CPI forecast errors will be topics for further research.