Visualization of distance measures implied by forecast evaluation criteria

Transcription

1 Visualization of distance measures implied by forecast evaluation criteria Robert M. Kunst Institute for Advanced Studies Vienna and University of Vienna Presentation at RSS International Conference, Manchester, September 06 /3

2 Outline Introduction Symmetric criteria 3 Asymmetric criteria Optimizing accuracy criteria? /3

3 The forecaster s problem The forecaster searches for values f t,t =,...,m, (the forecasts ) that are as close as possible to the unknown true values (or actuals ) y t,t =,...,m. In the customary stochastic framework, one may see y t as realizations of random variables, and f t as random variables measurable in an information set defined by (other) realizations of y. In a non-stochastic framework, one may see forecasts as simple functions of data. (Kunst & Jumah, 00) In any case, the forecaster s problem demands for a definition of closeness. 3/3

4 Distances between forecasts and actuals A typical measure of distance { R d(.) : m R + (Y,F) d(y,f) contains a local distance function d : R R R + and a link function R +m R +. For example, the mean squared error has a quadratic local distance function d(y,f) = (y f) and the arithmetic mean as a link function. /3

5 Conditions for a good distance measure Among the conditions for a good distance measure, some were named often (see, e.g., Armstrong and Collopy, 99): scale-free: appears to be important only if several variables are evaluated jointly; robust: untouched by isolated events of forecast failure; sensitive: reacts to improved forecast quality (contradiction to robust ); should provide similar rankings to other criteria. Not so often named were the following properties: monotonicity: should improve as a forecast approaches the actual or vice versa; should be consistent with the estimation procedure for continuous parameters within classes of prediction models. 5/3

6 Axioms of a metric Definition A metric is a distance function d : M R + that fulfills three axioms for any elements x,y M: d(x,y) = d(y,x) () d(x,y) = 0 x = y () d(x,z) d(x,y)+d(y,z). (3) Distance functions used in prediction evaluation often violate these conditions. Squared distance (MSE), absolute distance (MAE), and maybe maximum metric are the only metrics directly used. 6/3

7 Symmetric and asymmetric criteria Symmetric distance functions fulfill the first axiom of a metric. Approaching the actuals by the forecasts and the forecasts by the actuals has the same effect on d(y,f). MSE, MAE, smape are symmetric. Asymmetric distance functions violate the first axiom. In some cases, they deteriorate (increase) if actuals approach forecasts. Note that macroeconomic actuals often change (are revised) after the forecasts have been announced. MAPE is a popular asymmetric criterion. 7/3

8 The contour plots Consider contour plots connecting points Y = (y,y ) with identical distance d for a given F = (f,f ), or points F for a given Y in the R space. Issues with these contour plots: For symmetric d, only one contour plot is shown. It may be of interest to consider the plots for different values of the givens, as some d change across locations (MAPE). Some link functions can only be identified for m >. For example, the concept of a median is not recognizeable from the case m =. Alternatively, Kim and Kim (06) present coloured contours in an (Y,F) world, visualizing only distance, not the link. 8/3

9 The classical distance functions The criteria mean squared error (MSE) and root mean squared errror (RMSE) m m d (Y,F) = m (y t f t ), d RMSE (Y,F) = m (y t f t ), t= and mean absolute error (MAE) m d (Y,F) = m y t f t, t= t= are often disliked because of their scale dependence, which, however, is easily repaired. 9/3

10 5 Contours for MAE and MSE Figure: Points with identical forecast accuracy if the true value is (, ) or actuals yielding identical accuracy for a given forecast of (, ). MAE and MSE. 0/3

11 Weighting the observations A forecaster may be more interested in precision close to the end of the sample, as this may be a good indicator for the ability to predict the physical future. This idea is accommodated by weighted criteria such as m d (Y,F) = m w t (y t f t ), t= where m t= w t = m and w t increasing in t may be natural specifications. /3

12 Contours for weighted MAE and MSE Figure: Forecasts with identical accuracy if true value is (, ) or actuals yielding identical accuracy for given forecast value (, ). Weighted MAE and MSE with observation on x axis interpreted as the older observation. /3

13 Linking by maximum and minimum Whereas linking absolute errors by the maximum d max = max t=,...,m y t f t corresponds to L, a normed space and a metric, linking by the minimum d min = min t=,...,m y t f t does not correspond to a metric, as the triangle inequality is violated. The minimum criterion is very robust and ignores isolated forecast failure events. It violates two of the three axioms of a metric. 3/3

14 3 Contours for maximum and minimum criteria Figure: Points with identical forecast accuracy if the true value is (, ) or actuals yielding identical accuracy for given forecast (, ). Maximum of absolute differences on the left and minimum on the right. /3

15 Other arbitrary exponents Distance function d p (Y,F) with ( m ) /p d p (Y,F) = y t f t p t= are less attractive for p,. They are non-robust for p > and non-convex for p <. Non-convex contours imply failure of the triangle inequality and an implausible emphasis on linear combination of values rather than values proper. 5/3

16 5 Contours for L and L 0.5 distances Figure: Points with identical forecast accuracy if the true value is (, ) or actuals yielding identical accuracy for given forecasts (,). L criterion on the left and square root criterion on the right. 6/3

17 The symmetric MAPE The symmetric mean absolute percentage error (smape) smape(y,f) = m y t f t m y t +f t = f t y t +f t t= is scale-free and nonetheless symmetric (although its symmetry was disputed by Koehler, 00). It becomes undefined for f t = y t and improves beyond that value: non-monotonic reaction. Predicting a recession with growth rate % by a growth rate of % does not appear to be unusual in macro-economic forecasting. t= 7/3

18 Contours and d for the smape dsmape Figure: Forecasts with identical accuracy if true value is (, ) or actuals yielding identical accuracy for given forecasts of (, ) for smape criterion. Right graph shows the local distance function for smape. 8/3

19 The geometric MAE The concept of the the geometric mean absolute error (gmae) uses a geometric average as a link function (see, e.g., Fildes, 99): gmae = m m f t y t The competition can be won by few extremely precise forecasts. This is another non-convex criterion. It violates two of the three axioms of a metric. t= 9/3

20 Contours for the geometric MAE Figure: Actuals with identical forecast accuracy if the forecast value is (,), or forecasts with identical accuracy if actual is (,), if the criterion is the geometric MAE. 0/3

21 Mean absolute prediction error The mean absolute percentage error (MAPE) d MAPE (Y,F) = m y t f t y t t= m = f t is maybe the most popular criterion in applied forecasting. It is scale-independent and asymmetric. Its robustness is scale-dependent: robust for large values and positive errors, sensitive for small values and negative deviations. It is undefined for y t = 0 and becomes non-monotonic beyond that value. t= y t /3

22 Contours for the MAPE Figure: Points with identical forecast accuracy if the true value is (, ) on the left and actuals yielding identical accuracy for given forecasts (, ) on the right. Criterion is mean absolute prediction error. /3

23 Local distance function for MAPE Figure: Local distance function for the MAPE criterion. True value is. 3/3

24 The mean absolute scaled error The mean absolute scaled error (MASE) is due to Hyndman and Koehler (006). They expressed scales by increments m t= MASE = f t y t (m ) m t= y t y t, but it should also be possible to use the sum of absolute actuals, m t= MASE 0 = f t y t m m t= y t. These criteria only become undefined if the actuals are time-constant or identically zero. Nonetheless, even the MASE is asymmetric and non-convex. /3

25 Contours for MASE Figure: Actuals with identical forecast accuracy if the forecast value is (,), left graph for the MASE as suggested by Hyndman and Koehler (006), right graph for scales by levels instead of differences. 5/3

26 The mean arctangent absolute percentage error The mean arctangent absolute percentage error (MAAPE) is due to Kim and Kim (06). They subject the local distance to an arc-tangent transformation MAAPE = m m t= arctan f t y t. y t The MAAPE remains finite and well-defined even with y t = 0. The arctangent function can be motivated geometrically, but it is arbitrary. The arctangent is bounded, so the MAAPE is always defined, in contrast to the MAPE. 6/3

27 Contours for MAAPE Figure: Forecasts (left) and actuals (right) with identical forecast accuracy if the forecast value is (, ). Criterion is the MAAPE suggested by Kim and Kim (06). 7/3

28 Local distance function for MAAPE Figure: Local distance function for the MAAPE criterion. True value is. 8/3

29 Criteria not covered here Many researchers use relative criteria that compare forecast accuracy with a benchmark, such as a random walk forecast (Theil U etc.). Keeping the benchmark constant leads to comparable contours, so excluding them does not limit generality. A particular case with a peculiar link function is the percent better criterion. 9/3

30 Discrete and continuous choice Forecast evaluation criteria are typically used to compare rival models or institutions. Why are they not used to compare parameter values for a constant model structure? Most loss and distance functions can be regarded as bases for robust M estimation. In M estimation, a function is maximized that is not the likelihood for any distribution. Why not do this, for example, for the MAPE? The answer is simple: the results of doing this are abysmal. Are the results of accuracy comparisons across models or institutions so promising? Studies of this kind may be helpful. 30/3

31 Some cursory simulations Autoregressive coefficient selected as the minimum of smape criterion. Generating model is y t = φy t +ε t for φ = 0.9 (left) and φ = 0.5 (right), T = 00. 3/3

32 Tentative conclusions Classical criteria, such as MAE and MSE, are most attractive, as they provide a reasonable distance concept and the most logically consistent decision; Optimizing percentage criteria can lead to an undue emphasis on white-noise (forecast is constant) and random-walk (forecast is last value) forecasts; If insensitivity to scales is such a strong concern, why not scale all variables by classical or by robust criteria to location 0 and to variance ; Could we learn more from three-dimensional visualization? For more arguments critical of percentage criteria (MAPE, smape), see Franses (06). 3/3

33 Thank you for your attention 33/3

34 References Armstrong, J.S., and Collopy, F. (99) Error measures for generalizing about forecasting methods: Empirical comparisons, International Journal of Forecasting 8, Fildes, R. (99) The evaluation of extrapolative forecasting methods, International Journal of Forecasting 8, Franses. P.H. (06) A note on the Mean Absolute Scaled Error, International Journal of Forecasting 3, 0. Hyndman, R.J., and Koehler, A.B. (006) Another look at measures of forecast accuracy, International Journal of Forecasting, Kim, S., and Kim, H. (06) A new metric of absolute percentage error for intermittent demand forecasts, International Journal of Forecasting 3, Koehler, A.B. (00) The asymmetry of the sape measure and other comments on the M3 Competition, International Journal of Forecasting 7, Kunst, R.M., and Jumah, A. (00) Toward a Theory of Evaluating Predictive Accuracy, IHS Economics Series, 6, Institute for Advanced Studies, Vienna. 3/3