Percentage errors can ruin your day. Roland Martin, Stephan Kolassa June 27, 2012

Transcription

1 Percentage errors can ruin your day Roland Martin, Stephan Kolassa June 27, 2012 SAP

2 The problem Forecast Accuracy measurement Used to determine the success of the forecast process Forecasting method, parameters, model Overall performance of the system Helps finding weaknesses in the model, data or the business Often forecasters are paid according to the resulting forecasting error/accuracy Low forecast error High reward Goal (even unwillingly): Choose metric with low errors Overall goal: Find the best forecasts for the business (and nothing else!) 2012 SAP AG. All rights reserved. 2

3 Simplifying the problem Reduction to an easier scenario Businesses are so different but all share one common item: There is a forecast and there is a given The same holds even for all types of time series (sparse, seasonal, lumpy demand, ) Therefore, reduce the scenario to a very simple one: Find a forecast for some possible values The goal here is not to find the best forecast method but to illustrate what happens in terms of error interpretation 2012 SAP AG. All rights reserved. 3

4 MAPE as error measure Measures can be biased Why is the error measure of that importance? Isn t one as good as the other? No! Bias Not telling or misleading Example: MAPE (Mean Absolute Percentage Error) Biased: Rewards underforecasts Comparing different magnitudes: Does not weigh between different magnitudes Problem if one of the actuals is zero Why the (M)APE is chosen often as forecasting measure Easy interpretation as the percentage of the actual value Scale-free (not measured in units or currency) possible to compare time series with different sales levels (with the danger mentioned above) Usable across multiple products or services 2012 SAP AG. All rights reserved. 4

5 Rolling dice Why the heck using Rolling Dice as an example? Everyone understands rolling a die (you do not need forecasting expertise for that) A die produces a random series of numbers that is not biased Mean value is 3.5 (easy to understand even to non-statisticians) The chance that people understand that 3.5 is the best (error least) forecast one can make is very high and if people still do not believe you can show this in a real-time demo using a die, a pen and paper ( or leading-edge technology like excel ) Once people believe that 3.5 is the best forecast investigate error measures 2012 SAP AG. All rights reserved. 5

6 Rolling dice - explained How does it work Take a standard six-sided die The die rolls represent natural variations for an item with no trend, seasonality or causal factors Ask for the best forecast for the die rolls upfront Most certainly the answer will be 3.5 Why? Cause it is the mean Roll (or ask someone to roll) the die several times to simulate demands For ten successively roles there is a 80% chance that the MAPE for a forecast of 2 is lower than that of 3.5 (increase number of rolls to increase the chance) Lessons to learn: Although we know that the best forecast is 3.5 the MAPE mocks us with a best forecast of 2 If in such an easy example the MAPE mocks us, guess what he does in a more complicated scenario 2012 SAP AG. All rights reserved. 6

7 The problem with the MAPE Where does the problem come from Percentage errors explode when the actuals are very small compared to the forecast Comes from the asymmetry of percentage errors for under- vs. overforecasts o APE for underforecasts will be between 0-100% o APE for overforecasts has no upper bound Example for rolling dice (mean 3.5): Roll of 2 and forecast 3.5 APE of 75% Roll of 5 and forecast 3.5 APE of 30% If your performance is measured by MAPE bias forecasts downward with all negative consequences on the downstream planning for example Especially for time series with a high coefficient of variation this holds 2012 SAP AG. All rights reserved. 7

8 Do we want biasedness? Asynchronous costs Perishables vs never-out-of-stock items But under-forecasting should not be mistaken to solve overstocks The degree of underforecasting which results from minimizing the MAPE may not correspond to the desired degree of bias Use unbiased point forecasts and item specific safety stocks resulting from quantile forecasts This leads to considering the loss function and a Cost of Forecast Error (Goodwin, 2009) 2012 SAP AG. All rights reserved. 8

9 Alternatives to the MAPE Alternatives to the MAPE ASE (Absolute Scaled Error) (Hyndman & Koehler 2006) 1 n 1 y i y i n i=2 y i y i 1 wmape (weighted Mean Absolute Percentage Error) n i=1 n i=1 y i y i y i 2012 SAP AG. All rights reserved. 9

10 Conclusions Test your error measures Using the MAPE for forecast evaluation may lead you to underforecasting In general choose an error measure that reflects the business needs If Rolling dices visualizes where the flaws of some measures are think of what happens in your business when using these measures 2012 SAP AG. All rights reserved. 10

11 Thank You! Contact information: Roland Martin Title Address Phone number

12 Results for APE variants Original APE APE variant APEf (APE with respect to the forecast) Formula ŷ y y ŷ y ŷ Forecast that minimizes the expected error in rolling dice 2 5 sape (Symmetric APE) ŷ y 1 2 (ŷ + y) 4 maxape (Max of Actual and Forecast) ŷ y max ŷ, y 4 tape (Truncated APE) min ŷ y y, SAP AG. All rights reserved. 13