Managing forecast performance or My adventures in Errorland. Steve Morlidge

Transcription

1 Managing forecast performance or My adventures in Errorland. Steve Morlidge

2 Steve Morlidge Unilever ( ) roles include: Controller, Unilever Foods UK ($1 billion turnover) Leader, Dynamic Performance Management change project (part of Unilever s Finance Academy), Outside Unilever Chairman of the BBRT, BBRT Associate, 2007 to present Founder/director, Satori Partners Ltd., 2006 Ph.D., Hull University (Management Cybernetics), 2005 Visiting Fellow, Cranfield University, 2007 Coauthored book Future Ready: How to Master Business Forecasting, 2010 Editorial Board, Foresight magazine, 2010 Founder, CatchBull (forecasting performance management software), 2011

3 My perspective 1. Demand forecasting in the supply chain, characterized by: volume/volatility/velocity/variety/very short term 2. Forecasting is a business process that should add value 3. Forecasting should be theoretically robust but evidence driven 4. Any insight has to be operationalized before it adds value 5. I am not properly qualified so I can ask the dumb questions 6. Privileged access to data and insights

4 Some dumb questions 1. Is it possible to measure forecastability (i.e. assess forecast quality)? 2. Can we measure whether forecasting adds value? 3. Does forecasting add value? 4. How do we measure intermittent demand forecast quality? 5. How do we value the value that forecasting adds (part 1)? 6. Can forecast quality be managed?

5 Forecastability and Adding Value The Theory

6 Why forecast? 101(for the Supply Chain) Replenishment based on consumption (Kanban) Replenishment based on forecast so to avoid stock out we need safety stock based on the standard deviation of demand Changes in stock levels follow pattern of demand Errors from a good forecast (average demand) will be lower Replenishment based on consumption is equivalent to using the prior period s actual as a forecast, so the upper bound of forecast error should be the naive forecast error - any higher and forecasting adds no value and comparing errors to those from a naïve forecast also allows for forecastability because items with volatile demand (high levels of noise) are more difficult to forecast well.. leading to less safety stock as it is based on the (lower) standard deviation of error

7 Quality: a practical definition At the decision making level (e.g. low level stock replenishment point) forecasts should be: Better than simple replenishment (higher bound of forecast error = naïve forecast error) As close as possible to minimum avoidable error (lower bound of error =?) Produced at affordable cost (generate more value than the process consumes)

8 Lower Limit of error (Issue 30) Signal = totally forecastable Range of Naïve Forecast Errors (including noise) Goodwin: Foresight Summer Noise = totally unforecastable Signal Range of Actual (Perfect) Forecast Errors (excluding noise) Flat Signal Min RAE = 0.5 =0.71 Changing Signal Min RAE = < 0.71 The lower bound of forecast error is also related to the naïve forecast expressed as Relative Absolute Error (RAE) but how?

9 New thinking: new measures Total Error Increasing volatility = increasing difficulty of forecasting Relative Absolute Error (RAE) Destroying Value Forecast Error Zero Error simple replenishment = Naïve Forecast How low can you go? Theoretical limit of Forecastability (no trend) Practical limit of Forecastability? (any trend) Unavoidable Error? 1.0 (b/e) 0.7 (good) 0.5 (best) 0.0 Adding Value Value is added or destroyed at stock holding level (item/ location)

10 Forecastability Adding Value The Evidence

11 The evidence (Issues 32, 33) 9 samples from 8 businesses 330,000 data points Traditional measures unhelpful Median RAE Wtd Av RAE Median MAPE Forecast Accuracy M3 Competition Sample Rank Method RAE-wtd Type Sample 2 Sample Sample Sample Sample 7 Sample 7 Sample 8 Sample Mean % 49% 34% 77% 89% 34% 56% 35% 56% 45% 42% 8% 10% 35% 105% 53% 110% 51% 62% 43% 1 ForecastPro 0.67 Expert 2 B4J6automatic 0.68 Arima 3 Dampen 0.70 Trend 4 Comb6S4H4D 0.71 Trend 5 Winter 0.71 Trend 6 Forecast6X 0.72 Expert 7 Holt 0.72 Trend 8 Theta 0.73 Decomposition 9 Single 0.73 Simple 10 ARAMA 0.74 Arima 11 AAM Arima 12 PP4Autocast 0.76 Trend 13 Autobox Arima 14 Autobox Arima 15 Naïve Simple 16 Autobox Arima 17 Flores4Pearce Expert 18 Automat4Ann 0.81 Expert 19 AAM Arima 20 Robust4Trend 0.86 Trend 21 Theta4Sm 0.88 Trend 22 RBF 0.88 Expert 23 Flores4Pearce Expert 24 Smartfcs 0.96 Expert Average 0.78 Excl Outliers 0.96 Very little value added (4%)

12 Research samples from 8 businesses 330,000 data points M3 Competition Median Wtd Av Median Forecast RAE RAE What MAPE good Forecasts RAE >1.0 Accuracy looks like Sample 1 Sample % 49% % 77% Distribution of RAE < >1.0 0% 6% 52% 42% 1% 5% 33% 62% Sample Sample Sample Sample 7 Sample 7 Sample % 34% 56% 35% 56% 45% Few forecasts 42% can 8% beat RAE of 0.5 natural 10% 35% limit? 105% 53% 8% 12% 33% 47% 13% 11% 27% 49% 1% 9% 42% 48% 7% 10% 27% 56% 4% 11% 44% 40% 6% 2% 35% 57% Sample Mean Excl Outliers % 51% 62% 43% Half of all forecasts are destroying value 2% 3% 31% 64% 5% 8% 36% 52%

13 The distribution of RAE about as good as it gets 0.7 is good: Significant levels of value destruction

14 Forecastability Adding Value Operationalization Examples from ForecastQT

15 Key Concepts: Forecast Value Added 100% Error Forecast Error Value Added Score (VAS) -100 to 0 = Unacceptable simple replenishment = Naïve Forecast VAS 0-30 = Acceptable VAS = Good VAS = Excellent Limit of Forecastability Unavoidable Error D R I L L Relative Absolute Error (RAE) Destroying Value Adding Value Cost of Avoidable Error Zero Error Stock Holding Level 0.0

16 Performance Tracking European Packaging Manufacturer Performance of benchmark method (SES) shown as grey line Apart from first few months performance is excellent but now falling away League tables reflects different level of demand variability (Italy 41%, Sweden 19%)

17 Portfolio Management UK Spirits Business RAE 0.85, 49% value destructive European Tobacco Business RAE 0.80, 25% value destructive UK Wine Business RAE 0.90, 53% value destructive UK Beverages Business RAE 0.65, 25% value destructive

18 Segmentation analysis Increasing size (units/cost) à HORSES High volume, low variability. Optimise forecast algorithm and restrict judgemental input. Should be easy to beat naive. Aim for green DONKEYS Low volume, low variability. Use simple/cheap approaches with no judgemental intervention. Consider using naïve/kanban Aim for amber MAD BULLS High volume, high variability. Forecasting involves judgement in addition to algorithms. Difficult but possible to excel Aim for blue JACK RABBITS High volume, high variability. Use simple/cheap approaches with no judgemental intervention. Consider Make To Order Aim for amber Increasing volatility (COV) à

19 Segmentation Example (Issue 34) Scatter plot not aligned with performance aspirations

20 Segmentation Analysis High volume, low variability...often where the value is destroyed (and MAPE is best!) so consider using Kanban

21 Intermittent Demand and Valuing Adding Value The problem with measures

22 Remember this? (Issue 37) What is causing this? Period 1 Period 2 Period 3 Total Actual Forecast AE NAE RAE 0.5

23 Intermittent Demand (1) Q1: What is the best forecast for this data series? Q2: What is the mean absolute error for a forecast of 3 per period? Q3: What is the mean absolute error for a forecast of zero per period?

24 Intermittent Demand (2) Q1: What is the mean absolute error for a forecast of 4 per period and zero? Q2: What is the mean absolute error for a forecast of 5 per period?

25 Intermittent Demand (3) Q1: Why is the pattern of errors so (apparently) inconsistent? Q2: How much confidence do you now have in your metrics?

26 Asymmetric Demand Absolute error metrics optimize on the median not the mean MAE for forecasts with different levels of bias (Loss Function) As a result whenever the distribution of data is asymmetric, absolute error: is an unreliable guide to forecast performance cannot be used for selecting algorithms

27 Criteria for an alternative metric 1. Optimizes on the mean 2. Easy to calculate, aggregate and interpret 3. Can be used for symmetric demand as well 4. Reflects the business impact of forecast error

28 Candidates Metric Optimum Simple Symmetric Business Impact Squared Error Measures Direct Measures Mean Based Measures Mean No Yes Partly Mean? No Yes? Yes Mean Yes No No

29 Mean Based Measures e.g. Forecast vs Mean Actual Cumulative Forecast Error Measures of bias Loss Function optimizes on the mean and is linear

30 Mean Based Measures Forecast A Forecast B Forecast C Forecast D 1. Doesn t reflect all qualities of a forecast 2. OK for algorithm selection but 3. Can t be used for measuring non intermittent demand forecast quality Metric Optimum Simple Symmetric Business Impact Mean Based Measures Mean Yes No No

31 Bias Adjusted Error Metric Zero Forecast Bias (abs) + Variation + av (abs) Period'1 Period'2 Period'3 Period'4 Period'5 Mean'(abs) Actual Forecast Error'vs'Mean &3.0 &3.0 &3.0 &3.0 & Variation Bias'Adjusted 9.0 Forecast Error*vs*Mean Variation Bias*Adjusted 3.6 and RAE 0.5 à 1.3 RAE 0.6

32 Bias Adjusted Error Forecast A Forecast B Forecast C Bias = 0.0 Variation = 3.6 Bias = 0.0 Variation = 2.4 Bias = 0.0 Variation = 3.6 Forecast D Bias = 0.0 Variation = Reflects the two key qualities of a forecast: How well it captures the LEVEL of demand How well it captures the PATTERN of demand 2. OK for algorithm selection but 3. Can be it used for measuring non intermittent demand forecast quality?

33 Bias Adjusted Error Bias adjusted absolute error optimizes on the mean As shape of the loss function is linear it is easy to interpret

34 Criteria for an alternative metric 1. Optimizes on the meanþ 2. Easy to calculate, aggregate and interpretþ 3. Can be used for symmetric demand as well 4. Reflects the business impact of forecast error

35 Forecasting 101- The Ideal Replenish based on forecast No (cycle) stock at period end W1 W2 W3 W4 W5

36 Forecasting 101- The Reality Month 1 Month 2 Month 3 Month 4 Month 5 Month 3&5 = over forecast (Bias)à too much cycle stock Month 2&4 = under 1. The level of excess/shortfall in cycle stock is directly proportional forecasted to the (Bias)à level of bias. too little cycle 2. The level of safety stock is directly proportional to the level of stock variation (for a given target service level) 3. The the impact on the business of bias and variation (level and pattern) Month 2&3 = low are independent of each other variation requires less safety stock to meet service target Month 4&5 = high variation requires more safety stock to meet service target

37 Bias and traditional metrics (Issue 38) Period'1 Period'2 Period'3 Period'4 Period'5 Mean Actual Q1: What is the mean absolute error for a forecast of 2 per period? Q2: What is the mean absolute error for a forecast of 3 per period? = 0.0 = 1.0 Period'1 Period'2 Period'3 Period'4 Period'5 Mean Actual Q3: What is the mean absolute error for a forecast of 2 per period? Q4: What is the mean absolute error for a forecast of 3 per period? = 0.8 = 1.4

38 Bias Adjusted Measures Bias Bias Variation 1. Error increase is directly proportional to the increase in bias 2. The impact of bias are variation (level and pattern) are independent 3. BAMAE provides a consistent measure of the level of bias (and impact on cycle stock) and variation (impact on safety stock) irrespective of the distribution of demand

39 Traditional Measures The Loss Function for MAE is non linear Traditional error metrics underestimate the impact of bias especially when the level of bias is small compared to the level of variation. The outcome is sensitive to the distribution of demand. As a result changes in the size of error do not reflect the impact on the stock or the business

40 Example UK FMCG Business

41 Candidates Metric Optimum Simple Symmetric Business Impact Squared Error Measures Direct Measures Mean Based Measures Bias Adjusted Error Mean No Yes Partly Mean? No Yes? Yes? Mean Yes No No Mean Yes Yes Yes

42 Managing Forecast Quality Operationalising BAMAE and RAE

43 How can this be operationalized? BAMAE = Bias Adjusted Mean Absolute Error RAE = Relative Absolute Error BARAE BAMAE BAMAnE Robust measure of forecast performance compared to simple replenishment (Value Added) Value Added Contribution of bias Contribution of variation under forecast over forecast = stock outs = excess stock

44 Managing Forecast Performance Management ACTION Compare and rank Judgemental impact Optimise portfolio Real time issue resolution Analytical System ERROR ANALYSIS Forecast Value Added Allowing for forecastability Statistical alarms: signals from noise/ bias from variation Continuously updated Forecasters Judgement History INPUT JUDGEMENTAL INPUT Adjusting history Building forecast models Changing and tuning models Adjusting/overriding results Forecasting System(s) OUTPUT SAP/APO ForecastPro SAS Excel etc Performance Data (item level errors) HIGH COMPLEXITY 1000 s of forecasts running in parallel at multiple levels across many dimensions seasonality/intermittency USED TO MAKE DECISIONS replenishment Forecast

45 ForecastQT Home Page Clear summary of three key indicators Value Added Score (VAS), Bias and Variation Supports drill down into SKU level detail Easy to use tool with intuitive graphical interface based entirely in a web browser Value Added (compared to Kanban replenishment) and contribution of bias and variation Drill down by product (sorted by size of bias) Bias in aggregate and in detail (LEVEL OF ERRORS reflected in CYCLE STOCK) Variation demand vs error volatility (PATTERN OF ERRORS reflected in SAFETY STOCK)

46 Issue Management Example UK FMCG: < 1b revenue High level bias (blue) c 0%, but low level and high level bias (grey) = 15%... generating both over and under forecasting bias alarms

47 Valuing Value Added (2) the cost of error Forecast Error Stock impact of Avoidable Forecast Error (SAFE ) <0.25% of revenue Unavoidable variation Avoidable variation Avoidable bias Excess stock Overforecasting Underforecasting Excess warehousing costs Excess financing costs Excess stock write off Avoidable lost sales Expediting costs Total Cost of Avoidable Forecast Error (CAFE ) < 2% of revenue

48 What is this worth? Cost of Sales Per $1b revenue 1 Total Cost of Error 4%-6% $20m-$30m Forecast Value Added 0%-2% $0m-$10m Avoidable Error 1%-3% $5m-$15m Avoidable Inventory 2 $1m-$5m 1 Assuming 50% Gross Margin 2 Assuming 10 weeks stock cover

49 Contact details Thank you