FVA Analysis and Forecastability

Transcription

1 FVA Analysis and Forecastability Michael Gilliland, CFPIM Product Marketing Manager - Forecasting SAS

2 About SAS World s largest private software company $2.43 billion revenue in ,000 customer sites / offices in 56 countries Leader in advanced analytics software 35.2% market share (per IDC) more than double its nearest competitor Ranked #1 in Fortune s 100 Best Companies to Work For the past two years. Find more at 2

3 Agenda APICS - Chicago Forecast Value Added Analysis Recap and scientific basis Assessing Forecastability Various approaches Setting Forecast Accuracy Objectives 3

4 Forecast Value Added Analysis 4

5 What is Forecast Value Added? APICS - Chicago Forecast Value Added is defined as The change in a forecasting performance metric (such as MAPE, Accuracy, or Bias) that can be attributed to a particular step or participant in the forecasting process FVA is measured by comparing the results of a process activity to the results you would have achieved without doing the activity FVA can be positive or negative 5

6 FVA Analysis: The Null Hypothesis APICS - Chicago FVA Analysis is based on basic scientific method, starting with the null hypothesis: H 0 : The specific process activity has no effect on process performance Analogy: Evaluate a new drug for safety and efficacy by comparing performance to a placebo 6

7 FVA Analysis: The Naïve Forecast A naïve forecast serves as the placebo in evaluating forecasting process performance Something simple to compute, requiring the minimum amount of effort and manipulation to prepare a forecast Random Walk (using last known actual) Seasonal Random Walk (using year ago actual) Moving Average 7

8 What is FVA Analysis? APICS - Chicago Consider a very simple forecasting process: Demand History Statistical Model Analyst Override FVA Analysis compares the accuracy of the statistical forecast (generated by forecasting software) to the accuracy of the analyst s manually adjusted forecast FVA Analysis would also compare both to a naïve forecast 8

9 FVA Analysis: Comparing to Naïve Forecast APICS - Chicago The most fundamental FVA analysis is to compare results of your forecasting process to the results you would have achieved by just using a naïve forecast If you are doing better than a naïve forecast, your process is adding value If you are doing worse than a naïve forecast, then you are simply wasting time and resources Process Step MAPE FVA vs. Naive FVA vs. Stat Naive 30%.. Statistical 20% 10%. Override 25% 5% -5% 9

10 Why Use FVA Analysis: Eliminate Waste FVA Analysis is used to identify and eliminate non-value adding activities Streamline the process by eliminating wasted efforts Direct resources to more productive activities Potentially achieve better forecasts By eliminating those activities that are making the forecast worse, you get better forecasts for free! 10

11 FVA Analysis: Reporting the Results APICS - Chicago Be cautious in interpreting your FVA results Don t draw conclusions without sufficient evidence One period of data is not enough! Over short time periods, results may just be due to chance Use Donald Wheeler s book Understanding Variation: The Key to Managing Chaos to guide the analysis If you haven t conducted FVA analysis and know that you are beating a naïve forecast then maybe you aren t!!! 11

12 Forecastability 12

13 Objective of the Forecasting process The objective of the forecasting process is to generate forecasts as accurate and unbiased as you can reasonably expect them to be, and do this as efficiently as possible What accuracy is reasonable to expect is determined by forecastability 13

14 Forecastability: Lower and Upper Limits Boylen: Forecastability refers to the range of forecast errors that are achievable on average, in the long run Lower limit for forecast accuracy? Naïve model If the naïve model achieves 70% accuracy, then you should be able to achieve no worse than this Upper limit for forecast accuracy? Much more difficult problem!! 14

15 Forecastability: The Upper Limit In order to achieve highly accurate forecasts, we need several things There is a structure or rule guiding the behavior that is being forecast We understand the rule and express it correctly in our forecasting model The rule isn t changing over time There is little variation (randomness) in the behavior about the rule Accuracy is ultimately limited by the amount of randomness in the behavior about the rule 15

16 Forecast Accuracy APICS - Chicago Forecastability: The Comet Chart Volatility (CV) The forecast accuracy we achieve is largely dependent on the volatility of what we are trying to forecast 16

17 Forecastability: Additional Approaches See the Spring 2009 issue of Foresight for a special section of assessing forecastability ( Includes some fairly sophisticated approaches from physics, graphical decomposition, and information theory Critique of CV as being too simplistic and unreliable as an indicator of forecastability Criticisms not as applicable for the types of patterns we deal with in supply chain forecasting 17

18 Setting Forecast Accuracy Objectives 18

19 Wrong Way to Set Forecasting Objectives APICS - Chicago 100% Accuracy 19

20 Wrong Way to Set Forecasting Objectives Arbitrary (what management wants or needs to achieve) for example MAPE < 20% Accuracy is determined more by the nature of behavior you are forecasting (smooth & stable vs. wild & erratic) than by the method E.g. Achieve 60% accuracy in forecasting Heads or Tails in the toss of a fair coin 20

21 Wrong Way to Set Forecasting Objectives Based on industry benchmarks (achieving best in class accuracy) Accuracy is determined more by the nature of behavior you are forecasting (smooth & stable vs. wild & erratic) than by the method 21

22 Wrong Way to Set Forecasting Objectives Improve accuracy compared to prior year Assumes forecastability stays the same Bad assumption if you change the data generating process E.g. move from everyday low pricing to promotion driven 22

23 Right Way to Set Forecasting Objectives Achieve forecast accuracy no worse than a naïve model No specific number, since we don t know in advance what accuracy the naïve model will achieve Focus on process automation and efficiency, and the elimination of process waste 23

24 Additional Resources on Today s Topics Forecast Value Added Analysis: Step-by-Step SAS on-demand webcast SAS white paper Blog: The Business Forecasting Deal (blogs.sas.com/content/forecasting) Book: The Business Forecasting Deal (available on amazon.com) Contact: mike.gilliland@sas.com 24