Getting the Most out of Statistical Forecasting!

Transcription

1 Getting the Most out of Statistical Forecasting! Author: Ryan Rickard, Senior Consultant Published: September 2017

2 About SCMO 2 Founded in 2001, SCMO2 Specializes in High-End Supply Chain Consulting Work Focused on The Implementation and Better Use of SAP Applications, Including ERP ECC & S/4, SCM APO & IBP on HANA, Ariba, Among Others Featured in Publications and Regularly Present at SAP Conferences Globally, like SAP Insider, SAPPHIRE NOW and ASUG Annual Conference. Partnered with SAP s Supply Chain Group to Deliver Informative Sessions on Latest Tools and Functionality, like SAP Integrated Business Planning. Partnered with SAP Insider to Deliver Multi-Day Bootcamp Seminars. Company Statistics Delivering Strategic, Implementation, Enhancement, Migration/Upgrade and Outsourced Support Services across SAP s Execution and Supply Chain Planning Suite Including and Not Limited to: ERP ECC & S/4 HANA, SCM APO, IBP on HANA, SCIC/Control Tower, SNC, EIS (SmartOps), S&OP Powered by HANA and Ariba US-based Platinum Level Supply Chain Consultants With Deep Expertise in Both the Technical Tools and Functional Business Processes Delivering Projects and Services Across 20+ Different Countries in North and South America, Europe and Asia Since Our Inception 15+ Years Ago

3 Forecasting is a Core Competency We already offer programs specific to Demand Planning and S&OP

4 Upcoming Events for SCMO2 SCMO2 and SAPinsider IBP Bootcamp: SCMO2 presenting at Fall Focus (ASUG):

5 Q&A Questions throughout today s Webinar? Feel free to click on the Q&A. An SCMO2 panelist will answer questions throughout the Webinar. We will address any outstanding questions at the end of the session.

6 Session Leader Meet Ryan Rickard Ryan Rickard Senior Consultant 17 years Experience in Supply Chain Planning, Including Working as a Planner, IT Resource, and as a Business Process Re-design Lead Demand Planning and Statistical Forecasting Specialist in APO-DP and IBP-Demand Frequent Speaker at Many Premier Supply Chain Events Contact Info: Ryan Rickard, Sr. Consultant rrickard@scmo2.com (770) Follow SCMO2:

7 Webinar Series Getting the Most out of Statistical Forecasting! A multi-series webinar to explain How to Effectively Analyze & Model your Demand Session 1 Session 2 Session 3 Session 4 Variability Matters Calculating Variability & Segmenting to help drive the process How Much is Enough? How much Historical Data is Enough? How frequent to Run (Stat) & React? Super Model Forecasting The Optimal Level of Aggregation Weeks vs. Months can make a Difference FVA: The New Frontier Understanding how Forecast Value Add can enhance your forecasting value

8 Session 1, 2 and 3 Recap Session 1 Variability Matters All products are not the same. Their DNA and patterns are different. Calculating Variability can be done using the Coefficient of Variation methodology in Excel or APO/IBP Zeros matter in the CoV calculation, and when counting periods with historical values. Variability correlates to Forecastability Session 2 How Much is Enough Typically, the more data you have the better If you have Weekly and Monthly data, analyze both. Look for patterns weekly that are masked when aggregated Monthly. The frequency of generating Stat should align to your business process. Running Stat more frequently allows your Supply Chain to react the fastest. Considering both Variability & Historical Period counts is important to assigning appropriate models Session 3 Super Model Forecasting Patterns and periods of history are different at each level of aggregation What s most important for your business? The right Product/Customer forecast or Product/Location forecast? Running Stat at the same level that you measure Forecast Accuracy will give the best understanding of Stat performance For Seasonal Items, consider CoV 2 (CoV of Forecast Error)

9 Session 4 FVA: The New Frontier Understanding how Forecast Value Add can enhance your Forecast Value

10 Forecasting Deep Dive FORE CASTING Forecasting is a key part of achieving effective planning results. It s difficult to have an effective supply chain with poor forecasting. FVA is a Forecasting Deep Dive

11 Typical Forecasting Metrics & Expected Outcomes Forecasting Metrics: Forecast Accuracy Forecast Bias Expected Outcomes: Reduced Inventory levels Improved Customer Service Levels metrics Reduced Excess & Obsolete Inventory (% of Sales) Improved Budgeting and Financial Reconciliation Now we have another new metric FVA!

12 Introduction to FVA FVA is a metric that allows you to evaluate the performance of each step, each level, and even each participant/planner in the forecasting process It expresses the results of doing something versus doing nothing FVA can be either positive or negative, indicating whether your efforts are adding value to the forecast, or making it worse, and to what degree Measuring FVA allows you to identify waste and inefficiency in the forecasting process When FVA is negative, then the process activity is making the forecast worse and should be eliminated

13 A Simple Example Let s assume that you are forecasting a material and making inputs or adjustments at the customer level You have generated a Statistical forecast (at either the material or material/customer level) You have manual overrides to the Statistical forecast at various levels Suppose your Statistical forecast achieved a MAPE (or error) of 30% And, assume that your Consensus Demand (which included overrides) achieved a MAPE of 25% This would indicate that the extra analysis and adjustment to the Statistical forecast actually made the forecast better

14 Why Use FVA? Traditional forecasting metrics tell you about the Size, Direction or Tendencies of the Forecast Error Forecast Accuracy measures the absolute Forecast Error (aka MAPE) Forecast Bias tells you about the Forecast Tendency (the tendency to be too high, or too low) But neither tell if you have been improving or hurting the process FVA is an analysis that can help you determine: How efficient you are at forecasting If the adjustments or changes are actually adding value over time Which inputs are adding value and how much

15 The Big Question is Where do we add value during the forecasting process? That s the Key!

16 Forecast Value-Added Analysis (FVA) Where do we add value during the forecasting process? Forecast Accuracy Naïve Forecast Statistical Forecast Judgment Forecast 60% 65% 62% Simplest Forecast: Tomorrow will be like today. System Forecast: Generated using tools like SAP APO, SAP IBP Adjusted Forecast: Incorporates additional market/economic information (a.k.a Sales, Marketing ) +5% (value added to Naive) -3% (value subtracted from Stat) +2% (value added to Naïve)

17 Naïve Forecast A naïve forecast is something simple to compute. It requires minimum effort and manipulation to prepare. If you were asked to forecast the weather, the easiest and most predictable method would be to consider yesterday s weather (last month s weather, or same month last year s weather).

18 Goal or Purpose of a Naïve Forecast The Naïve forecast becomes your measuring stick It s the baseline measurement that all other forecasts are compared to If you can t beat the simplest forecast, then you need to eliminate the steps or waste and use the Naïve

19 Methods for Creating a Naïve Forecast Several commonly used methods to creating a naïve forecast are: 1. Sales History or Seasonal Random Walk uses the history from the same period a year ago as the forecast for the same period this year If you sold 50 units last June and 75 pieces last July, your future forecast would be 50 in July and 75 in August 2. Random Walk or last value uses the last known actual value as the future forecast (in every period) If you sold 15 units last month your future forecast in every time period would be 15 units. If your history drops to 10 units this month, then your future forecast shifts to 10 units. 3. Moving Average, or other very simple statistical formula Requires very minimal effort Doesn t require many data points Duration, or time periods, can be easily adjusted Typically smooths outliers and seasonal patterns 4. Blend or Average of all 3

20 Naïve Examples 1 2 3

21 Which is best to create the Naïve? As you can see, you can get different naïve forecasts based on the method chosen Which is best? It depends on the nature of the pattern you are trying to, or wish to forecast. One suggestion is to take a blend, or average of several. Or use one that is simple! Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Seasonal Random Walk Random Walk Moving Average Average Naïve Forecasts Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Seasonal Random Walk Random Walk Moving Average Average

22 FVA Steps Create a Naïve Forecast Compare the Statistical Forecast to the Naïve forecast Compare other input Forecasts to the Naïve forecast Compare the final Consensus Demand to the Naïve Forecast Review Results By Material Groupings By Materials By Customer Groupings By Customer Considerations: Which Lag or Snapshot do you want to Measure at (M, M-1, M-2, M-3)? Which Levels do you need to Store data and Measure a?. Create Naïve Forecasts for each. Where to Build FVA? Where to Store? Where to Analyze and Report? How to Display the Results?

23 Take the Forecasting Hippocratic Oath First, do NO harm! The easiest way to make the forecast better is to STOP making it WORSE. Dr. Gregory House

24 The Flaw of Averages An average may hide as much information as it reveals

25 Slice N Dice The FVA tool allows us to drilldown into the granular details of our forecasts We can break down the forecasting results by Month, Family, Product, Customer, Location, etc We can also break down the various forecasting inputs (Stat, Sales, Marketing, Demand Planning, Consensus) at the various levels

26 FVA: Slice & Dice Stat forecasting is not beating the Naïve forecast. Are we using the proper Stat Forecast Model? Are you using the right amount of Historical Data points? Is the history cleaned? Let s look at some REAL LIFE examples! May the Force be With You! Stat forecasting is adding value, but the Consensus Demand adjustments are hurting the accuracy. How can we improve? How should we use additional market and customer information to improve the forecast?

27 An Interlude on Metrics Philosophy of Metrics Forecast Accuracy Calculation Displaying Results Other Calculation Considerations

28 Why Do We Track a Metric? Monitor performance Learn if we need to improve Learn where we need to improve

29 The Philosophy of Metrics No metric is perfect Metrics are full of numbers All metrics are wrong, some metrics are useful Perfection is the enemy of the good The question is, Is the metric better than using our gut to make decisions Can the metric help me improve the forecast? Philosopher -Voltaire Philosopher Adam Smith Philosopher -Voltaire Numbers are Irrefutable Objective Lend Weight to Analysis & Understanding

30 Definition of Forecast Accuracy ForecastAccuracy 1 MAPE n i 1 n i 1 AbsError ActualDemand i i Notes: Forecast Accuracy usually considers a lagged or snapshot of the forecast (i.e. M-1, M-2, M-3 snapshot) Error is typically calculated for each Product/Location, then summed to get the total Error This calculation weights higher volume SKUs more heavily The Forecast Accuracy % result can be negative Be mindful of the Flaw of Averages

31 Displaying FVA Results Compared to the Naïve, each input forecast is tracked as a % Compared to Naïve FVA - M FVA - M FVA - M Statistical Forecast -16.5% -7.4% -8.5% Sales Mgr Forecast -36.9% -19.1% -30.0% Marketing Forecast -9.5% -17.5% -9.2% Demand Planning Forecast -6.5% 1.1% -7.6% Consensus Demand 2.0% -12.9% 1.7% The graph of FVA% is a little confusing. Will Management Understand this?

32 Displaying Forecast Accuracy Results Understanding and comparing Forecast Accuracy is often much easier! That s much better! Forecast Accuracy 2017-M M M08 Naïve Forecast 84.5% 81.2% 79.6% Statistical Forecast 68.0% 73.8% 71.1% Sales Mgr Forecast 47.5% 62.1% 49.6% Marketing Forecast 75.0% 63.7% 70.4% Demand Planning Forecast 78.0% 82.3% 72.0% Consensus Demand 86.4% 68.3% 81.3%

33 Other FA & FVA Calculation Considerations Timing of Forecasts Misses Should Forecast Accuracy go Negative? What s the Naïve Forecast for a New Product? What Lag should we use? What about forecast inputs made a multiple levels? What if we don t utilize Statistical Forecasting today?

34 Forecast Accuracy Doesn t Consider Timing Typical Forecasting Scenario: A Promotion is forecasted for the 1st week of June Most of the promotional orders arrive early and are shipped in May. The rest of the orders arrive and ship in early June Month Statistical Forecast Promotion Uplift Total Forecast Actuals Error MAPE Forecast Accuracy % May 50,000 50,000 75,000 25, % 66.7% June 50,000 30,000 80,000 55,000 25, % 54.5% You are a victim of the double-ding

35 Forecast Accuracy < 0% Some companies stop Forecast Accuracy at 0% Why? Because it is hard to explain negatives to Executives It s harder to understand how negative accuracies impact the Overall results So, should we allow the FA % to go negative? Yes! Products Total Forecast Actuals Error A 4,200 2,000 2,200 B 10,000 2,000 8,000 C 70,000 2,000 68,000 The magnitude of the miss is very important! Especially for New Products and Promotions (Correct) (Wrong) Fcst Accy % -10.0% % % 0 The difference between 3x and 30x IS something to sneeze at! There s BAD, and then there is REALLY BAD!

36 Naïve for New Products What is the Naïve for a New Product? If there is no Prior History, then the Naïve Forecast is simply 0 Remember the Naïve is just a reference forecast It s not the actual forecast or what we would actually order from the factory/supplier If the Naïve forecast is 0, we should always be able to beat it with either a Statistical Forecast or a Judgment Forecast right? Can we copy the History from a similar item as the Naive The Average of other New Products launches Wait for History Do we just use the initial Forecast Zero

37 What Lag? What lag should we use to measure Forecast Value Add and Forecast Accuracy? Considerations: What are your average Product Lead Times? Are some items Manufactured and other Purchased? When do you take your forecast snapshot (end of month, or beginning of month)? Select something Simple and Consistent (at least to begin) for all Products and Groupings What do we do if we re adding value at lag M-1, but not at lag M? Oh, I know! Use FVA to compare the results at M-1 vs M lags Drill down to determine which Levels and Inputs made the forecast worse as we got closer

38 Forecast Inputs & Changes at Multiple Levels If we have forecast inputs or adjustments at a detailed level (i.e. Product/Customer), then what do we use as the Naïve forecast? How do we know if detailed forecasts impact the aggregate? Material Customer Jun Jul Aug Jun Jul Aug Jun Jul Aug Jun Jul Aug FIT001 Statistical Forecast Adjustment to Stat Forecast Total Forecast Actuals Retail Retail Retail Dist Dist Web Club Comm Comm An Example for illustrative purposes!

39 What is the Naïve for Levels with Adjustments? If we have forecast inputs or adjustments at a detailed level (i.e. Product/Customer), then what do we use as the Naïve forecast? We use the Naïve at each of those Customer levels 1. We ll start with the Adjustments made at the customer level 2. We only consider the Actuals for the Customers with Adjustments 3. We pull the Naïve forecast at the Customers with Adjustments 4. Finally, we ll also need the Stat forecast and the Final Judgment forecast (Stat+Adjustments) so we can track the FVA Adjustments Only Adjustment to Stat Forecast Material Customer Jun Jul Aug Retail1 Retail2 Retail3 100 Dist FIT001 Dist2 Web 200 Club Comm1 250 Comm Actuals Jun Jul Aug Naïve (Adjustments) Jun Jul Aug Total Forecast (Adj) Jun Jul Aug

40 Calculating Naïve FA for Detailed Level Changes Now that we have the detailed Naïve information, we ll calculate the Naïve Forecast Accuracy for each Product/Customer The Naïve forecasts for the Customer specific Adjustments The Actuals for the Customers with Adjustments The Naïve Error & Forecast Accuracy for each Customer with Adjustments Adjustments Only Material Customer Retail1 Retail2 Retail3 Dist1 FIT001 Dist2 Web Club Comm1 Comm2 Naïve (Adjustments) Jun Jul Aug Actuals Jun Jul Aug ABS Error (Naïve Adjustments) Jun Jul Aug FA% Naïve 57.4% 77.0% 36.0% 29.2%

41 Calculating Input FA for Detailed Level Changes Next we ll calculate the Forecast Accuracy of the Stat and Adjustments to Stat Adjustments made for each Customer Actuals for the Customers with Adjustments Now we can calculate the Forecast Accuracy for the Stat and Total Forecast at the Customer level Adjustments Only Adjustment to Stat Forecast Material Customer Jun Jul Aug Retail1 Retail2 Retail3 100 Dist FIT001 Dist2 Web 200 Club Comm1 250 Comm Actuals Jun Jul Aug Statistical Forecast (Adj) Jun Jul Aug Total Forecast (Adj) Jun Jul Aug FA% Stat Stat+Adj 65.6% 98.4% 79.6% 67.3% 40.0% 0.0% 30.8% 92.3%

42 Comparison of FA s for the Detailed Changes Now we can compare the Forecast Accuracy % of the Naïve, Stat, and Adjusted (Judgment) Forecasts Adjustments Only Adjustment to Stat Forecast Material Customer Jun Jul Aug Retail1 Retail2 Retail3 100 Dist FIT001 Dist2 Web 200 Club Comm1 250 Comm For these Customers, the Stat forecast is better than the Adjustments which were made. The manual adjustments took away 9% and 36% points of value. Actuals Jun Jul Aug Now All the data is in place, and we re ready for the FVA FA% Naïve Stat Stat+Adj 57.4% 65.6% 98.4% 77.0% 79.6% 67.3% 36.0% 40.0% 0.0% 29.2% 30.8% 92.3% FVA Stat-Naïve Total-Naïve 8.2% 41.0% 2.7% -9.7% 4.0% -36.0% 1.5% 63.1% For these Customers, the Stat forecast is better than the Naïve, but the Adjustments to Stat added lots of value.

43 No Stat? Bad Stat? What if we don t utilize a Statistical Forecast today? Or, what if we don t do it (Stat) well? Can you create a Simple Model? Maybe even an Average in Excel Try different levels and time buckets Client Example: Created 4 reference Stat key figures (these didn t impact the business forecast in any way Used 4 different modeling scenarios Crostons Product/Monthly Seasonal Linear Regression Product/Monthly Pick Best/Composite (between the 2 above) Product/Monthly Pick Best/Composite (between the 2 above) Product/Customer Provided a good comparison when Stat Forecast wasn t created Provided a good comparison when the current Stat wasn t adding value

44 A More Modern Measurement Technique Forecasting Metrics: Forecast Accuracy Forecast Bias Does your business set Forecast Accuracy targets for Demand Planning, Sales, and Marketing? Instead basing performance off of Forecast Accuracy Why Not FVA? Instead of Measuring Forecasting Success via FA, maybe FVA is better? For example, Demand Planners don t control Inventory Planning or Product Availability. Nor do they control Customer Orders or Shipments. They can control the Accuracy of the Stat, and they can determine which forecast inputs are adding or hampering value. And they can drive communication and forecast adjustments to minimize the harm.

45 How has FVA been Proven Useful Reduction of Customers One business went from forecasting at 37 Key Customers, down to 6 As a result, the FA% went up by 10+% at 90 day lag Hit a 7 month consecutive high and achieved up to 80% FA Removed & Combined Customers to those where they could actually add value Shifted Focus from C & D products to A s and B s Put the C & D products on auto-pilot (aka Stat Forecast) In some cases, they chose to NOT forecast the C & D products at all Provided more time to focus on improving the A s and B s Recognition that the Naïve forecast is valuable Recognition that generating a Statistical Forecast is extremely useful Recognition that aggregate forecasting (Product) is better than the sum of the detailed forecasts Recognition that certain Sales input (aka Promotions) was never valuable Quit taking Promotional insight /uplift from certain Salespeople Reallocation of Resources (New Statistical Forecasting Team) After realizing that Stat and Naïve were valuable, shifted focus away from most inputs and overrides and focused on optimizing the Statistical Forecasting Process (Segmentation, Levels of Aggregation, etc.) Leadership recognition that Forecast Accuracy targets weren t realistic Forecastability! Realization that changing the forecast in the Short-Term wasn t adding value Shifted focus to the periods (lead time) where an updated forecast really impacted planning capabilities

46 FVA A Paradigm Shift Quote from a Supply Chain Vice President who s team embraced FVA With FVA, you realize that perhaps the only reasonable goals for forecasting performance are to beat a naïve model and continuously improve the process. Improvements can be reflected through a reduction in forecast errors, or a reduction in forecasting process (minimizing the use of company resources in forecasting). If good automated software can give you usable forecasts with little or no management intervention, why not rely on the software and invest management time in other areas that have the potential to bring more value to the organization? Let your production people produce and let your sales people sell don t encumber them with forecasting unless you truly must and it add value. You want to eliminate waste and streamline your process for generating forecasts as accurately and efficiently as possible.

47 Summary of Key Points Session 4 FVA is a new forecasting metric which allows you to Deep Dive and Slice n Dice to understand value A Naïve forecast is simple and easy, and it becomes the baseline measuring stick to base all other input comparisons against First, Do no Harm! The easiest way to improve the forecast is to STOP making it WORSE. No Metrics is Perfect, but Numbers are Irrefutable Be mindful of the Flaw of Averages dig into the weeds to understand detailed Value Add Forecast Accuracy can go Negative The Naïve for a new product is 0 Consider FVA as a new metric to measure forecasting importance to help eliminate waste in the process

48 Questions? Contact Info: Ryan Rickard, Sr. Consultant (770) Follow SCMO2:

49 Survey So that we can improve our presentation going forward, please take a couple of minutes to give us your feedback We have 4 brief questions that we d appreciate your response on as you exit the Webinar

50 Getting the Most out of Statistical Forecasting! Author: Ryan Rickard, Senior Consultant Published: September 2017