How Accurate is My Forecast?

Transcription

1 How Accurate is My Forecast? Tao Hong, PhD Utilities Business Unit, SAS 15 May 2012 PLEASE STAND BY Today s event will begin at 11:00am EDT The audio portion of the presentation will be heard through your computer speakers. This is an automatic setup and is preferred. If you would prefer to dial in, please call: US Toll-Free: Toll/International: Conference Code: # If you experience any technical difficulties, you may contact WebEx Technical Support at

2 How Accurate is My Forecast? Tao Hong, PhD Utilities Business Unit, SAS 15 May 2012

3 Poll #1 Which one of the following best describes your job? a) I develop or use short term load forecasts (horizon <= 2 weeks) b) I develop or use long term load forecasts (horizon > 2 weeks) c) I develop or use all load forecasts in my company d) I have nothing to do with load forecasts 3

4 How Accurate is My Forecast? Research Consulting Tao Hong, PhD Teaching 4

5 Outline Introduction Basic concepts Error statistics Worst practices Beyond 5

6 Introduction 6

7 Forecasting Principle #1 ALL the forecasts are WRONG!!! 7

8 Forecasting Principle #2 ALL the forecasts can be IMPROVED!!! 8

9 This Talk Answers the Questions Like Which error statistics shall I use? What are the MAPEs of other utilities? How is my accuracy comparing with other peers? Did I do anything wrong? (It looks very accurate already ) What shall I improve? How to improve my forecasting accuracy? Shall I continue improving my forecasting accuracy? 9

10 This Talk is about HOW TO Generate error statistics Evaluate forecasting accuracy Compare forecasts Use error statistics to improve load forecasts 10

11 This Talk in Analytics Roadmap How Accurate is My Forecast? Descriptive Summary Statistics Predictive Forecasting Prescriptive Optimization Analytics 11

12 Avg Understanding of Load Forecasting SUUG Load Forecasting Workshop CenterPoint Energy, Houston, TX, March 6 th, 2012 Descriptive Analytics 57 people showed up representing 15 companies 48 turned in evaluation forms 98% would like to recommend the workshop to their colleagues BEFORE AFTER 100 Years of Load Forecasting: Classics, Challenges and Best Practices with Smart Grid information 12

13 BASIC Concepts 13

14 Two Forecasts Ex ante forecast Before the event Using information available in advance The ONLY way to produce genuine forecasting accuracy Ex post forecast After the event Assuming the values of predictors are known Produce useful forecasts 14

15 Poll #2 A one day ahead load forecasting model Dynamic regression Load(t) = T(t) + T(t)^2 + T(t)^3 + Load(t-1) + Load(t-24) Assumption Today: 5/15/2012 t: Hour Ending15:00, 5/1/2012 T(t), Load(t-1) and Load(t-24) are all actual values What s this forecast? a) Ex ante b) Ex post c) Neither 15

16 Data Partition Training estimate parameters Validation select variables and models Test assess/confirm predictive power Artificial Neural Networks Be Careful!!! If you change the network architecture after assessing the results from Test data, you will need an addition dataset completely blind from parameter estimation and model selection to confirm forecasting accuracy. 16

17 Genuine Forecasting Accuracy Ex Ante + Test + 17

18 Error Statistics 18

19 R-Square Goodness of Fit Predictive Power 19

20 Criteria Error measures ME, MPE, MAE, MAPE, MSE Error spread: variance, Q3, max Residual diagnostics Line/scatter plots Normality: plot, skewness, kurtosis Autocorrelation: Durbin-Watson Change in mean and variance Qualitative methods Simplicity Model parameters Forecast appearance Goodness of Fit Predictive Power 20

21 Mean Absolute Percentage Error (MAPE) MAPE = 100% n n t=1 A t F t A t Scale-independent Not sensitive to sign of error Equal weights Not sensitive to large errors Pros Easy to calculate Easy to understand Widely used Cons o Can t handle At = 0 o Sensitive to small denominator o Asymmetric (MAPE>MEDAPE) 21

22 MAPE s Alternatives MAPE Scale-independent Not sensitive to sign of error Equal weights Not sensitive to large errors o Can t handle At = 0* o Sensitive to small denominator* o Asymmetric (MAPE>MEDAPE) Alternative MAE MPE, ME WMAPE RMSE MAE MAE SMAPE 22

23 MAPE in Load Forecasting Content MAPE of Hourly Load Daily/Monthly/Annual Peak Daily/Monthly/Annual Energy Daily/Monthly/Annual Min MAPE by Hour, Weekday, Month, Season, Regular/Special Days Spread of APE Max, 95 th percentile, 90 th percentile, Q3, Q2 Error Score MAPE1 + MAPE2 + + Spread1 + Spread2 + 23

24 Error Score for Combining Forecasting An example with MAPE Model #1 Model #2 MAPE (ALL) Sunday Monday Tuesday Wednesday Thursday Friday Saturday = 1.56? 24

25 Error Score for Combining Forecasting An example with MAPE Model #1 Model #2 Combine MAPE (ALL) Sunday Monday Tuesday Wednesday Thursday Friday Saturday = 1.56? 25

26 Poll #3 What s your genuine (ex ante) day ahead hourly load forecasting MAPE last YEAR? a) MAPE <1% b) 1% <= MAPE < 2% c) 2% <= MAPE < 3% d) 3% <= MAPE < 4% e) 4% <= MAPE < 5% f) MAPE >= 5% 26

27 Worst Practices 27

28 Worst Practices Irrelevant comparison Apple to orange I heard Giant State Electric is getting a MAPE at 0.5%, we need to do a better job! Always looking forward Stay with the best ONE but ignoring the rest That ANN model did a very bad job yesterday, I don t want to see it anymore. Let s remove it from our model repository. Poor documentation Missing key elements My gut feeling tells me that our model is underestimating tomorrow s peak. Let add 5% to it. 28

29 Model Life Cycle Model Life Cycle Born Production Retirement Forecaster s Job Development Maintenance Archiving Comparison Internal (same or different methodologies), external Decision In, update, tune, out Documentation Who, when, what, where, how, why, how much 29

30 Summary Genuine = Ex Ante + Test + Extend MAPE from all hours to selected hours Pay attention to the variance of the error Use Error Score to evaluate your forecasting accuracy Combining forecasts through in-depth error analysis Be very careful when comparing models Look backward from time to time Documentation, documentation, documentation! 30

31 Beyond 31

32 Beyond Forecasting in the utility industry Accurate = Good? 32

33 Business Analytics Technology Beyond Bucket Effect Honesty 33

34 Beyond Training Portfolio Category Title Length (hr) Course Electric Load Forecasting: Fundamentals and Best Practices 16 Workshop Assessment of Forecasting Practices 8 Workshop 100 Years of Load Forecasting 4 Workshop Power Distribution System Losses 4 Talk On the Holiday Effect of Electricity Demand 1 Talk From Load Forecasting to Demand Response 1 Talk How Accurate is My Forecast? 1 Coming Next Course Special Topics in Electric Load Forecasting 16 Workshop Advanced Analytics for Utilities 8 Talk Rating and Ranking My Models, Who s #1? 1 34

35 Beyond Electric Load Forecasting: Fundamentals and Best Practices Next offerings Jun 4-5, San Francisco, CA; Jul 12-13, Dallas, TX Outline 1. Introduction to electric load forecasting 2. Salient features of electric load series 3. Multiple linear regression 4. A naïve benchmark for short term load forecasting 5. Customizing the benchmarking model 6. Very short term load forecasting 7. Medium and long term load forecasting 8. Variables, methods, techniques and further readings 9. Frequently made mistakes 35

36 Beyond IEEE Working Group on Energy Forecasting Next meeting IEEE Power & Energy Society General Meeting, July 22-26, 2012, San Diego, CA Session 1: Load Forecasting Methodologies and Applications in Operations and Planning Session 2: Demand Response: Analytics, Practice, and Challenges in Smart Grid Environment Current Projects Benchmarking of short term load forecasting accuracy Literature review of load forecasting techniques and practice Global Energy Forecasting Competition 2012 IEEE Transactions on Smart Grid -- Special Issue on Analytics for Energy Forecasting with Applications to Smart Grid 36

37 Q & A 37

38 Tao Hong, PhD SAS Institute Thank you for joining us today. Please take the short survey your feedback is important to us!