Machine Learning with Neural Networks J. Stuart McMenamin, David Simons, Andy Sukenik Itron, Inc.
Please Remember» Phones are Muted: In order to help this session run smoothly, your phones are muted.» Full Screen Mode: To make the presentation portion of the screen larger, press the expand button on the toolbar. Press it again to return to regular window.» Questions: If you want to ask a question, type your question in the chat or Q&A boxes. We will address it as soon as we can.» Recording will be posted online. A link to download the recording and pdf when it is available.
Overview» Focus is on hourly load modeling» Question 1: How can neural network models be used to identify nonlinear relationships?» Question 2: How can we use this knowledge to build stronger hourly regression models?» Question 3: Can this approach be used to build accurate and robust hourly models?
Daily Energy vs Daily Average Temperature
Daily Weather Response HDD60 = Max(60 AvgDB, 0) CDD60 = Max(AvgDB 60, 0)
Specification 1. Base Model
Specification 1: Base Model
Degree-Day Variable Calculations (HR 17) Avg DB CDD HDD
Specification 1: Model Results
Neural Network Background
Linear Model Y Y a bx y 1 y 0 a Slope = Derivative dy y1 y b 0 dx x x 1 0 x 0 x 1 X
Nonlinear Model Y Local Slope = Derivative dy y1 y 0 dx x x 1 0 Y f (X) y 1 y 0 x 0 x 1 X
Neural Network Model Specific: where: General Form: Y G X, b u t 3 3 t 2 2 t 1 1 0 t 1 X a X a X a a e 1 1 H t 3 3 t 2 2 t 1 1 0 t 2 X X X b b b b e 1 1 H t t 2 2 t 1 1 0 t u H B H B B Y Specific Form 2 Nodes, 3 X s: Flexible nonlinear functional form Identifies important nonlinearities Identifies important variable interactions
Neural Network Specification t t 2 2 t 1 1 0 u H B H B B t Y t 3 3 t 2 2 t 1 1 0 t 1 X a X a X a a e 1 1 H t 3 3 t 2 2 t 1 1 0 t 2 X b X b b X b e 1 1 H Output Layer Hidden Layer Input Layer 2 3 1 H 1 H 2 Y X 1 X 2 X 3 Feed forward Neural Net with a Single Output (Y) One Hidden Layer with Two Nodes (H 1 and H 2 ) Logistic Activation Function in the Hidden Layer Linear Activation Function in the Output Layer
-5.00-4.50-4.00-3.50-3.00-2.50-2.00-1.50-1.00-0.50 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Logistic Activations are Nonlinear 1 exp Z HX where Z a0 a1x1 a2x 2... 1 exp Z 1 exp Z 1.00 0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00 H = exp(z)/(1+exp(z)) Z = Weighted Sum of X s
Why This Form is Useful
Why This Form is Useful
Derivatives h t h t h j h, h t j t 2 Z exp 1 Z exp a B X Ŷ t h t h t h t h h h t j t X Z Z H B X Ŷ t t h h 0 t u H B B Y h j t j h,j h,0 t h X a a Z Neural Network Logic Partial Derivatives t h t h t h exp Z H 1 exp Z
Model Derivatives in MetrixND (F Tab) Access using GetDeriv Function
Local Response Function in MetrixND
Terminology Mapping Econometric y = f (X,ß) X s Y s Transformation Sample Coefficients Constant/Slope Estimation Iteration Recursive LS Errors Dynamic Neural Net Artificial Neural Network Network Inputs Network Outputs/Targets Hidden Layer Training Set Network Weights/Connections Bias/Tilt Training/Learning Epoch Back Propagation Mistakes Recurrent
Building Weather Splines
Capped and Uncapped Degree Days
Daily Weather Response Multi-part slopes using Capped Degree-Days HDD60_Cap5 = Min(Max 60 AvgDB, 0, 5) CDD60_Cap5 = Min(Max AvgDB 60, 0, 5)
Daily Weather Response Multi-part slopes using Capped Degree-Days HDD55_Cap5 = Min(Max 55 AvgDB, 0, 5) CDD65_Cap5 = Min(Max AvgDB 65, 0, 5)
Daily Weather Response Multi-part slopes using Capped Degree-Days HDD50_Cap5 = Min(Max 50 AvgDB, 0, 5) CDD70_Cap5 = Min(Max AvgDB 70, 0, 5)
Daily Weather Response Multi-part slopes using Capped Degree-Days HDD45_Cap5 = Min(Max 45 AvgDB, 0, 5) CDD75_Cap5 = Min(Max AvgDB 75, 0, 5)
Daily Weather Response Multi-part slopes using Capped Degree-Days HDD40_Cap5 = Min(Max 40 AvgDB, 0, 5) CDD80 = Max AvgDB 80, 0
Daily Weather Response Multi-part slopes using Capped Degree-Days HDD35 = Max 35 AvgDB, 0 CDD80 = Max AvgDB 80, 0
Daily Energy Vs Daily Average Temperature Multi-part slopes using Capped Degree Days
Derivative of NNet w.r.t. Temperature vs Daily Average Temperature Cooling Heating
Averaging the Slopes Maximum Powered CDD Average slope between 75 and 80 8.0 GWh / Degree Average slope between 60 and 65 1.5 GWh / Degree Cooling
Daily Weather Response CDDSpline = 0.179 CDD60 Cap5 + 0.507 CDD65 Cap5 + 0.829 CDD70 Cap5 + 1.000 CDD75 Cap5 + 0.979 CDD80 Multi-part slopes using Capped Degree-Days
Averaging the Slopes Heating Maximum Powered CDD Average slope between 75 and 80-5.5 GWh / Degree Average slope between 55 and 60-0.6 GWh / Degree
Daily Weather Response Multi-part slopes using Capped Degree-Days HDDSpline = 0.103 HDD60 Cap5 + 0.376 HDD55 Cap5 + 0.581 HDD50 Cap5 + 0.750 HDD45 Cap5 + 0.871 HDD40 Cap5 +1.000 HDD35
Daily Weather Response CDDSpline = 0.179 CDD60 Cap5 + 0.507 CDD65 Cap5 + 0.829 CDD70 Cap5 + 1.000 CDD75 Cap5 + 0.979 CDD80 Multi-part slopes using Capped Degree-Days HDDSpline = 0.103 HDD60 Cap5 + 0.376 HDD55 Cap5 + 0.581 HDD50 Cap5 + 0.750 HDD45 Cap5 + 0.871 HDD40 Cap5 +1.000 HDD35
Specification 2. Hourly Regression With Daily Weights
Specification 2: Daily DD Weights
Degree-day Variable Calculations (HR 17) Avg DB Daily Weights Hour by Hour DD Variables CDD HDD
Specification 2: Model Results
Hourly Weather Response
Hour 00 Load Vs Temperature
Hour 01 Load Vs Temperature
Hour 02 Load Vs Temperature
Hour 03 Load Vs Temperature
Hour 04 Load Vs Temperature
Hour 05 Load Vs Temperature
Hour 06 Load Vs Temperature
Hour 07 Load Vs Temperature
Hour 08 Load Vs Temperature
Hour 09 Load Vs Temperature
Hour 10 Load Vs Temperature
Hour 11 Load Vs Temperature
Hour 12 Load Vs Temperature
Hour 13 Load Vs Temperature
Hour 14 Load Vs Temperature
Hour 15 Load Vs Temperature
Hour 16 Load Vs Temperature
Hour 17 Load Vs Temperature
Hour 18 Load Vs Temperature
Hour 19 Load Vs Temperature
Hour 20 Load Vs Temperature
Hour 21 Load Vs Temperature
Hour 22 Load Vs Temperature
Hour 23 Load Vs Temperature
Specification 3. Hourly Neural Network
Specification 3. Hourly Neural Network Node 1 (Linear) Node 2 (Sigmoid) Node 3 (Sigmoid)
Specification 3: Model Results
Using Machine Learning To Determine The Hourly Weather Response
Hour 00 Derivative Vs Temperature
Hour 01 Derivative Vs Temperature Cooling Heating
Hour 02 Derivative Vs Temperature Cooling Heating
Hour 03 Derivative Vs Temperature Cooling Heating
Hour 04 Derivative Vs Temperature Cooling Heating
Hour 05 Derivative Vs Temperature Cooling Heating
Hour 06 Derivative Vs Temperature Cooling Heating
Hour 07 Derivative Vs Temperature Cooling Heating
Hour 08 Derivative Vs Temperature Cooling Heating
Hour 09 Derivative Vs Temperature Cooling Heating
Hour 10 Derivative Vs Temperature Cooling Heating
Hour 11 Derivative Vs Temperature Cooling Heating
Hour 12 Derivative Vs Temperature Cooling Heating
Hour 13 Derivative Vs Temperature Cooling Heating
Hour 14 Derivative Vs Temperature Cooling Heating
Hour 15 Derivative Vs Temperature Cooling Heating
Hour 16 Derivative Vs Temperature Cooling Heating
Hour 17 Derivative Vs Temperature Cooling Heating
Hour 18 Derivative Vs Temperature Cooling Heating
Hour 19 Derivative Vs Temperature Cooling Heating
Hour 20 Derivative Vs Temperature Cooling Heating
Hour 21 Derivative Vs Temperature Cooling Heating
Hour 22 Derivative Vs Temperature Cooling Heating
Hour 23 Derivative Vs Temperature Cooling Heating
Computing The Hourly Degree-Day Weights Average slope of 5 degree bucket = Max = Min = Slope Max/MinSlope
Specification 4. Hourly Regression With Hourly Weights
Specification 4: Model Results
Adding Rolling Lag DD Variables Specification 5 Add 6 Period Lag Specification 6 Add 24 Period Lag
Specification 5: Model Results
Specification 6: Model Results
Specification 7 Model Results
Specification 8: Model Results
Specification 9: Model Results
Specification 10: Results
Specification 11: Results
Conclusions Cooling Heating
Conclusions» Neural networks provide a powerful framework for identifying nonlinear relationships.» Detection and quantification of nonlinearities is a form of machine learning.» Neural network derivatives can be processed to construct weights for degree day splines.» Regression models using these splines are equivalent to the comparable neural networks.» Adding additional interaction variables results in regression models that are rich, accurate, and robust.
FORECASTING BLOG http://blogs.itron.com/forecasting/
2018 SCHEDULED EVENTS forecasting@itron.com 858.724.2620 www.itron.com/forecastingworkshops http://blogs.itron.com/forecasting/ Workshops Dates Location Energy Forecasting 101 February 28 - March 1 Washington, DC Introduction to SAE April 24 Austin, TX Advanced Forecast Topics April 24 Austin, TX Fundamentals of Modeling Energy and Demand Forecasting October 24-26 Chicago, IL Fundamentals of Short-term Operational Forecasting September 18-19 San Diego, CA Meetings 16th Annual Energy Forecasting Meeting April 25-27 Austin, TX 12th Annual ISO Forecasting Summit May 15-17 San Diego, CA European User Meeting TBD TBD Itron Utility Week September 30-October 2 Scottsdale, AZ Webinars Using Neural Networks to Build Robust Hourly Models February 20 On-line Budget Forecasting A Practitioner s Handbook May 22 On-line 2018 Forecast Accuracy Benchmarking Survey and Energy Trends September 11 On-line Short-term Load Forecasting A Practitioner s Handbook December 4 On-line
CONTACT US forecasting@itron.com www.itron.com/forecasting http://blogs.itron.com/forecasting/