Proposed Changes to the PJM Load Forecast Model Load Analysis Subcommittee April 30, 2015 www.pjm.com
Agenda Overview Specific Model Improvements Usage & Efficiency Variables Weather Re-Specification Autoregressive (AR) Error term Forecast Impacts Accuracy & Stability Next Steps 2
Overview & Results www.pjm.com
Overview PJM sought to address the recent trend of over-forecasting through implementing model changes that both improve model fit and account for ongoing trends in customer usage and energy efficiency. PJM focus was on: Improving the portion of the error that can be explained by the parameters included in the model (i.e. not the forecast accuracy of the model inputs). Analyzing how the model would have performed had it been in place 4
Overview PJM has identified a new model specification that makes several significant changes. These are: Inclusion of customer usage & efficiency variables New weather specification Introduction of an autoregressive error term 5
Forecast Results New forecast specification affects the forecast in two major ways Lowers the starting point of the forecast due to the model more accurately capturing recent historical trends Lowers the growth rate as it takes into account usage & efficiency trends through the equipment indexes PJM 10-Year Growth Rate: 0.7% using New Specification versus 1% using Current Specification 6
Forecast Results 7
Summer NCP Growth Rates 8
Model Change: Inclusion of Customer Usage & Efficiency Variables www.pjm.com
Overview Incorporate customer usage & energy efficiency variables Through membership in the Itron Energy Forecasting Group (EFG), PJM obtains history and projections of appliance saturation and efficiency. The history and projections are derived from EIA s National Energy Modeling System and are consistent with the Reference Case of the EIA s Annual Energy Outlook. These can be used to account for trends that the current model does not adequately capture 10
EIA Reference Case Projections in the EIA Reference Case are based on "business as usual". EIA considers the historical behavior of appliance/equipment purchases and their associated efficiency and projects in the context of: Installed & operating costs Fuel choice decisions The probabilistic lifetime of appliances/equipment Acquisition of new appliances/equipment The expectation of future technological trends and future mandated efficiency standards Current laws and regulations 11
EIA Reference Case Discussion Points Projections in the EIA Reference Case do not explicitly take into account potential state or utility goals/programs. However, they are embedded in the history of equipment and thus past or ongoing programs are expected to continue in the forecast. PJM believes that an important step to creating a more accurate forecast is having a clearer understanding of recent history, which includes properly accounting for the trajectory of equipment and efficiency trends in the model. 12
Usage/Efficiency Implementation PJM uses the EIA/Itron data and projections to calculate equipment indexes for heating, cooling, and other. Each equipment index is a weighted average across equipment types of saturation (or share) normalized by efficiency levels. More information can be found in Appendix A These indexes are interacted with economics and/or weather variables, and used as independent variables in the econometric model. As a result, different zones can have different relationships with these drivers. 13
Usage/Efficiency Implementation New (or modified) variables with usage/efficiency interactions are Heat_IN2_HDD and Cool_IN2_CDD Heat_IN2_Lag1HDD and Cool_IN2_Lag1CDD Heat_SplineWWP and Cool_SplineTHI Heat_Shldr_WAT19_50lt and Cool_Shldr_THI Other_DailyIN2 14
Model Change: New Weather Specification www.pjm.com
Existing Model Review: Weather Several variables in this category have counterintuitive or difficult to explain signs on the coefficients. Other variables seem redundant www.pjm.com 16
New Weather Specification PJM investigated new ways of modeling weather with the main goals of Simplifying/reducing the number of weather variables Providing the forecast model the means to be more responsive during peak-type weather conditions Looked into seasonal splines which helps accommodate these overarching goals Summer: Spline based on THI Winter: Spline based on Wind-adjusted Temperature Shoulder: Spline based on Wind-adjusted Temperature and THI 17
Summer Weather Spline 18
Summer Weather Spline 19
Summer Weather Spline 20
Summer Weather Spline Coefficients from regressions within different THI ranges are used as the basis to create new variable SplineTHI. This new variable has a linear relationship with load, but has the properties of the spline-like relationship in the previous slide. More info can be found in Appendix B. 21
Winter Weather Spline 22
Winter Weather Spline 23
Winter Weather Spline 24
Winter Weather Spline Coefficients from regressions within different WWP ranges are used as the basis to create new variable SplineWWP. This new variable has a linear relationship with load, but has the properties of the spline-like relationship in the previous slide. More info can be found in Appendix B. 25
Shoulder Weather Specification Shoulder months require a different approach as they contain some days that behave like Winter days (where wind can play a factor) and other days that behave more like Summer days (where humidity can play a factor). Use wind-adjusted temperature or THI depending on the conditions of the day 26
Shoulder Weather 27
Shoulder Weather 28
Shoulder Weather 29
Shoulder Weather Wind-adjusted temperature is used for colder days and days with little to no weather sensitivity. Remaining shoulder days will use THI to pick up influence of humidity. More info can be found in Appendix B. 30
Weather Variables Weather variables are interacted with the Heat or Cool index to reflect that the effect of weather on load is a function of the weather itself and the equipment used to perform in that weather Heat_SplineWWP and Cool_SplineTHI Heat_Shldr_WAT19_50lt and Cool_Shldr_THI Heat_IN2_HDD and Cool_IN2_CDD Heat_IN2_Lag1HDD and Cool_IN2_Lag1CDD 31
Model Change: Autoregressive (AR) Error Term www.pjm.com
Investigating an Autoregressive Error Term Autocorrelation Statistical phenomenon common in time series analysis wherein a variable is correlated with a lag(s) of itself If present, the consequence can be that model coefficients do not have the minimum variance property. In other words, the estimated model may not provide the best possible fit. Detected via statistical test (Durbin-Watson) and lags identified based on statistical plots (i.e. autocorrelation function and partial autocorrelation function plots, a.k.a. ACF and PACF) Can be accounted for in a statistical model through the use of an AR error term. We chose to use AR(1). www.pjm.com 33
Investigating an Autoregressive Error Term Notice the persistence of negative residuals in recent years and notice the somewhat cyclical pattern www.pjm.com 34
Investigating an Autoregressive Error Term Notice the greater resemblance with the AR error term to white noise www.pjm.com 35
Forecast Impacts: Accuracy www.pjm.com
Forecast Accuracy The primary goal of the PJM forecast is to project the expected Summer peak of the RTO. The accuracy of that forecast is dependent on how the relationship of load with its drivers changes over time as well as the accuracy of its inputs (namely economics and now the equipment/efficiency variables). Accuracy will be tested by comparing actual unrestricted load on peak-type days (10 highest Summer load days) versus what the model would have projected for those days Under knowledge available at the time of the formation of the forecast (Vintage Runs) Under knowledge available today (Current Runs) 37
Forecast Accuracy: Vintage and Current Runs Vintage Runs Current Runs Forecast Year Estimation Through Economics* Equipment Index** Economics* Equipment Index** 2009 August 2008 December 2008 2008 October 2014 2014 2010 August 2009 November 2009 2009 October 2014 2014 2011 August 2010 December 2010 2010 October 2014 2014 2012 August 2011 December 2011 2011 October 2014 2014 2013 August 2012 November 2012 2012 October 2014 2014 2014 August 2013 November 2013 2013 October 2014 2014 *Economics applies to both the Current Specification and New Specification **Equipment Index only applies to the New Specification 38
Forecast Accuracy: Vintage Runs 39
Forecast Accuracy: Current Runs 40
Forecast Accuracy Using information that would have been available at the time, the new forecast model specification has an out-of-sample MAPE of 6.3% on the three-year out forecast. This compares with 10.1% under the current model specification. Were the forecast model to have up-to-date information on economics and equipment/index trends (the Current Runs), the new forecast model specification has an out-of-sample MAPE of 1.7% on the three-year out forecast versus 6.2% with the current specification. We consider this to be model error. 41
Forecast Accuracy Additional analysis has shown that a large portion of the error in the Vintage Runs is due to uncertainty in the economic forecast. In other words, if we were able to have current information on economics (but not on the equipment index) then much of the error would be resolved. 42
Forecast Stability PJM has conducted preliminary analysis to gauge the stability of the new forecast model specification versus the existing model. Evidence indicates that while the new model specification significantly improves accuracy, it does not have a significant impact on forecast stability. Forecast stability being defined as the percentage change in the forecast for a given year (RPM and/or RTEP) with each incremental forecast release. 43
Next Steps: Timeline & Additional Discussion www.pjm.com
Timeline Late May: Additional LAS meeting to further discuss stakeholder questions/concerns and any additional analysis June: PJM Staff will bring new M-19 language to LAS July: PJM Staff will bring new M-19 language to PC August: Second read/approval at PC; First read at MRC September: Second read/approval at MRC December: Load Forecast Report with New Model Specification 45
Discussion How does cleared EE fit into the equation? Is cleared EE a part of the forecast reduction attributable to equipment saturation & efficiency projections? Is cleared EE in addition to the reduction due to these projections? If it is the former, how is that reconciled with managing energy efficiency as a capacity resource? If it is the latter, a peak forecast incorporating the saturation & efficiency projections should be adequate. However, should the energy forecast then be further reduced by the energy equivalent of the EE capacity? www.pjm.com 46