Application of NARX based FFNN, SVR and ANN Fitting models for long term industrial load forecasting and their comparison

Application of NARX based FFNN, SVR and ANN Fitting models for long term industrial load forecasting and their comparison *Shahid M. Awan 1, 3, Member, IEEE, Zubair. A. Khan 1, 2, M. Aslam 3, Waqar Mahmood 1, Affan Ahsan 4 1 Al-Khawarizmi Institute of Computer Sciences, 2 Department of Electrical Engineering, 3 Department of Computer Science and Engineering, University of Engineering and Technology. Lahore, Pakistan. 4 Descon Integrated Projects (Pvt) Limited, Lahore, Pakistan. *shahidawan@kics.edu.pk Abstract Accurate load forecasting is essential for energy planning and load management. This paper presents long term industrial load forecasting (LTLF) using Nonlinear Autoregressive Exogenous model (NARX) based Feed-Forward Neural Network (FFNN) method, Support Vector Regression (SVR) and Neural Network models. It is applied to data sets obtained from National Transmission and Dispatch Company (NTDC) of Pakistan, ranging from to 21. Several influencing load factors are examined. Comparison of results obtained by all three techniques is presented which portray a high acceptable accuracy with 2.9% Mean absolute percentage error (MAPE) on monthly and yearly demand estimation for industrial sector. Keywords- Neural nonlinear systems, nonlinear estimation, modelling. I. INTRODUCTION Electric Load Forecasting (LF) is the estimation of future demand of energy based on historical data. Accurate load forecasting plays an important role in coping future energy requirement and hence and important component of energy management systems. It is generally categorized into three types for different time horizons: short term forecasts are from one hour to one week, medium term forecasts span from a week to a month or year and long term forecasts spans a year or more. It has many applications including energy purchasing, generation, transmission planning, decision making and infrastructure development based upon type of each forecasts. Each electric supply company serves different type of customers like domestic, agricultural, commercial and industrial. All of these consumers have different load consumptions patterns over different time intervals. Especially industrial customers, they have a special requirement that an un-interrupted supply is required to complete the whole cycle of production otherwise manufactured goods are wasted. Industrial sector plays an important role in economic development of a country and are critical for survival of a nation. In most countries industrial zones are kept apart from residential colonies and separate electricity infrastructure is developed to fulfill energy requirement of this sector. Industrial sector in Pakistan consumes around 27% of generated electrical energy, which is 2nd highest consumer after domestic users and 25.4% of overall Gross Domestic Product (GDP) of Pakistan is dependent on this sector. So long term electric load forecasting is very crucial for planning and scheduling electric supply. This helps in infrastructure development, distribution management and uninterrupted supply, specifically to industrial sector. During the past few decades a diverse number of techniques and models have been developed by researchers around the globe to address the challenge of load forecasting. Diversity in different modeling techniques is because of nature of data sets in hand, type of load forecasting and nature of influencing factors on load variation [1, 2]. In this research paper different techniques have been applied to solve the long term load estimation problem of industrial sector. Based upon data sets provided by National Transmission and Dispatch Company (NTDC) [24] of Pakistan, we have developed three different models with varying predictive performance and accuracy of results. Models are repeatedly evolved unless acceptable results are obtained. This work would help in power planning section of NTDC to make timely decision to fulfill energy requirement and to help in survival and growth of industrial sector. Other sections of this paper cover the short review of techniques used in our work and work done in this area using these techniques, while paying special attention to industrial load forecasting and use of NARX based model in LF domain. Uniqueness of the problem and proposed methodology is discussed in section 3, while results and conclusion are presented in section 4. II. TECHNIQUE USED IN THIS STUDY Since 196, a lot of work has been done in LF domain. From classical and statistical approaches to more recent and artificial intelligence based approaches has been applied to solve the problem of accurate load forecasting based on National ICT R&D Fund, Pakistan 978-1-4673-158-9/12/$31. 212 IEEE 83

priorities, custom features and varied input parameters [1, 2]. In this study modern techniques of forecasting have been used which include Artificial Neural Networks [3] and support vector machines for regression [17]. Artificial Neural Networks simulate biological inspiration from human brain s ability of learning and decision making. Based upon their predictive performance and capability to handle nonlinear and dynamic systems ANN are considered as powerful tool for time series prediction and modeling tasks. Fundamental component of ANN is neuron with inputs and output connections as shown in fig. 1. It computes the weighted inputs using the formula X 1 X 2 X 3 X 4 X n W 3 W 2 W 4 W 1 W n Inputs Figure 1. Artificial Neuron Activation function Where x i is input vector and w ij are corresponding weights for this input vector, y j is the response of neuron based upon transfer function. For continuous output, neuron uses sigmoid transfer function as given in eq. 2. Network can compute the error E, by taking mean squared error (MSE) that is expressed by eq. 3. Where y i is output obtained at neuron i and a i is the desired output. Two major neural network topologies exist based upon connection between neurons and direction of data propagation, they are: feedforward and feedbackward networks [3]. In feedforward network resultant output is calculated and passed to next hidden or output layer, while in backpropagation learning the error of subsequent output or hidden layer is back propagated to previous layer in a recursive manner for reestimation of weights. There is a special type of neural network known as recurrent neural networks (RNN s), [3] which have feedforward neural structure but they maintain an internal hidden state and contain a local or global feedback cycle in their structure. Introducing a time delay to RNN s give rise to an effective class of SRN model named as Nonlinear autoregressive with exogenous input (NARX) Neural Network model [4]. NARX is a non-linear model for time series prediction, derived from Autoregressive exogenous (ARX) model. It is a recurrent dynamic network, with feedback connections enclosing several layers of the network. It can be defined by following equation and shown in fig. 2. (1) Output (2) (3) Figure 2. NARX Model In this model the current value of the output y(t) is regressed upon the previous value of the output and exogenous inputs. The function F in NARX model is some nonlinear function and can be implemented by different techniques such as a polynomial, neural network, wavelet network, sigmoid network and so on, as shown in fig. 2, but using a feedforward neural network to approximate the function F is the most popular and efficient way. NARX can be implemented through ANN in two ways. The first architecture is called Parallel architecture. In which the predicted output is used as input to next output prediction. As during training we are available with both "true inputs and output" for better results it is efficient to use them instead. The second architecture is Series-Parallel architecture this architecture not only uses true outputs as regressor to next prediction point but also has feedback loop eliminated, hence simple ANN training algorithms such as backpropagation can easily be implemented. fig. 3, represents heir model, Where NARX network is designed as a feedforward Time Delay Neural Network [4]. NARX recurrent neural network is a powerful modeling technique for nonlinear systems. Better generalization capability, effective learning and faster convergence to global minima make it better than other network [5][6]. Beside NARX is a relatively new approach in LF domain [7], still some applications of this model in power systems have been reported in literature. Espinoza, et al, [8] have estimated five NARX models for short term load forecasting using fixed size least square support vector machines (LS-SVM) and two auto regressive NARX (AR-NARX) models. The later structures reduced the autocorrelation problem by exploration of residuals. As in [9] different ANN based hybrid models are utilized to predict the medium term peak load, exponential smoothing and ARIMA are used to model the load trend then ANN models are used for forecasting monthly peak demand of Jeddah city. Many applications of artificial neural network for load forecasting can be found in literature. In the work of D. Bassi et al,[1] they have presented a solution to problem of (4) 84

predicting monthly electric loads (medium term) based on historical load data and economic and demographic data. The neural network chosen for this work is the Time Lagged Feed forward Network (TLFN). ANN was selected based on methodological selection of variables, a prior study of the problem, the processing that the ANN can make in temporal aspect. comparable to its competitor and sophisticated technique of neural network. Basic idea behind support vector regression (SVR) is to map the data into higher dimensional space to map nonlinearity in original data [17], as to perform linear regression in higher dimensional space. The SVR function is expressed by following equation, Where represent the nonlinear mapping of data from input space x, and b is the threshold value. The performance of SVR is majorly dependent on kernel function being used. Kernel functions are used to perform operations in the input space rather than higher dimensional space. Different kernel functions are reported in literature, like polynomial, Radial Basis Functions, Multi Layer Perceptions [18]. The general form of radial basis kernel function is expressed in following, (5) (6) Figure 3. NARX Implementation through ANN Fuzzy set theory based Neural Network model for short term load forecasting is reported in [11]. It uses fuzzy inference to generate rule set from historical data. Then the parameters of rule set are tuned by 3 layered feed-backward network. This model is used to predict daily load profile of Greek Interconnected Power System. Papalexopoulos et al, [12] has presented an ANN based STLF system for Pacific Gas and Electric company (PG&E). Three layered feed-forward neural network with seventy-seven input neurons, twenty-four neurons with sigmoid transfer function in hidden layer and twenty four output neurons are used to represent 24 hour forecasted load of next day. This model outperformed to earlier linear regression model by handling rapidly changing weather conditions. Neural network trained with exogenous and endogenous variables is reported in [13] and termed as multi context artificial neural network (MCANN). It is modified form of multi context ANN as it has divided the hidden layer into two parts to speed up network training. Clustering based forecasting technique has been reported in [14]. Radial Basis Functions (RBF) has been used to train ANN model. Based on similarity, data is first partitioned into different clusters and then passed to two layered RBF-NN for estimation of future demand. Influence of real time energy price on power demand is modeled by [15], by using RBF based NN along adaptive fuzzy inference system. Fuzzy inference serves to establish relation in price fluctuation and energy demand. Based on historical data RBF-NN first estimate load for next day, then fuzzy rule set is used to adjust forecasted result by incorporating the price effect. Energy price based load demand estimation for Ontario Hydro System is performed by H. Chen[16]. To model nonlinear relationship of electricity price and load fluctuation, 3 layered feed-forward neural network with back-propagation training method is used. Support Vector Machine (SVM) proposed by Vapnik [17] is a statistical tool for classification and regression. SVM have greater ability of generalization while avoiding over fit to data. Thus resulting in a best trade-off between complexity and performance. Its computational performance and accuracy is Several applications of support vector regression for load forecasting are reported in literature, some of key approaches are discussed in [19, 2]. Industrial load forecasting has been discussed in [21], in which authors has presented a composite model of fuzzy based neural network to predict long term industrial load. First fuzzy inference is used to estimate yearly maximum and minimum load values, further these results are combined with ANN to predict the annual peak loads. Fuzzy inference and neural model were developed on 1 year data of the industrial city of Ramadan in Egypt. Problem of short term load forecasting for Spanish industrial customers is discussed in [22]. They presented neuro-fuzzy system with backpropogation (BP) learning algorithm as well as Autoregressive Integrated Moving Average (ARIMA) process. Industrial customers are classified into different sets based upon their time of use, and then these sets are passed to ANN models. Final results are obtained by averaging results of these models. III. FORECASTING BY NARX FFNN MODEL As stated before, high coupling is found between techniques being used to solve the problem of load forecasting and data set being used. Due to nature of data available, parameter set under consideration and behavior of different input factors on load fluctuation. This concludes that one technique that has shown best results on some data set cannot be assured to show the same results on other data sets [1],[23]. In our work we have evaluated 3 different techniques and based on results, we have selected the NARX based FFNN model to solve our Long Term Load Forecasting (LTLF) problem for industrial sector. Results obtained do not generalize the superiority of NARX based FFNN structure on other models tested in this study. Beside a large amount of work has been done in LF domain, uniqueness of our work lies in utilization of NARX based model to industrial sector of energy demand forecasting with greater accuracy, this type of modeling have never been done before. There are several factors that have been considered while formulating this LF model, such as industrial customer growth rate, industrial electricity tariff, Industrial Gross Development 85

Percentage (GDP). Rise of industrial load is directly associated with rise in industrial consumers, and electricity tariff for the industry. Industrial GDP is an indicator of industrial growth and depicts its role in economy growth of the country. The industrial load and affecting parameters data sets are obtained from NTDC [24], ranging from to 21. We have made three divisions of whole data, 7% of data is used for training the models where 15% of data is utilized for validation of different models and remaining 15% for model testing. In NARX model structure the load at given time interval is elucidated by load at past time intervals as well as by introducing exogenous variables that track the seasonal variation in input factors. As a result input data set is converted into new input set of past readings and output data set of future outputs. In our case industrial load for given month or year can be predicted better by incorporating previous load values. The time delayed feedforward neural network used in this approach consists of 5 neurons in hidden layer with 2 levels of time delays. The resultant network structure can be shown by the fig. 4 below. from different approaches. Graphs are also plotted for each model to demonstrate their forecasting outcome. This clearly shows that NARX based FFNN model with 2.9% outperforms than other two techniques. TABLE I. Technique COMPARISON OF MAPE MAPE RBF-SVR 4.4 % ANN 5.1 % NARX-FFNN 2.9 % Results of Neural network fitting model with 2 hidden layer and 5 neurons in first hidden layer and single neuron in output layer, are shown by following graph. (7) Figure 4. NARX Neural Network Model The correlation obtained between input factors and output is.99. The regression plot of output and target is shown in fig. 5 below. 1 Predict(ANN) Figure 6. Prediction with ANN Support vector machines with the ability to find a global minimum and less prone to overfitting makes this approach appropriate for different application. In this study we have used radial bases function as kernel function of SVR. The results obtained are shown in Fig. 7. IV. Figure 5. Regression Plot RESULTS AND CONCLUSIONS We have evaluated these modeling approaches on industrial data of Pakistan. Models are ranked on the bases of Mean Absolute Percentage Error (MAPE). MAPE expresses accuracy as a percentage, and is defined by the formula as in eq. 7. As discussed before 7% of data is used for model training and remaining data is used for validation and testing of models. Cumulative average MAPE on training and test data is presented here. The following table shows results obtained Finally results obtained from NARX based FFNN model are shown in Fig. 8. Graphs have been plotted against the cumulative yearly values of actual and regressed values. It is evident from this graph that results obtained by this model closely fit to target values. This clearly indicate that proposed hybrid model of NARX based FFNN is an attractive approach for predicting yearly and monthly peak demand of industrial sector. This model has shown very accurate results with mean absolute percentage error of 2.9%. Based upon predictive performance and better accuracy NARX based neural network model was espoused than support vector regression and neural network fitting models. 86

Powered by TCPDF (www.tcpdf.org) 1 1 Figure 7. Prediction with SVR Figure 8. Prediction with NARX FFNN Though we found good results on available data sets of country wide industrial customers, resulting in monthly and yearly prediction of peak load demand. Data collected is based upon monthly billing and usage history. This study could be expanded towards region wise weekly load demand estimation, if such data could be made available. ACKNOWLEDGMENT This research is co-sponsored by National ICT R&D Fund and NTDC (WAPDA) Pakistan. We specially appreciate Nisar A. Bazmi, Chief Engineer, NTDC. for providing data sets and co-operating us to carry out this study. REFERENCES Predict(SVR) Predict(NARX) [1] Hahn H., Meyer-Nieberg S., Pickl S. Electric load forecasting methods: Tools for decision making, European Journal of Operational Research, 199 (3), pp. 92-97, (December ). [2] Hesham K. Alfares; Mohammad Nazeeruddin, Electric load forecasting: literature survey and classification of methods, International Journal of Systems Science, Volume 33, Issue 1, pages 23 34, (January 22). [3] M. Bodén, A Guide to Recurrent Neural Networks and Backpropagation, in IN THE DALLAS PROJECT, SICS TECHNICAL REPORT T22:3, SICS, 22. [4] J. M. P. Menezes,Jr. and G. A. Barreto, Long-term time series prediction with the NARX network: An empirical evaluation, Neurocomput., vol. 71, no. 16-18, pp. 3335 3343, Oct. 28. [5] Tsungnan Lin, Bill G. Horne, Peter Tino, C. Lee Giles, Learning longterm dependencies in NARX recurrent neural networks, IEEE Transactions on Neural Networks, Vol. 7, No. 6,1996, pp. 1329-1351. [6] Yang Gao, Meng Joo Er, NARMAX time series model prediction: feedforward and recurrent fuzzy neural network approaches, Fuzzy Sets and Systems, Vol. 15, No. 2, 25, pp.331-35. [7] A. Muñoz, E. F. Sánchez-Úbeda, A. Cruz, and J. Marín, Short-term Forecasting in Power Systems: A Guided Tour, in Handbook of Power Systems II, pp. 129-16, 21. [8] Espinoza, M.; Suykens, J.A.K.; Belmans, R.; De Moor, B.;, "Electric Load Forecasting," Control Systems, IEEE, vol.27, no.5, pp.43-57, Oct. 27. [9] A.A. Desouky, M.M. Elkateb, Hybrid adaptive techniques for electricload forecast using ANN and ARIMA, IEE Proceedings of Generation, Transmission and Distribution,, 147(4), 213-217. [1] Danilo Bassi, Oscar Olivares, "Medium Term Electric Load Forecasting Using TLFN Neural Networks" International Journal of Computers, Communications & Control Vol. I (), No. 2, pp. 23-32. [11] A. G. Bakirtzis, J.B. Theocharis, S.J. Kiartzis, K.J. Satsios, Short term load forecasting using fuzzy neural networks, IEEE Trans, Power Syst, 1 (3) (1995) 1518-1524. [12] A. D. Papalexopoulos et al., An Implementation of a Neural Network Based Load Forecasting Model for the EMS, IEEE Trans. Power Systems. Vol. 9, No. 4, p. 1956-1962 (). [13] T. Rashid et al., A Practical Approach for Electricity Load Forecasting, World Academy of Science, Engineering and Technology (25). [14] A.S.Pandey et al., Clustering based formulation for Short Term Load Forecasting International Journal of Intelligent Systems and Technologies 4:2 (). [15] Zhang Yun, Zhou Quan et al. RBF Neural Network and ANFIS-Based Short-Term Load Forecasting Approach in Real-Time Price Environment. IEEE Transactions on Power Systems, Vol. 23, No. 3, (August 28). [16] H. Chen, et al, "ANN Based Short-Term Load Forecasting in Electricity Markets", Proceedings of the IEEE Power Engineering Society Transmission and Distribution Conference,2:411-415, (21). [17] V.Vapnik. The Nature of Statistical Learning Theory}. Springer Verlag, 1995.ISBN -387-94559-8. [18] Tutorial on Support Vector Machine (SVM) Vikramaditya Jakkula, School of EECS, Washington State University, Pullman 99164. [19] Yongli Wang et al."svm Model Based on Particle Swarm Optimization for Short-Term Load Forecasting", Proceedings of the 5th international symposium on Neural Networks, Pages: 642 649, ISBN:978-3-54-87733-2 (28).. [2] T D. Niu, Y. Wang, and D. D. Wu, "Power load forecasting using support vector machine and ant colony optimization," Expert Systems with Applications, August. [21] M. A. Farhat, Long-term industrial load forecasting and planning using neural networks technique and fuzzy inference method, in Proceedings of the 24 IEEE Universities Power Engineering Conference, pp. 368 372, 24. [22] Domingo A. Gundin, Celiano Garcia, Yannis A. Dimitriadis, Eduardo Garcia, Guillermo Vega, Short-Term Load Forecasting for Industrial Customers Using FASART and FASBACK Neuro-fuzzy Systems, Power Systems Computation Conference (PSCC), Seville, Spain, 22. [23] S. Rahman and O. Hazim, A generalized knowledge-based short-term load-forecasting technique, IEEE Trans. Power Syst., vol. 8, pp. 58 514, May 1993. [24] National Transmission & Despatch Company (NTDC), Pakistan. 87