The Effects of Pre-Processing Methods on Forecasting Improvement of Artificial Neural Networks

Transcription

1 Australia Joural of Basic ad Applied Scieces, 5(6): , 2011 ISSN The Effects of Pre-Processig Methods o Forecastig Improvemet of Artificial Neural Networks A. Azadeh, M. Sheikhalishahi, M. Tabesh, A. Negahba Departmet of Idustrial Egieerig ad Ceter of Excellece for Itelliget Based Experimetal Mechaics, College of Egieerig, Uiversity of Tehra, Ira Abstract: Alog other methods, Itelliget Methods ca be used i order to model the tred of chages of a certai variable. These methods require data to be preprocessed before beig used i the forecastig process. Geerally, the preprocessig step icludes omittig outliers, assessmet of the missig data, data smoothig, etc. I this paper, the effect of various smoothig methods o the fial forecasted results is studied. Furthermore, data from the electricity cosumptio i Ira over the past 20 years were used as actual data. After beig smoothed, these data are the icorporated ito a Artificial Neural Network i order to forecast the electrical cosumptio. The comparisos betwee several Seasoal Decompositio, icludig Seasoal Adjustmet Series (SAS) ad Seasoal Tred Cycle (STC), Expoetial Smoothig (Simple, Liear, Holt ad Witer) ad Box- Jekis (Movig Average, Auto Regressio, ad Auto Regressio Itegrated Movig Average) methods show the superiority of SAS i Decompositio categorizatio over other methods. The structure of this study may be used for other data sets for improvemet of data pre-processig. Key words: Artificial Neural Network; Pre-Processig; Forecastig; Improvemet INTRODUCTION It is a difficult task to forecast the electrical cosumptio due to its various seasoal ad mothly chages. Electrical cosumptio represets two importat attributes: o oe had, it shows mothly ad seasoal chages ad o the other had it shows a icreasig tred (Fig. 1). Fig. 1: The electricity cosumptio tred i a period of 132 moths. Correspodig Author: A. Azadeh, Departmet of Idustrial Egieerig ad Ceter of Excellece for Itelliget Based Experimetal Mechaics, College of Egieerig, Uiversity of Tehra, Ira Tel: , Fax: E~mail: aazadeh@ut.ac.ir or ali@azadeh.com 570

2 Several methods are used i order to solve forecastig problems. Forecastig methods ca be categorized ito two mai groups: statistical methods ad artificial itelligece methods. I statistical methods, after traiig the historical data, a equatio is used to preset the relatioship betwee load ad its correspodig factors. While, i artificial itelligece methods, huma way of thikig ad reasoig is copied. Artificial itelligecebased algorithms have bee itroduced based o expert system, evolutioary programmig, fuzzy system, artificial eural etwork, ad a combiatio of these algorithms. Statistical methods such as regressio models (Taylor ad Buizza, 2003; Goia et al., 2010), Box Jekis model (Abraham ad Nath, 2001), expoetial smoothig (Huag et al., 2002) ad Kalma filters (Park et al., 1991) caot represet the complex oliear relatioships amog factors. Therefore, Artificial Itelligece-based algorithms are beig widely used recetly. Neural Network has come to occupy a promiet place i forecastig problems, as well as regressio ad time series methods. Neural Networks are suitable for modelig problems with ukow factors. ANN have bee successfully used for short term load forecastig (Villalba ad Bel, 2000; Sejyu et al., 2004; kadil et al., 2006; Vahidiasab et al., 2008). Kadil et al. (2006) demostrated ANN capabilities i load forecastig without the use of load history as a iput. Xiao et al. (2009) preseted a approach of back propagatio eural etwork with rough set (RSBP) for complicated Short-term load forecastig with dyamic ad o-liear factors to develop the accuracy of predictios. Through attribute reductio based o variable precisio with rough set, the ifluece of oise data ad weak iterdepedecy data to BP is avoided; therefore, the time take for traiig is decreased. Ghiassi et al. (2006) preseted the developmet of a dyamic artificial eural etwork model (DAN2) for medium term electrical load forecastig. Yearly ad seasoal model were preseted that used past mothly system loads to forecast future electrical demads, ad both of them produce mea absolute percet error (MAPE) values below 1%, demostratig the effectiveess of DAN2 i forecastig medium term loads. The objective of this paper is to fid a appropriate smoothig method to be used with the Neural Network. Furthermore, differet preprocessig methods are used for smoothig ad ormalizig the iput data i order to acquire more accurate output data. Azadeh et al. (2007) tried to preprocess the data to get a better output. Movig average method was cosidered ad utilized i order to elimiate the tred of data. Azadeh et al. (2008) used time series-based model to study the impact of data preprocessig ad post-processig o the performace of a fuzzy system. Zhag ad Qi (2005) studied the effectiveess of data preprocessig, icludig deseasoalizatio ad detredig, o eural etwork modelig ad forecastig performace. It is fouded that eural etworks are ot able to capture seasoal or tred variatios effectively with the upreprocessed raw data. Moreover, it is fouded that detredig or deseasoalizatio ca dramatically reduce forecastig errors. Caas et al. (2006) ivestigated the effects of data preprocessig o model performace usig cotiuous ad discrete wavelet trasforms ad data partitioig. The results showed that etworks traied with preprocessed data perform better tha etworks traied with udecomposed, oisy raw sigals. To the best of our kowledge, o comprehesive preprocessig method has bee applied before usig ANN for the forecastig. The remaider of the paper is orgaized as follows: i sectio two, the Artificial Neural Network is reviewed. Sectio three presets differet data preprocessig methods. Error calculatio methods are itroduced i sectio four. I sectio five, the computatioal results of differet methods are compared with each other, ad fially coclusios are preseted i sectio six. 2. The Neural Network: Simply defied, ANNs are mathematical techiques desiged to deal with differet problems. The research i the field has a history of several decades, but the iterest started to grow i the early 1980s. Today, Neural Networks ca be cofigured i various arragemets to perform a rage of tasks icludig patter recogitio, data miig, classificatio, forecastig ad process modelig. ANNs are composed of attributes that lead to perfect solutios i applicatios where we eed to lear a liear or oliear mappig. Some of these attributes are learig ability, geeralizatio, parallel processig, ad error edurace. These attributes would eable ANNs to solve complex problem methods precisely ad flexibly. ANNs cosist of a iter-coectio umber of euros. There are may varieties of coectios uder study; however, here we will discuss oly oe type of etwork which is called Multi Layer Perceptio (MLP) etwork. I this etwork, the data flow forward to the output cotiuously without ay feedback. Fig. 2 shows a typical three-layer feed forward model used for forecastig purposes. The iput odes are the previous lagged observatios, while the output provides the forecast for the future value. I order to process the iformatio received by the iput odes, hidde odes with appropriate oliear trasfer fuctios are used. The model ca be writte as 571

3 Fig. 2: A three layer MLP etwork. m yt a0 j f ijyti 0 jt j1 i1 (1) Where m is the umber of iput odes, is the umber of hidde odes, f is a sigmoid trasfer fuctio 1 such as the logistic: f ( x) aj, j 0,1,..., is a vector of weights from the hidde to output 1exp( x). odes ad ij, i 1,2,..., m; j 0,1,..., are weights from the iput to hidde odes. a0 ad 0 j are weights of arcs leadig from the bias terms which have values always equal to 1. Note that Eq. (1) which idicates a liear trasfer futio is employed i the output ode as desired for forecastig problems. The MLP s most popular learig rule is the error back propagatio algorithm. The ANN models are researched i coectio with may power system applicatios. Most of the employed models use MLP etworks. The attractio of MLP has bee explaied by the ability of the etwork to lear complex relatioships betwee iput ad output patters, which would be difficult to model usig covetioal algorithmic methods. There are three steps i solvig a ANN problem: (1) traiig, (2) geeralizatio, ad (3) implemetatio. Traiig is a process that is leart by the etwork to recogize the preset patter from iput data set. We preset the etwork with traiig examples, which cosist of a patter of activities for the iput uits together with the desired patter of activities for the output uits. For this reaso, each ANN uses a set of traiig rules that defie traiig method. Geeralizatio or testig evaluates etwork ability i order to extract a feasible solutio whe the iputs are ukow to etwork ad are ot traied to etwork. We determie how closely the actual output of the etwork matches the desired output i ew situatios. I the learig process, the values of itercoectio weights are adjusted so that the etwork produces a better approximatio of the desired output. ANNs lear by example. They caot be programmed to perform a specific task. The examples must be selected carefully, otherwise useful time is wasted or eve worse the etwork might be fuctioig icorrectly. The disadvatage is that because the etwork fids out how to solve the problem by itself ad its operatio ca be upredictable (Schalkoff, 2007). The etwork preseted i this paper cosists of a 12-euro iput layer with a liear trasfer fuctio ad two hidde layers, with sigmoid fuctio, i which there are 16 euros i the first ad 6 euros i the secod layer. The output layer cosists of oly oe euro with liear trasfer fuctio. The learig method of the model is a Back Propagatio (BP), mometum, weight decay method. 3. Data Preprocessig Methods: I this step, we attempt to preprocess data i order to get better outputs. Preprocessig methods ca be categorized as statistical, data miig, metahueristics methods, etc. While workig with these techiques, it is of tremedous importace to take ito cosideratio the data post-processig, which eables us to covert the forecasted results to real data. I the statistical methods preseted i this paper, data ca ot be post-processed, 572

4 thus, test data will be preprocessed as well as the data from the learig period, ad the the etwork is assessed. Preprocessig i this paper is a two-stage process. First, elimiatig the icreasig tred of data, ad secod, data ormalizig. The mai goal of this paper is to evaluate the differet techiques of elimiatig the tred of data. Movig Average, Expoetial Smoothig, Seasoal Decompositio, ad Auto Regressio are discussed i this study Tred Elimiatig Methods: C Movig Average Method: I this method, the average of previous data is subtracted from the origial data. I this paper a 12-moth movig average is employed. C Expoetial Smoothig Method: This method was first itroduced by C.C.Holt (1958) ad was employed to trai useasoal time series with o tred. However, the method ca also be used for elimiatig tred ad estimatio of the seasoal chages of time series. I the expoetial smoothig method, greater weights are give the ew data rather tha the older data. The assiged weights decrease expoetially for older data. This forecastig method icludes Brow s Simple Expoetial Method, Brow s Liear Method, ad Holt-Witers Smoothig Method. C Seasoal Decompositio Method: I this method a set of data are decomposed ito seasoal compoets. This model icludes a composed tred ad cycle compoets ad a error fractio. C Auto Regressio Method: Box-Jekis forecastig method icludes Movig Average MA(q) processes, Auto Regressio AR(p) process, Auto Regressio Movig Average ARMA(pq) processes, Itegrated Models, ad Auto Regressio Itegrated Movig Average ARIMA (P,O,q) process, i which P is the order of Auto Regressio, q is the order of MA, ad d is the subtracted order Normalizatio: I this step, the data must be ormalized over the rage [0, 1]. This is ecessary for two reasos: First, it is ecessary to guaratee that all iput data have bee give the same weight. If the iputs of two euros lie i differet rages, the the euro with the larger absolute scale will be favored durig traiig. This is because of the distace-based rules used i the algorithms. Secod, the euros trasfer fuctio, either as a sigmoid fuctio or a hyperbolic taget (tah) fuctio, requires limited rage of values over [0, 1]. Various ormalizig methods are geerally employed to this ed. A commo feature of these methods is that a offset is deducted from the data items, after which it is coverted over the desired rage by meas of a scalig factor. The most commoly employed method for ormalizatio ivolves liear mappig over a specified rage, whereby each value of a variable x is trasformed as follows: x max x mi x.( xxmi ) x mi (2) x x max mi I this study, due to usig the sigmoid trasfer fuctio, x = 0.9 ad x = Calculatig the Estimated Error: For calculatig the estimated error, it is assumed that AT is the actual value of the variable at time t, max mi ad At is the estimated value obtaied by a forecastig method. The error ca be obtaied by subtractig the estimated value from the actual value. Therefore, estimatio error is calculated as follows: et At At (3) The estimatio error ca be used as a criterio for evaluatig the accordace of the forecastig method with the actual patter of data. For example, whe a method is capable of estimatig the tred ad seasoal ad periodic compoets of the time series, the estimatio error reflects the irregular compoet. Total sum of errors i a forecastig method i which periodic sets are observed, is equal to: se [ At At] i1 Due to probabilistic ature of the estimatio error, Absolute Error is used i order prevet positive ad egative errors from cacelig each other out. (4) 573

5 AE AT AT (5) Mea Absolute Deviatio is a error statistics that averages the absolute differece betwee each pair of actual ad fitted forecast data poits. Thus, MAD is calculated as follows: MAD et [ At At] i1 i1 Mea Squared Error also keeps positive ad egative errors from cacelig each other. MSE is a absolute error measure that squares the errors ad ca be calculated as follows: MSE 2 2 et [ At At] i1 i1 I some cases, especially whe the cost of the forecast error is more closely related to the percetage error tha the umerical size of the error, calculatig the forecast error as a average percet error of the historical data poits is most appropriate. MAPE is a relative error statistic measure as a average percet error of the historical data poits ad is obtaied as follows: (6) (7) et [ At At] i1 At i1 AT MAPE I this paper, MAPE is used for calculatig the forecast error. (8) 5. Computatioal Results: I this sectio, the results of usig differet preprocessig methods o data ad estimated methods from the 13 th to 120 th moths are used. This is due to the reaso that data of the first 12 moths is ot available i the Movig Average Method ad i some methods, icludig Seasoal Decompositio, these data cotai cosiderable error. Data of the last 12 moths are used to test the etwork. Figure (3) demostrates the data after beig smoothed ad before beig ormalized. The usuitability of the data from the first 12 moths ca be easily see i the figure 2. Fig. 3: Tred of chages after processig with each smoothig method. 574

6 Now results of smoothig ad preprocessig methods are show i figure 4-12: Fig. 4: Smoothed ad ormalized data from Movig Average Method. Fig. 5: Smoothed ad ormalized data from Auto Regressio Method. After smoothig ad ormalizig the data usig the above ie methods, the desiged Artificial Neural Networks are traied by the first 108 record ad the data of the last 12 moths are estimated. The error is calculated by Mea Absolute Percetage Error (MAPE) method. Table 1 presets the results associated with each method. 575

7 Fig. 6: Smoothed ad ormalized data from the Auto Regressio Itegrated Movig Average (ARIMA) with parameters (0, 12, 2). Fig. 7: Smoothed ad ormalized data from Brow Simple Expoetial Method. Accordig to the table, the SAS Seasoal Decompositio Method yields to the smallest value of estimatio error. Witer Expoetial Smoothig ad the STC Seasoal Decompositio methods take the secod ad third place regardig the smallest estimatio error value, respectively. The rest of methods accordig to the least value of estimatio error ca be ordered as follows: Holt Expoetial Smoothig, Movig Average, Auto Regressio, Liear Expoetial Smoothig, ARIMA, ad Simple Expoetial Smoothig. 6. Coclusio: Itelliget methods ca be used i order to model the tred of chage of a certai variable which require data preprocessig before usig i the forecastig process. I this paper the effects of various smoothig 576

8 Fig. 8: Smoothed ad ormalized data from Brow Liear Expoetial Method. Fig. 9: Smoothed ad ormalized data from Holt Expoetial Method. 577

9 Fig. 10: Smoothed ad ormalized data from Holt-witer Expoetial Method. Fig. 11: Smoothed ad ormalized data from SAS Seasoal Decompositio Method. methods have bee studied usig electricity cosumptio data i Ira from the past 20 years. After smoothig, a itegrated ANN has bee used i order to forecast the electrical cosumptio. Results showed that Seasoal Decompositio Method ad Expoetial Smoothig have resulted i the smallest values of error usig the Holt ad Holt-Witers Techiques. Alog these methods, Box- Jekis methods result i small values of error. Fially, Table 2 compares the features of the proposed approach with previous researches. As it ca be see, i most of the previous researches, a robust relative error estimatio method was used. While, most of the previous approaches are capable of hadlig complex ad o-liear relatioships, oly a few of them are able to hadle corrupted data. The distictive feature of the proposed approach is that it is the first research i which the effect of comprehesive preprocessig methods o the forecastig results of ANN is cosidered. 578

10 Fig. 12: Smoothed ad ormalized data from STC Seasoal Decompositio Method. Table 1: Ppreprocessig methods results Box-Jekis Expoetial Smoothig Seasoal Decompositio Movig Auto ARIMA Simple Liear Holt Witer STC SAS Average Regressio (MA) (AR) MAPE Table 2: The features of this study versus other methods Feature Comprehesive Hadle Robust relative Hadlig Itelliget Method preprocessig complex ad error estimatio corrupted modelig ad o-liear method (MAPE) data forecastig relatioships The proposed approach % % % % % Taylor ad Buizza(2003) % % Cheg ad Wei(2009) % % % Azadeh et al. (2007) % % % % Zhag ad Qi (2005) % % % Caas et al. (2006) % % Yag ad Stezel(2006) % ACKNOWLEDGMENT The authors are grateful for the valuable commets ad suggestio from the respected reviewers. Their valuable commets ad suggestios have ehaced the stregth ad sigificace of our paper. 579

11 REFERENCES Azadeh, A., S.F. Ghaderi ad S. Sohrabkhai, Forecastig electrical cosumptio by itegratio of Neural Network, time series ad ANOVA', Applied Mathematics ad Computatio, 186: Azadeh, A., M. Saberi, S.F. Ghaderi, A. Gitiforouz ad V. Ebrahimipour, 'Improved estimatio of electricity demad fuctio by itegratio of fuzzy system ad data miig approach', Eergy Coversio ad Maagemet, 49: Abraham, A. ad B.A. Nath, euro-fuzzy approach for modelig electricity demad i Victoria, Applied Soft Computig, 1(2): Caas, B., A. Fai, L. See ad G. Sias, 'Data preprocessig for river flow forecastig usig eural etworks: Wavelet trasforms ad data partitioig', Physics ad Chemistry of the Earth, 31: Chatfield, C., The Aalysis of Time Series, CHAPMAN & HALL. Cheg, C.H. ad L.Y. Wei, Oe step-ahead ANFIS time series model for forecastig electricity loads, optimizatio ad egieerig, 0(0): Goia, A., C. May ad G. Fusai, Fuctioal clusterig ad liear regressio for peak load forecastig, Iteratioal Joural of Forecastig, i press. Ghiassi, M., D.K. Zimbra ad H. Saidae, Medium term system load forecastig with a dyamic artificial eural etwork model, Electric Power Systems Research, 76: Huag, H., R. Hwag ad J. Hsieh, A ew artificial itelliget peak power load forecaster based o o-fixed eural etworks, Electrical Power & Eergy Systems, 24(3): Kadil, N., R. Wamkeue, M. Saad ad S. Georges, A efficiet approach for short term load forecastig usig artificial eural etworks, Electrical Power ad Eergy Systems, 28(8): Park, J.H., Y.M. Park ad K.Y. Lee, Composite modelig for adaptive short-term load forecastig, IEEE Trasactios o Power Systems, 6(2): Schalkoff, R.J., Artificial Neural Networks, McGraw Hill. Sejyu, T., P. Madal, K. Uezato ad T. Fuabashi, Next day load curve forecastig usig recurret eural etwork structure, IEE Proceedigs Geeratio, Trasmissio ad Distributio, 151(3): Taylor, J.W. ad R. Buizza, usig weather esemble predictios i electricity demad forecastig, Iteratioal Joural of Forecastig, 19(1): Villalba, S.A. ad C.A. Bel, Hybrid demad model for load estimatio ad short-term load forecastig i distributio electric systems. IEEE Tras Power Deliver, 15(2): Vahidiasab, V., S. Jadid ad A. Kazemi, Day-ahead price forecastig i restructured power systems usig artificial eural etworks, Electric Power Systems Research, Volume, 78(8): Xiao, Z., S.J. Ye, B. Zhog ad C.X. Su, BP eural etwork with rough set for short term load forecastig, Expert Systems with Applicatios, 36(1): Yag, J. ad J. Stezel, Short-term load forecastig with icremet regressio tree, Electric Power Systems Research, 76(9-10): Zhag, G.P. ad M. Qi, 'Neural etwork forecastig for seasoal ad tred time series', Europea Joural of Operatioal Research, 160: