Backpropagation Based Training Algorithm for Takagi - Sugeno Type MIMO Neuro-Fuzzy Network to Forecast Electrical Load Time Series

Transcription

1 Backpropagato Based Trag Agorthm for Takag - Sugeo Type IO Neuro-Fuzzy Network to Forecast Eectrca Load Tme Seres Aoy Kumar Pat, ember, IEEE, ad Gerhard Doedg deeg GmbH, Kurfuersteaee - 30, D-8 Breme, Germay Fax: , E-mas: aoypat@yahoo.com ; pat@deeg.de Water Aheer ad Dobrvoe Popovc Uversty of Breme, FB / NW, D-8359 Breme, Germay Abstract - The paper descrbes a Backpropagato based agorthm that ca be used to tra the Takag-Sugeo (TS) type mut-put mut-output (IO) euro-fuzzy etwork effcety. The trag agorthm s effcet the sese that t ca brg the performace dex of the etwork, such as the sum squared error (SSE), dow to the desred error goa much faster tha that the smpe backpropagato agorthm (BPA). Fay, the above trag agorthm s tested o euro-fuzzy modeg ad forecastg appcato of Eectrca oad tme seres. Idex Terms: Takag-Sugeo mode, IO-Neuro-fuzzy etwork, Backpropagato agorthm, Eectrca Load tme Seres, Forecastg. I. INTRODUCTION We propose here a trag agorthm for a mut-put mut-output (IO) Takag-Sugeo type Neuro-fuzzy (NF) etwork ad euro-fuzzy approach for modeg ad forecastg appcato of eectrca oad tme seres. The euro-fuzzy approach attempts to expot the merts of both eura etwork ad fuzzy ogc based modeg techques. For exampe, the fuzzy modes are based o fuzzy f-the rues ad are to a certa degree trasparet to terpretato ad aayss. Whereas, the eura etworks based mode has the uque earg abty. I ths paper TS type IO euro-fuzzy etwork s costructed by mutayer feedforward etwork represetato of the fuzzy ogc system as descrbed secto II, whereas ts trag agorthm s descrbed secto III. Neuro-fuzzy modeg ad forecastg etc., are descrbed secto IV, ad fay bref cocudg remarks are preseted secto V. II. NEURO-FUZZY NETWORK The fuzzy ogc system (FLS) () cosdered ths paper for costructg the mut-put ad mut-output (IO) euro-fuzzy etwork s based o Takag-Sugeo (TS) type fuzzy mode, ad wth Gaussa membershp fuctos (GF), product ferece rue, ad a weghted average defuzzfer. f ( x) y = z z, = = (a) where, =, 3, L, m; =, 3, L, ; y = θ 0 + θ x, wth =, 3, L, ; = =, 3, L,m; =, 3, L, ; { ( x c ) }, z = a exp = wth =, 3, L,. (b) σ (c) Here, we assume that =,, σ a c U > 0, ad y V, where U, ad V are the put ad output uverses of dscourse respectvey. The correspodg th rue from the above fuzzy ogc system ca be wrtte as foows: R : If x s G ad x s G ad. ad x s G The y = θ 0 + θ x + θ x + L+ θ x () Where, x, wth =, 3,, ; are the umber of puts to the system, whereas, f, wth =, 3,, m; are the m umber of outputs from the system, ad G, wth, =, 3,,, ad =, 3,,, are the Gaussa membershp fuctos (GFs) wth correspodg mea ad varace parameters as c, σ respectvey ad y s the output cosequet of the th rue. The rues () wth above cosequet are kow as Takag-Sugeo (TS) rue. I the fuzzy ogc system () the Gaussa membershp fucto (GF) s deberatey chose because the same membershp fucto s cotuousy dfferetabe at a pots, that s a esseta requremet to appy gradet method based trag agorthm. Furthermore, t s aso mportat to ote that the FLS () s capabe of uformy approxmatg ay oear fucto to ay degree of accuracy over a uverse of dscourse U = R. Now by carefuy observg the fuctoa forms of (), t ca be see that the above FLS ca be represeted as a three ayers mut-put mut-output feedforward etwork as show fgure-. Because of euro mpemetato of TS type fuzzy ogc system fgure- represets actuay a TS type mut-put mut-output euro-fuzzy (NF) etwork where, stead of coectg weghts ad bases as eura etwork - we have here the mea ( c ) ad varace ( σ ) parameters of the GFs, aog wth ( θ 0, θ ).e., y from the rues /0/$ IEEE

2 cosequet as the equvaet adustabe parameters of the etwork. Now based o ay gve set of put-output data f the adustabe parameters of the NF etwork are sutaby seected the above FLS ca correcty approxmate the uderyg oear system that geerated the gve set of put output data pars. III. TRAINING ALGORITH FOR NF NETWORK The fuzzy ogc system oce represeted as the equvaet mut-put mut-output feedforward etwork (fgure-), fact ca be traed usg ay sutabe trag agorthm such as the stadard backpropagato agorthm (BPA). However, because of sow covergece speed of pure BPA comparso to ay other secod order trag method, the foowg a more effcet trag method, amey the combato of BPA wth modfed error dex exteso based trag agorthm w be used. A. Backpropagato Trag for TS-IO- NF Network Gve a set of N umber of put-output data pars of the form ( x ) p, d p, such that p p x ( p x x p x ) U,, L, R, d m p V R ; the obectve s to determe a fuzzy ogc systems f ( p x ) the form of (), such that the performace fucto,.e., the sum squared error (SSE) N S p T = 0.5 e = 0.5 E E (3a) p= m ad S = S = ( S + S + L + Sm) (3b), = s mmzed, where, E = coum vector of errors for the th output from the fuzzy ogc system p { ( p e ) p = f x d }, p =,..., N. I addto, we have aso assumed that a =, ad, correspodg to the umber of mpemeted membershp fuctos ad aso the umber of rues, s gve. Therefore, the probem s reduced to the adustmet of the y.e., ( θ 0, θ ) parameters from the rues cosequet, ad the mea (c ) ad varace ( σ ) parameters of the GFs, so that the SSE s mmzed. For smpcty, we w use here S, f, d, ad e to deote SSE, f ( p x ) d p, p, ad e respectvey, so that the error ( ( p e f x ) d ) p =. The steepest descet gradet rue, used for euro-fuzzy etwork trag s based o the recursve expressos: η ( S θ ) ( S θ ) ( S c ) θ 0 ( k + ) = θ 0 ( k) 0 (4) θ ( k + ) = θ ( k) η (5) k c ( + ) = c ( k) η (6) ( S σ ) σ ( k + ) = σ ( k) η (7) where, S s the SSE, as the trag performace fucto, at the k th terato step ad ( k ), ( k ), c ( k) θ 0 θ, ad σ ( k) are the free parameters of the etwork at the k th terato step, ad ther startg vaues are geeray seected radomy, η s the costat step sze or earg rate ad usuay η <<, ad =, 3,..., ; where s the umber of puts to the FLS or euro-fuzzy etwork whereas, =, 3,..., m; ad m s the umber of outputs from the euro-fuzzy etwork, ad =, 3,..., ; that correspods to the umber of GFs seected as we as the umber of rues,, etc. Now from fgure-, t s see that the etwork output f, ad hece the S, ad therefore fay S = SSE, depeds o θ 0, ad θ oy through y. Smary, the etwork output f - ad hece S ad S - aso deped o c, ad σ oy through z, where, f, y, b, ad z are represeted by (8), (9), (0) ad () respectvey. f y = h = y = θ 0 + θ x + θ x + L+ θ x ( z b) ad b = (8) (9) h =, z (0) = x c z = exp = σ () Therefore, the correspodg cha rues w be as gve by (), (3), (4) ad (5) respectvey. ( θ ) = ( S f ) ( f y ) ( θ ) y S 0 0 () ( S θ ) = ( S ) ( ) ( f f y θ ) y (3) ( S c ) = S z ( z c ) = = = ( S f ) ( f z ) ( z c ) ( S σ ) = ( S z z σ ) = = = ( S f ) ( f ) ( z z σ ) (4) (5) /0/$ IEEE

3 Fay, the resuts of the cha rues are wrtte as foows: ( S ) = ( f d ) ( z b) θ 0 (6) ( S ) = ( f d ) ( z b) x θ (7) { } ( S ) ( c = A z x c ) ( ) σ (8) ( ) = ( σ ) ( 3 S A z x c ) Where, A = = σ (9) ( f d ) ( y f ) b ( f d ) θ + θ x f b = 0 = = (0) Therefore, wth the above resuts the fa update rues for the etworks free parameters w be as foows: ( k + ) = ( k ) ( f d ) ( θ η z b) ( k + ) = ( k) ( f d ) ( θ η z b) x ( k ) = ( ) η ( c k. A. z x c ) ( σ ) θ 0 0 () θ () c σ { } + (3) ( ) = ( ) η ( ) ( 3 k k A.. ) + σ z x c σ (4) The equatos (3) through (4) represet the backpropagato trag agorthm (BPA) for the Takag- Sugeo type mut-put mut-output euro-fuzzy etwork, or the equvaet fuzzy ogc system of form () whch the cosequet part of the fuzzy rues are ear such as: y = θ 0 + θ x + θ x + θ x + L+ θ x 3 3. I the above rues cosequet f the coeffcets θ = 0, for =, 3, L,; =, 3, L, ; ad m = ; the the resutg fuzzy ogc system or the equvaet euro-fuzzy etwork s exacty detca wth the mut-put sgeoutput euro-fuzzy etwork as descrbed the [], []. The resutg FLS ca be thought of as a speca case of both amda as we as Takag-Sugeo type FLS where rue cosequet s a sgeto fuzzy set. However, for =, 3,..., 4; =, 3,, ; ad for m = the resutg fuzzy ogc system s detca wth TS type mut-put ad sgeoutput (ISO) euro-fuzzy etwork as descrbed [3]. Geeray, the BPA, based o steepest descet gradet rue, uses a ow earg rate (Lr) η << for etwork trag order to avod the oscatos the fa phase of the trag, ad therefore, the backpropagato trag usuay requres a arge umber of recursve steps or epochs. The acceerato of the trag process wth cassca backpropagato s however achevabe f the adaptve verso of the earg rate s used, or the mometum verso of the steepest descet rue s used. The mometum verso of the BPA ca be wrtte as foows: θ 0 ( k + ) = ( k) η ( mo) θ 0 {( f )( d z b) } mo θ ( k ) θ c σ 0 ( k + ) = ( k) η ( mo) {( f )( b) } mo d z x θ ( k ) θ ( k + ) = ( k ) η ( mo) c { A ( ) ( ) } mo z x c σ c ( k ) ( k + ) = ( k) η ( mo) A z σ 3 ( ) ( ) mo x c σ ( k ) σ (5) (6) (7) (8) where, W ( k ) = W ( k) W ( k ), (9) ad mo = mometum costat <. For further vestgato purposes, we have fay modfed aso the mometum verso of the BPA by addg to t the modfed error dex term of (30) as [4]. ( ) e e N Sm ( x) = 0.5γ ( x) avg, (30) = N where, eavg = e ( x) (3) N = Where, e avg s the average error as defed (3). Now ths defes the ew error dex or the performace fucto as foows: ( x) S( x) ( x) Sew = + Sm, (3) where, S(x) s the umodfed performace dex of the etwork or SSE as defed secto-3. From ths ew gradet ca be defed as foows: ( ( e ) e( x) ( x) = e( x) + e( x) ( x) S γ (33) ew avg where, (gama) γ = Costat factor <<, that has to be chose appropratey. So that, wth the modfed error dex exteso as per equato (33) we eed to add oy a ew term γ ( e( x) eavg) wth the orga error vector e ( x). B. Adaptve Learg Rate ad Oscato Cotro The proposed BPA wth modfed error dex exteso of SSE fucto as ew performace fucto, ad aso wth a /0/$ IEEE

4 sma mometum term, geera, s very effcet, ad faster tha the pure BPA trag the NF etwork cosdered ths paper. However, the performace fucto (SSE) of the etwork s ot aways guarateed to reduce towards the desred error goa every epoch. Ths meas that trag ca some tme proceed to the egatve drecto gvg rse to cotuous crease SSE (stead of reducg t after every epochs) or eve oscato.e., some tmes the SSE may crease or decrease resutg og trag tme or eve udesrabe trag performace. I order to guaratee a satsfactory trag performace - three sets of adustabe parameters are recommeded to be stored. If the cosecutve two ew sets of adustabe parameters reduce the etwork SSE fucto the foowg epochs the same sets are used ad earg rate s creased by a factor of. tmes the earer earg rate for further trag. Otherwse f the SSE teds to crease beyod some factor say WF tmes the curret SSE wth the ew sets of parameters, the the same ew sets are dscarded ad trag s proceeded wth the od sets of adustabe parameters. Thereafter, a ew drecto of trag s sought wth the od sets of parameters ad wth ower vaues of Learg rate parameter for e.g. 0.8 or 0.9 tmes of od earg rate. By ths way, every epoch SSE s ether decreased steady or at east s mataed wth the certa rego of curret performace vaue. IV. IDENTIFICATION / ODELING OF NONLINEAR DYNAICS The proposed TS type mut-put mut-output eurofuzzy agorthm has bee tested tay for modeg ad forecastg the Eectrca oad tme seres. The tme seres predcto wth ead tme L predcts the tme seres vaues at (t + L), based o the avaabe tme seres data up to the pot t. To forecast the eectrca oad tme seres usg the eurofuzzy approach, the tme seres data X={X, X, X 3,., X q } has to be rearraged a mut-put-mut output (IO) XIO ke structure. XIO stads for the tme seres X s represeted Iput ad Output form. For the gve tme seres modeg ad forecastg appcato the IO euro-fuzzy predctor to be deveoped s supposed to operate wth oy four puts.e., ( = 4), ad wth three outputs.e., (m = 3) oy. Now f the sampg terva ad the ead tme of forecast both are take to be tme ut, the for each t 4, the put data ths case represet a four dmesoa vector ad output data a three dmesoa vector as descrbed beow. XI(t-3) = [X(t-3), X(t-), X(t-), X(t)], XO(t-3) = [X(t+), X(t+), X(t+3) ] Furthermore, order to have sequeta output each row the vaues of t ru as 4, 7, 0, 3.,(q-6); ad so o, so that the XIO matrx w ook ke (34). I (34) the frst four coums of the XIO matrx represet the four puts to the etwork whereas, ast three coums represet the output from the euro-fuzzy etwork. X, X X 3, X 4, X 5, X 6, X 7 X 4, X 5, X 6, X 7, X 8, X 9, X 0 XIO = X q 6, X q 5, X q 4, X q 3, X q X q, X q (34) I ths forecastg exampe 63 put-output data were geerated ad oy frst 0 put-output data sets from XIO matrx (.e., frst 0 rows from XIO matrx) were used for NF etwork trag whereas, remag 663 data were used for forecastg test. The trag ad forecastg performaces acheved wth the euro-fuzzy etwork are ustrated ad sted fgure- ad 3, ad tabe- respectvey. From fgure- ad tabe- t ca be see that the trag agorthm brgs the SSE as the performace fucto dow to smoothy from ts ta vaue oy 300 epochs. Furthermore, the SSE pot (fgure-) every epoch shows that the trag does ot exhbt much oscato. The resut ceary shows the exceet trag as we as forecastg performace of TS type euro-fuzzy etwork. As further exampes, the modeg of other eectrca oad tme seres were cosdered, ad t has bee aso observed that mut-put ad mut-output NF etwork traed wth proposed agorthm ca approxmate the other eectrca oad tme seres wth hgh accuracy. V. CONCLUDING REARKS I the paper a effcet trag agorthm based o combato of BPA ad modfed error dex exteso has bee deveoped to tra the Takag-Sugeo type mut-put mut-output (IO) Neuro-fuzzy etwork much speedy. Furthermore, t was aso observed that addto of a sma mometum term ad adaptve verso of the earg rate ad aso the modfed error dex exteso of the performace fucto mproves the covergece speed of the etwork sgfcaty. The traed NF etwork s foud to be very effcet modeg ad predcto of the varous oear dyamcs. However, the fuzzy rues geerated through IO or ISO euro-fuzzy trag s occasoay foud to be o-trasparet, meag that cear terpretato of a the tued fuzzy sets are some tmes mpossbe. Ths s due to the fact that the membershp fuctos fay tued through euro-fuzzy etwork trag s very ofte hghy smar or argey overappg o each other, gvg rse to a dffcut stuato to terpret. For sovg ths probem or to mprove the terpretabty of fuzzy rues set theoretc smarty measures [5] shoud be computed for each par of fuzzy sets ad the fuzzy sets whch are hghy smar shoud be merged together to a sge oe. If the tued membershp fucto happes to be a uversa fuzzy set wth the uverse of dscourse that shoud be aso removed as t does ot cotrbute aythg the rue base. Of-course, the mproved trasparecy s obtaed sometmes at the cost of sacrfcg the mode accuracy. Furthermore, sce the parameters of the Gaussa membershp fuctos are ucostraed, t s /0/$ IEEE

5 probabe that the fuzzy partto occasoay may ot ook ke usua fuzzy parttos. Therefore, such cases terpretato of the traed euro-fuzzy system may ot be possbe. oreover, the determato of optmum umber of fuzzy rues ad optmum umber of membershp fucto () are aso mportat as because uecessary arger rue base may over ft the osy data ad thereby, resutg poor geerazato or predcto abty wth vadato data set. Here, for the determato of optmum umber of rues or membershp fuctos the GA or geera evoutoary computato (EC) may be used as a proper support too. Furthermore, f aog wth BPA the Least squares estmators (LSE recursve / o recursve) woud have bee used to determe the rues cosequet parameters the covergece of the BPA coud be creased further. Ateratvey, t was observed that the Leveberg-arquardt agorthm deveoped for Takag-Sugeo type IO euro-fuzzy etworks ad mpemeted by the preset authors, though computatoay very heavy, ofte eads to better trag performace wth faster covergece ad esseta for og term predcto of eectrca oad tme seres [6]. Refereces [] L. X. Wag, ad J.. ede, Back-Propagato Fuzzy System as Noear Dyamc System Idetfers, ISBN: /9 IEEE, FUZZ-IEEE 99 pp [] A. K. Pat, ad D. Popovc, "Forecastg Chaotc Tme Seres Usg Neuro-Fuzzy Approach," IEEE-IJCNN 999, 0-6 th Juy, 999, Washgto D.C., USA, pp , paper o. - 88, CD RO proceedgs. [3] A. K. Pat, ad R.Babuška, "Effcet trag agorthm for Takag- Sugeo type Neuro-fuzzy etwork," FUZZ-IEEE 00, eboure, Austraa, d 5 th December, 00, paper o. P38 CD RO proceedgs. [4] D. Popovc, ad D. Xaosog, Structura Optmzato of Neura Networks for odeg, SYSID 997, th IFAC Symposum o System Idetfcato, Fukuoka, Japa, 8- Juy 997, pp (vted paper). [5].Setes, R.Babuška, U.Kaymark, et. a., Smarty measures fuzzy rue base smpfcato, IEEE Trasacto o System, a ad Cyberetcs, 998, Part-B, Vo. 8, Jue-998, pp [6] A. K. Pat, Trag Takag-Sugeo Type ut-iput ut-output Neuro-Fuzzy Networks wth the Leveberg-arquardt Agorthm to Geerate Log Term Forecasts of Eectrca Load Tme Seres, Techca Report - TR-NF0-00 Jauary 00 deeg GmbH, Breme, Germay. utpe Outputs f f m z / b y y m y m y b z / b G Z G G.... G G... z G x x x utpe Iputs x x..... x Fgure-: IO-TS type Neuro-Fuzzy etwork represetato of Fuzzy Logc System TABLE - Tabe-: descrbes the trag ad forecastg performace of TS type IO euro-fuzzy etwork S. No. Data sets from XIO Trag parameters Ita SSE (wth prescaed data, scag factor = 0.0 ) Fa SSE (wth prescaed-data) to 0 rows - trag data = 5, Lr = , gama = (Iput) = 4, m (output) = 3, 0.5, mo= 0.5, epoch =300 to 63 rows ( trag + forecastg) rows forecast /0/$ IEEE

6 Trfmmo:SSE-Vs-Epoch 300 Sum Squared Error NF(gr)-Actua(bue)-Error(red) No of Epochs Trfmmo: Neuro-fuzzy output-vs-actua Tme Fgure-: ustrates the trag performace of TS type IO euro-fuzzy etwork smfmmo.m: Neuro-fuzzy output vs. A ctua NF(gr)-Actua(b) tme smfmmo.m: Neuro-fuzzy predcto error NF Predcto Error (red) tme Fgure-3: ustrates the forecastg performace of TS type IO euro-fuzzy etwork. Note: the fgure-3 data from to 0 correspod to trag data ad data from to 3489 represet the forecastg performace wth vadato data set. It s mportat to ote that data wth the tme pots 00 to are dfferet from the trag data. St the TS type IO Neuro-fuzzy etwork ca predct ths data rego wth reasoaby hgh accuracy /0/$ IEEE