Two Activation Function Wavelet Network for the Identification of Functions with High Nonlinearity

Transcription

1 Interntonl Journl of Engneerng & Computer Scence IJECS-IJENS Vol:1 No:04 81 Two Actvton Functon Wvelet Network for the Identfcton of Functons wth Hgh Nonlnerty Wsm Khld Abdulkder Abstrct-- The ntegrton of wvelet theory nto soft computng hve recently ttrcted gret nterest. The frst rel ntegrton of wvelet nto soft computng led to the development of wveletnetworks. In ths work, robust Two Actvton Functon Wvelet Network (TAFWN) s proposed for the dentfcton of functons wth hgh nonlnerty. The TAFWN s further trned by the lest men squres (LMS) to mnmze the men-squred error. Smulton results re demonstrted to vldte the dentfcton blty nd effcency of the proposed network. Index Terms-- Wvelet Network, Neurl network, Identfcton. I. INTRODUCTION A neurl network s n nterconnected network of smple processng elements, e.g. sclng nd flterng. The processng elements nterct long pths of vrble connecton strengths whch when sutbly dpted cn collectvely produce complex overll desred behvor. Obvously neurl networks re well suted to solvng the sme types of problems s the humn s brn. Prtculrly, neurl networks excel t recognton, dentfcton, nd clssfcton types of problems [1]. Neurl networks hve been ppled very successfully n the dentfcton nd control of dynmc systems. The unversl pproxmton cpbltes of the multlyer perceptron (the bckpropogton lgorthm) mke t populr choce for modelng nonlner systems nd for mplementng generl-purpose nonlner controllers []. Neurl networks hve been employed successfully n dptve control system desgn problems of nonlner systems, but there re stll some dffcultes n neurl network-bsed Wsm dptve control desgns. The bss functons re generlly not orthogonl or redundnt;.e., the network representton s not unque nd s probbly not the most effcent one. Furthermore, the convergence of neurl networks my not be gurnteed. Even when t exhbts good convergence rte, the trnng procedure my stll be trpped n some locl mnm, dependng on the ntl settngs. Wsm Khld Abdulkder Unversty of Kuf, Irq wesmkhld@yhoo.com In ddton, pproxmton errors nd externl dsturbnces cnnot be effcently ttenuted. Hence, performnce nd even stblty my not be gurnteed. By combnng the de of neurl networks nd the merts of wvelets, wvelet network ws developed nd t's consdered s specl cse of RBF network [3]. Wvelet networks cn n some stutons be n ttrctve lterntve to other neurl networks such s sgmod feed-forwrd nd Rdl Bss Functon (RBF) networks [4]. In ths pper, Two Actvton Functon Wvelet Network lgorthm s proposed. It ntroduces n dentfcton model of Two Actvton Functon Wvelet Network nd compres smultons of Wvenet nd Two Actvton Functon Wvelet Network lgorthms for ther lernng bltes to dentfy sngle vrble nonlner functons. II. NEURAL NETWORK ADAPTIVE WAVELETS (Wvenets) Wvelet neurl networks (WN) re feed-forwrd neurl networks usng wvelets s ctvton functon. WN hve been used n clssfcton nd dentfcton problems wth some success. In wvelet networks, both the poston nd the dlton of the wvelets re optmzed besdes the weghts [5]. WNs hve recently ttrcted gret nterest, becuse of ther dvntges over RBF networks s they re unversl pproxmtors but cheve fster convergence nd re cpble of delng wth lrge dmenson nd non-sttonry sgnls. In ddton, WNs re generlzed RBF networks [6]. WNs hve wde vrety of pplcton res, lke speech recognton nd dgtl communcton. They were lso ppled n robot pplctons where robot knemtcs s contnuous functon nd opertes n bounded rnge [7]. III. THE PROPOSED TWO ACTIVATION FUNCTION WAVELET NETWORK Wvelet network represents the pplcton of wvelets flters to neurl networks. An pplcton of two wvelet flters to neurl networks s nvestgted n ths pper. Ths new technque clled Two Actvton Functon Wvelet network (TAFWN). It s n nterestng lterntve to wvelet networks tht bsorbs the dvntge of hgh resoluton of wvelets nd the dvntges of lernng feed-forwrd neurl networks. The Two Actvton Functon Wvelet Network (TAFWN) s very smlr to wvelet Network (WN) but hve some mportnt dfferences, wheres wvelets hs n ssocted sclng functon ( nd wvelet functon (, TAFWN hs two sclng 1(, ( nd two wvelet functons 1(, IJECS-IJENS August 01 IJENS

2 Interntonl Journl of Engneerng & Computer Scence IJECS-IJENS Vol:1 No:04 8 (. Accordngly, there re two sclng flters nd two wvelet flters. IV. THE TWO ACTIVATION FUNCTION WAVELET NETWORK ALGORITHM The TAFWN rchtecture pproxmtes ny desred sgnl y ( by generlzng lner combnton of two set of dughter wvelets h1,, b ( nd h,, b (, where the dughter wvelets h1,, b ( nd h,, b ( re generted by dlton,, nd trnslton, b, from two mother wvelets h ( ) nd h ( ), where t b. The network rchtecture 1 s shown n fgure (1): h1,, b h,, b where ( ( t h ( ) 1. (1) t b h ( ).. () : Dlton fctor, wth > 0. b : Trnslton fctor. t : Sgnl tme ntervl A TAFWN s 3-lyers feed forwrd neurl network. Frst the TAFWN prmeters, dlton 's, trnslton b's, nd weght w's should be ntlzed, nd the desred sets of dt, the nput sgnl x(, the desred output (trge y(, the number of sclng functons p (p= n ths work) nd the number of wvelons k re gven. Assumng tht the network output functon stsfes the dmssblty condton nd the network suffcently pproxmtes the trget. The pproxmted sgnl of the network yˆ ( cn be represented by equton: p k yˆ ( x( w h (. j 1 1, b (3) where x( s the nput sgnl. w, s the weght coeffcents between hdden nd j output lyers. j=1,,, p. p=: number of sclng functons. =1,,, k. k s number of wvelons. h s two set of dughter wvelets generted, b from two mother wvelets h (, h ( s n equtons (1) nd () respectvely. 1 Smlr to WN, fter constructng the ntl TAFWN nd fter clcultng output sgnl of the network, the trnng of TAFWN strts. It s further trned by the grdent descent lgorthms lke lest men squres (LMS) to mnmze the men-squred error. Durng lernng, the prmeters of the network re optmzed. The TAFWN prmeters w j,, j,, nd b cn be j, optmzed n the LMS lgorthm by mnmzng cost functon or the energy functon, E, over ll functon ntervl. The energy functon s defned by equtons (4) nd (5), y ( s the desred output (trge nd y ˆ( s the ctul output sgnl of TAFWN. E E 1 T e (. (4) t 1 1 T ( y( yˆ( ).(5) t 1 where T s the totl ntervl of functon, ( desred output (trge nd ˆ( of WN. y s the y s the ctul output sgnl To mnmze E the method of steepest descent s used, whch requres the grdents E, E, nd E for w updtng the ncrementl chnges to ech prtculr prmeter w,, nd b, respectvely. The, j, j, j grdents of E re: w T e( h( ) x(...(6) t 1 T e( x( w t 1 T e( x( w t 1..(8) where h( ) (7) h( ) IJECS-IJENS August 01 IJENS

3 Interntonl Journl of Engneerng & Computer Scence IJECS-IJENS Vol:1 No:04 83 t (9) The dervtves of the vrous wvelet flter wth respect to ts trnslton, h( ), re gven n [1]. The ncrementl chnges of ech coeffcent re smply the negtve of ther grdents. w.. (10) w b... (11) (1) Thus ech coeffcent w, b, nd of the network s updted n ccordnce wth the rule gven: w ( t 1) w ( t ) w w. (13) b ( t 1) b ( t ) b b. (14) ( t 1) ( t )... (15) Where s the fxed lernng rte prmeter [1]. At ech terton, the network prmeters re modfed usng the grdent descent lgorthm tll one of the stoppng condtons descrbed s stsfed. The trnng lgorthm of the proposed two ctvton functon wvelet network conssts of the followng sx steps: 1. Intlze TAFWN prmeters, dlton 's, trnslton b's, nd weght w's, p=, two mother wvelets flters t b t b, the desred sets of dt, the h, h 1 nput sgnl x(, the desred output (trge y(, nd the number of wvelons k re gven.. Set: the number of trnngs, ter = 0, the ncrementl chnges of ech coeffcent, ( w,, ) =0, nd the ntl squre error, E 0. 5 ter 3. Clculte the pproxmted sgnl of the network yˆ ( usng equton (3). 4. Clculte the grdents of ech coeffcent usng equtons (4), (5), (6) nd clculte the coeffcents ncrementl chnges whch re the negtve of ther grdents. 5. Choose constnt nd clculte the new coeffcents w, ter 1 b ter 1, nd of the network ter 1 n ccordnce wth the rules gven n equtons (13), (14) nd (15). 6. Clculte the squre error E usng equton (5). ter 1 If E s smll enough, then the trnng s good nd the ter 1 lgorthm s stopped. Otherwse, set ter = ter + 1 nd go to (3). V. TAFWN IDENTIFICATION MODEL Fgure () shows the TAFWN Identfcton Model confgurton of nonlner functon pproxmton nd ts essentl fetures. The functon f(x) nd the two wvelet flters re drven by the sme nput sgnl. The wvelet flters ctng on n nput sequence x(n) to produce n output sgnl y(n). The output sgnl d(n) supples the desred output sgnl for the wvelet flter tht pproxmtes the nonlner functon chrcterstc. The flter s desgned so tht the output should pproxmte trnng sgnl or desred output d(n). The estmton error e(n), whch s used for controllng the flter coeffcents, s the dfference between the desred output sgnl nd the ctul output sgnl of the dptve system. VI. SIMULATION CONFIGURATION AND RESULTS The TAFWN lgorthm, wth one hdden lyer of twenty [0 Morlet, 0 Slog1] wvelons n the hdden neurons (k = 0) nd fxed lernng rte of 1 s mplemented to dentfy ths hrd nonlner functon: f ( x) 005 (1 x) e... (16) x 1 sn(3 ( x 0.6) ) e for x [ 0,1], where the ntervl wdth s 0.01 [8]. 3( x 0.5) sn(4 ( x 0.9) ) Intl w's nd dltons 's re set to 0 nd 7 respectvely. b's re spced eqully prt throughout the trnng dt. The smulton result shown n Fgure (3-, b, c). Fgure (3-) shows the MSE gnst the number of tertons for off-lne trnng of the network. Fgure (3-b) llustrtes the performnce results of the network dentfyng the functon gven bove. The trned TAFWN s tested usng n nput sequence wth ntervl wdth of nsted of 0.01 wthout chngng the coeffcents (k, 's, b's, nd w's) produced n trnng. Wth MSE equls to the pproxmted sgnl s obtned s shown n fgure (4). VII. WAVELET NETWORK vs. TAFWN In the prevous sectons, TAFWN Networks re proven to be, s well s mny other neurl prdgms, specfc cse of the generc prdgm nmed RBF Networks. The lernng performnce of the conventonl Wvelet Networks s provded for comprson to the proposed TAFWN structures. Moreover, n the present secton IJECS-IJENS August 01 IJENS

4 Interntonl Journl of Engneerng & Computer Scence IJECS-IJENS Vol:1 No:04 84 smultons of Wvenets lgorthm nd the TAFW Network lgorthm wll be nvestgted nd compred for ther lernng bltes to dentfy the nonlner functons s t s presented n the followng two exmples. Ths secton wll confrm ths de by provdng severl observtons derved from the results of the MATLAB smultons. RESULTS Smultons of Wvenets lgorthm nd the TAFW Network lgorthm s compred for ther lernng bltes to dentfy the pecewse defned nonlner functon of sngle vrble gven by: -.186x f(x )= 4.46x for x [-10,-] for x [-, 0] 0.05x e sn((0.03x+0.7)x) for x [ 0,10].. (17) where the ntervl wdth s 0.1. Fgures (5-) nd (6-) show the MSE gnst the number of tertons for off-lne trnng of ech network. Fgures (5-b) nd (6-b) llustrte the performnce results of the two networks The WN wth one hdden lyer of forty [40 Morlet] wvelons n the hdden neurons (k = 40). All lernng rte prmeters for weghts, dltons, nd trnsltons re fxed t 0.01 nd ntl weghts w's nd dltons 's re set to 0 nd 10, respectvely. Intl trnslton prmeters b's re spced eqully prt throughout the trnng dt to provde nonoverlppng prttons throughout the neghborng ntervls. TAFWN wth one hdden lyer of twenty [0 Morlet, 0 Rsp] wvelons n the hdden neurons (k = 0), nd fxed lernng rte of Intl weghts w's nd dltons 's re set to 0 nd 10, respectvely. Intl trnslton prmeters b's re lso spced eqully prt throughout the trnng dt. Note, tht there re no prtculr resons governng the choce of these prmeters. These prmeters were obtned from the experence of mny smultons tht were found to be effcent. In fgure (5-), whch represents the WN trnng, t cn be noted tht the obtned MSE equl to n 150 tertons. Fgure (6-) show tht by usng TAFWN trnng the MSE vlue equl to e-004 lso n 150 tertons, whle the MSE equl to t only 8 tertons. Tht s provng how the dentfcton property of TAFWN s strkngly mproved nd the trnng speed s gretly ncresed over wvelet network. Tble (1) gves comprson between these two dentfer network trnng n number of tertons, Tme, nd MSE vlue. The trned TAFWN s tested usng n nput sequence wth ntervl wdth of 0.01 nsted of 0.1 wthout chngng the coeffcents (k, 's, b's, nd w's) whch hve been produced from trnng TAFWN. Wth MSE equls to.537e-004 the pproxmted sgnl of 0.01 ntervl wdths s obtned wth respect to the desred sgnl s shown n fgure (7). By usng both WN nd the proposed TAFWN to pproxmte nother nonlner functon gven n equton (18) s shown n fgures (8) nd (9), t cn be noted how the TAFWN trnng s reder thn WN trnng. Tble () gves comprson between the trnng (number of tertons, Tme, nd MSE vlue) of these two networks: 3 f ( x) x 0.3x 0. 4x.. (18) For x [-, ], where the ntervl wdth s 0.1. The WN wth one hdden lyer of forty [40 Polywog5] wvelons n the hdden neurons (k = 40), ll lernng rte prmeters for weghts, dltons, nd trnsltons re fxed t 0.0. All ntl weghts w's nd dltons 's re set to 0 nd 10, respectvely. Intl trnslton prmeters b's re spced eqully prt throughout the trnng dt to provde nonoverlppng prttons throughout the neghborng ntervls. The TAFWN wth one hdden lyer of twenty [0 Rsp1, 0 Polywog5] wvelons n the hdden neurons (k = 0), nd fxed lernng rte of All ntl weghts w's nd dltons 's re set to 0 nd 8, respectvely. Intl trnslton prmeters b's re spced eqully prt throughout the trnng dt. The trned TAFWN s tested usng n nput sequence wth ntervl wdth of 0.01 nsted of 0.1 wthout chngng the coeffcents (k, 's, b's, nd w's) whch hve been produced from trnng TAFWN. Wth MSE equls to e-005 the pproxmted sgnl of 0.01 ntervl wdths s obtned wth respect to the desred sgnl s shown n fgure (10). VIII. CONCLUSIONS An dvnced wvelet network, clled Two Actvton Functon Wvelet Network s proposed s n nterestng lterntve to wvelet networks. Ths technque bsorbs the dvntge of hgh resoluton of wvelets nd the dvntges of lernng nd feed-forwrd of neurl networks. It s shown how the dentfcton property of wvelet neurl network s strkngly mproved nd the trnng speed s gretly ncresed through the proposed Two Actvton Functon Wvelet Network (TAFWN). Severl lgorthms for functon dentfcton re desgned, mplemented, nd tested usng Mtlb tool. The Two Actvton Functon Wvelet Network (TAFWN) structure s mplemented nd severl exmples re crred out to verfy ths mplementton. It cn be concluded tht ths structure cheves: ) n pproxmton for nonlner functons, ssumng resonble choce of the number of wvelons nd mother wvelet bss functons. ) t needs much less number of tertons durng trnng or pproxmton of nonlner systems n comprson wth tht requred by Wvelet Network IJECS-IJENS August 01 IJENS

5 Interntonl Journl of Engneerng & Computer Scence IJECS-IJENS Vol:1 No:04 85 REFERENCES [1] Gvpht Lekut, "Adptve Self-Tunng Neuro Wvelet Network Controllers ",Ph.D. Thess.Vrgn Polytechnc Insttute nd Stte Unversty. Deprtment of Electrcl Engneerng, Blcksburg, Vrgn, Mrch 31, [] M. Mst, Y. Mst nd J. Pogg, "Neurl Network Toolbox for use wth MATLAB", MATLAB verson 6.5. [3] Yu-Mn Cheng, Bor-Sen Chen, nd Fu-Yun Shu, "Adptve Wvelet Network Control Desgn for Nonlner Systems", Deprtment of Electrcl Engneerng - Ntonl Tsng-Hu Unversty, [4] T. Kugrjh, Prof. P.S. Krshnprsd nd Prof. W.P. Dywns, "Adptve Control Usng Wvelet Networks", H. H. Szu, Ed., Proceedngs SPIE- The Interntonl Socety for Mechncl Engneerng, v4, pp , [5] Mrc Thullrd, "A Revew of Wvelet Networks, Wvenets, Fuzzy Wvenets nd Ther Applctons", Semens Buldng Technologes, Cerberus Dvson, 8708 Mennedorf, Swtzerlnd, Mrc.thullrd@cerberus.ch, September 000. [6] Sheng-Tun L1 nd Shu-Chng Chen, "Functon Approxmton usng Robust Wvelet Neurl Networks", Deprtment of Informton Mngement, Ntonl Kohsung Frst Unversty of Scence nd Technology, Kohsung, Twn, ROC, 001. [7] Dongbng Gu nd Huosheng Hu, "Neurl Predctve Control for Cr-lke Moble Robot", Deprtment of Computer Scence, Unversty of Essex Wvenhoe Prk, Colchester CO4 3SQ, UK, Interntonl Journl of Robotcs nd Autonomous Systems, Vol. 39, No. -3, My, 00. [8] Ycne OUSSAR, Gerrd DREYFUS, "Intlzton by Selecton for Wvelet Network Trnng", Lbortory of Electronc Superor School of Physcl nd Chemstry Industrl 10, rue Vuqueln F PARIS Cedex 05, FRANCE, [9] P. Pcton, "Neurl Networks", Second Edton 000, publshed by Plgrve. [10] Wley, Chchester, "Neuro-Fuzzy nd Hybrd Approches", Deprtment of Electrcl Engneerng, Unversty of Hong Kong, 1997, ISBN [11] Wen Yu nd Xoou L, "Fuzzy Identfcton Usng Fuzzy Neurl Networks Wth Stble Lernng Algorthms", IEEE Trnsctons on Fuzzy Systems, Vol. 1, No. 3, 004. Fg.. TAFWN Identfcton Model () Men-Squre Error per lernng terton Fg. 1. Adptve TAFWN Structure (b) Desred nd Identfed output sgnl per tme IJECS-IJENS August 01 IJENS

6 Interntonl Journl of Engneerng & Computer Scence IJECS-IJENS Vol:1 No:04 86 (c) TAFWN prmeter tme hstores Fg. 3. Smulton results of TAFWN trnng (b) Desred nd Identfed output sgnl per tme Fg. 4. Output sgnl of trned TAFWN (c) WN prmeters tme hstores Fg. 5. Smulton results of WN trnng () Men-Squre Error per lernng terton () Men-Squre Error per lernng terton IJECS-IJENS August 01 IJENS

7 Interntonl Journl of Engneerng & Computer Scence IJECS-IJENS Vol:1 No:04 87 (b) Desred nd dentfed output sgnl per tme () Men-Squre Error per lernng terton (b) Desred nd Identfed output sgnl per tme (c) TAFWN prmeters tme hstores Fg. 6. Smulton results of TAFWN trnng (c) WN prmeter tme hstores Fg. 7. Output sgnl of trned TAFWN Fg. 8. Smulton results of WN trnng IJECS-IJENS August 01 IJENS

8 Interntonl Journl of Engneerng & Computer Scence IJECS-IJENS Vol:1 No:04 88 Fg. 10. Output sgnl of trned TAFWN () Men-Squre Error per lernng terton TABLE I Comprsons between WN nd TAFWN of ther lernng bltes Network type No. of tertons Tme MSE vlue TAFWN sec WN sec TAFWN sec WN sec (b) Men-Squre Error per lernng terton TABLE II Comprsons between WN nd TAFWN of ther lernng bltes Network type No. of tertons Tme MSE vlue TAFWN sec WN sec TAFWN sec WN 50 9 sec (c) TAFWN prmeter tme hstores Fg. 9. Smulton results of TAFWN trnng IJECS-IJENS August 01 IJENS