NONLINEAR SYSTEM IDENTIFICATION USING RBF NETWORKS WITH LINEAR INPUT CONNECTIONS

Transcription

1 Malaysian Journal of Compuer Science, Vol. No., June 998, pp NONLINEAR SYSEM IDENIFICAION USING RBF NEWORKS WIH LINEAR INPU CONNECIONS Mohd Yusoff Mashor School of Elecrical Elecronic Engineering Universiy Science of Malaysia Pera Branch Campus 3750 ronoh, Pera Malaysia Fax: ABSRAC his paper presens a mofied RBF newor wih adional linear inpu connecions ogeher wih a hybrid raining algorihm. he raining algorihm is based on - means clusering wih square roo updaing mehod Givens leas squares algorihm wih adional linear inpu connecions feaures. wo real daa ses have been used o demonsrae he capabiliy of he proposed RBF newor archiecure he new hybrid algorihm. he resuls incaed ha he newor models adequaely represened he sysems dynamic. Keywords: Nonlinear sysem, Sysem idenificaion, Neural newor, RBF newor, Linear inpu connecions.0 INRODUCION Neural newors have heoreically been proved o be capable of represening complex non-linear mappings [], []. Mulilayered feed forward neural newors offer an alernaive o convenional modelling mehods, newors wih various raining laws have successfully been used o model non-linear sysems [3], [4], [5], [6], [7], [8]. In general, neural newor models are highly non-linear in he unnown-parameers. A drawbac of his propery is ha he raining algorihm mus be based on a non-linear opimisaion echnique which is associaed wih he problems of slow parameer convergence, inensive compuaion poor local minima. Raal Basis Funcion (RBF) newors ha overcome some of hese problems have been inroduced by several auhors [8], [9]. his newor offers a faser efficienlyrained-newor while parially avoing he problem of local minima. raining algorihms for RBF newors normally comprise of a procedure o posiion he RBF cenres a linear leas squares echnique o esimae he weighs. Various raining algorihms have been proposed o rain RBF newors [4], [5], [6], [7], [8]. In he presen sudy, he convergence properies of he hybrid algorihms are furher improved by proposing a RBF newor wih adional linear inpu connecions. As an alernaive o he hybrid algorihm [6], a new hybrid algorihm based on -means clusering using square roo updaing mehod Givens leas squares algorihm (wih adional linear inpu connecions feaures) is proposed o rain RBF newors. Givens leas squares algorihm has been seleced due o is superior numerical sabiliy accuracy..0 LINEAR-INPU-CONNECION-RBF NE- WORK A RBF newor wih m oupus n h hidden nodes can be expressed as: n h i i0 ij j j= y = w + w φ v c ; i =,..., m () where w ij, w i0 c bias connecion weighs RBF cenres respecively, v j are he connecion weighs, is he inpu vecor o he RBF newor composed of lagged φ inpu, lagged oupu lagged precion error is a non-linear basis funcion. denoes a sance measure ha is normally aen o be he Euclidean norm. he funcion φ ( ) can be seleced from a se of basis funcions such as linear, cubic, hin-plae-spline, muliquadraic, inverse muli-quadraic Gaussian funcions. In he presen sudy, he hin-plae-spline funcion has been seleced as he non-linear funcion because his funcion has a good modelling capabiliy [9]. hin-plae-spline is given by: φ( a) = a log ( a) () Since neural newors are highly non-linear, even a linear sysem has o be approximaed using he non-linear neural newor model. However, modelling a linear sysem using a non-linear model can never be beer han using a linear model. Considering his argumen, in he presen sudy a RBF newor wih linear inpu connecions is proposed. 74

2 Nonlinear Sysem Idenificaion Using RBF Newors wih Linear Inpu Connecions he proposed newor allows he newor inpus o be conneced recly o he oupu node via weighed connecions o form a linear model in parallel wih he non-linear original RBF model as shown in Fig.. he new RBF newor wih m oupus, n inpus, n h hidden nodes n l linear inpu connecions can be expressed as: nl nh i i0 ij ij j j= j= = + + y w λ vl w φ v c (3) where i =,,..., m, he λ s vl s are he weighs he inpu vecor for he linear connecions respecively. he inpu vecor for he linear connecions may consis of pas inpus, oupus noise lags. Since λ's appear o be linear wihin he newor, he λ's can be esimaed using he same algorihm as for he w s. As he adional linear connecions only inroduce a linear model, no significan compuaional load is added o he sard RBF newor raining. Furhermore, he number of required linear connecions are normally much smaller han he number of hidden nodes in he RBF newor. Fig. : he RBF newor wih linear inpu connecions 3.0 MODELLING NON-LINEAR SYSEMS here are a number of sues ha have been accomplished on modelling non-linear sysems using raal basis funcion newors [4], [5], [6], [7], [8], [9], [0]. Modelling using RBF newors can be considered as fiing a surface in a muli-mensional space o represen he raining daa se using he surface o prec over he esing daa se. herefore, RBF newors require all he fuure daa of he sysem o lie wihin he domain of he fied surface o ensure a correc mapping so ha good precions can be achieved. his is normal for he nonlinear modelling where he model is only valid over a cerain ampliude range. A wide class of non-linear sysems can be represened by he non-linear auo-regressive moving average wih exogenous inpu (NARMAX) model []. he NARMAX model can be expressed in erms of a non-linear funcion expansion of lagged inpu, oupu noise erms as follows: ( = ( ) L ( ) ( ) L ( ) y f y,, y n, u,, u n, s y u ( ), L, ( )) + e e n e e where y u e y = = M ; u = M e M ym u r em are he sysem oupu, inpu noise vecor respecively; ny, nu ne are he maximum lags in he oupu, inpu noise vecor respecively; m r are he number of oupu inpu respecively. he non-linear funcion, f s ( ) is normally very complicaed rarely nown a priori for pracical f s will be modelled sysems. In he presen sudy, using RBF newor expressed by equaion (3) where φ ( ) is chosen o be he hin-plae-spline funcion. he newor v is formed using lagged inpu, oupu inpu vecor, noise ha are denoed as u( ) u( n u ) L ( ) ( ) y y n y (4) L, e L e n e respecively in equaion (4). Anoher mehod o include a noise model in a RBF newor is o use only linear noise connecions as in Fig.. his approach can reduce he complexiy of he RBF newor hence accelerae he raining process. However, his approach only allows linear noise model may no be sufficien if he daa is highly corruped by nonlinear noise. raining algorihms for RBF newors normally comprise of a procedure o posiion he RBF cenres a linear leas squares echnique o esimae he weighs. In he presen sudy, a new hybrid algorihm based on -means clusering using square roo updaing mehod Givens leas squares algorihm wih adional linear inpu connecions feaures is proposed o rain RBF newors. Givens leas squares has been seleced because of he superior numerical sabiliy accuracy of his mehod []. 4.0 PROPOSED HYBRID ALGORIHM Given a se of inpu-oupu daa, u( ) y where =,, L N, he connecion weighs, cenres widhs may be obained by minimising he following cos funcion: N J = ( y y$ ) y y$ = (5) 75

3 Mashor where N $y ( ) are he number of daa used for raining he preced oupu generaed by using he RBF newor in equaion (3) respecively. Equaion (5) can be solved using a non-linear opimisaion or graen descen echnique. However, esimaing he weighs using such algorihm will desroy he advanage of lineariy in he weighs. hus, he raining algorihm is normally spli ino wo pars: (i) allocaion of he RBF cenres, c j (ii) esimaion of he weighs, w ij. his approach allows an independen algorihm o be employed for each as. he cenres are normally locaed using an unsupervised algorihm such as -means clusering, fuzzy clusering Gaussian classifier whereas he weighs are normally esimaed using a class of linear leas squares algorihm. Moody Daren [8] used he -means clusering mehod o posiion he RBF cenres a leas means squares algorihm o esimae he weighs, Chen e al. [6] used a -means clusering o posiioned he cenres a Givens leas squares algorihm o esimae he weighs. In he presen sudy, he -means clusering using square roo updaing mehod is used o posiion he RBF cenres a Givens leas squares algorihm wih adional linear inpu connecions feaures will be used o esimae he weighs. A deailed descripion of he -means clusering using square roo updaing mehod can be found in Daren Moody [3]. Afer he RBF cenres he non-linear funcions have been seleced, he weighs of he RBF newor can be esimaed using a leas squares ype algorihm. In he presen sudy, exponenial weighed leas squares were employed based on he Givens ransformaion. he esimaion problem using weighed leas squares can be described as follows: Define a vecor z() a ime as: z = [ z,..., z ] (6) n h where z( ) n h are he oupu of he hidden nodes vecor he number of hidden nodes of he RBF newor respecively. If linear inpu connecions are used, equaion (6) should be mofied o include linear erms as follows: z z L z zl L zl (7) [ nh nl ] = where zl s are he oupu of he linear inpu connecion nodes. Any vecor or marix of size n h should be increased o n h + n in order o accommodae he new l srucure of he newor. A bias erm can also be included in he RBF newor in he same way as he linear inpu connecions. Define a marix Z() a ime as: z z Z = : z an oupu vecor, y() given by: y = [ y,..., y] hen he normal equaion can be wrien as: y = Z Θ (0) where Θ( ) is a coefficien vecor given by: [ nh nl ] = + Θ w,..., w he weighed leas squares algorihm esimaes (8) (9) () Θ by minimising he sum of weighed squared errors, defined as: e = ( ) i i WLS β [ y Z Θ ] () i= where β, 0 < β <, is an exponenial forgeing facor. he soluion for he equaion (0) is given by Θ = [ Z Q Z] Z Q y (3) where Q() is n n agonal marix defined recursively by Q = [ β Q( ) ], Q = ; (4) β() n are he forgeing facor he number of raining daa a ime respecively. Many soluions have been suggesed o solve he weighed leas squares problem in equaion (3) such as recursive mofied Gram Schmid, fas recursive leas squares, fas Kalman algorihm Givens leas squares. In he presen sudy, Givens leas squares wihou square roos was used. he applicaion of he Givens leas squares algorihm o adapive filering esimaion have simulaed much ineres due o superior numerical sabiliy accuracy []. Givens leas squares wihou square roos is summarised below. Inroduce a n n agonal marix D() as: [ ( ) ] D = ag d ( ) L d n ( ) σ n = n h + n ε ; l + (5) where n, d's σ ε () are he number of he esimaed RBF weighs, he leas squares esimaion errors he sard deviaion a ime respecively. Define a n n mensional upper riangular marix R() as: r r3 K K r n 0 r r n 3 O O 0 O O O O M R = M O O O O M M O O O O r( n ) n 0 L L (6) 76

4 Nonlinear Sysem Idenificaion Using RBF Newors wih Linear Inpu Connecions where he r's are he leas squares esimaed coefficiens ( i afer esimaing x ) ( i from x ). he Givens Leas Squares algorihm solves for Θ() in equaion (3) by performing a Givens ransformaion: / D ( R / ) [ ( )] D [ R] ( 0) / ( 0) ( 0) [ x... xn ] (7) δ 0 [ ] where ( 0) ( 0) [ x L xn ] z zn y L = (8) δ (0) is called a maximum lielihood facor [], iniialised o /β(), where β() is updaed using equaion (4). Afer i-h seps, Givens ransformaion ransfers / / / 0 L 0 ( ) ( ) rii ( ) L ( ) rin ( ) + ( i ) / ( 0 L 0 x x i i ) ( i ) / ( i i ) ( L i ) / ( xn i ) ( δ ) ( δ ) + ( δ )... (9) ino / / / 0 L 0 rii L rin + ( i) / ( 0 L 0 0 xi i ) ( L i ) / ( xn i ) ( δ ) + ( δ )... (0) he explici compuaion of Givens ransformaion can be expressed as follows: ( i ) ( xi i ) = ( ) + δ ( ) c = ( ( i ) xi i ) () b = δ ( i) ( i ) δ = cδ ( i) ( i ) ( i ) x = x x ri ( ) = i+,..., n ( i ) () ri = cri ( ) + bx he algorihm is iniialised by seing σε ( 0) = 0 R( 0) = I D( 0) = ρ where σ ε (3) 0 ρ are he iniial sard deviaion of he esimaed parameers a large posiive scalar respecively; I is an n n ideniy marix. he forgeing facor β = β0β( ) + β0 where β0 β( 0) o one β 0 is normally larger han β is normally compued accorng o: (4) are ypically chosen o be less bu close β 0. Afer Givens ransformaion is compleed, he esimaed Θ can be calculaed by bac parameer vecor, subsiuion. w ( ) = r (5) n n n ( n ) i in ij j j= i+ w = r r w ; i = ( n ), ( n 3),..., (6) he RBF newor using he hybrid algorihm based on - means clusering using square roo updaing mehod Givens Leas Squares wih adional linear inpu connecions feaure can be used o model non-linear sysems. Iniially, he RBF cenres are posiioned using he -means clusering using square roo updaing mehod hen he Givens leas squares algorihm is used o esimae he weighs of he RBF newor, w s. 5.0 MODEL VALIDAION here are several ways of esing a model such as one sep ahead precions (OSA), model preced oupu (MPO), mean squared error (MSE), correlaion ess, chisquare ess. In he presen sudy, he firs four ess were used o jusify he performance of he fied models. OSA is a common measure of precive accuracy of a model ha has been considered by many researchers. OSA can be expressed as: y$ = fs( u( ), L, u( nu ) y( ), L, y( ny ) ε$, θ $, L, ε$ n, θ $ (7) ( ε ) he residual or precion error is defined as ε$, θ $ = y y$ (8) where f s ( ) is a non-linear funcion, in his case he RBF newor. Anoher es ha ofen gives a beer measuremen of he fied model precive capabiliy is he model preced oupu. Generally model preced oupu can be expressed as: = ( ( ) L ( ) ( ) L ( ) y$ d fs u,, u nu, y$ d,, y$ d ny, (9) 0, L, 0) he deerminisic error or deerminisic residual is $ = y y$ (30) ε d d MSE is an ieraive mehod of model validaion where he model is esed by calculaing he mean squared errors afer each raining sep. MSE es will incae how fas a precion error or residual converges wih he number of raining daa. he MSE a he -h raining sep, is given by ) 77

5 Mashor n d EMSE(,Θ ) = y( i) y$ ( i, Θ ) (3) nd i= where EMSE (, Θ ) y$ ( i, Θ ) are he MSE OSA for a given se of esimaed parameers Θ( ) afer raining seps respecively, n d is he number of daa ha were used o calculae he MSE. An alernaive mehod of model validaion is o use correlaion ess o deermine if here is any precive informaion in he residual afer model fiing []. he residual will be unprecable from all linear nonlinear combinaions of pas inpus oupus if he following hold []: Φεε ( τ) = E[ ε( - τ) ε ] = δ for all τ Φuε ( τ) = E[ u( - τ) ε ] = 0 for all τ Φε( ε )( τ) = [ ε( ε( τ) ( τ) )] = τ u E - u 0 for 0 (3) Φ ( τ) = [( ( τ) ) ε ε ] = τ ' E u - u 0 for all u Φ ' ( τ) = E[ ( u ( - τ) u ) ε ε ] = 0 for all τ u where u E [ ] are he mean value of u he expecaion respecively. In pracice, if he correlaion ess lie wihin he 95% confidence limis, ±96. N, hen he model is regarded as adequae, where N is he number of daa used o rain he newor. fied RBF newor model. he newor has been rained using he following specificaions: v = u( ) u( ) y( ) y( 4) wih a bias inpu vl = u u y y 4 e 3 e 4 e 5 ; n h = 0 Noice ha vl() denoes he linear inpu connecions vecor he e s represen he linear noise erms. OSA MPO generaed by he newor model over boh he raining esing daa ses are shown in Fig. 3 respecively. he plos show ha he model precs very well over boh he raining esing daa ses. Correlaion ess shown in Fig. 4 are quie reasonable τ plos are marginally ouside where Φ u ( τ ) ' ε Φ u ' ε he 95% confidence limis. he evoluion of MSE obained from he newor model is shown in Fig. 5. As he model precs very well has reasonable correlaion ess, he model was considered o be sufficien o represen he idenified sysem. 6.0 APPLICAION EXAMPLES he proposed RBF newor rained using he hybrid algorihm was used o model wo sysems. In all he examples, he cenres were iniialised o he firs few samples of he inpu-oupu daa. During he calculaion of he MSE, he noise model was excluded from he model since he noise model will normally cause he MSE o become unsable in he early sage of raining. In all examples, all he 000 daa were used o calculae he MSE he designing parameers were aen as = , = 0. 99, 0 = ρ β β 0 Fig. : OSA superimposed on acual oupu for example Example he firs daa se, S was aen from a hea exchanger sysem consiss of 000 samples. A deailed descripion of he process can be found in Billings Fadhil [4]. he firs 500 daa were used o idenify he sysem he remaining 500 daa were used o es he Fig. 3: MPO superimposed on acual oupu for example 78

6 Nonlinear Sysem Idenificaion Using RBF Newors wih Linear Inpu Connecions OSA MPO generaed by he fied model are shown in Fig. 6 7 respecively. he plos show ha he model precs very well over boh he raining esing daa ses. All he correlaion ess, shown in Fig. 8, are well τ inside he 95% confidence limis excep for Φ u which is marginally ouside he confidence limis a lag 7. he evoluion of he MSE plo is shown in Fig. 9. Since he model precs very well has good correlaion ess, he model can be considered as an adequae represenaion of he idenified sysem. ' ε Fig. 4: Correlaion ess for example Fig. 7: MPO superimposed on acual oupu for example Fig. 5: MSE for example Example S is a ension leg plaform, he firs 600 daa were used o rain he newor he res were used o es he fied newor model. he newor was rained using he following srucure v = u u 3 u 4 u 6 u 7 [ u( 8) u( ) y( ) y( 3) y( 4)] = ( ) ( ) ( ) ( ) ( 3) ( 5) vl y y e e e e wih bias inpu; n h = 40 Fig. 8: Correlaion ess for example Fig. 6: OSA superimposed on acual oupu for example Fig. 9: MSE for example 79

7 Mashor 7.0 CONCLUSIONS A RBF newor wih adional linear inpu connecions a hybrid raining algorihm based on square roos - means clusering Givens leas squares (wih adional linear inpu connecions feaure) has been inroduced. wo pracical daa ses were used o es he efficiency of he mofied RBF newors he hybrid raining algorihm. he examples showed ha he fied RBF newor models yield good precions correlaion ess. Hence, he proposed mofied RBF newors ogeher wih he hybrid raining algorihm is considered as an adequae represenaion of he idenified sysem. REFERENCES [] G. Cybeno, Approximaions by superposiion of a sigmoidal funcion, Mahemaics of Conrol, Signal Sysems, Vol., 989, pp [] K. Horni, Approximaion capabiliies of mulilayer feed forward newors, Neural Newors, Vol. 4, 99, pp [3] S. A. Billings, H. B. Jamaludn S. Chen, Properies of neural newors wih applicaions o modelling non-linear dynamical sysems, In. J. of Conrol, Vol. 55, 99, pp [4] S. Chen, S. A. Billings, C. F. N. Cowan P. M. Gran, Non-linear sysems idenificaion using raal basis funcions, In. J. Sysems Science, Vol., 990, pp [5] S. Chen, C. F. N. Cowan, P. M., Gran, Orhogonal leas squares learning algorihm for raal basis funcion newors, IEEE rans. on Neural Newors, Vol., 99, pp [6] S. Chen, S. A. Billings P. M. Gran, Recursive hybrid algorihm for non-linear sysem idenificaion using raal basis funcion newors, In. J. of Conrol, Vol. 55, 99, pp [7] M. Y. Mashor, Sysem idenificaion using raal basis funcion newor, PhD hesis, Universiy of Sheffield, Unied Kingdom, 995. [8] J. Moody, C. J. Daren, Fas learning in neural newors of locally-uned processing unis, Neural Compuaion, Vol., 989, pp [9] M. J. D. Powell, Raal basis funcion approximaions o polynomials, Proc. h Biennial Numerical Analysis Conf., Dundee, 987, pp [0] S. Chen, S. A. Billings, C. F. N. Cowan, P. M. Gran, Pracical idenificaion of NARMAX models using raal basis funcions, In. J. Conrol, Vol. 5, 990, pp [] S. A. Billings, W. S. F. Voon, Srucure deecion model valiy ess in he idenificaion of non-linear sysems, Proc. IEE, Par D, Vol. 7, 986, pp [] F. Ling, Givens roaion based on leas squares laice relaed algorihms, IEEE rans. on Signal Processing, Vol. 39, 99, pp [3] C. Daren, J. Moody, Fas adapive -means clusering: Some empirical resuls, In. Join Conf. on Neural Newors, Vol., 990, pp [4] S. A. Billings, M. B. Fadhil, he pracical idenificaion of sysem wih non-lineariies, Proc. 7h IFAC Symp. on Idenificaion Sysem Parameer Esimaion, Yor, U.K., 985, pp BIOGRAPHY Mohd Yusoff Mashor obained his B.Eng.(Hons.) in Conrol Compuer Eng. from Polyechnic of Cenral London in 990, M.Sc. in Conrol Informaion echnology from Sheffield Universiy in 99 PhD from Sheffield Universiy in 995. Currenly, he is a lecurer a Deparmen of Elecrical Elecronic Eng., Universiy Science Malaysia. His research areas include Neural Newor (heory applicaion), Sysem Idenificaion, Adapive Precive Conrol. He is also a senior member of Insrumenaion & Conrol Sociey of Malaysia (ICSM). 80