International Journal "Information Technologies & Knowledge" Vol.5, Number 1,

Transcription

1 Internationa Journa "Information Tecnoogies & Knowedge" Vo.5, Number, 0 5 EVOLVING CASCADE NEURAL NETWORK BASED ON MULTIDIMESNIONAL EPANECHNIKOV S KERNELS AND ITS LEARNING ALGORITHM Yevgeniy Bodyanskiy, Pau Grimm, Nataiya Tesenko Abstract: At present time neura networks based on Group Metod of Data Handing (GMDH-NN), nodes of wic are two-input N-Adaines, is we-known. Eac of N-Adaines contains te set of adjustabe synaptic weigts tat are estimated using standard east squares metod and provides quadratic approximation of restoring noninear mapping. On te oter and, for needed approximation quaity ensuring tis NN can require considerabe number of idden ayers. Approximating properties of GMDH-NN can be improved by uniting te approaces based on Group Metod of Data Handing and Radia-Basis-Functions Networks tat ave ony one idden ayer, formed by, so-caed, R-neurons. Suc networks earning reduces, as a rue, to te tuning of synaptic weigts of output ayer tat are formed by adaptive inear associators. In contrast to neurons of mutiayer structures wit poynomia or sigmoida activation functions R-neurons ave be-saped activation functions. In tis paper as activation functions mutidimensiona Epanecnikov s kernes are used. Te advantage of activation function is tat its derivatives are inear according a te parameters tat aows to adjust sufficienty simpy not ony synaptic weigts but aso centers wit receptive fieds. Proposed network combines Group Metod of Data Handing, Radia-Basis-Functions Networks and cascade networks and isn t incined to te curse of dimensionaity, is abe to rea time mode information processing by adapting its parameters and structure to probem conditions. Te mutidimensiona Epanecnikov s kernes were used as activation functions, tat aowed to introduce numericay simpe earning agoritms, wic are caracterized by ig speed. Keywords: evoving neura network, cascade networks, radia-basis neura network, Group Metod of Data Handing, mutidimensiona Epanecnikov s kernes. ACM Cassification Keywords: F. Computation by abstract devices Sef-modifying macines (e.g., neura networks), I..6 Learning Connectionism and neura nets, G.. Approximation Noninear approximation. Introduction At present artificia neura networks ave gotten a wide spread for extensive cass of pattern recognition, identification, emuation, inteigent contro, time series prediction etc. probems due to universa approximating properties and abiities to earn. As far as wen a number of practica tasks soving te voume of earning sampe is imited, ten to te foreground earning rate factor comes. At te same time not a te neura networks are abe to overcome arising probems and, first of a, so-caed overfitting. As one of te most ig-performance networks tat are earned based on optimization procedures of second order wit ig convergence rate is Radia Basis Functions Neura Network (RBFN). Output signa of tis network ineary depends on adjusting synaptic weigts. At te same time tese networks are incined to so-caed curse of dimensionaity, wen te number of radia-basis neurons of idden ayer (R-neurons) exponentiay grows wie input signas state grows. It is possibe to overcome tis probem by dividing te initia task in one or anoter way to a number of subtasks of ow dimensionaity and grouping obtained soutions to get required resut. From computationa point of view te

2 6 Internationa Journa "Information Tecnoogies & Knowedge" Vo.5, Number, 0 most convenient in tis case is Group Metod of Data Handing (GMDH) [Ivaknenko, 969; Ivaknenko, 975; Ivaknenko, Stepasko, 975; Ivaknenko, Madaa, 994] tat demonstrated its efficiency wen soving a great number of practica tasks. In [Pam, Liu, 995] muti-ayered GMDH-neura network was considered. It as two-inputs N-adaines as a nodes and output of eac node is quadratic function of input signas. At te same time te synaptic weigts of eac neuron are defined in patc mode using standard east squares metod. It can be needed some more quantity of idden ayers to provide necessary approximation quaity. Tat is wy on-ine earning becomes impossibe. Hybrid arcitecture of artificia neura network based on ideas of GMDH and consequent forming of cascade neura networks [Avedyan, Barkan, Levin, 999]. Nodes of tis network are compartmenta R-neurons wit activation functions ike mutidimensiona Epanecnikov s kernes [Epanecnikov, 969; Friedman, Hastie, Tibsirani, 003; Bodyanskiy, Capanov, Koodyazniy, Otto, 00], tat ave arge degree of freedom, and tus improved approximating properties in comparison wit conventiona Gaussians. Compartmenta R-neuron wit Mutidimensiona Epanecnikov Kernes and Its Learning Agoritm Let introduce te structure of compartmenta R-neuron presented on fig. and concurring wit simpified arcitecture of conventiona Radia Basis Functions Neura Network wit two inputs x i and x, i, j =,,..., n, were n dimensionaity of input space. j w 0 xi ( k) ϕ ( x, c, Σ ) w xj ( k) ϕ ( x, c, Σ ) w Σ yˆ ( k) ϕ ( x, c, Σ ) p p p w p Fig. Compartmenta R-neuron Compartmenta R-neuron contains p activation functions (conventionay in RBFN mutidimensiona Gaussians or oter be-saped functions are used) ϕ ( x, c, Σ ), p+ synaptic weigts tat are united to vector 0 p w = ( w, w,..., w ), p two-dimensiona vector of centers c = ( i, j c c ), p ( ) matrices of receptive

3 Internationa Journa "Information Tecnoogies & Knowedge" Vo.5, Number, 0 7 fieds of activation functions Σ,two-dimensiona inputs vector x = ( x, x ), one output y ˆ ; =,,..., p ; k =,,..., N number of observation in processing sampe or index of current discrete time. Mutidimensiona Epanecnikov s kernes are used as activation functions ϕ ( x, c, Σ ) i j ϕ (,, Σ ) = x c x c, () ( Σ ) tat ave be-saped by positive definite matrix of receptive fied Σ. Te advantage of activation function () in comparison wit conventiona ones is in inearity of its derivatives wit respect to a te parameters tat aows to adjust not ony synaptic weigts but aso centers wit receptive fieds sufficienty easy. At te same time, transformation tat is reaized by compartmenta R-neura as a form p ˆ = 0 + ϕ (,, Σ ) y = 0 + w w x c w w x c ( Σ ). = = Usuay RBFN earning comes to synaptic weigts p w adjusting, but centers and caracteristics of receptive fieds are defined priory. At te same time for two-dimensiona case it is sufficienty easy to ocate te centers at te reguar attice nodes and define receptive fied as circes. Learning process itsef consists of synaptic weigts vector w estimating by earning sampe containing N observations x ( k) = ( xi( k), xj( k )), yk ( ), k =,,..., N, were yk ( ) - externa earning signa. By introducing to te consideration ( p + ) -vector of activation functions ϕ ( k) = (, ϕ ( x ( k), c, Σ ),..., ϕ ( x ( k), c, Σ )) and earning criterion p p p N N N N = ˆ ( ( ) ( )) = ( ) = ( ( ) ( ) ϕ ( )) k= k= k= E yk y k e k yk w k, () using standard east squares metod it is easy to obtain required soution in te form + were () symbo of inversion by Moore-Penrose. + N N w = ϕ ( k)( ϕ ( k)) ϕ ( kyk ) ( ), (3) k= k= If te data are fed to te processing consequenty in on-ine mode, ten instead of (3) can be used its recurrent variant in te form P ( k)( yk ( ) ( w ( k)) ϕ ( k)) w ( k) = w ( k ) + ϕ ( k), + ( ϕ ( k)) P ( k ) ϕ ( k) P ( k ) ϕ ( k)( ϕ ( k)) P ( k ) P ( k) = P ( k ), P ( 0) = γι, γ >> 0. + ( ϕ ( k)) P ( k ) ϕ ( k) Agoritms (3) and (4) are effective ony in te cases, wen required soution is stationary, tat is, optima vaues of synaptic weigts aren t variabe in time. Since in many practica tasks it is not so, for exampe, adaptive identification of non-stationary objects or non-stationary time series prediction, ten ig-performance adaptive earning agoritm aving tracking and fitering properties can be used [Bodyanskiy, Koodyazniy, Stepan, 00]: (4)

4 8 Internationa Journa "Information Tecnoogies & Knowedge" Vo.5, Number, 0 w = + η ϕ ϕ = ( k) w ( k ) w( k)( y( k) ( w ( k )) ( k)) ( k) = ( ) + w k ηw( ke ) ( k) ϕ ( k), ηw ( k) = rw( k) = αrw( k ) + ϕ ( k), 0 α, were α smooting parameter tat defines compromise between fitering and tracking properties. For te purpose of compartmenta R-neuron approximating properties improving not ony synaptic weigts but aso centers wit receptive fieds can be adjusted. At te same owing to Epanecnikov s kernes using earning agoritms ave sufficienty simpe form. By using gradient procedure of criterion () minimization and its rate optimization tecnique [Otto, Bodyanskiy, Koodyazniy, 003] we obtain compartmenta R-neuron earning agoritm in te form w ( k) = w ( k ) + ηw( k) e( k) ϕ ( k), ηw ( k) = rw( k) = αrw( k ) + ϕ ( k), c( k) = c( k ) + ηc( k) e( k) w ( k) ( Σ( k ) ) ( x ( k) c( k )) = = c( k ) + ηc( ke ) ( kg ) ( k), ηc ( k) = rc( k) = αrc( k ) + g( k), ( Σ ( k )) = ( Σ( k ) ) ησ( k) e( k) w ( k)( x ( k) c( k))( x ( k) c( k)) = = ( Σ( k) ) ησ( ke ) ( kg ) ( k), ησ ( k) =Γ Σ( k) = αγσ( k ) + Tr( G( k) G( k)). Evoving Cascade Neura Network Uniting of GMDH and cascade neura networks ideas eads to arcitecture presented on fig.. x x CR-N [] CR-N [] 3 CR-N [] [ ] CR-N [3] [ 3] x n CR-N [] SB yˆ Cn [ Cn ] ˆ y CR-N [ Cn ] [ Cn ] ˆ y Fig. Evoving cascade neura network Te first idden ayer of te network is formed simiary to te first idden ayer of GMDH neura network [Pam, Liu, 995] and contains te number of neurons equa to quantity of combinations of n in, tat is C n. Seection

5 Internationa Journa "Information Tecnoogies & Knowedge" Vo.5, Number, 0 9 bock SB executes sorting by accuracy, for exampe, in te sense of variations, of a output signas te most accurate signa is [ ]* ˆ C n y so tat, ten and te worst is y. Outputs of SB and ten are fed to te one neuron of te second ayer-cascade CR-N [] tat computes signa combined wit 3 [] ˆ [ ] wic in te tird cascade is. Te process of cascades increasing asts ti required accuracy obtaining, at te same time, maxima neurons number of tis network is restricted by te vaue C n. Tus, neura network is abe to process information tat are fed in rea time by readjusting bot its parameters and its arcitecture in time [Kasabov, 003] and by adapting to te conditions of te soving task. Concusion Arcitecture of evoving cascade radia-basis neura network was proposed in tis paper. It is formed based on te idea of combining GMDH and cascade networks. Aso tis network is not disposed to te curse of dimensionaity and is abe to process information in rea time by adapting its parameters and structure to te soving task conditions. Using of mutidimensiona Epanecnikov s kernes as activation functions aowed to introduce numericay simpe earning agoritms tat are caracterized by ig-performance. Acknowedgements Te paper is pubised wit financia support by te project ITHEA XXI of te Institute of Information Teories and Appications FOI ITHEA ( ) and te Association of Deveopers and Users of Inteigent Systems ADUIS Ukraine ( ). Bibiograpy [Ivaknenko, 969] Ivaknenko A.G. Sef-earning systems for recognition and automatic contro. Tecnika, Kiev, 969 (in Russian). [Ivaknenko, 975] Ivaknenko A.G. Long-term prediction and compex systems contro. Tecnika, Kiev, 975 (in Russian). [Ivaknenko, Stepasko, 975] Ivaknenko A.G., Stepasko V.S. Modeing noise-immunity. Naukova Dumka, Kiev, 975 (in Russian). [Ivaknenko, Madaa, 994] Ivaknenko A.G., Madaa H.R. Inductive Learning Agoritms for Compex Systems Modeing. CRC Press, London-Tokio,994. [Pam, Liu, 995] Pam D.J., Liu X. Neura Networks for Identification, Prediction and Contro. Springer-Verag, London, 995. [Avedyan, Barkan, Levin, 999] Avedyan E. D., Barkan G.V., Levin I.K. Cascade neura networks // Avtomatika i Teemekanika P (in Russian). [Epanecnikov, 969] Epanecnikov V.A. Non-parametric estimate of mutidimensiona probabiity density // Probabiity teory and its appication P (in Russian). [Friedman, Hastie, Tibsirani, 003] Friedman J., Hastie T., Tibsirani R. Te Eements of Statistica Learning. Data Mining, Inference and Prediction. Springer, Berin, 003. [Bodyanskiy, Capanov, Koodyazniy, Otto, 00] Bodyanskiy Ye., Capanov O., Koodyazniy V., Otto P. Adaptive quadratic radia basis function network for time series forecasting // Proc. East West Fuzzy Co. 00. Zittau/Goeritz: HS, 00. P.64-7.

6 30 Internationa Journa "Information Tecnoogies & Knowedge" Vo.5, Number, 0 [Bodyanskiy, Koodyazniy, Stepan, 00] Bodyanskiy Ye., Koodyazniy V., Stepan A. An adaptive earning agoritm for a neuro-fuzzy network / Ed. by B.Reusc Computationa Inteigence. Teory and Appications. Berin-Heideberg- New York: Springer, 00. P [Otto, Bodyanskiy, Koodyazniy, 003] Otto P., Bodyanskiy Ye., Koodyazniy V. A new earning agoritm for forecasting neuro-fuzzy network// Integrated Computer-Aided Engineering P [Kasabov, 003] Kasabov N. Evoving Connectionist Systems: Metods and Appications in Bioinformatics. Springer Verag, London, 003. Autors' Information Yevgeniy Bodyanskiy Professor, Dr.-Ing. abi. Scientific ead of Contro Systems Researc Laboratory, KNURE, Senior member of IEEE, professor of Artificia Inteigence Department of KNURE. e-mai: bodya@kture.karkov.ua Major Fieds of Scientific Researc: ybrid systems of computationa inteigence. Pau Grimm Professor, Computer Grapics Department, Erfurt University of Appied Sciences. e-mai: grimm@f-erfurt.de Major Fieds of Scientific Researc: image processing using inteigent computer tecnoogies. Nataiya Tesenko P.D., Senior researcer of Contro Systems Researc Laboratory, KNURE. e-mai: ntntp@ukr.net Major Fieds of Scientific Researc: sef-earning evoving neuro-fuzzy modes and systems in te inteigent data anaysis tasks