Evolutionary Product-Unit Neural Networks for Classification 1

Transcription

1 Evoutionary Product-Unit Neura Networs for Cassification F.. Martínez-Estudio, C. Hervás-Martínez, P. A. Gutiérrez Peña A. C. Martínez-Estudio and S. Ventura-Soto Department of Management and Quantitative Methods, ETEA, Escritor Castia Aguayo 4, 4005, Córdoba, Spain (corresponding author, phone ; fax ; Department of Computing and Numerica Anaysis of the University of Córdoba, Campus de Rabanaes, 407, Córdoba, Spain Abstract. We propose a cassification method based on a specia cass of feedforward neura networ, namey product-unit neura networs. They are based on mutipicative nodes instead of additive ones, where the noninear basis functions express the possibe strong interactions between variabes. We appy an evoutionary agorithm to determine the basic structure of the product-unit mode and to estimate the coefficients of the mode. We use softmax transformation as the decision rue and the cross-entropy error function because of its probabiistic interpretation. The empirica resuts over four benchmar data sets show that the proposed mode is very promising in terms of cassification accuracy and the compexity of the cassifier, yieding a state-ofthe-art performance. Keywords: Cassification; Product-Unit; Evoutionary Neura Networs Introduction The simpest method for cassification provides the cass eve given its observation via inear functions in the predictor variabes. Frequenty, in a rea-probem of cassification, we cannot mae the stringent assumption of additive and purey inear effects of the variabes. A traditiona technique to overcome these difficuties is augmenting/repacing the input vector with new variabes, the basis functions, which are transformations of the input variabes, and then to using inear modes in this new space of derived input features. Once the number and the structure of the basis functions have been determined, the modes are inear in these new variabes and the fitting is a we nown standard procedure. Methods ie sigmoida feed-forward neura networs, proection pursuit earning, generaized additive modes [], and This wor has been financed in part by TIN C05-0 proects of the Spanish Inter- Ministeria Commission of Science and Technoogy (MICYT) and FEDER funds.

2 PoyMARS [], a hybrid of mutivariate adaptive spines (MARS) specificay designed for cassification probems, can be seen as different basis function modes. The maor drawbac of these approaches is to state the optima number and the typoogy of corresponding basis functions. We tace this probem proposing a noninear mode aong with an evoutionary agorithm that finds the optima structure of the mode and estimates the corresponding parameters. Concretey, our approach tries to overcome the noninear effects of variabes by means of a mode based on noninear basis functions constructed with the product of the inputs raised to arbitrary powers. These basis functions express the possibe strong interactions between the variabes, where the exponents may even tae on rea vaues and are suitabe for automatic adustment. The proposed mode corresponds to a specia cass of feedforward neura networ, namey product-unit neura networs, PUNN, introduced by Durbin and Rumehart [3]. They are an aternative to sigmoida neura networs and are based on mutipicative nodes instead of additive ones. Up to now, PUNN have been used mainy to sove regression probems [4], [5]. Evoutionary artificia neura networs (EANNs) have been a ey research area in the past decade providing a better patform for optimizing both the weights and the architecture of the networ simutaneousy. The probem of finding a suitabe architecture and the corresponding weights of the networ is a very compex tas (for a very interesting review on this subect the reader can consut [6]). This probem, together with the compexity of the error surface associated with a product-unit neura networ, ustifies the use of an evoutionary agorithm to design the structure and training of the weights. The evoutionary process determines the number of basis functions of the mode, associated coefficients and corresponding exponents. In our evoutionary agorithm we encourage parsimony in evoved networs by attempting different mutations sequentiay. Our experimenta resuts show that evoving parsimonious networs by sequentiay appying different mutations is an aternative to the use of a reguarization term in the fitness function to penaize arge networs. We use the softmax activation function and the cross-entropy error function. From a statistica point of view, the approach can be seen as a noninear mutinomia ogistic regression, where we optimize the og-ieihood using evoutionary computation. Reay, we attempt to estimate conditiona cass probabiities using a mutiogistic mode, where the noninear mode is given by a product-unit neura networ. We evauate the performance of our methodoogy on four data sets taen from the UCI repository [7]. Empirica resuts show that the proposed method performs we compared to severa earning cassification techniques. This paper is organized as foows: Section is dedicated to a description of product-unit based neura networs; Section 3 describes the evoution of product-unit neura networs; Section 4 expains the experiments carried out; and finay, Section 5 shows the concusions of our wor. Product-Unit Neura Networs In this section we present the famiy of product-unit basis functions used in the cassification process and its representation by means of a neura networ structure. This cass of mutipicative neura networs comprises such types as sigma-pi

3 networs and product unit networs. A mutipicative node is given by y i= wi i = x, where is the number of the inputs. If the exponents are {0,} we obtain a higherorder unit, aso nown by the name of sigma-pi unit. In contrast to the sigma-pi unit, in the product-unit the exponents are not fixed and may even tae rea vaues. Some advantages of product-unit based neura networs are increased information capacity and the abiity to form higher-order combinations of the inputs. Besides that, it is possibe to obtain upper bounds of the VC dimension of product-unit neura networs simiar to those obtained for sigmoida neura networs [8]. Moreover, it is a straightforward consequence of the Stone-Weierstrass Theorem to prove that productunit neura networs are universa approximators, (observe that poynomia functions in severa variabes are a subset of product-unit modes). Despite these advantages, product-unit based networs have a maor drawbac: they have more oca minima and more probabiity of becoming trapped in them [9]. The main reason for this difficuty is that sma changes in the exponents can cause arge changes in the tota error surface and therefore their training is more difficut than the training of standard sigmoida based networs. Severa efforts have been made to carry out earning methods for product units [9],[0]. Studies carried out on PUNNs have not taced the probem of the simutaneousy design of the structure and weights in this ind of neura networ, either using cassic or evoutionary based methods. Moreover, so far, product units have been appied mainy to sove regression probems. We consider a product-unit neura networ with the foowing structure (Fig. ): an input ayer with nodes, a node for every input variabe, a hidden ayer with m nodes and an output ayer with nodes, one for each cass eve. There are no connections between the nodes of a ayer and none between the input and output ayers either. The activation function of the -th node in the hidden ayer is given wi byb ( x, w ) = xi, where w i is the weight of the connection between input node i= i and hidden node and w = ( w,..., w ) the weights vector. The activation function of each output node is given by f ( x, θ) = β0 + βb ( x, w ), =,,...,, where β is the weight of the connection between the hidden node and the output node. The transfer function of a hidden and output nodes is the identity function. We consider the softmax activation function given by: = ( x θ ) ( x θ ) m = exp f, g ( x, θ ) =, =,,...,. exp f, where θ = ( β, w,..., w m ), θ = ( θ,..., θ ) and β = ( β 0, β,..., β m ). It interesting to note that the mode can be regarded as the feed-forward computation of a three-ayer t neura networ where the activation function of each hidden node is exp( t) = e and where we have to do a ogarithmic transformation of the input variabes x i, []. ()

4 g() x g() x () gx Softmax Softmax Softmax,,..., bias β 0 β0 β 0 β β β β β β β m β m β m,,...,m w w w w w m w m x x w w x w m,,..., Fig.. Mode of a product-unit based neura networ. 3 Cassification probem In a cassification probem, measurements x i, i =,,...,, are taen on a singe individua (or obect), and the individuas have to be cassified into one of the D = ( x, y ); n =,,..., N is casses based on these measurements. A training sampe { } avaiabe, where x n = ( x n,..., xn ) is the random vector of measurements taing vaues in Ω R, and y n is the cass eve of the n -th individua. We adopt the common technique of representing the cass eves using a -of- encoding vector () () ( ) ( ) y = y, y..., y, such as y = if x corresponds to an exampe beonging to ( ) cass ; otherwise, ( ) y = function C : {,,..., } n n 0. Based on the training sampe we try to find a decision Ω for cassifying the individuas. A miscassification occurs when the decision rue C assigns an individua to a cass, when it actuay comes from a cass N n =. We define the corrected cassified rate by N CCR = I( C( xn ) = y n ), where Ii ( ) is the zero-one oss function. A good cassifier tries to achieve the highest possibe CCR in a given probem. We define the cross-entropy error function for the training observations as: N N ( ) ( ) ( ) y og g (, ) y f ( ) og exp f θ = n xn θ = n n ( n ) N n= = N x,θ + x,θ () n= = =

5 The Hessian matrix of the error function ( θ ) is, in genera, indefinite and the error surface associated with the mode is very convouted with numerous oca optimums. Moreover, the optima number of basis functions of the mode (i.e. the number of hidden nodes in the neura networ) is unnown. Thus, the estimation of the vector parameters ˆθ is carried out by means an evoutionary agorithm (see Section 4). The optimum rue C( x ) is the foowing: C( x ) = ˆ, where ˆ = arg max g ( x, θ ˆ), for =,...,. Observe that softmax transformation produces positive estimates that sum to one and therefore the outputs can be interpreted as the conditiona probabiity of cass membership and the cassification rue coincides with the optima Bayes rue. On the other hand, the probabiity for one of the casses does not need to be estimated, because of the normaization condition. Usuay, one activation function is set to zero and we reduce the number of parameters to estimate. Therefore, we set f ( x,θ ) = 0. 4 Evoutionary Product-Unit Neura Networs In this paragraph we carry out the evoutionary product-unit neura networs agorithm (EPUNN) to estimate the parameter that minimizes the cross-entropy error function. We buid an evoutionary agorithm to design the structure and earn the weights of the networs. The search begins with an initia popuation of product-unit neura networs, and, in each iteration, the popuation is updated using a popuationupdate agorithm. The popuation is subected to the operations of repication and mutation. Crossover is not used due to its potentia disadvantages in evoving artificia networs. With these features the agorithm fas into the cass of evoutionary programming.the genera structure of the EA is the foowing: () Generate a random popuation of size N p. () Repeat unti the stopping criterion is fufied (a) Cacuate the fitness of every individua in the popuation. (b) Ran the individuas with respect to their fitness. (c) The best individua is copied into the new popuation. (d) The best 0% of popuation individuas are repicated and substitute the worst 0% of individuas. Over that intermediate popuation we: (e) Appy parametric mutation to the best 0% of individuas. (f) Appy structura mutation to the remaining 90% of individuas. We consider ( θ ) being the error function of an individua g of the popuation. Observe that g can be seen as the mutivauated function ( ) = ( ) ( ) g g g x, θ ( x, θ,..., x, θ ). The fitness measure is a stricty decreasing transformation of the error function ( ) mutation is accompished for each coefficient θ given by Ag ( ) ( ( )) w i, = + θ. Parametric β of the mode with Gaussian noise: wi ( t + ) = wi ( t) + ξ( t), β ( t + ) = β ( t) + ξ ( t), where ξ ( t) N(0, α ( t)),

6 =,, represents a one-dimensiona normay-distributed random variabe with mean 0 and variance α ( t ), where α( t) < α ( t), and t is the t -th generation. Once the mutation is performed, the fitness of the individua is recacuated and the usua simuated anneaing is appied. Thus, if A is the difference in the fitness function after and preceding the random step, the criterion is: if A 0 the step is accepted, if A < 0, the step is accepted with a probabiity exp( A/ T ( g)), where the temperature T ( g ) of an individua g is given by T ( g) = A( g), 0 T ( g) <. The variance α ( t ) is updated throughout the evoution of the agorithm. There are different methods to update the variance. We use the /5 success rue of Rechenberg, one of the simpest methods. This rue states that the ratio of successfu mutations shoud be /5. Therefore, if the ratio of successfu mutations is arger than /5, the mutation deviation shoud increase; otherwise, the deviation shoud decrease. Thus: ( + λ) α ( t) if sg > / 5 α ( t + s) = ( λ) α ( t), if sg < / 5 α ( t) if sg = / 5 (3) where =,, s g is the frequency of successfu mutations over s generations and λ = 0.. The adaptation tries to avoid being trapped in oca minima and aso to speed up the evoutionary process when searching conditions are suitabe. Structura mutation impies a modification in the neura networ structure and aows exporations of different regions in the search space whie heping to eep up the diversity of the popuation. There are five different structura mutations: node deetion, connection deetion, node addition, connection addition and node fusion. These five mutations are appied sequentiay to each networ. In the node fusion, two randomy seected hidden nodes, a and b, are repaced by a new node, c, which is a combination of both. The connections that are common to both nodes are ept, with weights given by: βc = βa + βb, wc = ( wa + wb ). The connections that are not shared by the nodes are inherited by c with a probabiity of 0.5 and its weight is unchanged. In our agorithm, node or connection deetion and node fusion is aways attempted before addition. If a deetion or fusion mutation is successfu, no other mutation wi be made. If the probabiity does not seect any mutation, one of the mutations is chosen at random and appied to the networ. 5 Experiments The parameters used in the evoutionary agorithm are common for the four probems. We have considered α (0) = 0.5, α (0) =, λ = 0. and s = 5. The exponents w i are initiaized in the [ 5,5] interva, the coefficients β are initiaized in [ 5,5]. The

7 maximum number of hidden nodes is m = 6. The size of the popuation is N = 000. The number of nodes that can be added or removed in a structura P mutation is within the [, ] interva. The number of connections that can be added or removed in a structura mutation is within the [,6 ] interva. The stop criterion is reached if the foowing condition is fufied: for 0 generations there is no improvement either in the average performance of the best 0% of the popuation or in the fitness of the best individua. We have done a simpe inear rescaing of the,, being X the transformed variabes. input variabes in the interva [ ] We evauate the performance of our method on four data sets taen from the UCI repository [7]. For every dataset we performed ten runs of ten-fod stratified crossvaidation. This gives a hundred data points for each dataset, from which the average cassification accuracy and standard deviation is cacuated. Tabe shows the statistica resuts over 0 runs for each fod of the evoutionary agorithm for the four data sets. With the obective of presenting an empirica evauation of the performance of the EPUNN method, we compare our approach to the most recent resuts [] obtained using different methodoogies (see Tabe ). Logistic mode tree, LMT, to ogistic regression (with attribute seection, SLogistic, and for a fu ogistic mode, MLogistic); induction trees (C4.5 and CART); two ogistic tree agorithms: LTreeLog and finay, mutipe-tree modes M5 for cassification, and boosted C4.5 trees using AdaBoost.M with 0 and 00 boosting interactions. We can see that the resuts obtained by EPUNN, with architectures (3::), (34:3:), (4:5:3) and (5:3:) for each data set, are competitive with the earning schemes mentioned previousy. Tabe. Statistica resuts of training and testing for 30 executions of EPUNN mode. CCR T CCR #connect G Data set #node Mean SD Best Worst Mean SD Best Worst Mean SD Heart-stat Ionosphere Baance Austraian Tabe. Mean cassification accuracy and standard deviation for: LMT, SLogistic, MLogistic, C4.5, CART, NBTree, LTreeLin, LTreeLog, M5', ABOOST and EPUNN method. Data set LMT SLogistic MLogistic C4.5 CART NBTree Heart-stat 83.± ± ± ± ± ±7. Ionosphere 9.99± ± ± ± ± ±5. Baance 89.7± ± ± ± ± ±5.3 Austraian 85.04± ± ± ± ± ±4.03 Data set LTreeLin LTreeLog M5' ABoost(0) ABoost(00) EPUNN W/L Heart-stat 83.5± ± ± ± ± ±6.90 5/6 Ionosphere 88.95± ± ± ± ± ±5.5 7/4 Baance 9.86± ± ± ± ± ±.36 /0 Austraian 84.99± ± ± ± ± ±3.90 0/ * i

8 6 Concusions We propose a cassification method that combines a noninear mode, based on a specia cass of feed-forward neura networ, namey product-unit neura networs, and an evoutionary agorithm that finds the optima structure of the mode and estimates the corresponding parameters. Up to now, the studies on product units have been appied mainy to sove regression probems and have not addressed the probem of the design of both structure and weights simutaneousy in this ind of neura networ, either using cassic or evoutionary based methods. Our approach uses softmax transformation and the cross-entropy error function. From a statistica point of view, the approach can be seen as noninear mutinomia ogistic regression, where optimization of the og-ieihood is made by using evoutionary computation. The empirica resuts show that the evoutionary product-unit mode performs we compared to other earning cassification techniques. We obtain very promising resuts in terms of cassification accuracy and the compexity of the cassifier. References [] T.. Hastie and R.. Tibshirani, Generaized Additive Modes. London: Chapman & Ha, 990. [] C. Kooperberg, S. Bose, and C.. Stone, "Poychotomous Regression," ourna of the American Statistica Association, vo. 9, pp. 7-7, 997. [3] R. Durbin and D. Rumehart, "Products Units: A computationay powerfu and bioogicay pausibe extension to bacpropagation networs," Neura Computation, vo., pp. 33-4, 989. [4] A. C. Martínez-Estudio, F.. Martínez-Estudio, C. Hervás-Martínez, et a., "Evoutionary Product Unit based Neura Networs for Regression," Neura Networs, pp , 006. [5] A. C. Martínez-Estudio, C. Hervás-Martínez, A. C. Martínez-Estudio, et a., "Hybridation of evoutionary agorithms and oca search by means of a custering method," IEEE Transactions on Systems, Man and Cybernetics, Part. B: Cybernetics, vo. 36, pp , 006. [6] X. Yao, "Evoving artificia neura networ," Proceedings of the IEEE, vo. 9 (87), pp , 999. [7] C. Bae and C.. Merz, " UCI repository of machine earning data bases," mearn/mlrepository.thm, 998. [8] M. Schmitt, "On the Compexity of Computing and Learning with Mutipicative Neura Networs," Neura Computation, vo. 4, pp. 4-30, 00. [9] A. Ismai and A. P. Engebrecht, "Goba optimization agorithms for training product units neura networs," presented at Internationa oint Conference on Neura Networs ICNN`000, Como, Itay, 000. [0] A. P. Engebrecht and A. Ismai, "Training product unit neura networs," Stabiity and Contro: Theory and Appications, vo., pp , 999. [] K. Saito and R. Naano, "Extracting Regression Rues From Neura Networs," Neura Networs, vo. 5, pp , 00. [] N. Landwehr, M. Ha, and F. Eibe, "Logistic Mode Trees," Machine Learning, vo. 59, pp. 6-05, 005.