Parallel layer perceptron

Transcription

1 Neurocomputing 55 (2003) Letters Parallel layer perceptron Walmir M. Caminhas, Douglas A.G. Vieira, João A. Vasconcelos Department of Electrical Engineering, Federal University of Minas Gerais, Belo Horizonte , MG, Brazil Abstract In this paper, both the architecture and learning procedure underlying the parallel layer perceptron is presented. This topology, dierent to the previous ones, uses parallel layers of perceptrons to map nonlinear input output relationships. Comparisons between the parallel layer perceptron, multi-layer perceptron and ANFIS are included and show the eectiveness of the proposed topology. c 2003 Elsevier B.V. All rights reserved. Keywords: ANFIS; Multi-layer perceptron; Parallel layer perceptron 1. Introduction In the last years the multi-layer perceptrons (MLP) gained popularity in a vast range of applications due to its universal approximation characteristic [1,2] and the popularization of the backpropagation algorithm [7]. Another popular network is the adaptive-network-based fuzzyinference system (ANFIS) [4]. This network has some advantage if compared with MLPs due to the linear dependence of its output in relation to the consequents; therefore, a more ecient learning algorithm can be used. However, the complexityof the ANFIS topologyincreases exponentiallyas the rules are generated using all possible combinations of premisses. The number of generated rules N for a system with n inputs and P premisses is N = P n, hence it is prohibitive to use an ANFIS for problems with several variables. In this paper is proposed a novel network, called parallel layer perceptron (PLP), which tries to combine the advantages of both MLP and ANFIS topologies. Moreover, as the learning for most problems needs a big Corresponding author. Tel.: ; fax: address: caminhas@cpdee.ufmg.br (W.M. Caminhas) /$ - see front matter c 2003 Elsevier B.V. All rights reserved. doi: /s (03)

2 772 W.M. Caminhas et al. / Neurocomputing 55 (2003) computational eort, training neural networks can be viewed as a natural application of parallelism. The topologyproposed here is a natural extension of articial neural networks to parallel environments. Several aspects of the PLP will be treated in this paper. Firstly, the architecture of the proposed topologyis presented. Also included is a particular case which the error surface for half of its parameters is a quadratic curve. Afterwards a learning procedure is discussed. Subsequently, the universal approximation theorem is presented. Lastly, some computational results are presented and discussed. 2. PLP architecture The output y t of the PLP considering n inputs, m perceptrons per layer is calculated as m y t = [(a jt )(b jt )] ; (1) j=1 where (:); (:) and (:) are activation functions (hyperbolic tangent, Gaussian, linear, etc.), a jt = n i=0 p jix it ; b jt = n i=0 v jix it ; v ji and p ji are components of the weight matrices P and V, x it is the ith input for the tth sample, where x 0t is the perceptron bias, and y t is the tth position of the vector output y. In Fig. 1 is shown the PLP topology. Similarly to the traditional multi-layer perceptron, all network parameters can be adapted using the backpropagation method. However, some dierences must be highlighted. Firstly, in the MLP case, the network uses function of functions to Fig. 1. PLP architecture.

3 W.M. Caminhas et al. / Neurocomputing 55 (2003) input output mapping. The PLP is mainlybased on product of functions. Moreover, as can be seen in Fig. 1, the proposed topologyis composed of parallel layers. This feature simplies the network implementation in parallel machines or clusters. One particular case of the topologyshowed in Fig. 1 is considering both (:) and (:) as identityfunctions. In this case, the network output is computed as m m y t = [a jt (b jt )] = [L j N j ]: (2) j=1 j=1 It is important to note that, the particular case described in Eq. (2), has some desirable characteristics. The error surface in relation to p ji, which for this particular case is a linear parameter, is a quadratical structure, hence a more ecient learning algorithm can be used. A hybrid learning that combines backpropagation and the least-squares estimate (LSE) is used to adapt the network parameters. In the next sections, just the network given byeq. (2) will be considered. Firstly, the learning algorithm for the proposed network is presented. It is important to remember that the learning for the most general case is similar to the traditional backpropagation, thus this discussion is not covered in this paper. 3. Hybrid learning algorithm When the particular network presented in Eq. (2) is employed, a method that uses a combination of gradient and LSE is more interesting. As the output y t is a linear function of the parameters p ji, their optimum values can be calculated using simple algorithm based on linear algebra. This feature is similar to the consequent characteristics for ANFIS. To simplifythe explanation about the LSE let l k = p ji, where k = n(j 1) + i, which is simplythe transformation of the matrix P to a vector l with the same components. First of all, the outputs of the nonlinear perceptrons are calculated. One matrix C, which is a combination of the nonlinear output and the inputs is generated. The components c tk of the matrix C are, c tk = x it (b jt ). The equivalence of Eq. (2) in matrix notation is y = Cl. The aim of the learning process for most supervised learning algorithms for a neural network is minimizing the sum of the squared error for the training data, that is e = 1 2 (Cl yd)t (Cl yd); (3) where yd is the desired output vector. The optimum value for l is obtained using the condition that the gradient of Eq. (3) is zero. So, l =(C T C) 1 C T yd: (4) After the evaluation of the l, its components return to the matrix form P. The output is calculated and the nonlinear weights, matrix V, are adapted according to the classical backpropagation method, v ji (iter +1)=v ji (iter) 9e=9v ji, where is the learning rate, which is calculated as described byjang [4], iter is the iteration number.

4 774 W.M. Caminhas et al. / Neurocomputing 55 (2003) PLP as a universal function approximation One of the most important features that are desired in this kind of network, is the possibilityof mapping a nonlinear input output model. Consider the universal approximation theorem stated as follows [1,2]: Let (:) be a nonconstant, bounded and monotone-increasing continuous function. Let Ip denote the p-dimensional unit hypercube. The space of continuous functions on Ip is denoted by C(Ip). Then, given any function and 0 there exists an integer M and sets of real constants i ; w ij and b i, where i =1;:::;M and j =1;:::;p such that we dene ( M p ) F(x 1 ;:::;x p )= i w ij x j + b i i=1 k=1 (5) as an approximate realization of the function f(:); that is F(x 1 ;:::;x p ) f(x 1 ;:::;x p ) for all {x 1 ;:::;x p } in I p. Byinspection it can be seen that Eq. (2) has the same form as Eq. (5). As the logistic and tanh functions are nonconstant, bounded and monotone increasing, both can be used in the nonlinear layer to satisfythe conditions imposed bythe theorem. For radial basis functions, the approximation theorem can be stated as in [6]. A graphical interpretation can be given to the PLP approximation property. In fact, this network can be understood as a linear combination of nonlinear functions or vice versa. One network with one pair of parallel layers, and two perceptrons per layer, m = 2, is capable to approximate two periods of a sin function. In Fig. 2, is shown the linear functions generated bythe linear perceptron. In Fig. 3, is shown the generated nonlinear functions considering as a Gaussian function. The linear nonlinear product is shown in Fig. 4 and, lastly, in Fig. 5 is shown the resulted approximation. Fig. 2. Linear parameters.

5 W.M. Caminhas et al. / Neurocomputing 55 (2003) Fig. 3. Nonlinear parameters. Fig. 4. Product of linear and nonlinear parameters. 5. Numerical problems Fig. 5. Approximated function. Two test problems were used to compare the PLP computational performance. The comparisons were done with ANFIS [4] and MLP trained with the Levenberg

6 776 W.M. Caminhas et al. / Neurocomputing 55 (2003) Marquardt algorithm [3]. All networks were trained during 50 epochs. The PLP activation function,, used in the examples was the Gaussian function. The rst test problem is a mapping of a nonlinear input output model. One sinc function with two variables ranging between [ 10; 10] [ 10; 10], was used. The training data were composed by121 sampled points, equallyspaced. The results for this problem are shown in Table 1. The numbers in front of the PLP topologyrepresent the perceptrons and parallel layers numbers (2 m); the number of perceptrons in the hidden layer for MLP, considering one hidden layer; the number of rules for the ANFIS topology. The second example is the chaotic Mackey Glass dierential delay equation [5], ẋ =(0:2x(t ))=(1 + x 10 (t )) 0:1x(t), which is a benchmark largelyused in the neural and fuzzycommunities [4]. This series was solved using fourth-order Runge Kutta method assuming x(0)=1:2; t=17 and x(t)=0 for t 0. From the Mackey Glass series was extracted 1000 pairs of input output data for t = The rst 500 pairs were used for training and the others 500 to validating the model. The results are presented in Table 2. Columns three and four represent the root mean squared error for training (Trn) and validation data (Val), respectively. Table 1 Sinc results TopologyTime RMSE PLP (2 5) : MLP (10) PLP (2 16) : MLP (32) : ANFIS (16) PLP (2 25) : MLP (50) : ANFIS (25) Table 2 Mackey Glass series results TopologyTime Trn (10 3 ) Val (10 3 ) PLP (2 16) ANFIS (16) MLP (32) PLP (2 20) MLP (40) PLP (2 30) MLP (60)

7 W.M. Caminhas et al. / Neurocomputing 55 (2003) Comparing the simulations presented in Table 1, for equivalent networks (equivalence was considered here as the total number of functions used in each network), the PLP topologywas faster and presented smaller errors, showing its eciency. For instance, PLP (2 5) outperform, it is better in both training time and error than MLP(10). The PLP (2 16) outperform the MLP(32) and ANFIS(16). This same analysis can be done for the others network showing the advantage of the proposed network. The same analysis done in the previous example, can be done for the second test problem (Table 2). In this example, the PLP outperform both ANFIS and MLP. What is more, for ANFIS the spatial exponential problem occurs in this example. Even though the validation error does not depend onlyof the topology, the PLP networks presented good results for unknown samples, as shown in the fourth column of Table Conclusions In this work, a novel neural network topologywas proposed, PLP, including the architecture and learning algorithms. One particular case was also proposed due to its special characteristics that enable a more ecient training algorithm. This network can be used in a wide varietyof applications, due to its universal function approximation characteristic. The numerical results presented in this paper show the computational eciencyof the proposed topology. Although its parallel characteristics were not deeply approached in this paper, it is an important feature of PLP networks that seems to be an open area for further researches. Acknowledgements The authors would like to thank CNPq, CAPES, FINEP and FAPEMIG for the nancial support. References [1] G. Cybenko, Approximation by superpositions of a Sigmoid function, Math. Control Signals Systems 2 (1989) [2] K. Funahashi, On the approximate realization of continuous mappings byneural networks, Neural Networks Signals Systems 2 (1989) [3] M.T. Hangan, M.B. Menhaj, Training feedforward networks with the Marquardt algorithm, IEEE Trans. Neural Networks 5 (6) (1994) [4] J.S.R. Jang, ANFIS: adaptative-network-based fuzzy inference system, IEEE Trans. Systems Man Cybernet. 23 (3) (1993) [5] M.C. Mackey, L. Glass, Oscillation and chaos in physiological control systems, Science 197 (1977) [6] J. Park, I.W. Sandberg, Universal approximation using radial-basis-function networks, Neural Comput. 3 (1991)

8 778 W.M. Caminhas et al. / Neurocomputing 55 (2003) [7] D.E. Rumelhart, G.E. Hinton, R.J. Williams, Learning internal representations byerror propagation, in: D.E. Rumelhart, J.L. McClelland (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1, Bradford Books, MIT Press, Cambridge, MA, 1986 (Chapter 8). Walmir Matos Caminhas is an Adjunct Professor at Department of Electrical Engineering at Federal Universityof Minas Gerais, Brazil. He holds a Doctarate degree in Electrical Engineering obtained from Universityof Campinas, Sao Paulo, Brazil, in His research interests include computational intelligence and control of electrical drives. Douglas Alexandre Gomes Vieira was born in Belo Horizonte, Brazil, in He is a student of electrical engineering course at Federal Universityof Minas Gerais, Brazil. His research interests include computational intelligence, multi-objective optimization and design, and stochastic and deterministic optimization methods. João Antônio de Vasconcelos was born in Monte Carmelo, Brazil. He obtained his Ph.D. in 1984 at Ecole Centrale de Lyon France. He is Professor at Electrical Engineering Department of the Federal Universityof Minas Gerais since His research interests include vector optimization (evolutionarymulti-objective optimization) and design, computational intelligence, computational electromagnetics (nite element methods, boundaryintegral equation methods and others).