Identification of Non-Linear Systems, Based on Neural Networks, with Applications at Fuzzy Systems

Proceedings of the 0th WSEAS International Conference on AUTOMATION & INFORMATION Identification of Non-Linear Systems, Based on Neural Networks, with Applications at Fuzzy Systems CONSTANTIN VOLOSENCU Department of Automatics and Applied Informatics Politehnica University of Timisoara Bd. V. Parvan nr. 2, 300223 ROMANIA constantin.volosencu@aut.upt.ro http://www.aut.upt.ro/~cvolos Abstract: - The paper presents a short review how to use feedforward neural networks for non-linear system identification, with application at the neural implementation of a fuzzy system. In this application the inputoutput transfer characteristics of the fuzzy system are used to evaluate the accuracy of the identification results expressed for a neuro-fuzzy model. This method could be used for identification of the most general fuzzy systems, which are non-linear systems, being developed with all kind of fuzzyfication methods, rule bases, inference methods and defuzzification methods. Using this method, accurate neural models for a large class of fuzzy systems may be obtained. The neuro-fuzzy model preserves all the properties of the fuzzy systems: the values of the transfer gain and the sector property. The neuro-fuzzy model obtained by identification is useful in all applications of the fuzzy systems, for example in control. Good values of the quality neural identification criteria are obtained. The optimized neuro-fuzzy model is given by its structur weights and biases, related to the most adequate training method. Key-Words: - Neural networks, neural network training methods, fuzzy systems, empirical neural network training quality criteria, neuro-fuzzy systems. Introduction Neural networks represent a possible solution to model mathematic applications between real sets, to approximate non-linear real function with real variables. These real functions may be models for non-linear systems. To obtain these models the users are starting from an initial neural network and train it with a training method, using training set of inputoutput data of the original system, obtained from practical measurements made upon it. There are many neural network structures and training methods developed in the last decades, which may be used in practical applications. So, feedforward neural networks with continuous values are used with success for nonlinear system model identification. Many books and scientific papers present theories related to this application in the recent world literature. For exampl the book [] presents some approaches for the identification of nonlinear static and dynamics systems, with some elements of optimization techniques, based on classic methods, neural networks and fuzzy systems, for engineering applications. The paper [2] presents some problems related to the identification with neural networks of non-linear dynamic systems: identification of mass of a system based on acceleration recordings and the usage of recurrent neural networks for identification with a good robustness and small error. An application of on-line identification based on neural networks, using the Levenberg-Marquardt training method is presented in [3]. The paper [4] presents an application of neural identification for adaptive digital control of fast systems. An application of physical objects identification based on neural network is presented in [5]. Basic theory was presented in older publications. Introductory elements to the theory of neural network developing are presented in [6]. General surveys of identification and control of dynamical systems using neural networks are presented in [7, 8]. Feedforward neural networks with continuous values and hidden layers may be used to approximate nonlinear function with a desired error [9]. The fuzzy systems, defined using all kinds of fuzzification methods, membership functions [0], are strong non-linear systems. Their input-output transfer characteristics show this aspect []. To use neural networks in identification of fuzzy systems ISSN: 790-57 387 ISBN: 978-960-474-064-2

Proceedings of the 0th WSEAS International Conference on AUTOMATION & INFORMATION means to obtain a neuro-fuzzy system what must implement all the properties of these systems []. Until today they are some impediments in practical application of neural identification due to the inexistence of precise methods to obtain accurate neural models fro non-linear systems with good values of the empirical quality criteria, with a high confidence in the repeatability of the obtaining of the same results with the same training method and neural network structure. This paper presents some theoretical and practical considerations, in an unified and structured approach, related to the general problem of nonlinear system identification using neural networks, with application to the fuzzy system identification. The proposed method and quality criteria are based on a comparative study over different neural network structures, training method and fuzzy system types. It shows the possibility of learning of the fuzzy system model with a desired accuracy. In this case it is possible to obtain neural models of the fuzzy systems, which will have all the properties, and the characteristics of the fuzzy systems. These neural models may be use to implement fuzzy systems in more complex systems as for example the fuzzy control systems. 2 Reason and Objective of Modeling Fuzzy Systems with Neural Networks The reason of modeling fuzzy systems, with the general structure from Fig. [0, ], with feedforward neural networks, with the general structure from Fig 2 [6], is to determine an optimal neural structure for implementation of such a fuzzy system, the establishment of the complexity of the modeling process and the establishment of the synthesis elements of the neural network used for modeling. network development, by particularization in the case of fuzzy system. Fig. 2 The general structure of a neural network To solve the problem of determination of the structure of the adequate neural network it must respond to the following questions: What neural network type must be chosen for modeling of the fuzzy systems? What activation functions must be chosen? What training method must be used? How many layers must be in the neural network? How many units must be used on each layer? How many samples must be used in a training set? How many training epochs must be passed? If an optimum structure must be reached, from a point of view of the compromise between the learning precision, its dimension and the number of training epochs? What are the neural connection weights? What are the biases of each neuron? The structure of the neural model of the fuzzy system is given by: the layer number, the neuron number on each layer, the activation function type for each neuron, the connection weights between neurons and the biases of each neuron. So, a structure from Fig. 3 may be used. Fig.. The general structure of a fuzzy system This paper is based on a large work of the author and it presents synthetically the study cases and the principles that stay at the basis of modeling and the way of the result selection. In this paper some responds are given at specific questions for neural Fig. 3 The detailed neural network general structure The answers at the above questions are given related to the desired quality criteria for the neural networks. ISSN: 790-57 388 ISBN: 978-960-474-064-2

Proceedings of the 0th WSEAS International Conference on AUTOMATION & INFORMATION In the development of neural models two levels can be emphasized [8] (Fig. 3). possibility to obtain a neural network, which can predict new values, different from the training sets. The global design is recommended in applications with fix databases and the training sequences can be repeated many times. This is the case of the study presented in this paper. Fig. 3 Levels in neural identification At the lower level is placed the testing of training methods for neural networks. The intention is to build a model of a fuzzy system, which is a supervised learning model. This model is a system with supervised learning, which is learning a non-linear function or an application from a vector to another vector. In the learning process we comput based on the training error e=y d -y m, which is a function of inputs f Γ (u, u 2 ) and the gradient E. As a result of learning we obtain the connection weights w and the neuron biases b. The network learns to give the output y m based on the inputs u and u 2. At the second level is the neural identification the determination of the final network, which implements the fuzzy system. For this purpose the training method with the best characteristics must be used to obtain the optimum neural structur after an iterative training. In fact, a neural network is determined, which is capable to model a non-linear function y m =f BN (u, u 2 ) the most closed, with the smallest error, to the fuzzy system. The obtaining of the neural network is made with a direct neural identification, to assure the best training quality criteria. The network is used to give the estimated output of the fuzzy system based on the input values u and u 2. It implements a function of two variables f NN (u, u 2 ; w, b), which has the weights w and the network biases b as parameters. Each level is taken account of the other and it is working one with the other. The practical application is framed in the genre of applications in which the neural network knowledge is used in combination with other knowledge to build complex systems for specific applications. So, the fuzzy system is built on the fuzzy logic concepts. The fuzzy systems are nonlinear systems. And the neural networks become models for the fuzzy systems, which are non-linear systems. The study case is framed in the supervised learning. Multi-layer feedforward neural networks global designed are used, in which all the weights affect the output at a time moment [8]. The global learning is a slow learning method, but it gives the 3 Neural Identification Method 3. Principle Usage of neural networks assumes existence of a model for training. The neural network must implement this model. As application in this paper, a fuzzy system is used as a non-linear system model. To obtain the neural network, which implements the fuzzy systems, the direct identification method is used. The principle of direct neural identification is presented as follows, with reference to Fig. 4. Figure 4.The block diagram of the direct neural identification of the fuzzy system The neural network NN is receiving the same inputs u and u 2 as the fuzzy system FS. The output of the fuzzy system y d is the desired output on the training period. The purpose of identification is to find a neural network with a response y m identical with the output of the fuzzy block y d for a given set of inputs [u, u 2 ] T. In the time of identification process the vector norm yd y m is minimized through a number of adjustments of weights, using a learning technique (training). From this structure a first notice is that the number of neurons on the input layer must be equal to the number of inputs of the fuzzy system m=2. The training is done based on the error e=y d -y m between the desired output y d and the actual output of the neural network y m. Training sets as (u k, u 2k ; y dk ) are used for training, were k=,,p, where u k and u 2k are taking values on their universe of discourse. The number of neurons on the output layer of the neural network is n=, equal to the number of the output of the fuzzy system. If the fuzzy system implements a non-linear function, of two variables ISSN: 790-57 389 ISBN: 978-960-474-064-2

Proceedings of the 0th WSEAS International Conference on AUTOMATION & INFORMATION y d =f FS (u, u 2 ), the neural network will present a nonlinear function y m =f NN (u, u 2 ). In the case of direct neural identification the neural network is trained to copy the direct inputoutput transfer characteristic of the fuzzy system. Because the fuzzy system is a non-inertial system, characterized by the fact that its input and output variables have real continuous values, imposes the usage of a feedforward multiplayer neural network with continuous values [6]. The general structure of the multi-layer neural network is presented in Fig. 2 and 3, with m=2 and n=. In this cas because the training model is a fuzzy system, we may say after trainings we will obtain a neuro-fuzzy system. A 33-mm-g system [] was chosen for training. It has the most powerful non-linear character. In this fuzzy system type there are used: 3 fuzzy values for inputs and outputs with triangular and trapezoidal membership functions, inference with the max-min method and center of gravity defuzzification method. This is a fuzzy system used in application of control, with the input variables u = the error, u 2 =d error derivative and the output variable the controller command y=i d. 3.2 Training conditions The study case were: as training method were tested back propagation (bp), fast back propagation (bpx) and the Levenberg-Marquardt method (lm); the structures tested were:. hidden layer with 5 neurons, 2. - hidden layer with 0 neurons, 3. - 2 hidden layers with 0 and 5 neurons, 4. - 2 hidden layers with 0 and 8 neurons. For training, a set with (2 7 +).(2 7 +)=6.64 training forms was chosen. This dimension corresponds to a memory dimension with locations for each form. This approach corresponds to a case in which the fuzzy system is implemented as a fuzzy memory in digital control equipment. Each location is in fact the plac where a training form is placed. So, the number of training forms p is equal to the number of locations of the fuzzy memory. A training form is given by the set: ( de; di d ) k, k=,,p. Such a training form is obtained from the discrete MISO transfer characteristic of the fuzzy block di dk =f BF (e k, de k ), k=, p. The values of the training form resulted from sampling of the universes of discourse of the input variables e and de in 256 small samples and the computation of the defuzzified value: di dij = fbf ( ei, de j ) () where em det ( e, de j ) = em + ( i ), det + ( j ), i, j =,..., p + p p i (2) In each training epoch the entire training set is passed through the neural network. This dimension is related to a 8 digit DAC. 3.3 Quality Criteria Some quality criteria are used to evaluate the results of training and identification. These quality criteria are: The training error at the output of a neuron from the output layer: ε = di dm di d (3) The quadratic criterion: p q 2 E = ( ε ) (4) q= where p is number of forms. The error has the following equation: ε = di q q dm di (5) d where ε q is the error at the q form. This is the criterion chosen to be minimized. The average error: E = E p The average error in absolute value: e m = E Number of training epochs n e. The speed of error decreasing: E ve = = En+ En, n = n (6) (7) (8) where E is the difference of quadratic error E n from the n training and the n+ training. 3.4 Characteristics of the neuro-fuzzy systems For the neuro-fuzzy systems we may used the same characteristics as for the fuzzy systems []:. The MISO characteristic, or the system surfac di dm =f BN ( de), with the inputs on the following ISSN: 790-57 390 ISBN: 978-960-474-064-2

Proceedings of the 0th WSEAS International Conference on AUTOMATION & INFORMATION universe of discourse: e U e, de U de. 2. The family of the SISO characteristics di dm =f en (e; de), with de {-, -0,75, -0,5, -0,25, 0, 0,25, 0,5, 0,75, }, as a parameter. 3. The family of the translated characteristics with a compound value di dm =f N (x t ; de). 4. The family of the gain characteristics K BN (x t ; de). The accuracy of training and identification is demonstrated with the characteristics of the errors, introduced by identification in the neuro-fuzzy system, defined as differences between the characteristics of the neuro-fuzzy system and of the fuzzy system are sued:. MISO errors ε BN ( de) Study case n e E ε m bp 500 248 0,2 bp2 500 26 0, bp3 4000 06 0,08 bpx 000 53 0,096 bpx2 500 24 0,086 bpx3 500 25 0,0866 lm 00 07 0,08 lm2 00 30 0,04 lm3 00 3 0,0 lm4 77 3 0,0 Table. Values of the quality criteria ε BN = fbf de) fbn ( de), e U de U de ( (9) 2. SISO errors: ε e (e; de): ε = fe( e; de) fen ( e; de), e U de {, 0,75, 0,5, 0,25,0,0,25,0,5,0,75,} e (0) 3. The errors on the translated characteristics ε N (x t ; de): ε = f ( xt; de) f N ( xt; de), e U de {, 0,75, 0,5, 0,25,0,0,25,0,5,0,75,} N () 4. The errors of the gain characteristics ε K (e; de): ε = K BF ( e; de) K BN ( e; de), e U de {, 0,75, 0,5, 0,25,0,0,25,0,5,0,75,} K (2) 4 Results 4. Values of the Quality Criteria The values of the quality criteria resulted after trainings are presenting in Tab.. From the quality criteria values we notice the training time is increasing for the Levenberg- Marquardt method, which is a great memory consuming. But, with this method the most significant decreasing of error was obtained. More powerful computation equipment must be used in neural trainings. The errors at the output of the tested neural networks are presented in Fig. 5. The speed of error decreasing depends on the training method. The number of training epochs depends on the training epochs. The smaller quadratic error was obtained for the Levenberg-Marquardt method, in the case lm3, for a network with two hidden layers. This method is the most adequate. Fig. 5 The evolution of the quadratic error 4.2 Neural Structure The most adequate neural network to implement a fuzzy system, resulted after the comparison of the values of the quality criteria obtained in the study cases has the following structure: an input layer, two hidden layer and an output layer with 2 neurons on the input layer, 0 neurons on the first hidden layer, 5 neurons on the second hidden layer and neuron on the output layer. The structure is presenting in Fig. 6. Fig. 6 The structure of the neuro-fuzzy model This structure is considered an optimum, after a compromise between its dimension, the quadratic ISSN: 790-57 39 ISBN: 978-960-474-064-2

Proceedings of the 0th WSEAS International Conference on AUTOMATION & INFORMATION error and the training time. Its dimensions are according to the considerations of Kolmolgorov s theorem in developing neural networks [9]. Each connection has a weight and each neuron has a bias. The activation function of the neurons from the hidden layers is the sigmoid function. The activation function of the output layer is the linear function. The neural network is an associative memory [6]. In this case it associates to an input form [e q, de q ] T, with discrete values in tim an output form di q d with discrete values. The basic problem is to memorize a set of forms (e q, de q ; di q d ), q=, p, in such a way that when at the inputs a new form [e', de'] T is presented the network to respond at its output with a memorized form most closed to [e', de'] T. The neural network is a feedforward one and it functioning in the following manner: If the form q is given at its inputs, the unit j from the first hidden layer is receiving at its input:, q q q h j = w, je + w 2, jde + b j and it is producing at its output: q q ( w e + w de b ) (3), q, q j = fat ( h j ) = fat, j 2, j j (4) v + The unit i from the second hidden layer is receiving: 0 0 2, q 2, q 2 2 q q 2 h i = w j, iv j + bi = w j, i fat ( w, je + w 2, jde + b j ) + b (5) i j= j= lm3 with the structure from Fig. 6. The resulted values of the quality criteria are: n e =00, v E =0,007, E=3,05, ε m =0,03. The variation of the summation of the quadratic errors on 00 training epochs is presented in Fig. 7. Fig. 7 Variation of the training error We may notice after 80 training epochs the error attends the vicinity of its limit. The variation speed of error is decreasing. It will be necessary a greater number of training epochs to reduce the error. The final value of the error is around 3,05. This value assures an average error in absolute value at a neuron around ε m =0,03. The repartition of the errors at the output of the neural network for all 664 training forms applied in the training process is presented in Fig. 8. and it is producing: 0 2, q 2, q q q 2 v f ( h ) = f w f ( w e + w de + b ) + b (6) 2 i = at i at j, i at, j j= 2, j j i The unit from the output layer is receiving: 3, q h = 5 i= f at j w 3 2, q i,vi 3 + b = q q 2 3 ( w e + w de + b ) + b 5 0 3 2 wi, w j, i f at, j 2, j j i + b i= = = (7) and it is producing: 5 0 q 3 2 q q didm = fal wi fat w j, i fat ( w, je + w 2, jde + b j ) 2 3, + + bi b i= j= (8) 4.3 Neuro-Fuzzy System Characteristics In this paragraph the results of the direct neuroidentification of the most non-linear fuzzy system 33- mm-g [] are presented. The final result is considered the neural network Fig. 8 The output errors in training process The MISO characteristic of the neuro-fuzzy system is presented is presented in Fig. 9. Notice: In the following figures the output y=di d, the inputs u = e and u 2 = de. The surface of the errors at the neural output is presented in Fig. 0. We may notice that the profile of the characteristic MISO of the neural network resulted ISSN: 790-57 392 ISBN: 978-960-474-064-2

Proceedings of the 0th WSEAS International Conference on AUTOMATION & INFORMATION at the end of the training process are very closed to the profile of the MISO transfer characteristic of the fuzzy system. The errors introduced by the neural network are presented in Fig. 4. Fig. 9 The surface of the neuro-fuzzy model Fig. 2 Errors of the SISO characteristics Fig. 0 Errors of the neuro-fuzzy surface The family of the characteristics y=f RN (u ; u 2 ) are presented in Fig.. Fig. 3 SISO translated characteristics Fig. 4 Errors in the translated characteristics Fig. SISO transfer characteristics of the neurofuzzy model The errors introduced by the neural network are presented in Fig. 2. The translated characteristics y=f trn (x t ; u 2 ) of the fuzzy system are presented in Fig. 3. Analyzing the SISO transfer characteristics of the neuro-fuzzy system we may notice that the neural network implements with a small error all parts of the transfer characteristics the most nonlinear and the linear. The neural network implements the condition of the permanent regime u =0, u 2 =0; y=0. The errors at the neural network outputs decreased finally under 0,6. ISSN: 790-57 393 ISBN: 978-960-474-064-2

Proceedings of the 0th WSEAS International Conference on AUTOMATION & INFORMATION The errors are uniformly reprised on the entire universe of discourse. They are smaller at the universe margins. The translated characteristics have the sector property, the condition to be only in the I-st and III-rd quadrants. They are presenting the same non-linear parts as the fuzzy system characteristics. The family of gain characteristics K RN (x t ; u 2 ) of the fuzzy system are presented in Fig. 5. Fig. 5 The variable gain of the neuro-fuzzy model The family of characteristics of the errors is presented in Fig. 6. Fig. 6 Errors in the variable gain of the neuro-fuzzy model The characteristics of the function K BN (x t ; de) have the same profile as the characteristics of the fuzzy system. The function K BN (x t ; de) takes the maximum value K MN,2 closed to the maximum of the function K BF (x t ; de). The function K BN (x t ; de) has positive values over the entire universe of discourse of he compound variable x t. The neural network introduces the maximum errors around the origin of the compound variable. The neural network introduces the minimum errors at the margins of the universes of discourse of the input variables. 5 Conclusion In this paper some considerations for non-linear system identification with application at the fuzzy systems are presented. In this application the input-output transfer characteristics of the fuzzy system are used to evaluate the accuracy of the training results. This method has a large application at the most general fuzzy systems, developed with all kind of fuzzyfication methods, rule bases, inference methods and defuzzification methods. Using this method we may obtain accurate neural models of a large class of fuzzy systems. The neural network obtained with this method will be the best approximation for the non-linear systems, in this case the fuzzy systems. The neural model will keep all the properties of the fuzzy system, as the values of the transfer gain, the sector property, characteristic proofed based on the comparison between all the input-output transfer characteristics. Preserving these properties the neural model is useful in all application of the fuzzy systems, especially for exampl in control. The training was made to attend an optimized solution related to the quality criteria for the neural network. Good values are obtained, and the optimized neural- fuzzy model is given by its structur weights and biases, related to the best quality training criteria and the most adequate training method. The structure of the neuro-fuzzy model is given, with 2 hidden layers and the number of units on each layer, according to Kolomogorov s theorem for non-linear function approximation, with a total of 8 neurons. This result may be used in many other neuro-fuzzy applications, without other trainings. In complex applications we know now the number of neurons needed to implement one fuzzy system with neural networks and we could design complex network to implement multi-neuro-fuzzysystems. Acknowledgement: This work was developed in the frame of PNII- IDEI-PCE-ID923-2009 CNCSIS - UEFISCSU grant. ISSN: 790-57 394 ISBN: 978-960-474-064-2

Proceedings of the 0th WSEAS International Conference on AUTOMATION & INFORMATION References: [] O. Nelles, Nonlinear System Identification: From Classical Approaches to Neural Networks and Fuzzy Models, Springer, 2000. [2] R. Le Richer, D. Gualandris, J.J. Thomas, F. Hemez, Neural Identification of Non-linear Dynamics Structures, Journal of Sound and Vibration, Volume 248, No. 22, November 200, pp. 247-265(9), Elsevier. [3] P. Pivoňka, J. Dohnal, On-line Identification Based on Neural Networks Using of Levenberg-Marquardt Method and Backpropagation Algorithm, WSEAS Transactions on Systems, Issue 2, Volume 3, April 2004, pag. 38-385. [4] V. Veleba, P. Pivonka, Adaptive Controller with Identification Based on Neural Network for Systems with Rapid Sampling Rates, WSEAS Transactions on Systems, Issue 4, Volume 4, April 2005, pag. 385-388. [5] A. Khashman, B. Sekeroglu, K. Dimililer, Coin Identification Using Neural Networks, Proceedings of the 5th WSEAS Int. Conf. on Signal Processing (SIP '06), Istanbul, Turkey, 27-29 May 2006. [6] J. Hertz, A., Krogh, R.G. Palmer, Introduction to the Theory of Neural Computation, Addison- Wesley Publishing Co., Redwood Cliffs, USA, 99. [7] K. S., Narendra, Identification and Control of Dynamical Systems Using Neural Networks, IEEE Trans. on Neural Networks, March, 990. [8] P. J. Werbos, Neural Networks for Control: An Overview; Proceedings of IEEE International Conference on Automatic Control, 990. [9] L. Jin, M.M. Gupta, P.N. Nikiforuk, Approximation Capabilities of Feedforward and Recurrent Neural Networks, Intelligent Control Systems. Theory Applications, IEEE Press, 996. [0] H. Buhler, Reglage par logique flou Press Polytechniques et Universitaires Romands, Lausann 994. [] C. Volosencu, On Some Properties of Fuzzy Systems, Recent Advances in Signal Processing, Robotics and Automation, Cambridg UK, Feb. 2-23, 2009, Proceedings of the 8 th WSEAS International Conference on Signal Processing, Robotics and Automation (ISPRA 09), Mathematics and Computers in Science and Engineering, A Series of Reference Books and Textbooks, WSEAS Press, 2009, pag. 78-86. ISSN: 790-57 395 ISBN: 978-960-474-064-2