Nonlinear System Identification Using Maximum Likelihood Estimation

Transcription

1 ISS (Print) : ISS (Online): (An ISO 397: 007 Certified Organizatio Vol., Special Issue, December 03 onlinear System Identification Using Maximum Lielihood Estimation Siny Paul, Bindu Elias Associate Professor, Department of EEE, Mar Athanasius College of Engineering, Kothamangalam, Kerala, India, Abstract: Different algorithms can be used to train the eural etwor Model for onlinear system identification. Here the Maximum Lielihood Estimation is implemented for modeling nonlinear systems and the performance is evaluated. Maximum lielihood is a well-established procedure for statistical estimation. In this procedure first formulate a log lielihood function and then optimize it with respect to the parameter vector of the probabilistic model under consideration. Four nonlinear systems are used to validate the performance of the model. Results show that eural etwor with the algorithm of Maximum Lielihood Estimation is a good tool for system identification, when the inputs are not well defined. Keywords: eural etwor, onlinear system, Mean square error, Modeling. I. IRODUCIO his paper concentrates on modeling problem which arise when we can identify a certain quantity as a definite measurable output or effect but the causes are not well defined. his is called time series modeling, where inputs or causes are numerous and not quite nown in addition to often being unobservable. his type modeling is also called stochastic modeling. In system identification we are concerned with the determination of the system models from records of system operation. he problem can be represented diagrammatically as below. Fig. System Configuration [] where t) is the nown input vector of dimension m z(t) is the output vector of dimension p w(t) is the input disturbance vector n(t) is the observation noise vector v(t) is the measured output vector of dimension p hus the problem of system identification is the determination of the system model from records of t) and y(t). Copyright to IJAREEIE 64

2 ISS (Print) : ISS (Online): (An ISO 397: 007 Certified Organizatio Vol., Special Issue, December 03 An artificial neural networ is a powerful tool for many complex applications such as function approximation, optimization, nonlinear system identification and pattern recognition. his is because of its attributes lie massive parallelism, adaptability, robustness and the inherent capability to handle nonlinear system. It can extract information from heavy noisy corrupted signals. System identification can be either state space model or input-output model []. II. IPU-OUPU MODEL An I/O model can be expressed as y( t) g( ( t, )) e( t), where, is the vector containing adjustable parameters which in the case of neural networ are nown as weights, g is the function realized by neural networ and is the regression vector. Depends on the choice of regression vector different model structures emerge. Using the same regressors as for the linear models, corresponding families of nonlinear models were obtained which are named ARX, ARMAX, etc. Different model structures in each model family can be obtained by maing a different assumption about noise. ( t, ) y( t ), y( t ),... y( t, t ),... t m) () ARX ( t, ) y( t ),... y( t, t ),... t m), e( t ),... e( t ) () ARMAX Where y(t) is the output, t),the input and e(t) is the error. For the implementation of the above system, Feed forward neural networ can be used. MLFF Fig.. ARX model [] Copyright to IJAREEIE 65

3 ISS (Print) : ISS (Online): (An ISO 397: 007 Certified Organizatio Vol., Special Issue, December 03 ARX Model is well suited for Input-Output modeling of stochastic nonlinear systems [3] So in the proposed wor, ARX model is chosen as the system model; in which the model structure is a Multi Layer Feed Forward eural etwor(mlff) as shown in Fig. For all the models (using different algorithms). III. MAXIMUM LIKELIHOOD ESIMAIO. he term maximum lielihood estimate with the desired asymptotic properties usually refers to a root of the lielihood equation that globally maximizes the lielihood function [3]. In other words the ML estimate x ML is that value of the parameter vector x for which the conditional probability density function P(z/x) is maximum [4].he maximum lielihood estimate x ML of the target parameters x is the mode of the conditional probability density function(lielihood functio: he log lielihood function Where r is the residual. p( z / x) ( ) r / exp( r r ) log( p( z / x)) () d z σ is the std. deviation, d =desired value, z = measurement(estimated value). Maximizing log lielihood function log(p(z/x)) is equivalent to minimizing the negative log lielihood function L(z,x). By using the negative log-lielihood function L(z,x) the ML problem is reformulated as a nonlinear least square problem: Minimize L( z, x) r (3) he ML estimate must satisfy the following optimality condition: xl ( z, xml ) xml ) xml ) 0 (4) Where x) is the dimensional residual vector and x) the x n Jacobian matrix. R ( x) [ r ( x)... r ( x)] (5) J ( x) x x) (6) he operator x is defined as x [ / x.. / x.. / x3... / x ] Since the problem is nonlinear it is not possible to solve analytically. But certain optimization methods seem suitable to find the ML Estimate. Gauss-ewton method is one such optimization technique. A. System Modeling Using Gauss-ewton Method. () Copyright to IJAREEIE 66

4 ISS (Print) : ISS (Online): (An ISO 397: 007 Certified Organizatio Vol., Special Issue, December 03 A Feed forward eural etwor model similar to earlier cases is designed for the identification of the same nonlinear systems and trained using Gauss ewton method [5]. he Gauss-ewton method is applicable to a cost function that is expressed as the sum of error squares. Let E( x) ) (7) he error signal ) is a function of adjustable state vector x. Given an operating point x(, we linearize the dependence of r on x by writing ' r (, x) ) [ ) / x] x x( x x(, =, n (8) Equivalently, by using matrix notation we may write r' (, x) ) x x( (9) he updated state vector x(n+) is then defined by ' x( n ) arg.min ) r ( n, x (0) squared Euclidean norm of r (n,x), ' r n Hence differentiating this expression with respect to x and setting the result equal to zero, we obtain (, ) n x ( x x( ) ( x x( ( x x( )) () Solving this equation for x, ( x x( ) 0 x( n ) x( ( ). (3) which describes the pure form of the Gauss-ewton method. Unlie ewton s method that requires nowledge of the Hessian matrix of the cost function E(, Gauss ewton method only requires the Jacobian matrix of the error vector. However, for the Gauss- ewton iteration to be computable, the matrix product must be nonsingular. Unfortunately, there is no guarantee that this condition will always hold. o guard against the possibility that is ran deficient, the customary practice is to add the diagonal matrix δi to the matrix. he parameter δ is a small positive constant chosen to ensure that; + δi : positive definite for all n. On this basis, the Gauss-ewton method is implemented in slightly modified form: x( n ) x( ( I) (4) Here x represents the state vector of the system. In the eural etwor model x becomes the weight vector w. hus the update equation of the weight vector becomes; w( n ) w( ( I) (5) where is the Jacobian matrix equal to w x ) (ie derivative of the error w.r.t weights). IV. RESULS AD DISCUSSIOS. he same four nonlinear systems are modeled using feed forward neural networ. he ARX model with 4 inputs is used similar to the earlier cases. he performance analysis is done by plotting the mean square error in each case. Copyright to IJAREEIE 67 ()

5 ISS (Print) : ISS (Online): (An ISO 397: 007 Certified Organizatio Vol., Special Issue, December 03 ( epoch : 00 samples) Fig 3. Superposition of model output and desired output onlinear System y = Sin(x + x) Fig. 4. MSE Vs data samples. Fig 5. Superposition of model output and desired output of Ambient oise. Fig 6. MSE Vs data samples. Fig. 7. Superposition of model output and Fig. 8. MSE Vs data samples. desired output of Accoustic source A. Copyright to IJAREEIE 68

6 ISS (Print) : ISS (Online): (An ISO 397: 007 Certified Organizatio Vol., Special Issue, December 03 Fig 9. Superposition of model output and desired output of Accoustic source B. Fig 0. MSE Vs data samples. ABLE I. MEA SQUARE ERROR FOR DIFFERE SYSEMS. System From the above results it is seen that the model is giving good performance for all the four nonlinear systems which proves that ARX Model is well suited for Input-Output modeling of stochastic nonlinear systems V. COCLUSIO In this paper, a comprehensive analysis for nonlinear system identification is done and its performance is compared by implementation of the same in a eural etwor ARX model using MALAB programs. he adaptive feature revealed by feed forward and recurrent neural networ as well as their ability to model nonlinear time varying process, provides a surplus value to the model based predictive control. When applied correctly, a neural or adaptive system may considerably outperform other methods. his is an attempt to provide guideline to the practitioners to choose the suitable method for their specific problem in the field of system identification especially in the stochastic modeling of nonlinear systems. It is proved that MLE can be reformulated as a minimization problem; he results show good performance of the models and it is proven that MLE is good for nonlinear system identification. Four different nonlinear systems are used to chec the consistency of the performance of algorithm. he performance of MLE is good in terms of mean square error. REFERECES Mean Square Error y = sin (x +x) Ambient noise Acoustic source A 0.08 Acoustic source B [].K Sinha and B Kuszta, Modeling and Identification of Dynamic systems, Vol., ew Yor,Van ostrand Reinhold Company, 983. [] Simon Hayin, eural etwors a comprehensive Foundation, nd ed.,india, Pearson Education, 999. [3]A.V. Balarishnan, Kalman Filtering heory, ew Yor, Optimization Software Inc. Publications Division,,99. Copyright to IJAREEIE 69

7 ISS (Print) : ISS (Online): (An ISO 397: 007 Certified Organizatio Vol., Special Issue, December 03 [4] Ben James, Brian D.O, Anderson, and Robert. C. Williamson Conditional Mean and Maximum Lielihood approaches to Multi harmonic frequency estimation., IEEE ransactions on Signal Processing, Vol 4, o.6, pp , June 994. [5] Roy L Streit and od E Luginbubl, Maximum Lielihood raining of Probabilistic eural etwors, IEEE ransactions on eural etwors, Vol.5, o.5, pp , September 994. [6] sung-an Lin, C.Lee Giles and Sun-Yaun Kung, A Delay Damage Model Selection Algorithm for ARX eural etwors, IEEE ransactions on Signal Processing Vol.45,o., pp ,ovember 997. [7] Wen Yu and Xiaoou Li Some ew Results on System Identification with Dynamic eural etwor, IEEE ransactions on eural etwor, Vol, o., pp 4-47, March 00. [8] X.M. Ren, A.B. Rad, P.. Chan and Wai Lan Lo, Identification and control of continuous time onlinear Systems via Dynamic eural etwors IEEE ransactions on Industrial Electronics, Vol 50,o.3, pp , June 003. [9] Songwu Lu and amer Basar, Robust onlinear System Identification using eural etwor Models, IEEE ransactions on eural etwors, Vol.9, o.3, pp , May 998. [0] Mohamed Ibnabla, Statistical Analysis of eural etwor Modeling and Identification of onlinear Systems with Memory, IEEE ransactions on Signal Processing, Vol. 50, o.6, pp , June 00. [] ader Sadegh, A Perceptron etwor for Functional Identification and Control of onlinear Systems, IEEE ransactions on eural etwors, Vol.4, o.6, pp , ovember 993. []Octavian Stan and Edward Kamen, A localized Least Squares Algorithm for raining Feed forward eural etwors, IEEE ransactions on eural networs, Vol., o., pp , march 000. [3] Si Zhao Qin, Hong e Su and homas J McAvoy, Comparison of four eural etwor Learning Methods for Dynamic System Identification, IEEE ransactions on eural etwors, Vol. 3, o., pp -30, January 99. [4] G.V. Pusorius and L.A Feldamp, euro Control of nonlinear dynamical systems with Kalman Filter trained recurrent networs IEEE ransactions on neural networs, Vol 5, o., pp 79-97,March 994. [5] K.S. arendra and K Parthasarathy, Identification and control of Dynamical systems using neural networs, IEEE ransactions on eural etwors, Vol, o., pp 4-7, March 990 [6] Shuhi Li, Comparative Analysis of bac propagation and Extended Kalman filter in Pattern and Batch forms for training eural etwors, IEEE ransactions on eural etwor, Vol, o.4, pp44-49, June 00. Copyright to IJAREEIE 630