Determining the Model Order of Nonlinear Input-Output Systems by Fuzzy Clustering

Transcription

1 Determining the Model Order of onlinear Input-Output Systems by Fuzzy Clustering Balazs Feil, Janos Abonyi, and Ferenc Szeifert University of Veszprem, Department of Process Engineering, Veszprem, P.O. Box 158, H-8201, Hungary Abstract. Selecting the order of an input-output model of a dynamical system is a ey step toward the goal of system identification. By determining the smallest regression vector dimension that allows accurate prediction of the output, the false nearest neighbors algorithm (F) is a useful tool for linear and also for nonlinear systems. The one parameter that needs to be determined before performing F is the threshold constant that is used to compute the percentage of false neighbors. For this purpose heuristic rules can be followed. However, for nonlinear systems choosing a suitable threshold is extremely important, the optimal choice of this parameter will depend on the system. While this advanced F uses nonlinear inputoutput data based models, the computational effort of the method increases along with the number of data and the dimension of the model. To increase the efficiency of the method this paper proposes the application of a fuzzy clustering algorithm. The advantage of the generated solutions is that it remains in the horizon of the data, hence there is no need to apply nonlinear model identification tools. The efficiency of the algorithm is supported by a data driven identification of a polymerization reactor. 1 Introduction In recent years a wide range of model-based engineering tools have been developed. However, most of these advanced techniques require models of relatively low order and restricted complexity. Since most of the current data-driven identification algorithms assume that the model structure is a priori nown, the structure and the order of the model have to be chosen before identification. Several information theoretic criteria have been proposed for the selection of the order of input-output models of linear dynamical systems. A technique based on prediction-error variance, the Final Prediction-Error (FPE) criterion, was developed by Aaie [1]. Aaie also proposed another well nown criterion, Aiaie s Information Criterion (AIC), that is derived from information theoretic concepts, but do not yield consistent estimates of the model order. To avoid this problem, the Minimum Description Length (MDL) criterion has been developed by both Schwartz and Rissanen, and its ability to produce consistent estimates of model order has been also proven [8]. With the usage of these tools, determining the model order of linear systems is not a problematic tas. While there is extensive wor in determining the proper

2 2 B. Feil, J. Abonyi and F. Szeifert model order for linear systems, there is relatively little wor in the filed of nonlinear systems. For the determination of the order of nonlinear models deterministic suitability measures [3] and false nearest neighbors (F) based algorithm [9] have been wored out and applied in the chemical process industry. These ideas build upon similar methods developed for the analysis of self-driven chaotic time series [7]. The idea behind the F algorithm is geometric in nature. If there is enough information in the regression vector to predict the future output, then any of two regression vectors which are close in the regression space will also have future outputs which are close in some sense. Hence, the model order identification is reformulated as a determination of a distance measure and the calculation a problemspecific threshold that is used to compute the percentage of false neighbors for all combinations of the possible input variables. The one parameter that needs to be determined before performing F is the threshold constant. For this purpose heuristic rules can be followed [3]. However, for nonlinear systems choosing a suitable threshold is extremely important, the optimal choice of this parameter will depend on the system [9]. While this advanced F uses nonlinear input-output data based models, the computational effort of the method increases along with the number of data and the dimension of the model. To increase the efficiency of the method this paper proposes the application of a fuzzy clustering algorithm. The advantage of the generated solutions is that it remains in the horizon of the data, hence there is no need not to apply nonlinear model identification tools. The paper is organized as follows. Section 2 presents the idea behind the F algorithm. In Section 3, the application of fuzzy clustering for improvement of this algorithm is proposed. An application example is given in Section 4. Conclusions are given in Section 5. 2 F Algorithm Many non-linear static and dynamic processes can be represented by the following regression model y = f (x ) (1) where f (.) is a nonlinear function and x represents its input vector. Among this class of models, the identification of discrete-time, on-linear Auto- Regressive models with exogenous inputs (ARX) is considered in this paper. In the ARX model, the model inputs are past values of the process outputs y() and the process inputs u(). x = [y( 1),...,y( m),u( 1),...,u( n)] T (2) while the output of the model is the one-step ahead prediction of the process, y = y().

3 Fuzzy Clustering for Model Order Detection 3 The number of past outputs used is m and the number of past inputs is n. The values m and n are often referred to as model orders. The above SISO system representation can be assumed without a loss of generality since the extension to MISO and MIMO systems is straightforward. The method of false nearest neighbors (F) was developed by Kennen [7] specifically for determining the minimum embedding dimension, the number of time-delayed observations necessary to model the dynamic behavior of chaotic systems. For determining the proper regression for input/output dynamic processes, the only change to the original F algorithm involves the regression vector itself [9]. The main idea of the F algorithm utilized in this article stems from the basic property of a function. If there is enough information in the regression vector to predict the future output, then any of two regression vectors which are close in the regression space will also have future outputs which are close in some sense. For all regression vectors embedded in the proper dimensions, for two regression vectors that are close in the regression space and their corresponding outputs are related in the following way: y y j = d f ( x m,n ) [ ] ([ ]) x m,n x m,n j + o x m,n x m,n 2 j (3) where d f ( x m,n ) is the jacobian of the function f (.) at x m,n. Ignoring higher order terms, and using the Cauchy-Schwarz inequality the following inequality can be obtained: y y j d f ( x m,n ) 2 x m,n x m,n j (4) 2 x m,n y y j x m,n ( d f x m,n) 2 (5) 2 j If the above expression is true, then the neighbors are recorded as true neighbors. Otherwise, the neighbors are false neighbors. Based on this theoretical bacground, the outline of the F algorithm is the following. 1. Identify the nearest neighbor to a given point in the regressor space. For a given regressor: = [y(),...,y( m),u(),...,u( n)] T x m,n find the nearest neighbor x m,n j such that the distance d is minimized: d = x m,n x m,n j 2 2. Determine if the following expression is true or false y y j x m,n R j 2 x m,n where R is a previously chosen threshold value. If the above expression is true, then the neighbors are recorded as true neighbors. Otherwise, the neighbors are false neighbors.

4 4 B. Feil, J. Abonyi and F. Szeifert 3. Continue the algorithm for all times in the data set. 4. Calculate the percentage of points in the data that have false nearest neighbors J(m,n). 5. Continue the algorithm for increasing m and n using the percentage of false nearest neighbors drops to some acceptably small number. Because the model order is determined by finding the number of past outputs m and past inputs n, the J(m,n) indices become a surface in two dimensions. It is possible to find a global solution (or solutions) for the model orders by computing the desired index over all values of m and n in a certain range and determining which points satisfy the order determination conditions. The smallest m and n values such that J(m,n) is zero lie in the corner of this that is nearest to the origin, ˆm and ˆn. This corner is easily identified since J(m,n) 0 for m ˆm and n ˆn. When the noise is not zero, J(m,n) will not be zero if m and n are chosen as m ˆm and n ˆn, but it will tend to remain relatively small and flat. Therefore, we calculate table of J(m, n) and then search for the corner where J(m,n) drops quicly similarly to the MDL based method suggested in [8]. A more heuristic local solution is also possible. In this case, initial guesses for m and n are used, and the optimum model order is computed competitively; ate each iteration, either m or n is increased by one, depending on which reduces the index the greatest amount [3]. In cases where the available input-output data set is small, the algorithm is sensitive to the choice of the R threshold. In [3] the threshold value was selected by trial and error method based on empirical rules of thumb, 10 R 50. However, choosing a single threshold that will wor well for all data sets is impossible tas. In this case, it is advantageous to estimate R based on (5) using the following expression R = max d f ( x m,n ) as it has been suggested by Rhodes and Morari []. While ( the method uses input-output data based models for the estimation of d f x m,n), the computational effort of F increases along with the number of data and the dimension of the model. To increase the efficiency of the method this paper proposes the application of a fuzzy clustering algorithm that will be introduced in the following section. 3 Application of Fuzzy Clustering to F The available input-output can be clustered. The main idea of the paper is that when the appropriate number of regressors are used, the collection of the obtained clusters will approximate the regression surface of the model of the system. In this case the clusters can be approximately regarded as local linearizations of the system and can be used to estimate R. Clusters of different shapes can be obtained by different clustering algorithms by using an appropriate definition of cluster prototypes (e.g., points vs. linear varieties) or by using different distance measures. The Gustafson Kessel clustering algorithm [6] has been often applied to identify Taagi Sugeno fuzzy systems that

5 Fuzzy Clustering for Model Order Detection 5 are based on local linear models [2]. The main drawbacs of this algorithm are that only clusters with approximately equal volumes can be properly identified which constrain maes the application of this algorithm problematic for the tas of this paper. To circumvent this problem, in this paper Gath Geva algorithm is applied [5] that will be described in the following subsection. 3.1 Gath-Geva Clustering of the Data The objective of clustering is to partition a data set Z into c clusters, where the available identification data, Z T = [Xy] formed by concatenating the regression data matrix X and the output vector y x T 1 x T X = 2... x T y = y 1 y 2. y This means, each observation consists of m + n + 1 variables, grouped into an -dimensional column vector z = [x 1,,...,x n+m,,y ] T = [x T y ] T. Through clustering, the fuzzy partition matrix U = [µ i, ] c is obtained, whose element µ i represents the degree of membership of the observation z in the cluster ı = 1,...,c. In this paper, c is assumed to be nown, based on prior nowledge, for instance. For methods to estimate or optimize c in the context of system identification refer to [2]. The GG algorithm is based on the minimization of the sum of the weighted squared distances between the data points,z and the cluster centers, v i, i = 1,...,c. J(Z,U,V) = c i=1 j=1 µ m i, D2 i, (7) where V = [v 1,...,v c ] contains the cluster centers and m [1, ) is a weighting exponent that determines the fuzziness of the resulting clusters and it is often chosen as mw = 2. The fuzzy partition matrix has to satisfy the following conditions: U R c with µ i, [0,1], i,; c i=1 µ i, = 1, ; 0 < =1 (6) µ i, <, i (8) The minimum of (7) is sought by the alternating optimization (AO) method given below: Initialization Given a set of data Z specify c, choose a weighting exponent m > 1 and a termination tolerance ε > 0. Initialize the partition matrix such that (8) holds. Repeat for l = 1,2,...

6 6 B. Feil, J. Abonyi and F. Szeifert Step 1 Calculate the cluster centers. v (l) i = =1 µ (l 1) i, z µ (l 1) i, =1, 1 i c (9) Step 2 Compute the distance measure D 2 i. The distance to the prototype is calculated based the fuzzy covariance matrices of the cluster F (l) i = =1 ( µ (l 1) i z v (l) i )( µ (l 1) i =1 The distance function is chosen as z v (l) i n+1 D 2 i, (z (2π)( 2 ) det(f i ),v i ) = exp α i with the a priori probability α i α i = 1 =1 Step 3 Update the partition matrix µ (l) ) T ( 1 ( 2, 1 i c (10) z v (l) i ) T ( F 1 i z v (l) i ) ) (11) µ i, (12) i, = 1 ( Di (z,v i )/D j (z,v j ) ), 1 i c, 1. (13) 2/(mw 1) c j=1 until U (l) U (l 1) < ε. 3.2 Estimation of the R Threshold Coefficient The collection of c clusters approximates the regression surface as it is illustrated in Figure 1. Hence, the clusters can be approximately regarded as local linear subspaces. This is reflected by the smallest eigenvalues λ i, of the cluster covariance matrices F i that are typically in orders of magnitude smaller than the remaining eigenvalues [2] (see Figure 2). The eigenvector corresponding to this smallest eigenvalue, t i, determines the normal vector to the hyperplane spanned by the remaining eigenvectors of that cluster (t i )T (z v i ) = 0 (14) Similarly to the [ observation ] vector z = [x T y ] T, the prototype vector and is partitioned as v i = (v x i )T v y i into a vector v x corresponding to the regressor x, and

7 Fuzzy Clustering for Model Order Detection 7 Fig. 1. Example for clusters approximating the regression surface. a scalar v y i corresponding to the output y. The smallest eigenvector is partitioned in [ ( ) T T the same way, t i = t i,x i,y t ]. By using this partitioned vectors (14) can be written as [ ( ) ] t i,x T T ( [ ]) i,y t [x T y ] T (v x i ) T v y i = 0 (15) from which the parameters of the hyperplane defined by the cluster can be obtained: y = 1 ( ) t i,x T x + 1 ( ) t i T vi = a T i x + b i (16) t i,y } {{ } a T i t i,y } {{ } b i Fig. 2. Example for clusters approximating the regression surface. Although the parameters have been derived from the geometrical interpretation of the clusters, it can be shown [2] that (16) is equivalent to the weighed total leastsquares estimation of the consequent parameters, where each data point is weighed by the corresponding µ i.

8 8 B. Feil, J. Abonyi and F. Szeifert The main contribution of this paper is that it suggests the application of an adaptive threshold function that taes into account the nonlinearity of the system. This means, based on the result of the fuzzy clustering, for all input-output data pairs different R values are calculated. Since, the optimal value of R is R = ( d f x m,n) and the d f ( x m,n ) partial derivatives can be estimated based on the shape of the clusters from (16) d f ( x m,n ) c i=1 the threshold can be calculated as c 1 ( R = µ i i=1 t i,y t) i,x T 2 1 ( µ i t i,y t i,x T ) (17) 4 Application for Continuous Polymerization Reactor The following example illustrates identification using data from a model of a continuous polymerization reactor. The model describes the free-radical polymerization of methyl methacrylate with azobisisobutyronitrile as an initiator an toluene as a solvent. For further information on the details of this model and how it is derived, see [4]. The reaction taes place in a jaceted CSTR, and after some simplifying assumption are made the first-principles model is given by (23). (18) ẋ 1 = 10(6 x 1 ) x 1 x2 (19) ẋ 2 = 80u x 2 (20) ẋ 3 = x 1 x x 2 10x 3 (21) ẋ 4 = x 1 x2 10x 4 (22) ẋ 5 = x 4 x 3 (23) The dimensionless state variable x 1 refers to the monomer concentration, and x 4 /x 3 is the number-average molecular weight (an also the output y). The input u is the dimensionless volumetric flow rate of the initiator. Since a model of the system is nown, large amounts of data can be collected for analysis. For this example we apply a uniformly distributed random input over the range to with a sampling time of 0.2. By driving the system with this input signal, an output that is roughly in the range of 26,000 to 34,000 is produced, which is the desired operating range of the system. The model with output order m = 1 and input order n = 2 should give an accurate estimate of future outputs, because the MARS algorithm constructs an accurate model for this problem [9]. The Table 1 shows the results of the proposed algorithm with c=6 cluster. The number with m=2 and l=1 is enough small, but larger input and output orders are acceptable, too [9]. The clustering algorithm has an parameter: c, the

9 Fuzzy Clustering for Model Order Detection 9 Table 1. F results for polymerization data when R is obtained by fuzzy clustering Input Delays (n) Output Delays (m) % F number of the clusters. The increasing of this parameter increases the accuracy of the model as a general rule. For the purpose to avoid the overfitting and the increasing of the calculation requirement it is recommended to determine the number of the clusters automatically. For this purposes the method of Gath and Geva can be applied [5] and [2]. For comparisons the next table shows the results when constant threshold has been used. In this case the value of R has been estimated based on the parameters of a linear ARX model identified based on the data used for clustering purposes. Table 2. F results for polymerization data when R is obtained based on the parameters of a linear ARX model Input Delays (n) Output Delays (m) % F We can allocate that this linear model based method does not give conspicuously incorrect results, but induces larger error for high nonlinear systems, because it results in more inaccurate approximation. Hence, the application of the proposed clustering based approach is much more advantageous.

10 10 B. Feil, J. Abonyi and F. Szeifert 5 Conclusions By determining the smallest regression vector dimension that allows accurate prediction of the output, the F algorithm is a useful tool for linear and also nonlinear systems. It reduces the overall computational effort, simplifies and maes more effective the nonlinear identification which becomes difficult and gives not certainly accurate results by false regression vector. To increase the efficiency of F this paper proposed the application of clustering algorithm. The advantage of our approach is that it remains in the horizon of the data and there is need not to apply nonlinear model identification tools to determine the threshold parameter of the F algorithm. Acnowledgement The financial support of the Hungarian Ministry of Culture and Education (FKFP- 0073/2001) and the Hungarian Science Foundation (T037600) is greatly acnowledged. Janos Abonyi is grateful for the financial support of the Janos Bolyai Research Fellowship of the Hungarian Academy of Science. References 1. H. Aaie. A new loo at the statistical model identification. IEEE Trans. on Automatic Control, 19: , R. Babuša. Fuzzy Modeling for Control. Kluwer Academic Publishers, Boston, J.D. Bomberger and D.E. Seborg. Determination of model order for ARX models directly from input output data. Journal of Process Control, 8: , Oct Dec F.J. Doyle, B.A. Ogunnaie, and R. K. Pearson. onlinear model-based control using second-order volterra models. Automatica, 31:697, I. Gath and A.B. Geva. Unsupervised optimal fuzzy clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 7: , D.E. Gustafson and W.C. Kessel. Fuzzy clustering with fuzzy covariance matrix. In Proceedings of the IEEE CDC, San Diego, pages M.B. Kennel, R. Brown, and H.D.I. Abarbanel. Determining embedding dimension for phase-space reconstruction using a geometrical construction. Physical Review, A: , G. Liang, D.M. Wiles, and J.A. Cadzow. Arma model order estimation based on the eigenvalues of the covariance matrix. IEEE Trans. on Signal Processing, 41(10): , C. Rhodes and M. Morari. Determining the model order of nonlinear input/output systems. AIChE Journal, 44: , 1998.