Commun. Theor. Phys. (Beijing, China) 43 (2005) pp. 1056 1060 c International Academic Publishers Vol. 43, No. 6, June 15, 2005 Discussion About Nonlinear Time Series Prediction Using Least Squares Support Vector Machine XU Rui-Rui, BIAN Guo-Xing, GAO Chen-Feng, and CHEN Tian-Lun Department of Physics, Nankai University, Tianjin 300071, China (Received September 16, 2004; Revised October 19, 2004) Abstract The least squares support vector machine (LS-SVM) is used to study the nonlinear time series prediction. First, the parameter γ and multi-step prediction capabilities of the LS-SVM network are discussed. Then we employ clustering method in the model to prune the number of the support values. The learning rate and the capabilities of filtering noise for LS-SVM are all greatly improved. PACS numbers: 05.45.-a Key words: least squares support vector machine, nonlinear time series, prediction, clustering 1 Introduction Nonlinear time series prediction exists broadly in many fields such as natural science, social science, economics, national defense, and so on. [1] It remains an attractive problem for a long time to analyze the nonlinear time series and mine the information of data. Support vector machine (SVM) based on statistical learning theory and structural risk minimization was first introduced by Vapnik et al. [2 4] It can solve many problems that are often encountered in application of artificial neural network including over-fitting, curse of dimensionality, local minima of energy, and so on. Some experimental results show that SVM is an effective model to classifying patterns and to estimating functions. However, the amount of calculation becomes larger and the learning rate will be greatly cut down with increasing of the amount of data for training. This is mainly because SVM typically follows the solution to a quadratic programming (QP) problem. Therefore, Suykens proposed a modified version of SVM, the least squares support vector machine (LS-SVM). [5] In this case the solution is given by a linear system instead of a QP problem. Thus the complexity of calculation decreases and the learning rate is raised at the same time. In this work, the prediction of time series that is produced from Mackey Glass equation is studied using LS- SVM. This paper is organized as follows. In the second part we describe the basic ideas of LS-SVM for prediction. In the third part, at first LS-SVM is tested for the Mackey- Glass equation. Then the parameter γ and the capabilities of multi-step prediction using LS-SVM are discussed. In the end we put clustering method into pruning the support value spectrum. In the last part conclusion of this model is presented. 2 Least Squares Support Vector Machine An LS-SVM model can be designed as Fig. 1. Fig. 1 Scheme of support vector machine. Suppose that we have a training set {x i, y i } N with input pattern x i R n for the i-th example and y i R for the corresponding desired output pattern, where N is the number of vectors for learning. In the feature space, LS-SVM can be described as y(x) = w T φ(x) + b, (1) where the nonlinear mapping φ(x) maps the input data into a higher-dimensional feature space. In LS-SVM, the objective function is min J(w, e) = 1 w,e 2 wt w + 1 N 2 γ e 2 i, (2) subject to the constraint y i = w T φ(x i ) + b + e i for i = 1,..., N. (3) The project supported by National Natural Science Foundation of China under Grant No. 90203008 and the Doctoral Foundation of the Ministry of Education of China
No. 6 Discussion About Nonlinear Time Series Prediction Using Least Squares Support Vector Machine 1057 Equality constraint is taken in LS-SVM instead of inequality constraint in SVM. Furthermore, the error e i has been changed to e 2 i in the object function. [3] These will simplify the solution to this problem. The Lagrangian can be defined as L(w, b, e; α) = J(w, e) α i {w T φ(x i )+b y i +e i }, (4) where α i denotes Lagrange multipliers. According to the Karush Kuhn Tucker (KKT) condition, we partially differentiate L and obtain formulas as follows: N w = 0 w = α i φ(x i ), B = 0 n α i = 0, e i = 0 α i = γe i, for i = 1,..., N, α i = 0 w T φ(x i ) + b + e i y i = 0, fractal dimension. In this work, we choose s = 17. The nonlinear time series based on Mackey Glass equation are regarded as a criterion for comparing ability of different predicting methods. The data of Mackey Glass equation are initially normalized as the following formula, x t = x t min(x), for t = 1, 2, 3,... (9) max(x) min(x) (i) Prediction Using LS-SVM We get 1500 data produced by Mackey Glass equation. The initial 200 data are used to train the LS-SVM network and then to predict the following 1300 data. The parameters of LS-SVM are taken as γ = 10 7, σ = 1.7, and the predicting error is measured with the root-meansquare error (RMSE): RMSE = (x i x i) 2 /N, where x i stands for the predicting value, x i is the desired value, and N denotes the number for prediction. Figure 2 illustrates the predicting results with RMSE = 6.58 10 4 (the dotted line is the predicting values and real line is the desired ones). for i = 1,..., N. (5) By eliminating e i and w, we obtain the system ( ) 0 1 T ν [ b ] [ 0 ] 1 ν Ω + 1 γ I =, (6) a y where y = [y 1 ;... ; y N ], 1 ν = [1;... ; 1], α = [α 1 ;... ; α N ], and Ω ij = φ(x i ) T φ(x j ) = K(x i, x j ), i, j = 1,..., N. K(x i, x j ), which is called the inner-product kernel should satisfy the case of Mercer s condition. After application of the Mercer condition, we finally get the following regressive LS-SVM model: y(x) = α i K(x, x i ) + b. (7) where α i and b refer to the solutions to the linear equations (6). The kernel function K(.,.) has several choices such as polynomial function, radial-basis function, and multilayer perceptrons, etc. In all simulations we employ the radial-basis kernel with the form, K(x i, x j ) = exp ( x i x j 2 ) 2σ 2. Two parameters (γ, α) are considered for the LS-SVM. 3 Simulation Results Mackey Glass equation is a time-delayed differential equation that was first proposed as a model of white blood cell production, [6] αx t s x = 1 + (x t s ) 10 + (1 β)x t s, (8) where α = 0.2, β = 0.1, and s is an adjustable parameter. When s 17, equation (8) yields chaotic behavior with a Fig. 2 Predicting results using LS-SVM. (ii) Parameter γ It is shown in formula (5) that γ can affect the precision of prediction. We thus investigate the variation of the error with different γ. Table 1 Variation of predicting errors with different γ. γ RMSE 10 2 1.11 10 3 10 3 5.50 10 3 10 4 2.70 10 3 10 5 1.30 10 3 10 6 8.30 10 4 10 7 6.58 10 4 10 8 7.75 10 4
1058 XU Rui-Rui, BIAN Guo-Xing, GAO Chen-Feng, and CHEN Tian-Lun Vol. 43 Table 1 demonstrates that the error first decreases with increasing γ, however, it increases again when γ reaches a limit. In order to get the best predicting result, we choose γ = 10 7 for the least error in this problem. γ is also an important parameter to control the complexity of network and the balance between inseparable data. [3] In general, the complexity level of calculation increases with higher γ, and the result of prediction gets better at the same time. Moreover, there will be more inseparable data with increasing γ, which will lead to over-fitting and further make the error larger. Therefore, we need a best γ value to simulation experiment. (iii) Multi-step Prediction Multi-step prediction plays a significant role in testing the capability of prediction. We apply LS-SVM network and neural network in which the chaotic learning mechanism is introduced [7] to multi-step prediction of the nonlinear time series based on Machey Glass equation, then compare the corresponding results for N = 200. Table 2 Results of multi-step prediction with step from 1 to 4. Number of step RMSE of LS-SVM RMSE of chaotic algorithm [11] 1 6.58 10 4 0.021 2 2.0 10 3 0.031 3 4.50 10 3 0.052 4 6.60 10 3 0.146 Table 3 Variation of the error with different parameter N of LS-SVM. N 200 400 600 800 1000 RMSE 6.58 10 4 5.63 10 4 3.35 10 4 2.61 10 4 2.19 10 4 Time (s) 111 244.5 397.6 610.8 817.2 Table 2 and figure 3 show that better results are obtained from LS-SVM than those from the neural network with increasing prediction step. Fig. 3 Comparison between the errors for LS-SVM and those for chaotic algorithm. The dotted line is the errors using LS-SVM, and the dashed line denotes the errors using neural network with chaotic algorithm. (iv) Comparison of Two Approaches Pruning the Support Value Spectrum LS-SVM is a modified version of SVM in a least square sense. In this case the solution is given by a linear system instead of a QP problem. The computational complexity accordingly increases with more training data, which lowers the learning rate of the network. Table 3 displays the effect of the quantity of training data on the prediction time. In order to solve this problem, J.A.K. Suykens [5] proposed an approach that can leave out the least important data from training set. (i) Train LS-SVM based on the initial N support values. (ii) Remove a small amount of points (e.g. 5% of the set) with the smallest values in the sorted α i spectrum. (iii) Retrain the LS-SVM based on the reduced training set. (iv) Go to (ii) unless the user-defined performance index degrades. Though this method has hastened the prediction, it still needs to calculate a big matrix for several times before the ultimate support values are decided, and therefore the rate is limited again. In this work we apply k-means clustering to LS-SVM to pruning the support values (CLS- SVM), [8,9] (i) Take the initial N training data as support values, and at the same time they are the initial centers of clustering (C). (ii) Compute all the distances between the next N training data and the initial centers of clustering. If X i c j is the minimum, put X i into the center c j, then compute the new c j = (X i + c j )/2. (Note that X i is the datum that has been raised dimension.) (iii) Sort all the training data again, then compare the new centers with the ones obtained from step (ii), if no difference, stop; otherwise return to step (ii). We test the two approaches to prune support values by the following methods.
No. 6 Discussion About Nonlinear Time Series Prediction Using Least Squares Support Vector Machine 1059 Non-noisy Nonlinear Time Series Predic- (i) tion Fig. 4 Adjustment of support values and prediction of the time series based on Mackey Glass equation by using CLS-SVM. The dotted line is predicting values and solid line the desired ones. The compared predicting results are shown in Table 4 and Fig. 4. In this process the length of the nonlinear time series is set to 1500, in which 400 data are used to train the network, and the ultimate pruned number of support values is 200. It can be clearly seen from Table 4 that the two methods to prune support values have succeeded in saving the predicting time with comparable errors. The predicting result using CLS-SVM is illustrated in Fig. 4. CLS-SVM obtains the better result in reducing time, while the other is more suitable for decreasing errors. (ii) Noisy Nonlinear Time Series Prediction The capabilities of filtering noise are important to predict practical data. Here we investigate both methods. The noisy time series can be expressed as [7,10] x t = x t + ν t, (10) where ν t = βkµ, β denotes an adjustable parameter that controls the degree of noise, µ is uniformly distributed in region [ 1, 1], k stands for the signal-to-noise ratio, namely the standard deviation of the Mackey Glass time series divides the standard deviation of noise component. We discuss the variation of the noise filtering capabilities for the two approaches with different noise degrees, which is significant to practical data, as displayed in Table 5. Table 4 Comparison of three predicting models. LS-SVM Pruning the least important data CLS-SVM RMSE 5.63 10 4 5.66 10 4 4.8 10 3 Time (s) 244.5 146.6 104.9 Table 5 Comparison of noise filtering capabilities for the two methods with different levels of noise. Noise degree Pruning the least important data CLS-SVM 0.1 0.0280 0.0267 0.2 0.0563 0.0482 0.3 0.0949 0.0766 0.4 0.1355 0.1000 One can find that noise filtering capabilities of CLS-SVM are better than those of the other method, mainly because the internal characters of clustering work, which analyze the structure of data, and find out the inner information. This is the uppermost difference between the two approaches. The data that have been clustered can contain more information of the whole system than those that have not. We employ an RBF kernel with σ = 1.7, γ = 10 7 in all simulations, in which the dimension of reconstruction of the phase space is taken as 7. 4 Conclusion In this work, we apply LS-SVM to nonlinear time series prediction. The precision has been improved by taking the effects of different values of parameter γ into account. Multi-step prediction using LS-SVM that leaves out the least significant support values yields better results than those using artificial neural networks with chaotic learning mechanism. It is particularly mentioned that we introduce clustering into LS-SVM to prune support values. This new approach reduces the number of support values with comparable errors to LS-SVM, and greatly improves the capabilities of filtering noise.
1060 XU Rui-Rui, BIAN Guo-Xing, GAO Chen-Feng, and CHEN Tian-Lun Vol. 43 What we are going to focus on is to improve the clustering method used in LS-SVM to get better predicting result and less predicting time. It is sure that there will be wider foreground for LS-SVM in the prediction of practical data. References [1] Neural Networks in Finacial Engineering, eds. A.N. Referes, Y. Abu-Mostafa, J. Moody, and A. Weigend, World Scientific, Singapore (1996). [2] C. Cortes and V. Vapnik, Machine Learning 20 (1995) 273. [3] Simon Haykin, Neural Networks, Tsinghua University Press, Beijing (2001). [4] V.N. Vapnic, The Nature of Statistical Learning Theory, Springer, New York (2000). [5] J.A.K. Suykens, L. Lukas, and J. Vandewalle, Sparse Least Squares Support Vector Machine Classifiers, in Proceedings of European Symposium on Artificial Neural Networks, Belgium (2000), pp. 37 42. [6] M. Mackey and L. Glass, Science 197 (1977) 287. [7] LI Ke-Ping, Chen Tian-Lun, and Gao Zi-You, Commun. Theor. Phys. (Beijing, China) 40 (2003) 311. [8] Zhang Xue-Gong, Acta Automatica Sin. 26 (2000) 32. [9] Zheng Xin and Chen Tian-Lun, Commun. Theor. Phys. (Beijing, China) 40 (2003) 165. [10] Li Ke-Ping and Chen Tian-Lun, Commun. Theor. Phys. (Beijing, China) 35 (2001) 759. [11] Li Ke-Ping, Ph.D. thesis, Chaotic Neural Networks and Nonlinear Time Series Prediction, Nankai University, p. 30.