Evaluating nonlinearity and validity of nonlinear modeling for complex time series

Transcription

1 Evaluating nonlinearity and validity of nonlinear modeling for complex time series Tomoya Suzuki, 1 Tohru Ikeguchi, 2 and Masuo Suzuki 3 1 Department of Information Systems Design, Doshisha University, 1-3 Tatara-Miyakodani, Kyotanabe, Kyoto , Japan 2 Graduate School of Science and Engineering, Saitama University, 225 Shimo-Ohkubo, Sakura-ku, Saitama-city , Japan and 3 Graduate School of Science, Tokyo University of Science, 1-3 Kagurazaka, Shinjuku-ku, Tokyo , Japan (Dated: July 9, 2007) Even if an original time series exhibits nonlinearity, it is not always effective to approximate the time series by a nonlinear model because nonlinear models have higher complexity from the viewpoint of information criterion. Then, we proposed two measures to evaluate both nonlinearity of a time series and validity of nonlinear modeling to it by nonlinear predictability and information criterion. Through numerical simulations, we confirmed that the proposed measures work very well for detecting nonlinearity of an observed time series and evaluating the validity of the nonlinear model. The measures also have a robustness against observational noises. We also analyzed some real time series: the number of chickenpox and measles patients, sunspots numbers, the five Japanese vowels, and chaotic laser. we can confirm that the nonlinear model is effective for the Japanese vowel /a/, the measles patients and the chaotic laser. PACS numbers: r, a, Tp I. INTRODUCTION Analysis methods based on the linear theory, such as frequency-spectrum analysis, are used for prediction and control of complex phenomena[1]. However, discoveries of deterministic chaos indicate that linear modeling is not necessarily suitable to analyze such complex behavior. When we observe complex phenomenon as a time series, we are often asked to determine whether the phenomenon is linear or nonlinear, or possibly chaotic, only from the time series. Evaluating nonlinear predictability is one of the popular methods to identify deterministic chaos from the viewpoint of sensitive dependence on initial conditions[2]; if the time series is chaotic, prediction accuracy worsens exponentially. Although nonlinear prediction methods are indispensable for nonlinear time series analysis, nonlinearity of the time series does not always imply the validity of applying a nonlinear model to the time series. This is because nonlinear models are more complex and require a greater number of parameters than the linear model. This fact indicates that it is very important to consider both the evaluation of nonlinearity of the time series and the validity of application of the nonlinear modeling to the time series. However, conventional studies(ex. Ref.[3]) discussed either the nonlinearity or the validity so that these methods could not recognize the case that the nonlinear modeling is excessive even if the data have nonlinearity. In this paper, we propose two measures. The first measure quantifies the nonlinearity of the time series by comparing nonlinear-prediction errors with an optimum linear-prediction error using a statistical inference of cross-validation (CV) method[4]. Thus, we estimated how the nonlinearity of the time series contributes to the improvement of prediction accuracy[5]. However, nonlinear modeling generally increases the modeling complexity. Thus, nonlinearity does not always imply efficiency of a model. In other words, the modeling efficiency depends on a trade-off between complexity of the model and accuracy of data fitting. To discuss the efficiency, we introduced information criterion, or minimum description length (MDL)[6 9] to propose the second measure. The second measure evaluates the advantage of a nonlinear model to an optimal linear model by MDL. Different from conventional studies[3], we evaluate nonlinearity of the data and validity of nonlinear modeling by comparing nonlinear modeling and linear modeling in terms of the measures. To confirm the validity of the two proposed measures, we adapt the proposed measures to a mathematical model of the Rössler system[18] with several observational noises. As a result, we confirm that the proposed measures are valid not only for testing nonlinearity of the original data but also for quantifying the efficiency of nonlinear modeling to the optimal linear modeling even if the observed time series is contaminated by heavy observational noise. In addition, in terms of application of the proposed measures to real data analysis, we analyze NH 3 - far infrared (FIR) laser[10], annual sunspot number,the number of measles and chickenpox patients[2], and the Japanese vowels[11, 12]. II. THE PROPOSITION OF OUR MEASURES We first draw an outline of the linear and nonlinear modeling we used in this paper. To examine whether the original data have nonlinearity, it is natural to compare

2 2 fitting accuracy by a linear model and that by a nonlinear model. By the Takens embedding theorem[13] and its extension[14], we denote a point in a reconstructed state space as X(t) = {x(t), x(t τ),, x(t (d 1)τ)}. Here, d is an embedding dimension and τ is a delay time. Constructing a model corresponds to estimation of a function F in x(t) = F [X(t τ)]. If we consider X(t τ) as an input set and x(t) as an output, then the linear modeling implies an approximation of the function F with a hyperplane, that is an auto regressive (AR) model[1], and the nonlinear modeling with a hypersurface. In this paper, we use the radial basis function (RBF) network[7 9, 15] as a nonlinear part of the nonlinear model. The RBF(d, m) network is denoted as follows: x(t+τ) = + dx a i x(t (i 1)τ)+a 0 i=1 mx j Φ( X(t) C j )+η(t), (1) j=1 Φ( X(t) C j ) = exp( α j X(t) C j 2 ), η(t) N(0, 2 ), (2) where Φ is an RBF that forms a hypersurface, C j is a center point of Φ. Namely, these terms locally bend the hyperplane to form a hypersurface just like pushing a soft film at C j, and α j = 1 Xd+1 C j C i, where C i is the d + 1 i=1 i th nearest point of C j ; α j corresponds to a pressure to locally bend the hyperplane. Thus, the RBF model is intuitively a simple extension of the AR model. The modeling implies the estimation of the sets: {a i } d i=0, { j} m j=1, {α j } m j=1 and {C j} m j=1. The parameters of the model were estimated by a least-mean-squared error fitting. The fitting error corresponds to 2 in Eq. (2). If m = 0, Eq. (1) is reduced to the AR(d) model. In general, a linear model is much simpler than a nonlinear model. Moreover, let us consider an optimal model. Though we used the RBF terms as the nonlinear part in Eq.(1), the chosen RBF model is just an approximation, and no one can justify this approximation is sufficient, because no one has correct information of a nonlinearity class of the data. In addition, even if the information is given explicitly, finding an optimal model in the class needs a huge amount of computation, possibly it becomes an NP-complete problem. However, we can examine the advantage of nonlinear model if we give a first priority to optimize AR(d) linear terms in Eq.(1) for fitting linear noise and then the RBF terms are optimized to fit nonlinear noise which linear terms could not fit. We propose that the centers of the RBF network C j (j = 1,, m) were determined as the worse fitting points by the optimal AR(d) model. If the model fitting at X(T ) is the worst, a new center point is located at X(T ) and a new RBF is set at X(T ). Then, this process was repeated and the errors were calculated. By estimating the reduction of information criteria, we can examine the advantage of nonlinear modeling. Then, if we do not consider the modeling complexity, that is, the number of modeling parameters, we can use as many RBFs as required to compose a complex hypersurface for fitting the data to a model. If the fitting accuracy is improved by increasing the number of RBFs, it can be concluded that the original data has nonlinearity. To estimate the fitting errors by each model, we used the CV method[4], which is one of the resampling schemes[16]: the original data x(t) are divided into k (k = 1, 2,, K) parts, then, a part is considered as testing data to estimate a modeling accuracy and the remaining parts are considered as learning data for its modeling. For a test, we predicted the testing part by the AR(d) model or by the RBF(d, m) network estimated from the learning data of the other K 1 parts. Then, we can get a prediction accuracy. Next, changing the testing part to each of remaining parts and considering the other parts as each learning part as well, we estimated these prediction accuracies k 2 (k = 1,, K). And then we considered their mean value as a final modeling error 2 = 1 P K K k=1 2 k. By this method, we estimated a general fitting error without overfitting the original data. In our study, we set K = 2 for the sake of simplicity. Using this method, we selected d for the optimum AR model and m (d) for the RBF network. To quantify the nonlinearity of the original data, we proposed the first measure as follows: E(d) = 2 RBF(d,m (d)) AR(d 2, (3) ) where is the fitting error by the RBF network for the case m = m (d), and AR(d 2 ) is the fitting error by the optimum AR model for the case d = d. In Eq. (3), we could use the same embedding dimension d for the AR model and the RBF network as a simple comparison of the linear and the nonlinear models. However, we compared each optimum model based on the complexity of the model. Moreover, when d is unsuitable value to reconstruct an attractor from the observed time series, it is possible that the linear terms and the nonlinear terms in Eq.(1) interfere each other, for example, the nonlinear terms model linearity of the time series. Moreover, the non-uniform embedding is proposed in Ref.[7 9] to reconstruct an attractor more accurately in a state space. It is possible that the non-uniform embedding reduces the problem of interference between linear and nonlinear models. We aim to discuss the possibility as a future work. However, if the time series does not have any nonlinearity, any nonlinear model must not be more accurate than the optimum linear model. That is, it is satisfied that AR(d 2 ). On the other hand, even if the observed time series has nonlinearity and the linear terms model nonlinearity, the linear model cannot become more accurate than the nonlinear model. Then, it is satisfied that AR(d 2 ). Thus,

3 3 by examining whether the measure E(d) is smaller than one, we can evaluate whether the original time series has nonlinearity. In addition, although we cannot eliminate the modeling interference completely, we can minimize it by changing the embedding dimension d and can detect the largest nonlinearity as E(d). Even if an original time series exhibits nonlinearity, employing a nonlinear model is not always a viable option as it increases modeling complexity[17]. Thus, nonlinearity does not always imply validity of nonlinear modeling. To measure the balance between the complexity and the fitting error of a model, we adopted a major criterion, namely, the MDL[6 9]: M(n) = N log 2 + n log N, (4) where N is a data length and n is the number of modeling parameters. In this paper, to estimate n, we removed ineffective terms whose contribution rates are smaller than 1[%]. The contribution rate of each term of Eq.(1) is defined as a ratio of the difference between the residual of the estimated model and each modeling residual estimated by removing a term from the estimated model. Here, if we remove a linear i th term, a new modeling residual is denoted as i. If we remove a nonlinear j th term, a new modeling residual is denoted as j. Then, the contribution rate of the i th term of Eq.(1) is defined as the ratio of i (= i 0) to P d i i. The contribution rate of the nonlinear j th term is defined as the ratio of j to P m j j. If we do not introduce the contribution rate, we cannot avoid a possibility that a higher order AR model where all terms are non-zero, because the MDL counts every term equally regardless of its contribution. Then, the number of parameters of an optimum model was determined by minimizing the criteria. For the AR(d) model, M(n 1 (d)) with n 1 (d) = d+1 φ 1 (d) was minimized to decide the optimum embedding dimension d, where φ 1 (d) is the number of removed linear terms. For the RBF(d,m) network, M(n 2 (d, m)) with n 2 (d, m) = (d + 2)(m φ 2 (d)) + d + 1 φ 1 (d) was minimized to decide the optimum number of RBFs m (d), where φ 2 (d) is the number of removed nonlinear terms. To quantify the efficiency of nonlinear modeling and estimate the optimum embedding dimension, we used a different strategy from Refs.[7 9]: P (d) = M(1) M(n 2(d, m (d))) M(1) M(n 1 (d, (5) )) where M(1) is the most simple model, that is, d = 0 and m = 0. Equation (5) implies an improvement ratio by each optimum model. As the measure P (d) is larger than one, nonlinear modeling will be more suitable for the original time series from the viewpoint of modeling efficiency. The MDL considers all the term of an estimated model in a same weight. Then, even if the contribution rate of a term is small, it is counted equally. On the other hand, the CV method does not have the concept of the number FIG. 1: (Color online) Estimation of (a) the nonlinearity with Eq. (3) and (b) the efficiency of the nonlinear modeling with Eq. (5), for the mathematical model and its FT surrogate. of terms for modeling, and positively adopts the terms which can improve the fitting accuracy of the modeling even if the size of the model becomes very large. Namely, the CV method directly confirm whether a hypersurface (nonlinear model) or a hyperplane (linear model) fits data more accurately with avoiding the overfitting problem. In other words, the MDL estimates the efficiency of model from the viewpoint of Occam s razor, and the CV method reveals hidden nonlinearity in the original data. III. THE VALIDITY OF THE PROPOSED MEASURES To confirm the validity of the proposed measure, we applied the two measures to the first variable of the Rössler equations[18]: ẋ = y z, ẏ = x + ay, ż = b + z(x c), (a = 0.36, b = 0.4, c = 4.5) as a nonlinear time series, where we set the data length N = 8, 000. Then, we normalized the time series x and set τ = 15, which was decided by the auto-correlation function of the observed time-series. We also used the Fourier transformed (FT) surrogates[19] of the Rössler equations for producing a linear time series. The results are shown in Fig. 1. For the linear data (FT surrogate: dashed dotted lines), because E(d) is always larger than one and P (d) is always smaller than one (dotted-dashed lines in Fig. 1), no nonlinearity and no validity of the nonlinear model is confirmed. Solid lines with stars, crosses, pluses and closed circles denote the results of the examination of the robustness of the proposed measure to observational noise. We added Gaussian noise of R[dB] signal-to-noise ratio to the Rössler equations. Here, we defined R[dB] = 10 log σx/σ 2 2, where σx 2 and σ 2 are the variances of x and. In the case of nonlinear data, we found the regions where E(d) < 1 and P (d) > 1: the data have nonlinearity and nonlinear modeling is more effective. Besides, we evaluated an optimum embedding dimension for modeling at which P (d) is maximized. Importantly, in the case of the noisy Rössler data with a large amount of observational noise, the proposed measures can es-

4 4 FIG. 2: (Color online) The same as Fig. 1, but for the real data described in the text. timate the degree of nonlinearity and the validity of nonlinear model of the original data embedded in d- dimensional state space according to the degree of how much E(d) and P (d) leave one. It is natural that these measure are more close to one as the amount of noise is larger because nonlinearity are weakened by linear noise. It has been reported that in the case of modeling the data disturbed by observational noise with the least squares parameter estimation, nonlinear model cannot be estimated accurately, an estimated nonlinear model tends to have some extra terms having much smaller coefficients[22, 23]. One of the reasons why we can obtain reasonable results as shown in Fig. 1 is that contribution rates introduced above might work effectively. As a future work, we need to examine the influence of the model degeneracy problem on the proposed methods in detail and discuss how to clear the problem from different viewpoints. IV. APPLICATION TO REAL DATA ANALYSIS In terms of application to real data, we analyzed NH 3 - FIR laser (N = 1, 000)[10], first difference of the number of chickenpox patients (N = 533)[2], first difference of the number of measles patients (N = 432)[2], and the sunspots annual numbers (N = 294) by τ = 1, which is again decided by the auto correlation of the data. The results are shown in Fig. 2. We confirm that NH 3 - FIR laser and measles patients data exhibit nonlinearity because E(d) < 1, and the nonlinear modeling was more effective because P (d) > 1. Moreover, appropriate embedding dimensions are small with a maximum value of P (d), as shown in Fig. 2(b). As d is larger, E(d) and P (d) are more close to one. That is, the nonlinearity and the validity of nonlinear model weaken. On the other hand, we could not confirm nonlinearity for the chickenpox patients data because E(d) 1 for all d. Besides, the validity of nonlinear modeling is not confirmed because P (d) 1 for all d, especially in small d. Although the above results are consistent with the conventional results, the application to sunspot numbers reveals an interesting result. The results for the sunspots indicate that the time-series has nonlinearity, because FIG. 3: (Color online) Reconstructions of the five Japanese vowels (N = 8, 000): (a) the vowel /a/ (τ = 4), (b) the vowel /i/ (τ = 15), (c) the vowel /u/ (τ = 13), (d) the vowel /e/ (τ = 10), and (e) the vowel /o/ (τ = 8) [12]. E(d) < 1 (2 d 8). However, the validity of its nonlinear model is not confirmed because P (d) is always less than one. Finally, we analyzed five Japanese vowels[12]: /a/, /i/, /u/, /e/, and /o/ shown in Fig. 3. The data length is N = 8, 000. Results are shown in Fig. 4. Conventional studies[11, 12, 20, 21] reported that the Japanese vowels have a nonlinear fluctuation, which is an essential property of their naturalness. These studies indicate that it is produced by a nonlinear, possibly chaotic dynamics. In Fig. 4(a), we have the same results because E(d) < 1, especially in large d. However, we could not confirm the validity of the nonlinear model for the vowels /i/, /u/, /e/, and /o/ because P (d) is always less than one for these data as shown in Fig. 4(b). Only the vowel /a/ shows P (d) larger than one in large d. V. CONCLUSIONS In this paper, we proposed two measures: the first one (Eq. (3)) does not consider complexity but nonlinearity of data, which allows us to use as many modeling parameters as required to fit a model to the data. If the fitting accuracy is improved, the original data have nonlinearity. On the other hand, the second measure (Eq. (5)) considers complexity and efficiency of a model from the viewpoint of information criterion. We confirm the efficiency of such proposed measures by numerical

5 for the proposed methods. Moreover, Ref.[22] indicates the so-called error in variables problem that least square parameter estimation for nonlinear modeling has significant bias. The residual 2 in Eq.(2) may not be estimated accurately. As a result, selected optimal model by information criteria often tends to be over parametrized[23]. As future works, it is important to investigate the influence of such problems on the proposed method in order to develop more effective algorithm. 5 FIG. 4: (Color online) The same as Fig. 1, but for the five Japanese vowels. simulations. In addition, the measures can support the existence of observational noise and are useful for deciding the optimum embedding dimension for a compact modeling. In fact, we have discussed several examples of real data which have a property that nonlinear modeling is not always suitable even if the data have nonlinearity. Finally, as future works, it is important to discuss the availability of non-uniform embedding[7 9] Acknowledgments The authors thank George Sugihara for providing data on measles and chickenpox patients, and thanks the reviewers of Physical Review E for giving us several valuable comments. The research of T.I. was partially supported by a Grant-in-Aids for Scientific Research (C) (nos and ) from the Japan Society for the Promotion of Science. [1] P. J. Brockwell and R. A. Davis, Introduction to time series and forecasting, Springer-Verlag, [2] G. Sugihara and R. M. May, Nature 344, , [3] H. Tong, Non-Linear Time Series: A Dynamical System Approach, Oxford university Press, [4] D. M. Allen, Technometrics 13, , [5] T. Miyano, et al., Physica D 135, , [6] J. Rissanen, Stochastic complexity in statistical inquiry, World Scientific, Singapore, [7] K. Judd and A. Mees, Physica D 82, , [8] M. Small and K. Judd, Physica D 117, , [9] T. Nakamura, K. Judd, and A. Mees, Int. J. of Bifurcation and Chaos 13(5), , [10] Time Series Prediction: Forecasting the Future and Understanding the Past, eds. A. S. Weigend and N. A. Gershenfeld, Addison-Wesley, [11] I. Tokuda, et al., J. Acoust. Soc. Am., 110, , [12] T. Ikeguchi and K. Aihara, Journal of Intelligent and Fuzzy Systems 5(1), 33 52, [13] F. Takens, Detecting strange attractors in turbulence, in Dynamical System and Turbulence, Lecture Notes in Mathematics, 898, eds. D. A. Rand and L. S. Young, , Springer Berling, [14] T. Sauer, J. A. Yorke, and M. Casdagli, Journal of Statistical Physics 65, , [15] A. Mees, Int. J. of Bifurcation and Chaos 3(3), , [16] B. Efron, Ann. Statist., 7, 1 26, [17] M. Small, et al, Physical Review E 66(6):066701, [18] O. E. Rössler, Physics Letters 57A(5), , [19] J. Theiler, Physics Letters A 155(8,9), , [20] T. Ifukube, et al., J. Acoust. Soc. Jpn., 47, , [21] T. Miyano et al., Int. J. Bifurcation and Chaos 10, , [22] P. E. McSharry and L. A. Smith, Physical Review Letters 83(21), , [23] K. Judd and T. Nakamura, Chaos 16:033105, 2006.