Analyzing the State Space Property of Echo

Transcription

1 Proceedings of International Joint Conference on Neural Networks, Montreal, Canada, July 31 August 4, 25 Analyzing the State Space Property of Echo State Networks for Chaotic System Prediction AbstractFor chaotic system prediction, ESNs (Echo State Networks) are realization of neural state reconstruction, in which the reconstructed state variable is from the internal neurons' activation, rather than the delay vector obtained from delay coordinate reconstruction. In the framework of the neural state reconstruction, some quantitative analyses can be further made on the issues such as the network structure configuration and initial state determination. Based on the simulation study on chaotic data from Chua's circuit, it is shown that the ESN is a nonminimum state space realization of the target time series, and the initial state can be freely chosen in the training process, and in the phase of prediction, ESN needs to know where the prediction begins by being set a proper initial state through a process of teacher forcing. Index TermsEcho State Networks, Chaotic Time Series, State Space 1. INTRODUCTION In realworld system, lots of time series from practical problems belong to nonlinear chaotic time series. It has been proved in practice that the linear models can not be used to describe nonlinear chaotic time series, therefore many efforts have been given to the investigation of nonlinear model of the measured time series. In practice, however, the nature and structure of the state space is obscure and the actual variables that contribute to the state vector are unknown or debatable. For a given measurements of one component of state vector in a dynamic system, Takens' theorem implies that for a wide class of deterministic system there exists a diffeomorphism (onetoone differential mapping) between a finite window of the time series and underlying state of the dynamic system which give rise to the time series [1]. This implies that there exists, in theory, a nonlinear autoregression function which models the series exactly. Neural network is a universal approximator, so neural network can be used for the modeling and prediction of the chaotic system. There are dozens of literatures based on this idea of combining theory of delay coordinate embedding and neural network, for example, used in feedforward neural network [2][3][4], recurrent neural network [2][5][6], and Jianhui Xi, Zhiwei Shi, Min Han, Member, IEEE School of Electronic and Information Engineering, Dalian University of Technology, Liaoning, 11623, China minhan@dlut.edu.cn support vector machine [7][8], etc. Common character of this kind method is that the structure of the neural network is closely related with the embedding parameters, such as embedding dimension and delay time. As an alternative to Takens' embedding theorem, some efforts have been attached to modeling a chaotic system without the reliance of the embedding parameters [913]. These methods try to reconstruct the time series directly into a neural state space rather than leaning from a reconstructed trajectory by Takens' theorem. The technique used, at present, is to build a recurrent neural network which produces the target time series. ESN (Echo state network) [13] is a kind of recurrent neural network which produces target time series, once it is trained on observed time series. Echo state networks provide a new training approach to recurrent neural network, in the training process of ESN, the determination of optimal output weights becomes a linear, uniquely solvable task of output error minimization. Similar learning technique is also used by "Liquid State Machines" [14], in which target dynamics of interest are also read out by trainable mechanisms. Firstly, this paper highlights the conception of the neural state reconstruction, as an alternative to the delay coordinate reconstruction. We address the differences of the two types of reconstructed state variables: the state using delay coordinate is come from "time", and the state in the neural state space is come from "space". Secondly, we use ESN to identify the chaotic system by the observed data measured from a real realization of a Chua's circuit. The rich dynamical behavior of Chua's circuit was confirmed by computer simulation [15] and experiment [16]. Complex time series produced by Chua's circuit is a benchmark problem for system identification [17][18] and the time series prediction [19] and still a challenge for the neural network community [2]. The differences from the existing method are: First of all, the data is the measured data rather than simulated data [9][11][19][2], and secondly only one variable is available from the measurement instead of knowing all the state information of the circuit [9][I1], and then the initial state of the circuit is also unknown. Lastly, we view the trained ESN from a framework of state space. Compare with the state variables in the circuit, the state variables in the trained ESN is rather huge and IEEE 1412

2 redundant, we will show that any three state variables in the well trained ESN has a similar space structure as the phase space structure obtained from the circuit or from the delay coordinate embedding. Another interest is the analysis of the initial state of the well trained ESN. Because chaotic system is initial state sensitively, focus is on the proper initial state of the ESN when it is used to simulate and predict a chaotic time series. The organization of this paper is as follows: Section II highlights the conception of neural state reconstruction as an alternative to delay coordinate reconstruction. In Section I1I, ESN is used to identify the chaotic data from Chua's circuit and the prediction result is given by the detection of the two transitions between the two scrolls. Section IV is our further analysis of the trained ESN, including the nonminimum realization property of the ESN and the initial state determination. Section V is the conclusions. II. DELAY COORDINATE RECONSTRUCTION AND NEURAL STATE SPACE RECONSTRUCTION According to Takens' theorem, the geometric structure of the multivariable dynamics of the system can be unfolded from the observable ya,k) in a Ddimensional space constructed from the new vector: SR(k)=[yd(k), yd(k)..*yd (k(d)r)]t (1) where r is a positive integer called the normalized embedding delay. That is, given the observable yxk) for varying discretetime k, which pertains to a single component of an unknown dynamical system, dynamic reconstruction is possible using the Ddimensional vector SR(k) provided that D > 2n + 1 where n is state dimension the system. The procedure for finding a suitable D is called embedding, and the minimum integer D that achieves dynamic reconstruction is called the embedding dimension. Based on Takens' theorem, neural network can be trained as a onestep predictor to identify the unknown mapping f: RD + RI, which is defined by Yd(k + 1) = f(sr (k)) (2) Once it is determined, the evolution SR(k) Yyd(k +1) becomes known. The intention of the neural state space reconstruction is to build up a dynamic neural system to simulate system output yd(k) of the system, and the neural system can be generally written as: X(k + 1) = NN, (X(k), X(O)) y(k) = NNY (X(k)) where X(k) is the internal state variables of the neural system. It is well known that a dynamic system can be described fundamentally by state [21]. The state of a dynamic system is defined as a set of quantities that summarizes all the (3) information about the past behavior of the system that is needed to uniquely describe its future behavior, except for the pure external effects arising from the applied input (excitation). To properly represent a dynamic system one has to determine how the internal state transit from the previous one to the current one, and how current system output is measured from the internal state. From equation (3) we know that NNW defmes the state equation and NNy defines the measurement equation. Ify(k) in neural system (3) can simulate ya,(k), X(k) is called a reconstructed neural state variables. The differences between the delay coordinate reconstruction and the neural state space reconstruction are as follows: 1) The state variables reconstructed have different time or space components, the state variables SR(k) is composed of time components from Yd (k (D I)r) to Yd (k), however, the neural state variable X(k) only have the space components at time k. 2) The state variables reconstructed have different physical meaning, the state variables SR (k) is composed of the history values of the observed time series, however, the state variables in the neural state space are the activation values of the internal neurons, which have nothing direct relationship with the observed output. III. USING ESN TO LEARN CHAOTIC DATA FROM CHUA'S CIRCUIT Chua's circuit is famous device with nonlinear character. The nonlinear element in this circuit is the twoterminal piecewiselinear resistor denoted "Chua's diode". The equations governing the circuit dynamics are: C, dv' (V2RVI) id(vi) dv2 (VI V21) dt R L dil = V dt 2 where vi is the voltage across capacitor C,, il is the current through the inductor and the current through Chua's diode is given by Imv + Bp (mo ml) v. < B, id (vim) = I vm B_ m Iv I < Bp (5) [ mov, + Bp (m, mo ) v, > Bp The data used in the simulation were collected using a digital oscilloscope [18]. Of particular interest in this paper will be the double scrolls attractor generated by the circuit. The data in Fig. 1 is the voltage across capacitor Cl, which were sampled at TS=12us and 5 samples were recorded (4) 1413

3 with a resolution of 13 bits. For details about the data refer [18]. After the ESN is disconnected form the teacher yd(k) at the prediction origin(=23), the iterated ESN structure can be seen in the Fig.3, and the network starts from the initial state X(23). y(k) y~~~~~(k + I) $ m 3= * Fig. I k The Measured Data Sets In timeseries prediction, the prediction origin (denoted by ) is the time from which the prediction is generated. The time between the prediction origin and the predicted data point is the prediction horizon. The training sequence is 123 in the samples, and so the prediction origin is 23. Our interest is focused on the two transitions before 24 and around 25(the two transitions between two scrolls after 23 as shown in Fig.2), and it is checked if the ESN network can detect the transitions and make a proper prediction about future. Fig.3 The Structure of ESN for the Iterative Prediction Fig.4 is the continue prediction after the prediction origin (=23). As it is shown in the figure, the trained ESN detects the transitions between the two scrolls well, the accurate prediction is available until 2 steps after the prediction origin (until 25 steps of the original sequence). It is also interesting to give the corresponding logi of the absolute prediction error in Fig.5, where yadk) is the target sequence and y(k) is the sequence generated by ESN after the prediction origin. ;g " O Fig.2 The prediction origin and scrolls after the prediction origin the transition between the two A network of 5 units was created with spectral.93, connectivity 1/2, output feedback connection weights sampled from uniform distribution over (4, 4). The refined version method was used for the learning of the connection weight WO. The refined version method for the network learning can see [13], in our simulation, the refined version of the learning method is more stable than the method with trick of "wobbling". For the prediction task, the training sequence are teacherforced on the trained network before the prediction origin, then the network is left running freely and the network's continuation is compared to the true continuation k Fig.4 The Comparison of the Observed Output and the Predicted Output after the Prediction Origin 1414

4 ~z, 2 I 3.,. where X(k) is the internal state variables of the ESN at time k, and y(k) is the network output at time k, a(.) is a sigmoid function. A. The redundancy ofthe state variables in the ESN By Takens' theorem, a phase space is reconstructed from the observed v1, as is shown in the Fig.6. We know that it is similar with the portrait ofx(k)(= [v, (k); v2 (k); il (k)]) k Fig.5 The Prediction Error log1(abs(yd(k) y(k))) Versus the Prediction Horizon after the Prediction Origin IV. THE ANALYSIS OF THE STATE SPACE IN THE TRAINED ESN For each internal unit xi (X =[x,x2,...,xn]t) there exists an "echo function" e, [22], such that, if the network has been driven by an signal y(k) for a long time, the current state can be written as xi (k) = ei (y(k), y(k 1), y(k 2),...). Consider the NAR (Nonlinear AutoRegressive) system, the output of ESN is written as: y(k) = f(y(k 1), y(k 2),...) _ E, wiei (y(k 1), y(k 2),) (6) From equation (6), we know that the learning approach of ESN is to approximate the nonlinear system functionfby a linear combination of the echo functions, where the linear combination weights are the trained output connection weights. NAR system can be converted into a state space model. For the Chua's circuit, we know that the state dimension is three, and there is discrete map with the form (the discrete form of system shown as equation (4)): X(k + 1) = F(X(k)) Yd(k) = h(x(k)) where X(k) = [v, (k); v2 (k); il (k)], h(x(k)) is the observed fimction, in our simulation, Yd (k) = h(x(k)) = v,. The data are stationary in the sense that while the measures were taken, and there is no reason to believe that the circuit dynamics and parameters changed in any significant way, so F(.) and h() are assumed not to change with time. When the ESN is used for the prediction (as shown in Fig.3), the equations govern the ESN can be written as: X(k + 1) = a((w + W * WO) X(k)) (8) y(k) = W. X(k) 3i X t 3 S,, ' /.3I / < X 32/ Fig.6 The reconstruction state space by delay embedding The modeling process is carried out by letting system (8) to simulate system (7), that is to say, system (8) is a realization of system (7). However, the state dimension of the chaotic system (7) is 3, and ESN in our simulation has the dimension of 5. In other word, the ESN realization is a nonminimum realization. xg.4.~5 X^66) Fig.7 The evolution ofx5x145x255 in the neural network 1415

5 The evidence for the nonminimum realization can be checked by the portrait of any three of the 5 internal state variables in the ESN. Fig.7, Fig.8 and Fig.9 are the state evolution of x5x145x255, x9x187x418 and x9x29x495 (xi denotes the ith internal neuron of the trained ESN), respectively. Any of these portraits have the similar structure with the portrait of Fig.6, which comes from the delay embedding..2.1 ' i Fig.8 The evolution ofx9xi87x418 in the neural network A,2~~~~~~~~~~~~~~~~~, ~ ~ ~ ~ ~ Fi Teevltin X X9 i n Fig.9 The evolution of x9x29x495 in the neural network Another evidence for the nonminimum realization can refer[9], where the neural state space is a minimum realization of the system identified. The neural network in[9] is given as: X(k + 1) = W s(v X(k) + 8) (9) where the dimension of X(k) is three according to the Chua's circuit. However, this kind of recurrent neural network can learn and behave like Chua's double scrolls, though the learning of recurrent networks with gradient is i.15 difficult. B. The initial state determination ofthe ESN When the network is used for prediction from the prediction origin, it is run as an autonomous dynamic system. There is lots of evidence that the prediction system dynamics are chaotic. For a chaotic system, an important character is initial state sensitive, that is to say, a small difference in the initial state eventually will end up with a big difference between their states. For ESN, which is the proper initial condition when it is used for the iterative prediction? To this problem, first of all, consider the minimum realization in [9]. The state space realization in [9], however, is aimed to model the three state variables of the Chua's circuit, so the initial state of the neural network must be identical to the initial state of original system. It is usual case when the scalar variable is available, so we are interested to add a measurement equation to the minimum realization in [9], and we have the neural system: X (k+l)=w(vx* (k)+i3) /IN lu) y (k) = C.a(DX*(k)+) Now introduce a transform 1'( ) into the state variables of system (7). X(k) = W'(X(k)) (1 1) and system (7) can be rewritten as: X(k + 1) = V(F(W (X(k)))) (12) y(k) = h(w' (X(k))) To let system (1) approximate system (12) is obvious, if (F(v'())) and h(x'()) are continuous function defined on a compact set. The initial state of system (1) can be chosen freely to model system (7) if we can find a proper transformy() so that X*(O) = X(O) = %V(X()). This result can be easily extended to a nonminimum realization of the neural system. In the training of ESN, there is no worry to choose the initial state. In fact, one can train the ESN from the random initial state. Another question, as shown in equation (8), there is not a hidden layer between X(k +1) andx(k), how can the ESN approximate some general state equation? First of all, when the output layer is a sigmoid function, the transition between X(k +1) andx(k) could be universal; Secondly, it is not necessary to add such a layer because there are so many combination of the state variables to be chosen by the output dynamics, in fact, ESN leams well using linear regression and obtains a low error in the training sequence. Once the network training is completed, the ESN are ready for the prediction task. For a given prediction, the trained ESN need to know where the prediction begins. To make a proper prediction, it is necessary to tell where is the prediction origin (=23 in our simulation) for a given task, because the prediction is start from there. The 1416

6 evolution of the ESN network is based on its state variables, so to tell the prediction origin is to set a proper initial state for the network. In other word, once the initial state in system (8) is proper for the given prediction task, the proper prediction can be made. In our analysis, the initial state is set by the phase of teacher forcing. When the trained connection weights are in place, the network is teacher forced by the target series before the prediction origin : X(k + 1) = G(W X(k) + Wft yd(k)), k < (13) and then the ESN is disconnected form the teacher y,(k) at the prediction origin. In this process, proper initial state is set gradually and then the prediction begins: the output signal y(k)(k.) is created from the internal neuron X(k) through the weight WO, by y(k) = W X(k). Conversely, the internal signals were "echoed" from that output signal through the fixed output feedback connections. V. CONCLUTIONS When ESN is used to model nonlinear chaotic time series, it uses a linear combination of the echo function to approximate the nonlinear system function. This paper views the ESN from a perspective of state space, and the chaotic time series modeling becomes the problem of the neural state space reconstruction. We apply ESN to the problem of identification and prediction of observed data from a real implementation of Chua's double scrolls, and ESN learns the data well and makes a desired prediction performance. It detects the two transitions between the two scrolls. Based on the simulation study on chaotic data from Chua's circuit, we make a further analysis of the trained ESN in the framework of state space. It is shown that the ESN is a nonminimum state space realization of the target time series, and the initial state can be freely chosen in the training process, and in the phase of prediction, ESN needs to know where the prediction begins by being set a proper initial state through a process of teacher forcing. ACKNOWLEDGMENT This research is supported by the project (637464) of the National Nature Science Foundation of China. It is also supported by the project (51392) of the National Natural Science Foundation of China. All of these supports are appreciated. REFERENCES [1] F. Takens, "Detecting Strange Attractors in Fluid Turbulence", in Dynamical Systems and Turbulence, eds. D. Rand and L.S. Young, Springer, Berlin, 1981 [2] S. Haykin and J. Principe, "Making sense of a complex world," IEEE Signal Processing Magazine, Vol. 15, No.3, pp. 6668, [3] E. A. Wan, "Time series prediction by using a connectionist network with internal delay lines (data set A)," Proceedings of the NATO Advanced Research Workshop on Comparative Time Series Analysis, May , pp , [4] M.R. Cowper, B. Mulgrew, and C. P. Unsworth, "Nonlinear prediction of chaotic signals using a normalised radial basis function network," Signal Processing, Vol. 82, No.5, pp , 22. [5] J.C. Principe, A. Rathie, and J.M. Kuo, "Prediction ofchaotic Time Series with Neural Networks and the Issue ofdynamic Modeling", Int. J. Bifurcation and Chaos, Vol. 2, pp , [6] M. Han, J. H. Xi, S. G. Xu, and F. L. Yin, "Prediction of chaotic time series based on the recurrent predictor neural network," IEEE Transactions on Signal Processing, Vol. 52, No.12, pp , 24. [7] S. Mukherjee, E. Osuna, and F. Girosi, "Nonlinear prediction of chaotic time series using support vector machines," presented at Neural Networks for Signal Processing [1997] VII. Proceedings of the 1997 IEEE Workshop, pp.51152, [8] L. Cao, "Support vector machines experts for time series forecasting," Neurocomputing, Vol. 51, pp , 23. [9] J. A. K. Suykens and J. Vandewalle, "Learning a Simple Recurrent Neural StateSpace Model to Behave Like Chua's Double Scroll," IEEE Transactions on Circuits and Systems IFundamental Theory and Applications, Vol. 42, No.8, pp , [1] M. Han, Z. W. Shi, and W. Wang, "Modeling dynamic system by recurrent neural network with state variables," in Advances in Neural Networks ISNN 24, Pt 2, Vol. 3174, Lecture Notes in Computer Science. Berlin: SPRINGERVERLAG BERLIN, 24, pp [11] B. Cannas, S. Cincotti, M. Marchesi, and F. Pilo, "Learning of Chua's circuit attractors by locally recurrent neural networks," Chaos Solitons & Fractals, Vol. 12, No. I 1, pp , 21. [12] B. Cannas and S. Cincotti, "Neural reconstruction of Lorenz attractors by an observable," Chaos Solitons & Fractals, Vol. 14, No. 1, pp. 8186, 22. [13] H. Jaeger and H. Haas, "Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication," Science, Vol. 34, No.5667, pp. 788, 24. [14] W. Maass, T. Natschlager, and H. Markram, "Realtime computing without stable states: a new framework for neural computation based on perturbations," Neural Computation, Vol. vol.14, no.1 1, pp , 22. [15] T. Matsumoto, "A chaotic attractor from Chua's circuit," Circuits and Systems, IEEE Transactions on, Vol. 31, No.12, pp , [16] G. Q. Zhong and F. Ayrom, "Experimental confirmation of chaos from Chua's circuit," Int. J. Circuit Theory Appl., Vol.13, No. 11, pp.9398, [17] M. H. Petrick and B. Wigdorowitz, "A priori nonlinear model structure selection for system identification," Control Engineering Practice, Vol. 5, No.8, pp , [18] L. A. Aguirre, G. G. Rodrigues, and E. Mendes, "Nonlinear identification and cluster analysis of chaotic attractors from a real implementation of Chua's circuit," International Journal of Bifurcation and Chaos, Vol. 7, No.6, pp , [19] J. McNames, J. A. K. Suykens, and J. Vandewalle, "Winning entry of the K.U. Leuven timeseries prediction competition," International Journal ofbifurcation and Chaos in Applied Sciences and Engineering, Vol. vol.9, no.8, pp , [2] J. A. K. Suykens and J. Vandewalle, "The K.U. Leuven competition data: a challenge for advanced neural network techniques," 8th European Symposium on Artificial Neural Networks. ESANN"2. Proceedings, pp , 2. [21] R. E. Kalman, "A New Approach to Linear Filtering and Prediction Problems," Transaction ofthe ASMEJournal ofbasic Engineering, Vol. 82, pp. 3545, 196. [22] H. Jaeger, "The echo state approach to analysing and training recurrent neural networks" (GMDReport 148, German National Research Institute for Computer Science 21). 1417