A New Look at Nonlinear Time Series Prediction with NARX Recurrent Neural Network. José Maria P. Menezes Jr. and Guilherme A.

Transcription

1 A New Look at Nonlinear Time Series Prediction with NARX Recurrent Neural Network José Maria P. Menezes Jr. and Guilherme A. Barreto Department of Teleinformatics Engineering Federal University of Ceará, Centro de Tecnologia Fortaleza-CE, Brazil October 23-27, 2006

2 Contents

3 Motivation Objectives Theoretical Foundations Time Series Prediction (TSP) Tasks Recurrent Neural Networks (RNN) NARX Architecture Simulations VBR Video Traffic Laser Time Series Conclusion

4 Motivation 1. Long Term Dependence occurs very often in real-world time series (e.g. traffic series). 2. Theory of Dynamical Systems provides the theoretical bases to analyzing nonlinear systems with chaotic behavior. 3. Recurrent Neural Networks are capable of representing arbitrary nonlinear dynamical mappings, such as those commonly found in nonlinear time series prediction. 4. NARX Model is a recurrent neural network capable of modeling efficiently time series with long-term dependences.

5 Objectives of this Work 1. To evaluate the performance of standard dynamic neural networks in difficult time series prediction tasks. 2. To propose a new field of application for NARX networks: prediction of univariate time series with long range dependencies.

6 Theoretical Foundations

7 Time Series Prediction (TSP) Tasks TSP One-step-ahead prediction: Neural network models are commonly used to estimating only the next value of a time series. Multi-step-ahead prediction: If the user is interested in a wider prediction horizon. The model s output should be fed back to the input regressor for a fixed but finite number of time steps. Dynamic modeling: If the prediction horizon tends to infinity, the neural network will act as an autonomous system, modeling the long-term dynamics of the system that generated the oberved time series.

8 Recurrent Neural Networks (RNN) RNN Feedforward MLP-like networks can be easily adapted to process time series through an input tapped delay line (e.g. FTDNN model). Recurrent neural networks (RNN) have local and/or global feedback loops in their structure (e.g. Elman, Jordan and NARX models) [1]. RNN are capable to represent arbitrary nonlinear dynamical mappings, such as those commonly found in nonlinear time series prediction tasks.

9 Recurrent Neural Networks (RNN) Takens Embedding Theorem Takens [3] has shown that the state of a deterministic dynamic system can be accurately reconstructed by a time window of finite length sliding over the observed time series as follows: where x 1 (n) [x(n) x(n τ) x(n (d E 1)τ)] T, x(n) is the value of the time series at time n, d E is the embedding dimension and τ is the embedding delay.

10 Recurrent Neural Networks (RNN) FTDNN Focused Time Delay Neural Network

11 Recurrent Neural Networks (RNN) Elman Network

12 NARX Architecture NARX Model in System Identification Nonlinear Autoregressive with exogenous input (NARX) [2]: y(n + 1) = f [y(n),..., y(n d y + 1); u(n), u(n 1),..., u(n d u + 1)]. = f [y(n); u(n)], where u(n) and y(n) denote, respectively, the input and the output of the model at discrete time n. The parameters d u 1 and d y 1, d u d y, are memory delays.

13 NARX Architecture NARX Neural Network Architecture

14 NARX Architecture NARX Network in Nonlinear Time Series Prediction Using Takens Theorem to build the input regressor: u(n) = [x(n) x(n τ) x(n (d E 1)τ)] T, where we set d u = d E. The output regressor y(n) can be written in two different modes, depending on the training modes of the NARX network: y p (n) = [ x(n),..., x(n d y + 1)], y sp (n) = [x(n),..., x(n d y + 1)], where the P-mode contains d y past values of the estimated time series, while the SP-mode contains d y past values of the actual time series.

15 NARX Architecture Paralell Mode (NARX-P)

16 NARX Architecture Series-Parallel Mode (NARX-SP)

17 Simulations

18 Evaluated Networks NARX-P, NARX-SP, FTDNN and Elman networks. All networks have two-hidden layers and one output neuron. All neurons use the hyperbolic tangent activation function. The standard backpropagation algorithm is used to train the networks.

19 Summary Table - Training Parameters number of neurons number of neurons 1st hidden layer 2st hidden layer learning rate epochs (N h,1 ) p (N h,2 ) TASK 1 2d E + 1 p Nh, TASK 2 2d E + 1 Nh,

20 Performance Evaluation Metric The networks are evaluated in multi-step-ahead prediction tasks. Quantitatively, we compute the Normalized Mean Squared Error (NMSE): NMSE(N) = 1 N N σx 2 e 2 (n) where n=1 N is the horizon prediction, σ x 2 is the sample variance of the actual time series and e(n) = y(n) ŷ(n) is the prediction error at time n.

21 Task 1: Long-term prediction of VBR video traffic Variable bit rate (VBR) video traffic (Jurassic Park) [4]. This video traffic trace was encoded with MPEG-I. VBR video traffic typically exhibits burstiness over multiple time scales [5],[6] sample points, rescaled to the range [ 1, 1] samples for training and 500 samples for testing.

22 VBR Video Traffic Empirical Sensitivity Analysis - 1 Embedding dimension 1 FTDNN Elman NARX P NARX SP 0.8 NMSE Order

23 VBR Video Traffic Empirical Sensitivity Analysis - 2 Number of training epochs FTDNN Elman NARX P NARX SP NMSE Epochs

24 VBR Video Traffic Multi-Step-Ahead Predictions - 1 FTDNN Predicted Original 0 Bits Frame number

25 VBR Video Traffic Multi-Step-Ahead Predictions - 2 Elman Predicted Original 0 Bits Frame number

26 VBR Video Traffic Multi-Step-Ahead Predictions - 3 NARX-SP 0.5 Predicted Original 0 Bits Frame number

27 VBR Video Traffic Task 2: Long-term prediction of chaotic laser intensities Chaotic laser time series: comprises measurements of the intensity pulsations of a single-mode Far-Infrared-Laser NH3 in a chaotic state [7]. Available worldwide since a TSP competition organized by the Santa Fe Institute [8] sample points which have been rescaled to the range [ 1, 1] samples for training and 500 samples for testing.

28 Laser Time Series Dynamic Modeling - 1 FTDNN Elman Network Predicted Original Predicted Original P 0 P Time Time

29 Laser Time Series Dynamic Modeling - 2 NARX-SP Network Predicted Original P Time

30 Laser Time Series Sensitivity Analysis Length of the prediction horizon FTDNN Elman NARX P NARX SP 2 Arv Prediction Horizon (N)

31 Contents Motivation Objectives Theoretical Foundations Simulations Conclusion Laser Time Series Laser Time Series j j Recurrence Plot: original series, NARX-SP, FTDNN, Elman i j j i i i

32 Conclusion

33 Conclusion The results has shown that NARX network can be succesfully applied to complex univariate time series modelling and prediction tasks. The proposed approach consistently outperforms standard neural network based predictors, such as the FTDNN and Elman architectures.

34 References [1] J. F. Kolen and S. C. Kremer, A Field Guide to Dynamical Recurrent Networks, Wiley-IEEE Press, [2] T. Lin, B. G. Horne, P. Tino, and C. L. Giles, Learning long-term dependencies in NARX recurrent neural networks IEEE Transactions on Neural Networks, vol. 7, no. 6, pp , [3] F. Takens, Detecting strange attractors in turbulence, in Dynamical Systems and Turbulence, D. A. Rand and L.-S. Young, Eds. 1981, vol. 898 of Lecture Notes in Mathematics, pp , Springer. [4] O. Rose, Statistical properties of MPEG video traffic and their impact on traffic modeling in ATM systems, in Proceedings of the 20th Annual IEEE Conference on Local. [5] J. Beran, R. Sherman, M. S. Taqqu, and W. Willinger, Long-range dependence in variable-bit-rate video traffic, IEEE Transactions on Communications, vol. 43, no. 234, pp , [6] D. Heyman and T. Lakshman, What are the implications of long-range dependence for VBR video traffic engineering, IEEE/ACM Transactions on Networking, vol. 4, no. 3, pp , [7] U. Huebner, N. B. Abraham, and C. O. Weiss. Dimensions and entropies of chaotic intensity pulsations in a single-mode far-infrared NH3 laser. Physical Review A, 40(11): , [8] A. Weigend and N. Gershefeld. Time Series Prediction: Forecasting the Future and Understanding the Past. Addison-Wesley, Reading, 1994.

35 A New Look at Nonlinear Time Series Prediction with NARX Recurrent Neural Network