Nonlinear singular spectrum analysis by neural networks. William W. Hsieh and Aiming Wu. Oceanography/EOS, University of British Columbia,

Transcription

1 Nonlinear singular spectrum analysis by neural networks William W. Hsieh and Aiming Wu Oceanography/EOS, University of British Columbia, Vancouver, B.C. V6T 1Z4, Canada tel: (64) , fax: (64) Submitted to Neural Networks, April, 21 revised, July, 21 1

2 Abstract Singular spectrum analysis (SSA), a linear univariate and multivariate time series technique, is essentially principal component analysis (PCA) applied to the time series and additional copies of the time series lagged by 1 to K time steps. Neural network theory has meanwhile allowed PCA to be generalized to nonlinear PCA (NLPCA). In this paper, NLPCA is further extended to perform nonlinear SSA (NLSSA): First, SSA is applied to the data, then the leading principal components of the SSA are chosen as inputs to an NLPCA network (with a circular node at the bottleneck), which performs the NLSSA by nonlinearly combining all the input SSA modes into a single NLSSA mode. This nonlinear spectral technique allows the detection of highly anharmonic oscillations, as illustrated by a stretched square wave imbedded in white noise, which shows NLSSA to be superior to SSA and classical Fourier spectral analysis. NLSSA is also applied to the Southern Oscillation Index (a monthly time series of the equatorial Pacific air pressure conditions from 1866 to 2), which reveals a first mode with period of 52 months, and a second mode of 39 months. 2

3 1 Introduction Principal component analysis (PCA) is a classical multivariate statistical technique commonly applied to a set of variables {y i } to reduce the dimensionality of the dataset, or to extract patterns from the data (Mardia et al., 1979). By the 198s, interests in chaos theory and dynamical systems led to further extension of PCA to singular spectrum analysis (SSA): A common class of dynamical systems contains a set of variables, where the time derivative of each variable satisfies a first order differential equation. If there are K variables, it is possible to replace the K first order differential equations by a single Kth order differential equation. Furthermore, one can approximate the information contained in a continuous variable and its (1st to Kth) derivatives by a discrete time series and the same time series lagged by 1,..., K time steps. Thus a given time series and its lagged versions can be regarded as a set of variables {y i } (i = 1,..., K + 1), which can be analyzed by the PCA. This resulting method is the SSA with window L (= K + 1) (Elsner and Tsonis, 1996; Ghil et al. 21). In the multivariate case where there is more than one time series, one can again make lagged copies of the time series, treat the lagged copies as extra variables, and apply the PCA to this enlarged dataset resulting in the multichannel SSA (MSSA) method. For brevity, we will use the term SSA to denote both SSA and MSSA, unless there is confusion. In PCA, a straight line approximation to the dataset is sought which accounts for the maximum amount of variance in the data. There are now several types of neural network (NN) models which will use, instead of the straight line, a continuous curve to approximate the data. The nonlinear PCA (NLPCA) method of Kramer (1991) is capable of extracting 3

4 open curve solutions to approximate the data, but not closed curve solutions. Kirby and Miranda (1996) introduced an NLPCA with a circular node at the bottleneck (henceforth NLPCA.cir), capable of extracting closed curve solutions. The relative merits of NLPCA and NLPCA.cir have been studied by Hsieh (21). Since PCA can be nonlinearly generalized by neural networks, it is also tempting to nonlinearly generalize SSA. The objective of this paper is to develop a nonlinear SSA (NLSSA) method, a fully nonlinear spectral technique, based on neural networks. The outline of this paper is as follows: Sec. 2 gives a concise introduction to SSA, while Sec.3 describes the NLSSA method. In Sec.4, a test problem consisting of a stretched square wave plus white noise is used to illustrate the advantage of NLSSA over SSA (and classical Fourier spectral analysis) in detecting anharmonic periodic signals. NLSSA is also tested with real data, the Southern Oscillation Index, a renowned climate time series from 1866 to 2, in Sec.5. 2 SSA Given a time series y j = y(t j ) (j = 1,..., N), lagged copies of the time series are stacked to form the augmented matrix Y, Y = y 1 y 2 y N L+1 y 2 y 3 y N L y L y L+1 y N. (1) 4

5 This matrix has the same form as the data matrix produced by L variables, each being a time series of length n = N L + 1. Y can also be viewed as composed of its column vectors y j, forming a vector time series y(t j ), j = 1,..., n. The standard PCA (usually by the singular value decomposition) can be performed on Y, resulting in y(t j ) = i x i (t j ) e i, (2) where x i is the ith principal component (PC), a time series of length n, and e i is the ith eigenvector (or loading vector) of length L. Together, x i and e i, represent the ith SSA mode. When the time series contains an anharmonic wave, SSA is often superior to the classical Fourier spectral analysis, as the wave forms extracted from the SSA eigenvectors are not restricted to sinusoidal shapes, and hence can capture an anharmonic wave with fewer modes. In the multivariate case, with M variables y k (t j ) y kj, (k = 1,..., M; j = 1,..., N), the augmented matrix can be formed by letting Y = y 11 y 12 y 1,N L y M1 y M2 y M,N L y 1L y 1,L+1 y 1N.... y ML y M,L+1 y MN. (3) PCA can again be applied to Y to get the SSA modes. 5

6 3 NLSSA Assume SSA has been applied to the dataset, and after discarding the higher modes, we have retained the leading PCs, x(t) = [x 1,..., x l ], where each variable x i, (i = 1,..., l), is a time series of length n. The variables x are the inputs to a feed-forward neural network, mapping through a bottleneck to the output x (Fig. 1). There are two configurations, the first for the NLPCA (Fig. 1a) and the other for the NLPCA.cir (Fig. 1b). Hsieh (21) noted that the NLPCA.cir, with its ability to extract closed curve solutions, is particularly ideal for extracting periodic or wave modes in the data. In SSA, it is common to encounter periodic modes, each of which had to be split into a pair of modes (Elsner and Tsonis, 1996), as the underlying PCA technique is not capable of modelling a periodic mode (a closed curve) by a single mode (a straight line). Thus, two (or more) SSA modes can easily be combined by NLPCA.cir into one NLSSA mode. Also Hsieh (21) pointed out that the general configuration of the NLPCA.cir can model not only closed curve solutions, but also open curve solutions like the Kramer (1991) NLPCA. Because the NLPCA.cir is more general than NLPCA, it is used here to perform the NLSSA. For the NLPCA.cir network (Fig. 1b), we begin by mapping from x, the input column vector of length l, to the first hidden layer (the encoding layer), represented by h (x), a column vector of length m, with elements h (x) k = tanh((w (x) x + b (x) ) k ), (4) where (with the capital bold font reserved for matrices and the small bold font for vectors), W (x) is an m l weight matrix, b (x), a column vector of length m containing the bias parameters, and k = 1,..., m. It is common, but not necessary, to use the hyperbolic 6

7 tangent function as the transfer function (Bishop, 1995). Next, a linear transfer function maps from the encoding layer to the bottleneck layer containing a pair of coupled neurons p and q. The pre-states p o and q o are first calculated as p o = w (x) h (x) + b (x), and q o = w (x) h (x) + b (x), (5) where w (x), w (x) are weight parameter vectors, and b (x) and b (x) are bias parameters. Let r = (p 2 o + q 2 o) 1/2, (6) then the circular node is defined with p = p o /r, and q = q o /r, (7) satisfying the unit circle equation p 2 + q 2 = 1. Thus, even though there are two variables p and q at the bottleneck, there is only one angular degree of freedom from θ (Fig. 1b), due to the circle constraint. The mapping from the bottleneck layer to the decoding layer is h (u) k = tanh((w (u) p + w (u) q + b (u) ) k ), (8) (k = 1,..., m). Finally, the network output is given by x i = (W (u) h (u) + b (u) ) i. (9) When implementing NLPCA.cir, Hsieh (21) found that there are actually two possible configurations: (i) A restricted configuration where the constraint p = q = is applied (with denoting a sample or time mean), and (ii) a general configuration where no constraint on p and q is applied. The general configuration can actually model open 7

8 curve solutions like a regular NLPCA. The reason is that if the input data mapped onto the p-q plane covers only a segment of the unit circle instead of the whole circle, then the inverse mapping from the p-q space to the output space will yield a solution resembling an open curve. We will use the general configuration, which has 2lm + 6m + l + 2 free (weight and bias) parameters. The cost function J = x x 2 is minimized by finding the optimal values of the free parameters. The MSE (mean square error) between the NN output x and the original data x is thus minimized. In Fig.1, only a single hidden layer is used in the mapping from the input layer to the bottleneck, and also in the mapping from the bottleneck to the output layer, since given enough hidden neurons, any continuous function can be approximated to arbitrary accuracy by one hidden layer (Cybenko, 1989). The choice of m, the number of hidden neurons in both the encoding and decoding layers, follows a general principle of parsimony. A larger m increases the nonlinear modelling capability of the network, but could also lead to overfitted solutions (i.e. wiggly solutions which fit to the noise in the data). For nonlinear solutions, we need to look at m 2. If we can penalize the use of excessive weights, we can limit the degree of nonlinearity in the NLPCA.cir solution, thereby reducing the risk of overfitting. This is achieved with a modified cost function J = x x 2 + p ki (W (x) ki ) 2, (1) where p is the weight penalty parameter. A large p increases the concavity of the cost function, and forces the weights W (x) to be small in magnitude, thereby yielding smoother 8

9 and less nonlinear solutions than when p is small or zero. We have not penalized other weights in the network. In principle, w (u) and w (u) also control the nonlinearity in the inverse mapping from (p, q) to x. However if the nonlinearity in the forward mapping from x to (p, q) is already limited, then there is no need to further limit the weights in the inverse mapping. After the first NLSSA mode has been extracted, it can be subtracted from x to get the residual, which can be input again into the same network to extract the second NLSSA mode, and so forth for the higher modes. The nonlinear optimization was carried out by the MATLAB function fminu, a quasi- Newton algorithm. Because of local minima in the cost function, there is no guarantee that the optimization algorithm reaches the global minimum. Hence an ensemble of 3 NNs with random initial weights and bias parameters was run. Also, 2% of the data was randomly selected as test data and withheld from the training of the NNs. Runs where the MSE was larger for the test dataset than for the training dataset were rejected to avoid overfitted solutions. Then the NN with the smallest MSE was selected as the solution. 4 Anharmonic wave example To compare the NLSSA and the SSA, we use an anharmonic wave of the form 3 for t = 1,..., 7 f(t) = 1 for t = 8,..., 28 (11) periodic thereafter. 9

10 This is a square wave with the peak stretched to be 3 times as tall but only 1/3 as broad as the trough, and has a period of 28. Gaussain noise with twice the standard deviation as this signal was added, and the time series was normalized to unit standard deviation (Fig. 2). The time series has 6 data points. SSA with window L = 5 was applied to this time series, with the first eight eigenvectors shown in Fig. 3. The first 8 modes individually accounted for 6.3, 5.6, 4.6, 4.3, 3.3, 3.3, 3.2, and 3.1 % of the variance of the augmented time series y. The leading pair of modes displays oscillations of period 28, while the next pair manifests oscillations at a period of 14, i.e. the first harmonic. The anharmonic nature of the SSA eigenvectors can be seen in mode 2 (Fig. 3), where the trough is broader and shallower than the peak, but nowhere as intense as in the original stretched square wave signal. The PCs for modes 1-4 are also shown in Fig. 4. Both the eigenvectors (Fig. 3) and the PCs (Fig. 4) tend to appear in pairs, each member of the pair having similar appearance except for the phase difference. The first 8 PC time series were served as inputs to the NLPCA.cir network, with the penalty parameter p = 1, and m (the number of hidden neurons in encoding layer) ranging from 2 to 8 in a series of runs. The MSE dropped with increasing m, until m = 5, beyond which the MSE showed no further improvement. The resulting NLSSA mode 1 (with m = 5) is shown in Fig. 5. Not surprisingly, the PC1 and PC2 are united by the approximately circular curve. What is more surprising, are the nonlinear curves found in the PC1-PC3 plane (Fig. 5b) and in the PC1-PC4 plane (Fig. 5c), indicating nonlinear relations between the first SSA mode and the higher modes 3 and 4. In contrast, there was no relation found between PC1 and PC5, as PC5 appeared independent of PC1 (Fig. 5d). The nonlinear relations found are robust, as additional runs were made with reduced noise 1

11 in the data, and the nonlinear relations were even more clearly manifested (not shown). With less noise, nonlinear relations can be found between PC1 and PC5, and even higher PCs. The NLSSA mode 1 loading can be plotted as a function of the lag for various values of θ (Fig. 6), where it is clear that the highly anharmonic nature of the signal is much better revealed here than in the SSA modes 1 and 2 (Fig. 3, top panel). The θ time series (Fig.4, bottom panel) shows the phase generally increasing with time. The NLSSA reconstructed component 1 (NLRC1) is the approximation of the original time series by the NLSSA mode 1. The neural network output x are the NLSSA mode 1 approximation for the 8 leading PCs. Multiplying these approximated PCs by their corresponding SSA eigenvectors, and summing over the 8 modes allows the reconstruction of the time series from the NLSSA mode 1. As each eigenvector contains the loading over a range of lags, each value in the reconstructed time series at time t j also involves averaging over the contributions at t j from various lags. In Fig. 2, NLRC1 (top curve) from NLSSA is to be compared with the reconstructed component (RC) from SSA mode 1 (RC1) (curve c). The anharmonic nature of the oscillations is not revealed by the RC1, but is clearly manifested in the NLRC1, where each strong narrow peak is followed by a weak broad trough, similar to the original stretched square wave. Also the wave amplitude is more steady in the NLRC1 than in the RC1. Using contributions from the first 2 SSA modes, RC1-2 (not shown) is rather similar to RC1 in appearance, except for a larger amplitude. In Fig. 2, curves (d) and (e) show the RC from SSA using the first 3 modes, and the first 8 modes, respectively. These curves, referred to as RC1-3 and RC1-8, respectively, 11

12 show increasing noise as more modes are used. Table 1 compares how well the NLRC1 and the various RCs have retrieved the signal by giving their correlation and root mean square error (RMSE) relative to the stretched square wave signal. Among the RCs, RC1-3 attains the most favorable correlation (.849) and RMSE (.245), but remains behind the NLRC1, with correlation (.875) and RMSE (.225). The stretched square wave signal accounted for only 22.6% of the variance in the noisy data. For comparison, NLRC1 accounted for 17.9%, RC1, 9.4%, and RC1-2, 14.1% of the variance. With more modes, the RCs account for increasingly more variance (Table 1), but beyond RC1-3, the increased variance is only from fitting to the noise in the data. When classical Fourier spectral analysis was performed, the most energetic bands were the sine and cosine at a period of 14, the two together accounting for 7.% of the variance. In this case, the strong scattering of energy to higher harmonics by the Fourier technique has actually assigned 38% more energy to the first harmonic (at period 14) than to the fundamental period of 28. Next, the data record was slightly shortened from 6 to 588 points, so the data record is exactly 21 times the fundamental period of our known signal this is to avoid violating the periodicity assumption of Fourier analysis and the resulting spurious energy scatter into higher spectral bands. The most energetic Fourier bands were the sine and cosine at the fundamental period of 28, the two together accounting for 9.8% of the variance, compared with 14.1% of the variance accounted for by the first two SSA modes. Thus even with great care, the Fourier method scatters the spectral energy considerably more than the SSA method. 12

13 5 Southern Oscillation Index The Southern Oscillation Index (SOI) is defined as the normalized air pressure difference between Tahiti and Darwin. There are several slight variations in the SOI values calculated at various centres; here we use the SOI calculated by the Climate Research Unit at the University of East Anglia, based on the method of Ropelewski and Jones (1987). The SOI measures the seesaw oscillations of the sea level air pressure between the eastern and western equatorial Pacific. When the SOI is strongly negative, the eastern equatorial Pacific sea surface temperatures also become warm (i.e. an El Niño event occurs); when SOI is strongly positive, the central equatorial Pacific becomes cool (i.e. a La Niña event occurs) (Philander, 199). The SOI is known to have the main spectral peak at a period of about 4-5 years (Troup, 1965). For SSA, the window L needs to be long enough to accommodate this main spectral period, hence L = 72 months was chosen for the SSA. The monthly SOI time series from January, 1866 to December, 2, and the SOI smoothed by a 13-month moving average are shown in Fig. 7. Next SSA is performed on the (unsmoothed) SOI, with the first eight eigenvectors shown in Fig. 8. These first eight modes account for 11., 1.5, 9., 8.1, 6.4, 5.6, 3.3 and 2. %, respectively, of the variance of the SOI. The first pair of modes displays oscillations with a period slightly above 5 months, while the next pair of modes manifests a period slightly above 3 months (Fig. 8). The oscillations also display anharmonic features. The four leading PCs are plotted in pairs in Fig. 9, revealing the longer oscillatory time scale in the first pair and the shorter time scale in the second pair. The first 8 PC time series were served as inputs to the NLPCA.cir network, with the 13

14 penalty parameter p = 1, and m ranging from 2 to 6 in a series of runs. The MSE dropped with increasing m, until m = 3, beyond which the MSE showed no further improvement. The resulting NLSSA mode 1 (with m = 3) is shown in Fig. 1. Again the PC1 and PC2 are united by the approximately circular curve. Nonlinear curves found in the PC1-PC3 plane (Fig. 1b) and in the PC1-PC4 plane (Fig. 1c), indicate nonlinear relations between the first SSA mode and the higher modes 3 and 4. Between PC1 and PC5 (Fig. 1d), there was no relation. The NLSSA mode 1 loading as a function of the lag is shown for several values of θ (Fig. 11). The θ 1 time series (Fig.9, bottom panel) shows the phase generally decreasing with time. Next we perform a least squares fit between the NLSSA mode 1 loading and the sinusoidal form s(t) = A sin(ωt + φ) + C, (12) where the fitting parameters are A, ω, φ, and C. The fit was performed separately for θ =, 15, 3,..., 345. After the fitted frequency ω has been averaged over all the θ values, we found an average period of 52 months for the NLSSA mode 1. After the NLSSA mode 1 solution had been removed from the data (i.e. the eight PCs), the residual was input into the same NLPCA.cir network to yield NLSSA mode 2. Again separate runs were made with m varying from 2 to 6. The MSE for m = 3,..., 6 were quite similar, so m = 3 was chosen as the solution. As NLSSA mode 1 had extracted the dominant relation between PC1 and PC2, NLSSA mode 2 (Fig. 12) mainly revealed the next dominant relation, that between PC3 and PC4, 14

15 as shown by the circular relation in Fig. 12c. It also revealed moderate nonlinear relation betwen PC3 and PC2 (Fig. 12b), and a weaker nonlinear relation between PC3 and PC1 (Fig. 12a). The projected solution in Fig. 12d, a gently sloping line, suggests perhaps a very weak linear relation between PC3 and PC5. The NLSSA mode 2 loading as a function of the lag is shown for several values of θ (Fig. 13), revealing more anharmonic forms than the NLSSA mode 1 loading (Fig. 11). The θ 2 time series (Fig.9, bottom panel) shows the phase generally decreasing with time. Fitting the NLSSA mode 2 loading to the sinusoidal form (12), and then averaging over θ, yielded an average period of 39 months. This period is somewhat longer than the periods found for SSA eigenvectors 3 and 4 (Fig. 8). Presumably, by combining PC3 (and PC4) with PC2 and PC1 (modes with longer period fluctuations) (Fig. 12), the resulting NLSSA mode has a longer period than SSA eigenvector 3 (and 4). In Fig. 7, the RC from SSA modes 1 and 2 is shown as curve (c), the NLRC from NLSSA modes 1 and 2 as (d), the NLRC1 as (e), and the NLRC2 as (f). These 4 curves can be compared against curve (b), the SOI smoothed by a 13-month moving average, in terms of the correlation, the RMSE and the percentage of variance explained, as shown in Table 2. Clearly, the first two NLSSA modes are able to represent the SOI better than the first 2 SSA modes. 6 Summary and Conclusion Inspired by the fact that PCA can be nonlinearly generalized to NLPCA by an NN approach, we have nonlinearly generalized the singular spectrum analysis (SSA) method to 15

16 NLSSA. First SSA is performed on the data, then the leading PCs from the SSA are input into an NLPCA.cir (NLPCA with a circular node at the bottleneck). While the PCs are uncorrelated with each other, they may have nonlinear relations. The NLPCA.cir tries to nonlinearly combine the PCs by a closed (or open) curve solution in the PC space, which is the NLSSA mode 1. By subtracting this mode from the data, the residual can be input into the same network to extract the NLSSA mode 2, and so forth for the higher modes. This new method was tested with a highly anharmonic periodic signal (a stretched square wave) imbedded in white noise, where the noise has twice the standard deviation of the signal. The classical Fourier spectral analysis scatter energy strongly into the higher harmonics. With SSA, the energy scattering was much reduced, but the leading pair of SSA modes failed to detect the strongly anharmonic characteristics of the signal. The NLSSA mode 1 found nonlinear relations among the PCs of the SSA, thereby recovering the strongly anharmonic signal. The stretched square wave signal is not an entirely artificial example. Consider surface gravity waves over the ocean. Away from the forcing region, the wave dynamics is essentially linear, and the waves have a sinusoidal form. As the waves reach shallower water, the nonlinear terms in the governing equations become increasingly important, and the wave form develops sharper peaks and broader troughs (LeBlond and Mysak, 1978). In general, when stronger nonlinear dynamics increases the departure of the wave form from the sinusoidal shape, the advantage of NLSSA over SSA and Fourier spectral analysis should become more evident. To test the NLSSA method with real data, we chose the renowned climate time series, the Southern Oscillation Index (SOI) from 1866 to 2. The first two NLSSA modes 16

17 gave a much better approximation to the SOI than the first two SSA modes. From fitting sinusoidal curves to the NLSSA eigenvectors, we found that the NLSSA mode 1 has a period of 52 months, and mode 2 has a period of 39 months. The 39-month period of the second mode, which is significantly longer than that reported by previous studies using linear techniques (Troup, 1965; Ghil et al. 21), appears to result from nonlinearly combining the SSA modes 3 and 4 (with quasi-biennial periods) and the SSA modes 1 and 2 (with periods of slightly over 5 months). So far, the NLSSA technique has been applied to a single time series. The theory developed here is fully applicable to the multi-variate case (i.e. nonlinear multichannel SSA), which is demonstrated with tropical Pacific climate data in our next paper (Hsieh and Wu, 21). Acknowlegments Support through strategic and research grants from the Natural Sciences and Engineering Research Council of Canada is gratefully acknowledged. 17

18 References Bishop, C. M. (1995). Neural Networks for Pattern Recognition. Oxford: Clarendon Pr. Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 2, Elsner, J. B., & Tsonis, A. A. (1996). Singular Spectrum Analysis. New York: Plenum. Ghil, M., Allen, M. R., Dettinger, M. D., Ide, K., Kondrashov, D., Mann, M. E., Robertson, A. W., Saunders, A., Tian, Y., Varadi, F., & Yiou, P. (21). Advanced spectral methods for climatic time series. Rev. Geophys., (submitted). Hsieh, W.W. (21). Nonlinear principal component analysis by neural networks. Tellus (in press). [Downloadable from william/pubs.html] Hsieh, W.W., & Wu, A. (21). Nonlinear multichannel singular spectrum analysis of the tropical Pacific climate variability using a neural network approach. Journal of Geophysical Research, (submitted). Kirby, M. J., & Miranda, R. (1996). Circular nodes in neural networks. Neural Computation, 8, Kramer, M. A. (1991). Nonlinear principal component analysis using autoassociative neural networks. AIChE Journal, 37, LeBlond, P.H., & Mysak, L.A. (1978). Waves in the Ocean. Amsterdam: Elsevier. Mardia, K. V., Kent, J. T., & Bibby, J. M. (1979). Multivariate Analysis. London: Academic Pr. 18

19 Philander, S. G. (199). El Niño, La Niña, and the Southern Oscillation. San Diego: Academic Pr. Ropelewski, C. F., & Jones, P. D. (1987). An extension of the Tahiti-Darwin Southern Oscillation Index. Monthly Weather Review, 115, Troup, A. J. (1965). The southern oscillation. Quarterly Journal of Royal Meteorological Society, 91,

20 Table 1: A comparison between the reconstructed component (RC) from NLSSA mode 1 (NLRC1), and RC from SSA (using mode 1, modes 1 and 2,..., modes 1 to 8) in terms of the correlation and the RMSE between the RC and the stretched square wave signal. The percent variance of the data (signal + noise) accounted for by the RCs are listed in the final column. RC correlation RMSE % variance NLRC RC RC RC RC RC RC RC RC

21 Table 2: A comparison between the SOI (smoothed by the 13-month moving average) and the reconstructed components from SSA modes 1 and 2, and NLSSA modes 1 and 2, in terms of the correlation and the RMSE between the RC and the SOI, and the percent variance of the SOI accounted for by the RC. RC correlation RMSE % variance SSA 1& NLSSA 1& NLSSA NLSSA

22 Figure 1: The NN model for calculating (a) the nonlinear PCA (NLPCA), and (b) the NLPCA with a circular bottleneck node (NLPCA.cir). The model is a standard feedforward NN (i.e. multi-layer perceptron), with 3 hidden layers of variables or neurons (denoted by circles) sandwiched between the input layer x on the left and the output layer x on the right. Next to the input layer (with l neurons) is the encoding layer (with m neurons), followed by the bottleneck layer (containing one neuron u in the NLPCA), then the decoding layer (with m neurons), and finally the output layer (with l neurons). In NLPCA.cir, the bottleneck contains two neurons p and q confined to lie on a unit circle, i.e. only 1 degree of freedom as represented by the angle θ. Thus a nonlinear function maps from the higher dimension input space to the lower dimension bottleneck space, followed by an inverse transform mapping from the bottleneck space back to the original space, as represented by the outputs. To make the outputs as close to the inputs as possible, the cost function J = x x 2 is minimized. Data compression is achieved by the bottleneck, yielding the nonlinear principal component u in NLPCA or θ in NLPCA.cir. 22

23 (f) 12 (e) 1 8 (d) 6 (c) y (b) 4 2 (a) time Figure 2: The bottom curve (a) shows the noisy time series y containing a stretched square wave signal, whereas curve (b) shows the stretched square wave signal, which we will try to extract from the noisy time series. Curves (c), (d) and (e) are the reconstructed components (RC) from SSA leading modes, using 1, 3 and 8 modes, respectively. Curve (f) is the NLSSA mode 1 RC (NLRC1). The dashed lines show the means of the various curves, which have been vertically shifted for better visualization. 23

24 eigenvector 1,2 eigenvector 3,4 eigenvector 5,6 eigenvector 7, lag Figure 3: The first eight SSA eigenvectors as a function of time lag. The top panel shows mode 1 (solid curve) and mode 2 (dashed curve); the second panel from top shows mode 3 (solid) and mode 4 (dashed); the third panel from top shows mode 5 (solid) and mode 6 (dashed); and the bottom panel shows mode 7 (solid) and mode 8 (dashed). 24

25 5 PC1, PC PC3, PC θ time Figure 4: The PC time series of SSA mode 1 (solid curve) and mode 2 (dashed curve) (top panel), mode 3 (solid) and mode 4 (dashed) (middle panel), and θ, the nonlinear PC from NLSSA mode 1, in the bottom panel. Note θ is periodic, here bounded between π and π radians. 25

26 5 (a) 5 (b) PC 2 PC PC PC 1 5 (c) 5 (d) PC 4 PC PC PC 1 Figure 5: The first NLSSA mode indicated by the (overlapping) circles, with the input data shown as dots. The input data were the first 8 PCs from the SSA of the time series y containing the stretched square wave. The NLSSA solution is a closed curve in an 8- dimensional PC space. The NLSSA solution is projected onto (a) the PC1-PC2 plane, (b) the PC1-PC3 plane, (c) the PC1-PC4 plane, and (d) the PC1-PC5 plane. Due to projection effects, the curve appears to intersect itself in the waist of the figure 8 curve in (b), whereas in the 8-dimensional space, there is no intersection at the waist. 26

27 1 (a) θ = (9 ) 1 (b) θ = 18 (27 ).5.5 loading loading lag lag Figure 6: The NLSSA mode 1 loading as a function of the lag, for (a) θ = (solid curve), θ = 9 (dashed curve), (b) θ = 18 (solid), and θ = 27 (dashed). 27

28 12 (f) 1 (e) 8 (d) 6 (c) SOI 4 (b) 2 (a) year Figure 7: (a) The monthly Southern Oscillation Index (SOI) time series from January, 1866 to December, 2. (b) The SOI smoothed by a 13-month moving average. (c) The RC from SSA modes 1 and 2. The NLRC from NLSSA (d) modes 1 and 2, (e) mode 1, and (f) mode 2. The dashed lines show the means of the various curves, which have been vertically shifted for better visualization. The tick mark besides a year label indicates the start of the year. 28

29 eigenvector 1,2 eigenvector 3,4 eigenvector 5, eigenvector 7, lag (month) Figure 8: The first eight SSA eigenvectors of the SOI as a function of time lag. The top panel shows mode 1 (solid curve) and mode 2 (dashed curve); the second panel from top shows mode 3 (solid) and mode 4 (dashed); the third panel from top shows mode 5 (solid) and mode 6 (dashed); and the bottom panel shows mode 7 (solid) and mode 8 (dashed). 29

30 1 PC1, PC PC3, PC θ 1, θ year Figure 9: The PC time series of SSA mode 1 (solid curve) and mode 2 (dashed curve) (top panel), mode 3 (solid) and mode 4 (dashed) (middle panel) for the SOI, and θ 1 (solid) and θ 2 (dashed), the nonlinear PCs from NLSSA mode 1 and mode 2, in the bottom panel. Note θ 1, θ 2 are periodic, here bounded between π and π radians. 3

31 1 (a) 1 (b) 5 5 PC 2 PC PC PC 1 1 (c) 1 (d) 5 5 PC 4 PC PC PC 1 Figure 1: The NLSSA mode 1 of the SOI indicated by the (overlapping) circles (which form a thick curve), with the input data shown as dots. The NLSSA solution is a closed curve in an 8-dimensional PC space; here the NLSSA solution is projected onto (a) the PC1-PC2 plane, (b) the PC1-PC3 plane, (c) the PC1-PC4 plane, and (d) the PC1-PC5 plane. 31

32 1 (a) θ = (9 ) 1 (b) θ = 18 (27 ).5.5 loading loading lag (month) lag (month) Figure 11: The NLSSA mode 1 loading for the SOI, plotted as a function of the lag, for (a) θ = (solid curve), θ = 9 (dashed curve), (b) θ = 18 (solid), and θ = 27 (dashed). 32

33 1 (a) 1 (b) 5 5 PC 1 PC PC PC 3 1 (c) 1 (d) 5 5 PC 4 PC PC PC 3 Figure 12: The NLSSA mode 2 of the SOI indicated by the (overlapping) circles (which form a thick curve), with the input data shown as dots. The NLSSA solution is projected onto (a) the PC3-PC1 plane, (b) the PC3-PC2 plane, (c) the PC3-PC4 plane, and (d) the PC3-PC5 plane. 33

34 (a) θ = (9 ) (b) θ = 18 (27 ) loading loading lag (month) lag (month) Figure 13: The NLSSA mode 2 loading for the SOI, plotted as a function of the lag, for (a) θ = (solid curve), θ = 9 (dashed curve), (b) θ = 18 (solid), and θ = 27 (dashed). 34