A study of singular spectrum analysis with global optimization techniques

Transcription

1 DOI 0007/s A study of singuar spectrum anaysis with goba optimization techniques Moody T Chu Matthew M Lin Liqi Wang Received: 7 November 0 / Accepted: 4 October 03 Springer Science+Business Media New York 03 Abstract Singuar spectrum anaysis has recenty become an attractive too in a broad range of appications Its main mechanism of aternating between rank reduction and Hanke projection to produce an approximation to a particuar component of the origina time series, however, deserves further mathematica justification One paramount question to ask is how good an approximation that such a straightforward apparatus can provide when comparing to the absoute optima soution This paper reexamines this issue by expoiting a natura parametrization of a genera Hanke matrix via its Vandermonde factorization Such a formuation makes it possibe to recast the notion of singuar spectrum anaysis as a semi-inear east squares probem over a compact feasibe set, whence goba optimization techniques can be empoyed to find the absoute best approximation This framework might not be immediatey suitabe for practica appication because goba optimization is expectedy more expensive, but it does provide a theoretica baseine for comparison As such, our empirica resuts indicate that the simper SSA agorithm usuay is amazingy sufficient as a handy too for constructing exporatory mode The more compicated goba methods coud be used as an aternative of rigorous affirmative procedure for verifying or assessing the quaity of approximation Moody T Chu s research was supported in part by the Nationa Science Foundation under grant DMS Matthew M Lin s research was supported in part by the Nationa Science Counci of Taiwan under grant NSC 0-5-M MY3 M T Chu Department of Mathematics, North Caroina State University, Raeigh, NC , USA e-mai: chu@mathncsuedu M M Lin (B) Department of Mathematics, Nationa Chung Cheng University, Min-Hsiung, Chia-Yi 6, Taiwan e-mai: mhin@ccuedutw L Wang Department of Mathematics, Daian University of Technoogy, Daian, China e-mai: izzy@maiduteducn

2 Keywords Singuar spectrum anaysis Time series Hanke operator Semi-infinite matrix Low rank approximation Vandermonde factorization Goba optimization Introduction Singuar spectrum anaysis (SSA) has been attracting considerabe interest in recent years as a usefu technique for extracting critica information from genera time series This nove method, based on principes of mutivariate statistics, accounts for dynamicay improving the covariance structure of the underying time series [9] But what makes its wide appicabiity is the non-parametric and mode-free nature that enabes practitioners to depoy it without a prior knowedge of any underying structure [0,58] Its utiization is more as a handy exporatory mode-buiding too than an affirmative procedure The deveopment of the SSA is usuay attributed to Broomhead and King [8] whose origina intention was to identify quaitative information in the anaysis of dynamica systems Actuay, simiar ideas aso came to surface independenty at about the same time in other discipines, such as that in [3] for the dimensiona anaysis of weather and cimate attractors Over years, the methodoogies and appications of the SSA have been refined and broadened significanty in the iterature A quick Googe search for the term singuar spectrum anaysis shows up a wide range of appications Far from being compete, we mention a few representative appications incuding smoothing, fitration [57], noise reduction [55], trend and moduated harmonics extraction [0,39], change-point detection [4], signa parameter estimation, cimatic and meteoroogica forecasting [,6,54], economic and financia appications [4,8,9,46], causaity discovery [46], phase reconstruction [4,7,,53] For an eementary introduction to this subject we suggest the two monographs [,9] For a synopsis of more recent advances in this fied, we suggest the brief treatise [0] The goa of this paper is to propose a mathematica framework aowing us to assess the effectiveness of the SSA from a theoretica point of view So that this paper is sef-contained, we briefy review the basic SSA as foows There are quite a few variations of the basic SSA nowadays, incuding the mutivariate SSA [,5,46] and the minimum-variance based SSA [9,3], but we wi focus on the basic scheme to convey the idea Given a finite time series z = (z 0,,z n ) of ength n +, the basic SSA consists of four basic parts [9]: Embedding Choose an integer p < n,referredtoastheembedding dimension or window ength,and et q = n p + Define p-agged vectors z 0,,z q by z i := [z i,,z i+p ] C p and the associated trajectory matrix Z by z 0 z z q z z z q Z := [z 0,,z q ]= () z p z p z n Note that Z is a p q Hanke matrix This process of embedding z into Z, fundamenta in time series anaysis, creates a hande for manipuating rank reduction For the sake of characterizing the Vandemonde parameters more easiy for the Hanke operator, which wi be discussed subsequenty, we begin the index with 0

3 The choice of p affects the quaity of the approximation See discussions in [,9] A more recent work [6,7] shows that the optima vaue of p is reated to the separabiity between signa and noise components in the origina series It is recommended both theoreticay and empiricay that the threshod shoud be somewhere between n 4 and n Rank reduction Suppose that Z = U V = r σ i u i vi () is the singuar vaue decomposition (SVD) of the trajectory matrix Z, where nonzero singuar vaues are arranged in descending order σ σ σ r andr = rank(z) A fundamenta mechanism used for rank reduction is the truncated singuar vaue decomposition A truncated SVD of Z is a partia sum of the form Z := i= σ i u i vi, (3) i= where d r is a preseected positive integer serving as the target rank for the anaysis There is a statistica meaning of the truncated singuar vaue decomposition With regard to the ag-covariance matrix Z Z when coumns of Z have been centered, it can be argued that Z represents the best, unbiased, inear, minimum-variance estimate of Z among a possibe d-dimensiona sampe subspaces in C p Because of this, if d is propery chosen, then those components of Z corresponding to sma singuar vaues are considered as ess significant and often are disregarded 3 Group seection In order to identify possiby disparate physica behaviors of the time series, the above process can further be refined Without being specific, we simpy mention that different rankone matrices from the SVD of Z can be chosen and coected together, and be processed separatey, with the hope that propery partitioned groups woud refect different dynamica components of the origina time series See [9,40] for some practica concerns and appications 4 Signa recovery It is amost aways true that the truncated SVD of a structured matrix cannot preserve the origina structure [9,45] As such, Z generay is not a Hanke matrix The nearest Hanke matrix to Z can be obtained by diagonay averaging entries of Z aong the anti-diagonas In doing so, however, the rank of the resuting Hanke matrix approximation is no onger d So the above four steps have to be repeated with the hope that a imit point which is a Hanke matrix of rank d is finay determined This procedure is simiar in spirit to the genera ift-and-project agorithm deveoped for structured ow rank approximation [9] In short, the basic idea of the SSA is to decompose the origina time series into the sum of a number of interpretabe components, each of which shoud be easiy identified as part of a certain moduated signas or, in reciprocity, the random noises The workings of the

4 SSA are by repeatedy sifting the time series through the truncated SVD of the embedded trajectory matrix with the hope that a ceaner, more representative time series showing critica traits, such as ow-frequency trends, high-frequency variabiity, cycic components, or other structures of the origina series, can be extracted The notion of the SSA is attractive because of its simpicity in impementation and its broadness in appications The main mechanism whereby the scheme aternates between the rank reduction and the Hanke projection, however, deserves further understanding We can think of at east three reasons for this investigation Firsty, since the method is within the genera frame of ift-and-project, at best the inear rate of convergence is expected [9] How woud the strength of noise or the inherent intricacy of the origina signa affects the convergence? Secondy, the iteration may not achieve a Hanke approximation of rank d because the aternating projection gets stagnant at a oca minimizer of the distance between the manifod of matrices of rank d and the subspace of Hanke matrices What does the imit point of the SSA reay represent? Thirdy, there might be mutipe oca minimizers How does this oca soution affect the interpretation of physica properties being investigated? The purpose of this paper is to study, by means of goba optimization techniques on parameters of the Vandermonde factorization, what the absoutey best and ow rank Hanke approximation can be achieved for a genera time series In return, we are interested in using this information as a reference point for assessing the quaity of the SSA soutions This paper is organized as foows In Sect we expain the cause and effect of the notion of the ow rank Hanke structure The SSA usuay appies ony to finite series which actuay is no more than a segment of a ong term behavior So as to take into account a possibe intrinsic characteristics, we describe a natura inkage between the ow rank Hanke structure and a sinusoida signa which, in turns, eads to an infinite series whose trajectory matrix is necessariy semi-infinite We raise the question of how much information is contained in a segment or submatrix of the infinite case In Sect 3 we derive the fact that any finite rank Hanke operator, of finite dimension or not, aways enjoys a Vandermonde factorization Reated to Sect, this argument aso shows that, in a broad sense, a infinite series with finite rank trajectory matrix can be cast as a sinusoida signa Equipped with the Vandermonde parameters, we formuate the SSA as a east squares minimization probem over a compact set in Sect 4 Sinusoida signa To motivate the connection of a time series to its trajectory matrix, we first consider a noisefree time-domain signa comprising d components of exponentiay decaying sinusoids s(t) = a e αt e ı(πν t+φ ), (4) = where a,α,ν,andφ are rea numbers denoting the magnitude, the decay rate, the frequency, and the phase ange, respectivey Starting with t 0 = 0 and samping this signa at uniformy spaced nodes t 0, t, with fixed interva ength Δt (so /Δt is the so caed samping rate), we obtain an infinite sequence s(t k ) = a e αkδt e ı(πν kδt+φ ) = = a e ıφ = ( e ( α +ıπν )Δt ) k

5 For simpicity, denote s k := s(t k ) and define β := a e ıφ, (5) λ := e ( α +ıπν )Δt (6) Then the time series {s 0, s,} enjoys the reationship s k = β λ k (7) = We can rewrite the corresponding trajectory matrix S of the series {s 0, s,} as λ 0 λ λ λ λ S = β = λ λ 0 λ 0 λ 0 d β 0 0 λ 0 λ λ λ λ λ d 0 β λ 0 λ = λ, (8) 0 β d λ 0 d λ d which easiy shows that the matrix S is of rank d For stricty decaying signas, we shoud expect that α > 0 and, hence, λ <, impying that S is a bounded operator The decomposition in the ast equation of (8) is known as the Vandermonde factorization of S Note that the trajectory matrix S is naturay a semi-infinite matrix If we take ony finitey many sampes, resuting a finite time series as is typicay assumed in the appication of the SSA, then it means ony a eading principa submatrix of size p q extracted from S In this case, the corresponding finite trajectory matrix sti enjoys a Vandermonde factorization after appropriatey truncating that of (8) In other words, the formation of the basic SSA is a specia case of our genera theory Since the set of infinitey many sampes {s 0, s,} is supposedy carrying more or even compete information, a cassica question that immediatey arises is how much information of the origina signa can be retrieved from a finite number of sampes Keys to our goba optimization approach, as wi be expained beow, are the variabes β,λ,=,,d, which serve as a parameterization of any Hanke matrices of rank d The converse of the above characterization is worth noting Given a Vandermonde factorization in the form of (8), then by (5) the poar form of the compex number β determines the magnitude a and phase ange φ In the meantime, by (6) the poar form of the quantity λ determines the decay rate α and the frequency ν up to a scaing by Δt The dependence on the scaing Δt is due to the fact that whenever a measurement of decay rate and frequency is referred to, the meaning of a unit time ought to be defined first Once Δt is specified, then a composite signa is competey determined from a given Vandermonde decomposition In other words, we can go back and forth interchangeaby between a bounded ow rank Hanke matrix and a sinusoida signa through the reationship (7) In practice, the true signa s(t) is unknown and often the observed data {z k } are contaminated Since the sampes are taken independenty, we may assume the popuar AWGN channe mode,

6 z k := s k + ɛ k, (9) ie, white Gaussian noise ɛ k with specified signa-to-noise ratio (SNR) is added ineary per sampe Through embedding, the corresponding trajectory matrix Z can be forced to maintain the Hanke structure But the property of rank d is generay ost Indeed, Z coud easiy be of fu rank In the SSA, the idea is to approximate Z by a Hanke approximation of rank d, whence the signa information of s(t) is partiay retrieved As we have pointed out earier, the SSA does not even make use of the semi-infinite matrix Z in its entirety Ony a segment of finite ength n is fed into the procedure To what extent this process of aternating projection can recover the essentia information is the question we want to investigate in this study 3 Vandermonde parameterization In the preceding section, we have used a sinusoida signa to introduce the Vandermonde factorization of a trajectory matrix In this section, we want to bring forth the fact that any Hanke operator, finite dimensiona or not, aways enjoys such a factorization so ong as it is of finite rank So suitabe for the SSA, our theory can be appied regardess of wherever the time series might arise The Vandermonde factorization offers an effective characterization of a Hanke operator in d parameters, provided the operator is of rank d Later on, we sha make use of these parameters to set up a goba optimization framework We mention in passing that the importance of Hanke operators goes beyond its invovement in the SSA They appear frequenty in many other seemingy disparate areas of cassica mathematics [,33,47] Their appications cover a wide range of discipines outside mathematics See, for exampe, [4,8] and the many references in [48] It is thus of practica significance to characterize a genera ow rank Hanke operator in the east possibe parameters To derive Vandermonde parametrization which wi be used by goba optimization to estabish basis of comparison for assessing the effectiveness of the SSA, we now reca severa cassica resuts in the iterature We begin with the characterization of a bounded Hanke operators over the space of square summabe (semi-infinite) sequences [6,4,50] The foowing theorem states that whether the time sequence {h 0, h,} determines a bounded operator H on is equivaent to whether the sequence itsef represents the Fourier coefficients of an essentiay bounded function over the unit disk [47, Theorem ] Theorem The Hanke matrix H represents a bounded operator over if and ony if there exists a function ψ L on the unit circe such that its h m = π π 0 ψ(θ)e imθ dθ, m = 0,, (0) In this case, the operator norm of H is bounded above by ψ We have earned from the preceding section that the the number of components of a sinusoida signa is the same as the rank of the corresponding trajectory matrix What can be said about the converse, which is needed for the rank reduction step in the SSA? Given a time series {h 0, h, }, the associated generating function is defined by the forma power series

7 G(z;{h n }) := h n z n () The foowing criterion is a cassica resut by Kronecker [47, Theorem 4] Theorem The Hanke matrix H has finite rank if and ony if the power series () determines a rationa function In this case, n=0 rank(h) = deg(zg(z;{h n })), () where the degree of a rationa function is the maximum of the degrees of its minima constituent poynomias Finding the imit of the generating function G(z;{h n }) is not aways an easy task An equivaent but more direct observation of finite rank is through a inear recursive reationship (LRR) among eements of H [5, Chapter XV, Theorem 7] Theorem 3 The Hanke matrix H is of finite rank d if and ony if there exist constants γ 0,,γ d such that h i = γ d h i + γ d h i + + γ o h i d, i = d, d +, (3) and d is the east integer having this property Whie Theorem 3 is ong known as a necessary and sufficient condition concerning the finite rank of a semi-infinite Hanke matrix, it is interesting to note that the LRR (3) has been redeveoped by practitioners of the SSA from the finite-dimensiona point of view as a forecasting scheme [9,4,9] It is worth noting the interchange between finite and infinite cases as foows Assume the scenario that the semi-infinite Hanke matrix H in Theorem 3 is aready known to be of rank d Then it can be argued that the d d eading principa submatrix Ĥ, ie, the trajectory matrix of the finite series {h 0,,h d }, is necessariy nonsinguar The finite difference equation (3) can be written in the form of a inear system h 0 h h h d h h h 3 h d h h 3 h d+ h d h d h d γ 0 γ γ d = h d h d+ h d, (4) known as Yue-Waker equations in signa processing, for the coefficients γ 0,γ,, γ d We thus see that the parameters γ 0,,γ d in the LRR and consequenty the entire semiinfinite time series are uniquey determined by Ĥ and one extra eement h d Inother words, to uniquey determine a Hanke matrix of rank d, we need to know exacty d many eements {h 0, h,,h d } Obviousy, inear systems simiar to (4) can aso be formed for the same set of parameters γ 0,γ,, γ d by taking up any segment of d successive eements in the infinite time series, so ong as the corresponding trajectory matrix is nonsinguar Once the parameters γ 0,,γ d are in hand, we can take one step further That is, we can express the entry of H at any ocation in term of roots of the characteristic poynomia p(λ) := λ d γ d λ d γ λ γ 0, (5)

8 without the need of the recursive reference to other entries This formuation eads to a natura parameterization of H which wi be used to reassess the rationaes behind the SSA To fix the ideas, et λ,=,,r, denote the distinct roots of p(λ) in (5) and each of which has mutipicity ρ So r = ρ = d A genera soution to the difference equation (3) can be formuated as foows Let V p(λ) := [V (),,V (r) ] C d d, be a bock matrix whose bock V () =[v () ij ], =,,r, isofsized ρ,where v () ij := c ij λ i j, i = 0,,,d, j = 0,,,ρ, (6) and c ij = 0ifi < j;andc ij = (i i! j)! j! otherwise A typica bock ooks ike λ 0 λ λ λ 3 3λ 3λ V () := λ ρ λ ρ λ d (d )λ d (d )(d ) λ d 3 ρ λ ( d ρ ) λ d ρ C d ρ For convenience, note that in the above we start counting indices of the entries from (0, 0)In order to enforce the initia vaues h 0,,h d on the genera soution of (3), we introduce the inear system V p(λ) β (0) β (ρ ) β (0) ṛ β (ρ r ) r = h 0 h h h d, (7) known as the confuent Vandermonde system [34], for the undetermined coefficients β (0),,β(ρ ),,β r (0),,β (ρ r ) r Triviay, the equations in (7) are equivaent to h i = r = ρ j=0 β ( j) c ij λ i j, i = 0,,,d

9 It can further be verified recursivey that d h i = γ s h i d+s = = = s=0 r = r = ρ j=0 ρ j=0 β ( j) d γ s s=0 ( d ( s=0 r = ρ j=0 β ( j) c i d+s, j γ s λ i d+s j c i d+s, j λ i d+s j β ( j) c ij λ i j, i = d, d +, (8) For any given time series {h 0, h,}, we have thus shown that if the corresponding trajectory matrix H is of rank d, then each h i may be represented by the parameters β ( j) and λ, referred to as the Vandermonde parameters, according to the formua (8) In the iterature, the so caed Vandermonde factorization for a semi-infinite Hanke matrix H of rank d is simpy a rewriting of the same parametric representation (8) More specificay, et V denote the confuent Vandermonde matrix of size d obtained by extending the matrix V p(λ) defined in (6) downward to infinite ength We can rearrange the expression (8) and estabish the foowing identity [5,] Theorem 4 Suppose H is a semi-infinite Hanke matrix of rank d Then there exists a d d bock diagona matrix D whose -th bock is of size ρ ρ and is Hanke and upper anti-trianguar such that ) ) H = V D V (9) Assume for simpicity the generic case that a roots of p(λ) are distinct Then the soution to the LRR (3) can be written as ( d d ) ( d ) h i = γ s h i d+s = γ s β (0) λ i d+s = β (0) γ s λ i d+s s=0 s=0 = = s=0 ( d ) = β (0) λ i d γ s λ s = β (0) λ i, i = d, d +,, (0) = s=0 = which, in turn, estabishes the very same Vandermonde factorization as that in (8), even though the series {h 0, h,} may not geminate from a sinusoida signa at a Together with the remarks made in the second to the ast paragraph of Section,wemaysaythatany time series with finite rank trajectory matrix is equivaent to a sinusoida signa Truncating a semi-infinite series (or trajectory matrix) into a finite series (or trajectory matrix) does preserve a the properties we have discussed thus far The reaity is that we usuay begin with a finite series (or trajectory matrix) in practice, such as the SSA appication So can the finite dimensiona probem contains enough information about the underying infinite dimensiona probem? The answer, known as the singuar extension theory [33, II9], is not trivia Padding a given m k Hanke matrix with zeros to expand it into a semi-infinite Hanke matrix usuay resuts in a semi-infinite matrix with higher rank One feasibe way of extension is aready manifested in the course of our discussion That is, given any d d nonsinguar Hanke matrix, we embed this matrix as the eading principa submatrix of a nontrivia semi-infinite Hanke matrix H by specifying an extra vaue for h d Onyafter

10 having soved the corresponding Eq (4), we wi have created the recursive reationship (3) and the resuting H is of rank d 4 Least squares approximation We have argued that amost a time series with a rank-d trajectory matrix can be interpreted as discrete sampes of a noise-free sinusoida signa with d components in the form (4) For contaminated series {z 0, z,}, however, the rank condition is usuay ost Seeking out the noise-free signa (4) means finding suitabe vaues {β,,β d } and {λ,,λ d } for its Vandermonde parameters In this section, we set up the east squares approximation for this purpose Consider the scenario that the SSA is appied to the finite time series {z 0, z,, z p } with a square Hanke embedding, that is, suppose that the corresponding trajectory matrix Z is of size precisey p p Without knowing any specfic parametrization, the SSA aternates between the best rank d approximation and the best Hanke approximation with respect to the Frobenius matrix norm In contrast, since we now know that an eement in a generic Hanke matrix of rank d shoud ook ike (0), the SSA appication amounts to soving the minimization probem min β,λ C λ, =,,d p (i + ) z i i=0 β λ i = p + i=p (p i) z i β λ i () As p, the scheme eventuay is deaing with the probem (i + ) z i β λ i () min β,λ C λ, =,,d i=0 Noticeabe in both formuations is that the term-wise errors are not equay weighted Does this make any sense? Take the extreme case () as a point of argument We see that the weight is increased at a constant rate even when the signa graduay decays to zero In a sense we are imposing more penaty on the noise for weaker signas There are two consequentia concerns One is that the weights grow unboundedy and there is no satisfactory theory to guarantee the convergence of such a weighted infinite series in genera The other is that the weight scheme might have biased the outcome of the optimizers (β,λ ) for overy correcting the higher order terms which are composed mosty of noises On the other hand, we think that the be-shape weight distribution,,,p, p, in the finite case () ismorean artifact due to our obstinacy in maintaining the Hanke structure than any other mathematica reason [9,45] In what foows, we consider to fit the data by means of soving min β,λ C λ, =,,d p i=0 z i = β λ i from the first p sampes, that is, we regard a noises in (9) as being uncorreated, independent, and having equa variance For the SSA computation, it is preferred to keep p as sma as is necessary because the size of the associated trajectory matrix does affect the overhead in the sequence of matrix decomposition But in our formuation, the size p matters = = (3)

11 ony to the summation and there are a fixed number of ony d parameters to be estimated Note that we aow λ to range over a cosed unit disk The foowing arguments assure that the minimum in (3) does exist Define I I I Λ := Λ(λ,,λ d ):= λ λ λ d λ λ λ d λ p λ p λ p d Triviay, we can rewrite (3) in vector form, β := β β β d, z 0 z:= z z p min β,λ C λ, =,,d Λβ z (4) This is a noninear east squares probem over the compex fied For each given Λ of fu coumn rank, the unique optima soution for β is given by [38] β := L(Λ; z) := (Λ Λ) Λ z (5) where stands for the conjugate transpose The probem is thus reduced to finding λ for the minimization probem min λ,, λ d Λ(Λ Λ) Λ z z (6) An exampe of such an objective function for the case d = with randomy generated noise is iustrated in Fig, showing the presence of mutipe minimizers Most importanty, in a cases a minimizer over the compact d-dimensiona torus must exist Since we are deaing with a compact set, it is possibe to expoit goba optimization techniques to search for the absoute minimizer, which wi be deineated ater We are interested in using the goba soution to compare the resut constructed from the SSA agorithm Fig An exampe of objective function for (6) (The surface has been rotated with AZ = 4 and EL = 38)

12 5 Absoute soution versus SSA soution The feasibe set in our east squares formuation for the SSA is compact, so the absoute minimizer aways exists In this section, we report some numerica experiments on empoying the goba optimization techniques to find the absoute minimizer We stress again that our goa is not to propose a new method for the SSA Rather, because our parametrization makes it possibe to ocate the absoutey best optimizer, we want to empoy this more expensive method to yied an absoute baseine for comparison with resuts obtained from the SSA agorithm As wi be seen, our discovery is rather surprising 5 Goba optimization Any existing goba optimization software shoud suffice to meet our need for buiding a baseine of comparison For competion, we briefy review ony the genera notion of goba optimization and describe how we set up the experiments The objective of goba optimization is to find the gobay best soution in the presence of mutipe oca optima This is an extremey important too for amost a fieds of appications where optimization is needed Over the years, many strategies for goba optimization have been proposed Research resuts are too numerous to count and the research is ongoing Far from being compete, we mention books [3,44] for introduction and [37,43,49,59] for comprehensive survey on continuous goba optimization Over a, it might be fair to say that one prevaiing scheme that aows genera structura constraints and makes possibe straightforward generaizations and extensions of existent oca agorithms is the notion of adaptive partition A partia ist of practica methods with a view towards efficient computer-based impementations incudes mutistart framework equipped with sophisticated scatter search agorithm [5]; generic agorithms motivated by the process of natura seection and the surviva of the fittest principe [30,56]; simuated anneaing based upon the physica anaogy of cooing crysta structures that spontaneousy attempt to arrive at some stabe and, hopefuy, goba equiibrium [5,36]; and pattern search methods which compare instantiations and expore new regions with the goa of conducting a goba search [3,35] Depending on the appications, there are aso other more speciaized techniques such as the stochastic goba optimization See, for exampes, the discussions in [43,58] and the references contained therein It is not the intention of this paper to propose a new goba optimization method Rather, we empoy goba optimization techniques to hep to estabish an absoute baseine to evauate the performance of the SSA Toward that goa, our effort has been to set up the framework for goba optimization cacuation Once this framework is estabished, any existent goba optimization software can serve our purpose For demonstration, we take advantage of the goba optimization toobox avaiabe in Matab and chooseto use the MutiStartmethod with sover fmincon to carry out our experiments As the optimization sover makes use of derivatives, exact gradient information has the benefit of improving efficiency and precision We suppy such a cacuation of gradient in the Appendix We carry out our experiment by setting up test data in the foowing way Our design is based on two assumptions First, as we have advanced in this discourse, it suffices to use noise-free sinusoida signas as the contro group Second, we assume that the expected rank d is known a priori, which itsef is a difficut probem in practice We generate a sinusoida signa s(t) in the form of (4) with d components as the basis for comparison This exact

13 Iteration Overhead with SNR = ± 05 db Iteration Overhead with SNR = ± 3 db Numbers of Iterations Numbers of Iterations Expected rank d Expected rank d Fig Overhead of SSA in terms of iterations needed for convergence SNR > 0(red compact box stye); SNR < 0(bue empty box stye) (Coor figure onine) signa is random in the sense that its parameters a [0, 6], α [0, 5],andφ [0, ] are uniformy distributed pseudorandom numbers, whie the frequencies ν are pseudorandom integers from a uniform discrete distribution over [0, 400] Samping rate is set at 9 and we take 0 sampes s k := s(t k ) over the time interva [0, ) Additive white Gaussian noises based on the AWGN channe mode with fixed SNR are then added to the signa in the way of (9) as the observed signa We appy both the SSA agorithm and the MutiStart method to the observed signas The goa is to examine the coseness of the reconstructed signas by the agorithms to the origina signa 5 Overa effect of error strength We first experiment with the effect of error strength on the performance of the SSA method We want to examine the impact on both the overhead and the accuracy For simpicity of demonstration, we choose not to perform the group seection and consider ony the case with embedding dimension 5, ie, the trajectory matrix Z is aways of size 5 5 With each fixed d =,,5, we randomy generate 00 signas The aternating projection scheme is appied to each randomy perturbed time series unti two consecutive Hanke projections differ by ess than 0 when we regard that convergence has been achieved Each iteration invoves the SVD of a 5 5 Hanke matrix, which is not cheap Depicted Fig are the boxpots for numbers of iterations needed by the SSA to attain a rank d Hanke approximation of the origina Z subject to four different strengths of noise ±05 and ±3 db It seems to suggest that the strength of noise is not critica for convergence However, from Fig 3, it is quite obvious that stronger noise (negative SNR) does affect the quaity of approximation We caution that, because the manifod of rank d matrices is not convex, the aternating projection scheme, centra to the SSA agorithm, may not return at its convergence a rank d Hanke approximation to the given trajectory matrix Z Even if it does, the imit point may not be the nearest rank d Hanke approximation to Z See a counterexampe in [9] In contrast, our formuation directy searches for the best fitting signa in the sense of (4) We measure the strength by the ogarithm unit db Reca that one decibe is ten times the base-0 ogarithm of the ratio of the measured quantity to the reference eve Positive db means that the signa is stronger than the noise; otherwise, the noise is stronger

14 Errors of Reconstructed Signas with SNR = ± 05 db Expected rank d Errors of Reconstructed Signas with SNR = ± 3 db Expected rank d Fig 3 Errors between the true time series {s k } and the reconstructed time series {h k } by SSA SNR > 0(red compact box stye); SNR < 0(bue empty box stye) (Coor figure onine) 3 0 Comparison of Errors with SNR = 3 db Expected rank d Comparison of Errors with SNR = 3 db Expected rank d Fig 4 Contrast of errors between the true time series {s k } and the reconstructed time series {h k } by the goba optimization method (red compact box stye) and by the SSA agorithm (bue empty box stye) (Coor figure onine) In yet another experiment with randomy generate signas, we contrast the errors of the signas reconstructed by the SSA agorithm with those by our goba optimization approach for each d =,,5 with AWGN noises at SNR =±3dBinFig4 Reca that this is the primary purpose of this paper We see that in genera the goba optimization approach gives rise to better approximation to the origina time series {s k } than the SSA agorithm The advantage of the goba optimization approach to the SSA agorithm becomes more obvious when the noise is stronger (eg, db = 3) For weaky perturbed signas, the SSA performs amost compatiby to that by goba optimization approach 53 Sampe by sampe comparison The boxpots in Fig 4 indicate to us the overa trend It is informative to aso check out the errors produced by each independent random test We take the ratio of the SSA error to the goba optimization error sampe by sampe and pot the histogram of ogarithm of these

15 Percentage of Group 70% 60% 50% 40% 30% 0% 0% Sampe Wise SSA/Goba Ratio Histogram (SNR = 3 db) d= d= d=3 d=4 d=5 0% og of SSA Error to Goba Error Ratio Percentage of Group 60% 50% 40% 30% 0% 0% Sampe Wise SSA/Goba Ratio Histogram (SNR = 3 db) d= d= d=3 d=4 d=5 0% Fig 5 Histogram of sampe-by-sampe SSA error to goba optimization error ratios og of SSA Error to Goba Error Ratio ratios in Fig 5 For comparison, we pot the histogram for a ranks together, distinguished by coors, whie containers in each custer share the same bin vaue which is the midpoint of the corresponding custer range shown in the drawing A negative ogarithm of the ratio indicates that the SSA reconstruction is superior to the goba optimization reconstruction From the histogram we observe that, except for the case d =, the goba optimization usuay has a higher percentage of producing a better reconstruction when the noise is strong In contrast, when the noise is reativey weak (eg, db = 3), the simper SSA performs amazingy we, comparing to the more compicated goba optimization 54 Effect of weight scheme One might wonder why sometimes the SSA agorithm gives better approximation to {s k } Shoud not the goba optimization does the best? It is worth noting the mutipe factors that might cause this to happen Firsty, the quantity being minimized in the goba optimization is the sum p i=0 z i d = β λ i (see 3), not the absoute error p i=0 s i d = β λ i Because the true time series {s k } is practicay not avaiabe in reaity, the goba optimization offers at its best an optima approximation to the series {z k }, not {s k } Secondy, the machinery we empoyed in the goba optimization is the scheme Mutistart which uses uniformy distributed start points within predefined bounds In our experiment we aow merey 40 start points It is possibe to improve the objective vaue, or to confirm the goba soution, by more start points Thirdy, we have mentioned that not necessariy the SSA agorithm wi produce the best rank d Hanke approximation at its convergence Even if it does, the SSA is a speciay weighted east squares probem (see ), whereas our formuation (3) is not So we are comparing nearness to a backbox {s k } from two different objective functions If we reay want to consider min β,λ C λ, =,,d p i=0 ω i z i β λ i = (7)

16 Percentage of Group 00% 90% 80% 70% 60% 50% 40% 30% 0% 0% Weighted SSA/Goba Ratio Histogram (SNR = 3 db) d= d= d=3 d 4 d=5 0% og of SSA Error to Goba Error Ratio Percentage of Group 00% 90% 80% 70% 60% 50% 40% 30% 0% 0% Weighted SSA/Goba Ratio Histogram (SNR = 3 db) d= d= d=3 d 4 d=5 0% og of SSA Error to Goba Error Ratio Fig 6 Histogram of weighted sampe-by-sampe SSA error to goba optimization error ratios with weights ω i > 0, the norma Eq (5) per fixed Λ becomes Λ W Λβ = Λ W z, (8) with W := diag{ω 0,ω 0,,ω p,ω p } Equivaenty, with the change of variabes by diagona scaing Υ := W Λ, c := W z, (9) the weighted east squares probem (7) can be reformuated as min λ,, λ d Υ(Υ Υ) Υ c c, (30) which is anaogous to (4) and our code can easiy be modified accordingy If in particuar we use the specia SSA weight scheme, a rather surprising reversa situation then is observed As can be seen in Fig 6, there are many more sampes showing a negative vaue in its ogarithm of SSA error to goba error ratio than those in Fig 5, indicating that the SSA generay performs better than the goba optimization The contrast is especiay strong at the presence of stronger noise 6 Forecasting An important appication of the SSA is to predict future vaues based on the underying mode constructed from existent data In this regard, our goba optimization framework has the advantage of working directy with the Vandermonde parameters β s and λ s That is, once these parameters are estimated, the forecasting is a natura consequence as is described in (0) Though it is not the primary goa of this paper, we demonstrate this capacity in this section by comparing forecast vaues obtained from the goba optimization and the basic SSA For competion, we briefy review how forecasting by the basic SSA is accompished More detais can be found in [9,3,9] Suppose that u,,u d are the eft singuar vectors at convergence of the SSA appied to the given time series {z 0,,z n } Ideay, we woud prefer to see that a rank-d Hanke matrix Z has been found Otherwise, the fina reconstructed series

17 {h 0,,h n } is obtained by diagonay averaging entries of Z Let each singuar vector u i be partitioned as [ ] ũi u i = with ũ i denoting the first p entries and π i is the ast entry of u i Assuming ν := π + +π d <, define a p a := π i i= π i ũ i ν (3) Then it has been proposed [9, Section 3] (See aso [9, Theorem 5]) that the finite difference scheme p h k := a i h k i, k = n +, (3) i= be used as a forecasting scheme In contrast, the Vandermonde parameters naturay give rise to any future vaues via the cosed form (0) If necessary, we can generate the LRR (3) via (4) In that case, note the fundamenta difference that the LRR (3) uses ony d starting vaues {h 0,,h d } and is of minima order 6 Trend detection We use the rea-vaued signa s(t) = 3() t + sin(πt) (33) to demonstrate the forecasting capacity of both approaches We sampe the signa at the rate 9 over interva [0, ) It can be verified that the exact time series corresponds to the case d = 3 with exact Vandermonde parameters β =[3, 5ı,5ı] and λ =[, e πı 9, e πı 9 ] We then add rea-vaued AWGN noises with specified SNR to create artificia observed data {z k } Out of the 03 sampes, we use the first n = 5 to construct the basic mode which then is used to predict the remaining 5 future vaues For the SSA, the window ength is set at p = n =56 For the goba optimization, 40 initia vaues are taken In the atter, we must point out that the optimization takes pace over a compex domain, so the computed Vandermonde parameters and the reconstructed signa coud be compex-vaued To retrieve rea-vaued approximation, we simpy project the signa to the rea ine Two possibe scenarios are depicted in Fig 7 Note that in both cases, we have used ony the information over the interva [0,) to construct the mode As is expected, the Vandermonde parameters obtained from goba optimization are fairy consistent in predicting vaues over the interva [,) by using (0) The trend of the origina signa is reasonaby identifiabe, though in the picture on the right of Fig 7 the prediction begins to deviate We notice that in both scenarios the SSA resuts suffer from considerabe fuctuations around the true curve Conceivaby, its forecasting of vaues over the interva [,) wi not be as accurate as that from the goba optimization Instead of dispaying the forecast vaues by the SSA, we report the foowing measurement

18 5 4 3 Trend Detection and Forecasting observation pure SSA resut Goba resut Trend Detection and Forecasting observation pure SSA resut Goba resut sampes sampes time t time t Fig 7 Detecting trends and making forecasting under strong noise SNR = 3 by SSR(red) and goba optimization (green) (Coor figure onine) To gauge the effectiveness of the forecasting in genera, we empoy the notion of rootmean-square errors (RMSE) defined by [3] RMSE := 03 (z k h k ) 5, (34) k=5 where z k and h k stand for the observation and the forecasting vaues at time t k, respectivey We repeat the experiment 50 times for each SNR =±3 by generating different AGWN noises and coect the resuting RMSE for each test The resuts are depicted in Fig 8 The figure on the eft indicates that the average RMSE by the goba optimization is about 66 % of that by the SSA under weak noise (SNR = 3dB); whereas the ratio is about 80 % under strong noise (SNR = 3dB) Aso indicated is that the RMSE for SSA has ots of outiers, making the mean cacuation biased toward the higher end The figure on the right is the histogram of sampe to sampe future SSA error to the goba optimization error A positive ogarithm of this ratio indicates that the goba optimization forecasting is more accurate the the SSA forecasting Together with phenomenon observed in Fig 7, we think the forecasting capacity of the goba optimization shoud be quite convincing 6 Rea data appication Thus far, a our experiments have been done under the controed condition that the rank of the underying signa is precisey known a priori These experiments are designed to provide an absoute base to evauate the performance of the basic SSA In rea appications, the determination of a suitabe rank is not easy and pays a critica roe in the effectiveness of the reconstructed mode We reaize that for each individua rea data set, a ot endeavors need to be taken to seect as appropriate rank, to fine tune the parameters, and to interpret the resuting mode We notice that most of the many research artices have had to dea with one individua case each time In the section, we consider two rea data sets and demonstrate how the goba optimization method performs under these uncertainties Depicted by the bue curve in Fig 9 are the daiy cosing god prices for 97 successive trading days recorded by Hipe and Mceod in 994 [9] Needess to say, many

19 og (RMSE) SNR = 3dB Comparison of RMSE SNR = 3dB Percentage of Groups 35% 30% 5% 0% 5% 0% 5% Sampe Wise Forecast SSA/Goba Error Histogram SNR=3dB SNR= 3dB 0% og of Forecast SSA Error to Goba Error Ratio Fig 8 Comparisons of the RMSE and forecast errors between the goba optimization method (red compact box stye) and the SSA agorithm (bue empty box stye) with respect to the signa (33) (Coor figure onine) 800 d=3 800 d=4 800 d= d=6 800 d=7 800 d= Fig 9 Approximation of 97-day god price (bue) by using the first 89 daiy prices via SSA (red) and goba optimization (green) Prices for day 90 to 97 are by forecasting (Coor figure onine) factors woud have affected some abrupt changes of god price in the series The fundamenta question is whether the god price can be forecast based on past vaues For our experiment, we take the first 89-day prices and wish to compare the forecast the ast 8- day prices We take the suggestion in [9] by setting the window ength at p = 45 Aso suggested is the rank at about d = 5 or 6, but we choose to try a ranks from 3 to 8 For the goba optimization, we try 50 starting vaues Approximations by both methods at d = 3 are neary identica The SSA approximations for d = 4andd = 5 is distinguishabe, but for higher ranks the SSA approximations are amost identica It is worth noting that for d 5, the SSA approximations fuctuate amost at the same rhythm as the observed daiy prices This behavior is criticay important and show the amazing abiity

20 d = d = d = d = d = d = Fig 0 Approximation of monthy accidenta deaths (bue) in the USA by the first 63 monthy data via SSA (red) and goba optimization (green) Data after the 64th month are by forecasting (Coor figure onine) of the SSA method In contrast, the goba optimization method is abe to provide smoothing of the otherwise fuctuant prices, but it fais to pick up the fuctuant point On the other hand, the SSA method fais to predict the deep dive of the god price at the ast 8 days, whie on two occasions (d = 6, 7) the goba optimization predicts a continuing down side of the god price We have stressed the determination of a suitabe rank d is itsef a difficut probem This experiment strongy suggests that in rea appications a ot more factors, some might be even unknown, wi affect the fina construction of the mode Simiary, depicted in Fig 0 are the monthy accidenta deaths in the USA from 973 to 978 This data set has been used by many studies for testing the efficiency of agorithms [7,,3] We set the window ength at p = 3 and use the first 63 monthy data to construct the mode We try a ranks from d = 0 to 5 Except for the case d = where the mode from the goba optimization gives the wrong prediction, both methods perform reasonaby we, whereas we think that the simpe and fast SSA method is superior to more expensive goba optimization approach It is important to point out one possibe expanation on why the SSA method can outperform the goba optimization method in both experiments above We notice that the data stay far away from the unit disk and change rapidy, whereas in our setup for the goba optimization we have restricted λ (See 6) to be within the unit disk As such, the goba optimization method is more suitabe if the data inherit some asymptotic behavior when the powers of λ die out When the data simpy fuctuate as we have seen in the two rea data sets, we rey on β to compensate the decine of the powers of λ We may aso need more terms in the summation (7), which might then bring in more noise than the ower rank approximation 7Concusion Aternating projection between two manifods with the aim of seeking out a point of intersection, or a pair of points yieding minimum distance, has ong been empoyed across the fieds

21 in a variety of appications Starting with a Hanke matrix embedded with a contaminated time series, the SSA expoits this notion to find, in particuar, a ow rank Hanke approximation The idea is simpe enough for quick impementation and has gained popuarity among practitioners This paper investigates the effectuaity of the SSA agorithm by comparing its reconstructed time series with the absoute best approximation obtained from goba optimization techniques The framework is buit upon Vandermonde factorization of Hanke operators, which provides natura parameters over compact feasibe set and, hence, guarantees the existence of a goba soution Setting the stopping criterion at the nearness of two successive iterates within a uniformy specified bound, we find that the convergence behavior of the SSA agorithm is generay insensitive to the strength of the added noise That is, so ong as the size of the embedding is fixed, it usuay takes about the same order of iterations for the SSA to converge, regardess of the SNR On the other hand, as is expected, stronger noise does degrade the quaity of the fina approximation to the origina, uncontaminated signa What is most interesting is the empirica evidence that, despite of its simpicity, the SSA does indeed perform remarkaby we when comparing to the resuts from the more compicated goba methods This is especiay so when the SNR is reativey high Since the goba methods, if successfu, produce the best possibe approximation to the Vandermonde parameters from the contaminated data, such a comparison justifies therefore that, for exporatory mode-buiding purpose, the simper SSA agorithm might be sufficient as a handy too For more rigorous affirmative procedure, however, we must stress that the SSA does not aways give rise to an optima soution 8 Appendix In this section we expain how the gradient information can be derived for the goba optimization code MutiStart Let the objective function (6) be rewritten in the form f (λ,,λ d ) = ΛL(Λ; z) z,λl(λ; z) z, with inner product p, q := i p iq i for compex vectors Empoying poar coordinates (ρ i,θ i ) when λ i = ρ i e ıθ i as variabes in our goba method, we identify Λ = Λ(λ,,λ d ) = Λ(ρ,,ρ d,θ,,θ d ) We now cacuate the gradient of f with respect to the rea variabes ρ i and θ i Using the fact that Λ (ΛL(Λ; z) z) = 0, the action of the Fréchet derivative of f with respect to Λ at a compex matrix H is given by f Λ H =R( H, (ΛL(Λ; z) z) L(Λ; z) ), where R stands for the rea part of a compex-vaued quantity and the same notation, denotes the generaization to the Frobenius inner product of compex matrices On the other hand, the action of the Fréchet derivative of Λ with respect to variabes ρ =[ρ,,ρ d ] and θ =[θ,,θ d ] at vectors h, k R d can be expressed as