Using Artificial Neural Networks and Support Vector Regression to Model the Lyapunov Exponent

Transcription

1 Usng Artfal Neural Networks and Support Vetor Regresson to Model the Lyapunov Exponent Abstrat: Adam Maus* Aprl 3, 009 Fndng the salent patterns n haot data has been the holy gral of Chaos Theory. Examples of haot data nlude the flutuatons of the stok market, weather, and many other natural systems. Real world data has proven to be extremely dffult to predt due to t hgh dmensonalty and the potental for nose. It has been shown that artfal neural networks have been able to aurately alulate the Lyapunov Exponent, a feature that determnes the dvergene of two ntally lose trajetores and thus haos. A varaton of the support vetor mahne alled support vetor regresson s used n funton approxmaton and applyng ths tool to haot data may yeld another model that an also be used n the predton of the Lyapunov Exponent. Sne support vetor mahnes have been found to arrve at a predton muh faster than the artfal neural network, t may prove to be the model to use n tme senstve applatons. Introduton Nonlnear predton attempts to desrbe the nherent qualtes of a gven dynamal system usng mathematal models. Models an range n omplexty from a smple set of equatons suh as the logst equaton used to model the arryng apaty of a spees n an eosystem [] to the reaton of artfal neural network models wth hundreds of parameters used n foreastng exhange rates []. These dynamal systems as well as the rse and fall of populatons, the stok market, and the weather exhbt haos where there s a senstve dependene on ntal ondtons. Sne two ntally lose ondtons wll dverge exponentally fast, haot dynams pose a formdable hallenge for long term foreastng. * Unversty of Wsonsn Madson Contat Emal: amaus@ws.edu Asde from long term foreasts, one goal of nonlnear predton s to reate a mathematal model that aurately represents data take from a system. The hope s that by reatng an aurate model, underlyng features n the system wll be replated as well. One suh feature, the largest Lyapunov exponent (LE) s valuable n determnng whether or not a gven system s haot. Its value desrbes the average rate at whh predtablty s lost. The largest Lyapunov exponent s usually geometrally averaged along the orbt and s always a real number. Ths Lyapunov exponent s alulated numerally usng the proedure n [3]. When alulatng the LE for a model, we run fxed length foreasts usng the models reated by the artfal neural network and support vetor regresson model on tranng data. These foreasts are fxed at 5000 tme steps, 5000 was hosen beause t gves a good approxmaton of the true LE and s

2 quker to alulate than one mllon tme steps to fnd the true LE of the models and systems beng studed. The foreast from a model an onverge to a number of dfferent attrators that represent varyng LEs. A foreast that onverges exponentally fast to a fxed pont wll have an LE wth a value of negatve nfnty. The lmt yle s a ase that s often hard to dstngush wthout the alulaton of the LE beause the data may be followng a strt path wth a large number of perods or steps before repeatng. The LE of a lmt yle ranges between negatve nfnty and 0. An LE of zero denotes a system suh as x ( x ) n+ = 3xn n, requrng an nfnte number of ponts to onverge to an attrator. Dynamal systems suh as the Hénon map [4] have a postve LE and onverge to what are alled strange attrators. These attrators appear to follow an orbt but t never repeats. The Hénon map whose form x, n+ =.4xn 0. 3xn an be ntalzed wth x = (.,.) and produes a strange attrator plotted by x n versus x n as n Fgure. The LE alulated for the Hénon map usng 5000 tme steps equals.4093 and the goal of ths paper s to desrbe how neural networks and support vetor regresson models an be used to alulate the LE for ths map and other smlar dynamal systems. Models Studed Artfal neural networks have been used n past for modelng tme seres [5], nonlnear predton [6], and analyzng the underlyng features of data [7]. Hornk et al. [8] lamed that these models were unversal approxmators wth the ablty to represent any funton wth an arbtrary number of hdden unts. Ths model, seen shematally n Fg., used to replate the LE uses a sngle layer of hdden unts to perform funton approxmaton and takes the form, n d xˆ k = b tanh a0 + aj xn j, = j= where a s an n d matrx of oeffents, and b s represented by a vetor of length the number of neurons n. The a matrx represents the onneton strengths of the neurons, and the b vetor s used to modulate the strength of a partular neuron. In ths model, a 0 represents the bas term that s ommonly used n neural network arhtetures. FIG : Strange Attrator of the Hénon Map FIG. : Sngle Layer, Feed-Forward Neural Network

3 The neural network uses the d last ponts from a salar tme seres x n the predton of the next step k. In ths model, d s onsdered the dmenson or embeddng of the neural network. It represents the number of nputs or prevous tme lags used to predt eah subsequent value n the tme seres. The weghts n a and b are updated usng a varant of the smulated annealng algorthm where temperature s held onstant at, effetvely removng the term. For eah oeffent, a Gaussan neghborhood s searhed whose sze s hanged based on progress n tranng. The oeffents are hosen to mnmze the mean-square predton error, e = = ( xˆ x ) The other model studed, Support Vetor Mahnes have tradtonally been used n lassfaton problems; n 996 Vapnk et al. [9] extended ths model to perform robust funton approxmaton that s nsenstve to small hanges n the data. The LbSVM Toolbox for Support Vetor Mahnes by Chang and Ln [0] was used n ths projet. Vapnk lamed that ε-insenstve Support Vetor Regresson would provde effetve estmaton of funtons [] and SVR has also been used n foreastng the stok pre hange wth the help of analysts []. As Smola stated n [] SVR s used to approxmate a funton f (x) that has at most ε devaton from the targets x k. Ths effetvely removes ertan data ponts from the model s approxmaton of the data. A funton s found that an approxmate just a subset of the atual data and beause of the haot nature of the data beng studed; usng a subset of ponts should hange the model s ablty to replate the same haot dynams of the system beng studed. However, we found that t performs just as well as neural networks n approxmatng the LE. The funton that the SVR model produes takes the form f ( x) = w, x + b, where w, x denotes the dot produt of w and x. Calulaton of w and b nvolves mnmzng w through a onvex optmzaton problem [3] that mnmzes a Lagrangan funton [4]. The omputaton of b an be performed n dfferent ways. LbSVM omputed b by fndng the average value of the support vetors wth Lagrange multplers between 0 and the user-defned varable C. One a model was produed, the foreast of p ponts took the form xˆ = b + w K, p s = where s represents the number of support vetors used n the model. In the LbSVM toolbox, ths was automatally alulated. K represents the kernel used n the model and the approxmaton. In ths experment, K took the form of a Gaussan Radal Bass funton wth an adjustable parameter σ. The tranng data and foreast were ntalzed wth data from the Hénon map. Two tme seres were used to tran the model. One tme seres served as tranng data for the model, the other tme seres served the test tme seres that predted the foreastng error. After obtanng a model wth a low foreastng error ompared to the system beng studed, foreastng was used to alulate the LE. Durng the tranng of the SVR model, we optmze the model on the set of

4 data but there are a number of parameters to tran to reate a low test error. The sze of ε, the ost of vetors on the wrong sde of the funton C, the kernel parameters, et, must all be hosen. Sne eah parameter an be any real number, a tranng shedule was reated that was smlar to the neural networks tranng. The parameters traned C, ε, σ, p, were ntalzed wth values.00,.0,., and., respetvely. To tran the parameters, the parameters were vared aordng to a Gaussan neghborhood whose sze depends on the progress beng made n tranng the model. To objetvely hoose one model over another the mean-square predton error was used to ompare the foreast of the SVR and test data wth one aveat. The sze of the test tme seres used was effetvely shortened by a weghtng sale reated so the model foused on replatng the frst ponts n the test tme seres. e ' = = ψ ( xˆ x ) The ψ was hosen arbtrarly for ths projet and set to seven and for eah addtonal k, ψ k was halved, though further researh may fnd that ψ an be optmzed for the system beng studed. Ths sheme was reated beause of the haot nature of both the model reated and the data beng modeled, senstve dependene on ntal ondtons would ause the two sets of data to be nomparable after just tens of steps. If we tred to mnmze the mean square error between the entre foreast and the test data, t eventually led to a model that found the average between the hgh and low peaks n the test data. Numeral Results After tranng the SVR model on 400 ponts, the LE was alulated and found to be.488 for the Hénon map wth a test error of x 0-5. The strange attrator of the SVR model produed was vsually ndstngushable. It s mportant to note for all ases that dfferent tranng nstanes lead to dfferent results and these were hosen beause they were the lowest mean square errors found. The neural network was traned for one mllon epohs wth = 400 ponts taken from the Hénon Map. After tranng, the LE of the model was alulated. The LE was found to be.3843 wth a mean-square error of x 0-5. Fgure 3 shows the strange attrator produed by ths network, as seen, the neural network had some trouble reprodung the attrator for the Hénon map exatly though other better traned neural networks may perform better. Lkewse, the dfferene between the true LE of the system and the neural network may arse from stoppng the tranng too soon. A more aurate model seems to orrelate wth a more aurate LE of the model to the test system. FIG. 3: The Hénon map attrator reprodued by the Neural Network 3

5 Two other systems were studed usng the Neural Network (NN) and SVR models, the Logst map (LM) [5], x = 4x ( x n+ n n and the delayed Hénon map (DHM) [6], ) x n+ =.6xn 0. xn 4 whh led to the followng table of LEs. FIG. 5: The delayed Hénon map attrator reprodued by the SVR model Atual NN SVR LM DHM Table : The LE approxmatons usng 5000 steps n the foreast of eah system and model The neural network and SVR model both produed strange attrators for the delayed Hénon map, Fg. 4 and Fg. 5, respetvely that were vsually smlar to the atual strange attrator, Fg. 6. The Neural Network and SVR model aheved a mean square error of x 0-6 and x 0-6, respetvely. FIG. 6: Strange Attrator for delayed Hénon map The SVR model dd not reonstrut the exat struture of the strange attrator for the Logst map but ths ould be due to nadequate tranng, the error was The neural network was able to reonstrut the attrator and the LE wth an error of x 0-6. The attrator of the logst map and the attrator produed by the SVR model are seen n Fg. 7 and 8, respetvely. FIG. 4: The delayed Hénon map attrator reprodued by the Neural Network FIG. 7: Strange Attrator for Logst map 4

6 FIG. 8: The Logst map attrator reprodued by the SVR model Though both models produe mean square errors, t s not far to ompare them usng these values. The weghtng on the errors n SVR model was dfferent from that of the Neural Network. These objetve funtons were smply used to tran eah model. If a omparson was needed, a more aurate measurement would be to dentfy the dfferene between the true Lyapunov exponent of the system beng studed and the Lyapunov exponent produed by eah model. Dsussons and Conluson Artfal neural networks serve as exellent models for reonstrutng the data and studyng the LE for the Hénon Map. If we appled these models on data taken from a system whose LE s unknown, they would perform well gven that they are adequately traned. Certan measures ould be taken to ensure that the orret LE s found whh nlude ross valdaton and ensembles of neural networks among other ways to provde that the neural network does not overft the data t s gven. One drawbak of ths neural network model s that they requre many more alulatons than SVR and unlke SVR (pror to tranng the parameters), neural networks often fal to fnd the global optmum or absolute best way to model the data therefore t s mperatve to tran many dfferent networks on the same data. Support Vetor Regresson performed muh better than expeted. Mattera et al. found that reonstrutng the strange attrator and the haot dynams for a gven system usng SVR s not always possble [7] but by usng a weghtng system on the foreast of the data, t s possble to tran the parameters for these models to produe an aurate representaton of the data. The number of free parameters reated a hallenge for tryng to alulate the LE, a tranng shedule was reated to alter these parameters and reonstrut both the haot dynams and the strange attrator. Lke the neural network, usng ths type of tranng auses the model parameters to fall nto loal optmums, one way to overome ths s to use an ensemble of SVR models to fnd an average Lyapunov exponent. One mportant aspet of tme seres analyss nvolves determnng the embeddng dmenson for a model. For the attrator to be reonstruted t must be embedded n an approprate dmenson so t an be unfolded so one premage does not produe two dfferent mages. A suffent ondton for the embeddng dmenson was provded by Takens [8] who showed that omplete unfoldng s guaranteed f the tme-delayed embeddng spae has a dmenson at least one greater than twe the dmenson of the orgnal state spae. For the purpose of ths projet, the data was embedded n a spae that was guaranteed to unfold t, by lookng at the tme-delayed equatons of eah system, the embeddng ould be dedued. For the Hénon map, the embeddng was, for the delayed Hénon map, 4, and for the Logst map,. For an unknown system, embeddng an be alulated usng an algorthm known as false nearest neghbors [9]. 5

7 Support Vetor Regresson has shown to be useful n funton approxmaton, foreastng, and reonstruton of haot dynams for a gven system. Support Vetor Regresson reates a model that uses far less ponts than the artfal neural network. Tranng the SVR model parameters takes less tme than the neural network and n some ases t arrves at a better approxmaton of the LE for the system beng studed. SVR serves as another model to approxmate the LE for a dynamal system and t would be nterestng to test SVR more rgorously on ontnuous systems and real-world data. Referenes [] E. Allman and J. Rhodes, Mathematal Models n Bology: An Introduton (Cambrdge, New York, 004). [] L. Yu, S. Wang, and K. K. La, Foregn-Exhange-Rate Foreastng wth Artfal Neural Networks (Sprnger, New York, 007). [3] J. C. Sprott, Chaos and Tme-Seres Analyss (Oxford, New York, 003). [4] M. Hénon, Comm. Math. Phys., (976). [5] S. Trona, M. Gonab, and R. Baratta, Neuroomputng 55, 3-4 (003). [6] G. Zhang, B. Patuwo, and M. Hu, Int. J. Foreastng 4, (998). [7] J. Prnpe, A. Rathe, and J. Kuo, Int. J. Bfuraton Chaos, 4 (99). [8] K. Hornk, M. Stnhoombe, and H. Whte, Neural Networks, 5 (989). [9] V. Vapnk, S.E. Golowh, and A. Smola, n Advanes n Neural Informaton Proessng Systems, San Mateo, 996, edted by M. Mozer, M. Jordan, and T. Petshe, p. 55. [0] C. Chang and C. Ln, LbSVM: a lbrary for support vetor mahnes (00). [] V. Vapnk. The Nature of Statstal Learnng Theory. (Sprnger, New York, 995). [] Z. Zhang, C. Sh, S. Zhang, and Z. Sh, n Advanes n Neural Networks -ISNN, Chengdu, 006, edted by J. Wang, Z. Y, J. Zurada, B. Lu, H. Yn. [3] A. Smola and B. Sholkopf. Stat. Comput. 4, 3 (004). [4] S. Haykn Neural Networks, a Comphrensve Foundaton. (Prente Hall, New Jersey, 99). [5] R. May, Nature, 6 (976). [6] J. C. Sprott, Eletron. J. Theor. Phys. 3, (006). [7] D. Mattera and S. Haykn n Advanes n Kernel Methods: Support Vetor Learnng, edted by B. Shölkopf, C. J. Burges, and A. J. Smola, p.. [8] F. Takens, Detetng strange attrators n turbulene (Sprnger, Berln, 98). [9] H. Abarbanel, Analyss of Observed Chaot Data (Sprnger, New York, 996 6