Fusion of Neural Classifiers for Financial Market Prediction

Transcription

1 Fuson of Neural Classfers for Fnanal Market Predton Trsh Keaton Dept. of Eletral Engneerng (136-93) Informaton Senes Laboratory (RL 69) Calforna Insttute of Tehnology HRL Laboratores, LLC Pasadena, CA Malbu, CA ABSTRACT: Foreastng fnanal urreny markets s an extremely hallengng problem beause of the omplex and hghly haot nature of suh markets. Motvated by the substantal profts that ould be ganed by havng a system that ould aurately predt large trends n the market, fnanal nsttutons are lookng on advanes n mahne learnng, neural networks, and statsts to provde them wth another analyss tool. Researhers are nvestgatng the use of bak-propagaton neural networks for fnanal tme seres predton, due to ther suess on other pattern reognton problems suh as mahne & handwrtten harater reognton. However, to date ther performane has been onsderably lower than that aheved on the harater reognton problem doman. Ths s due n large part to the tremendous amount of nose nherent n the data, whh hnders the learnng of good mappng funtons. We beleve that redundant foreastng through the synergst use of multple neural network predtors n ombnaton wth an ntellgent deson aggregaton sheme, may be the key to nreasng the suess rate of omputer-aded foreastng systems. In ths paper, we ondut an empral and omparatve study on the use of alternatve methods for data preproessng, ftness evaluaton, and deson fuson. We demonstrate the advantage of our multple lassfer approah n predtng hanges n the foregn exhange rate of the U.S. Dollar versus the German Mark over 250 days of tradng. Keywords: fnanal market analyss, tme seres predton, lassfer fuson, evdental reasonng, neural networks 1. Introduton Reently, the dea of ombnng multple neural networks has beome an area of great nterest amongst pattern reognton researhers [3], [4], [5]. The ratonale behnd ths urrent dreton s that often real-world problems are far too omplex for any sngle method to generate the best results for all possble types of nputs. Instead, an ensemble onsstng of multple models s learned, and then the lassfaton deson s made by ombnng the lassfatons of the ndvdual models. Ths approah has led to mproved reognton rates over any of the ndvdual onsttuents of the ensemble. However, the amount of mprovement n auray has been found to be dretly related to the error ndependene of the ndvdual lassfers. Hene, ths sheme has typally been appled to the fuson of omplementary or orthogonal feature sets, suh as strokes and avtes for harater reognton, sne lassfers based on very dfferent feature sets often make errors n an unorrelated manner. In the future, we ntend to explore the aggregaton of foreastng models based on multple feature sets suh as wavelet and Fourer oeffents. However, n ths paper we study the benefts of ombnng sngle layered feedforward neural networks traned by bakpropagaton on an dental data set. In ths ase, network dversty was aheved by the nherent randomness assoated wth the bak-propagaton algorthm s ntalzaton of a network s weghts. Pattern lassfers traned n ths manner an be vewed as approxmatons from dfferent dretons to the same goal, somewhat lke reahng the peak of a mountan from dfferent startng ondtons. Hene, eah lassfer may behave dfferently wth eah ndvdual nput pattern, however n the long run ther error rates wll be nearly the same. Under these rumstanes, the fuson of redundant lassfers

2 an potentally mprove the overall performane of the system by redung the unertanty assoated wth the lassfaton, just as n everyday lfe we often onsult more than one expert before makng an mportant deson. Ths paper s organzed as follows. In seton 2, we provde an overvew of our mult-lassfer system for predtng fnanal markets. Seton 3 dsusses the desgn of the Artfal Neural Network (ANN) lassfer used for tme-seres predton, nludng the alternatve ost or error funtons utlzed durng the bak-propagaton learnng of the network parameters. Seton 4 desrbes the fuson methodologes explored for the aggregaton of the predton desons. The latter setons present expermental results onduted on the U.S Dollar vs. German Mark fnanal urreny data and the onlusons that may be drawn from ths study. 2. The System Sne dfferent pattern lassfers wll exhbt dfferent strengths and weaknesses, we propose a multple neural-based lassfer system for fnanal tme-seres predton whh ontans an ntellgent deson makng sheme that fuses the predtons, suh that eah lassfer s defenes are ompensated for whle preservng ts strengths. A number of dfferent strateges exst n ombnng lassfer desons. Two or more lassfers may be onatenated so that the output of one of them beomes the nput to another, or they may be operated n parallel. We hoose the later varant, where the group of lassfers to be ombned an be vewed as a group of experts lookng at the same problem from ther ndvdual ponts of vew and statng ther ndvdual predton about the future trend. The task performed by the deson module s to ombne the predtons n a manner suh that the overall unertanty assoated wth the fnal deson s redued. A blok dagram of the proposed system s shown n Fgure A Neural Network for Foreastng Fgure 1. Fuson of multple neural lassfers for mproved fnanal market analyss. The most suessful Artfal Neural Network (ANN) to be appled to pattern reognton tasks s the standard fully onneted mult-layered pereptron net. It learns a mappng between nput and output pars by adaptng ts weghts through bak-propagaton learnng algorthm. Fgure 2 shows the one-layer arhteture used n our fnanal tme-seres predton experments. The model assumes there exsts an underlyng omplex relatonshp between the urrent return and the pror returns over a twenty-day perod, hene the nput layer ontans 20 neurons. Generally, a sngle output neuron havng a nonlnear hyperbol tangent atvaton funton s used to produe values wthn the range of [- 1,1], where ts sgn ndates the dreton of hange n the market [1], [2]. However, n our multple lassfer system we would lke to be able to nterpret the network outputs as Bayesan a posteror probablty estmates, whh an then be easly ombned usng evdental reasonng methods. Rhard and Lppmann [6] showed that Bayesan probabltes are estmated when the desred network outputs are 1 of M lasses (one output s unty for the orret lass, all others are zero), and the network s traned by mnmzng the expeted mean square error (MSE) or the ross-entropy ost funton. Thus, we utlze two neurons n the output layer havng a sgmodal atvaton funton [0-1]. One of the output neurons s used to ndate the predton of an upward trend, whle the other ndates a downward trend n the market as shown n Fgure 2.

3 In stuatons n whh ths s true, the Gaussan probablty dstrbuton s approprate and the error term to be mnmzed s the mean square error. The error funton and ts dervatve used for weght update are: E p = k (t k y k ) 2 2σ 2 E p y k = (t k y k ) σ 2 Fgure 2. A sngle-layered pereptron used for fnanal foreastng. The network parameters were optmzed usng bak-propagaton learnng wth the followng varants. Frst, the weght update rule was modfed to nlude a momentum term α, whh along wth the learnng rate η were adapted durng tranng from ther ntal values of 0.9, and 0.1, respetvely. At eah tranng epoh, the tranng patterns were presented sequentally to the network for weght updatng. In addton, we norporated the oddness symmetry hnt nto the learnng proess by presentng eah tranng nstane followed by ts negaton of both the nput vetor and ts target value. In [2], t was shown that the symmetry hnt mproves the generalzaton ablty of the network, by preventng overfttng, and by restrtng the number of solutons the network may settle nto. Fnally, we expermented wth usng both the squared-error and the ross-entropy ost funtons for optmzng the network weghts. 3.1 Mean Square Error The tradtonal mean square error funton s the most popular ost funton used n the majorty of applatons for optmzng the weghts of a neural network. It has demonstrated good performane on real-world problems, and an be used for predton, lassfaton and regresson. It assumes that the network s error wll be normally dstrbuted about the predted values. where, σ 2 s typally fxed to be 1 and t k s the k th neuron s target value and y k ts output. When the Bayesan a posteror probabltes are estmated orretly, the lassfaton error rate wll be mnmzed, and the outputs sum to one suh that they an be nterpreted as probabltes. 3.2 Cross-Entropy Error Another popular ost funton measures the ross-entropy between atual outputs and desred outputs, whh are treated as Bayesan probabltes. Motvaton for ts use les n the assumpton that the desred outputs are ndependent, bnary, random varables, suh that the network s error wll be bnomally dstrbuted. Therefore, gven bnary target values of 0 and 1, we an wrte the learnng objetve n terms of the relatve or ross-entropy of the target value to the atual output of the network. The mnmzng error funton beomes: [ t log( y ) + (1 t ) log(1 y ] E = ) p k k k k k wth ts dervatve easly expressed as: Ep = y k t k y k 1 t k 1 y k where, t k s the k th neuron s target value and y k ts output. Ths ost funton an be nterpreted as mnmzng the Kullbak-Lebler probablty dstane measure. It weghts errors more heavly than the squared-error term, and thus the traned

4 network tends do a better job of predtng large hanges n the market at the expense of mslassfyng smaller varatons. For our applaton ths bas s desrable sne a falure to detet large shfts n the market s far more ostly than falng to detet smaller movements. 4. Combnng Classfer Predtons The fuson methodologes nvestgated for ombnng the predtons of the dfferent neural lassfers ranged from smple tehnques, requrng lttle omputaton suh as majorty votng and averagng, to the more omputatonally ntensve evdental reasonng tehnques: Bayesan, Dempster-Shafer s rule, and fuzzy ntegral fuson. In ths seton, we desrbe eah method of ombnaton employed n generatng the fnal predton deson. 4.1 Majorty Vote Ths sheme talles the lassfaton votes from all networks, then hooses the predton yeldng the maxmum number or that whh was ndated by the majorty (e.g., at least 3 out of 5 lassfers). 4.2 Arthmet Mean In ths ombnaton sheme, we smply average the ndvdual lassfer outputs. The maxmum of the averaged values s hosen as the orret predton lass. y = 1 n y, where, n s the number of lassfers. 4.3 Bayesan Evdental Reasonng The Bayesan evdental reasonng tehnque s strongly founded upon the framework of Fgure 3. Informaton fuson through Bayesan evdental reasonng. probablty theory, however the underlyng assumptons t requres for the propagaton of belefs may or may not be true n pratal stuatons. For example, Bayesan reasonng assumes that the pees of evdene E to be aggregated are statstally ndependent. Ths assumpton may not be true n ases where ausal or ontextual relatonshps exst, however for the purposes of fusng multple neural foreasters we wll assume that the evdene soures are ndependent wth respet to the errors they make. Fgure 3. shows the nformaton fuson proess under an evdental reasonng framework. Bayesan theory uses an Odds-Lkelhood Rato formulaton of Bayes rule to aggregate the evdene from multple soures. The a pror odds O(H) of a gven lass hypothess H (e.g., upward trend, downward trend) s related to ts a pror probablty P(H) by the followng relatons: O(H) = P(H) P(~ H) and P(H) = O(H) 1+ O( H) where ~H means not H. The lkelhood of the evdene E, gven that the hypothess H s true, s: L(E H) = P(E H ) P(E ~ H)

5 The lass probabltes for eah hypothess may be estmated from tranng data, and the neural network outputs dvded by these probabltes to produe saled lkelhoods, where the salng fator s the reproal of the unondtonal nput probablty. The formula for updatng the a posteror odds of a hypothess H, gven the evdene E observed s:,..., ) ( ) n O( H E 1, E E 2 n = O H L ) 1 ( E H. = and, the belef or a posteror probablty for a hypothess s smply: P(H E, E,..., E 1 2 n ) = O(H E 1,E 2,...,E n ). 1+ O( H E, E,..., E 1 2 n ) The fnal predton s hosen to be that hypothess H havng the greatest probablty gven the aumulated evdene. 4.4 Dempster-Shafer s Evdental Reasonng Dempster-Shafer s (DS) theory of evdene s another tool for representng and ombnng evdene, whh s onsdered to be a generalzaton of Bayesan theory. It s more flexble than Bayesan when our knowledge s nomplete, by permttng the assgnment of an gnorane term rather than forng an overommtment towards belef or dsbelef n a hypothess. Rather than representng the probablty of a hypothess H by a sngle value P(H), DS theory bnds the probablty to a subnterval [Bel(H),Pl(H)] of the nterval [0,1], where Bel(H) - belef and Pl(H) - plausblty represent the lower and upper bounds on the probablty, suh that: Bel(H) P(H) Pl(H). When Bel(H)=Pl(H), Dempster-Shafer theory redues to Bayesan. Aordng to D-S theory, the set of all possble outomes (.e., the sample spae) n a random experment s alled the frame of dsernment (FOD) denoted by Θ. For our problem, the frame of dsernment would be Θ={upward trend, downward trend}. Assoated wth eah of the neural network lassfers s a bas probablty assgnment (bpa), whh expresses the degree to whh the evdene onfrms or supports a hypothess. It s assgned aordng to the neural network output y k, and s estmated from the statsts of the tranng set. Gven two bpa s m 1 ( ) and m 2 ( ) dserned n the same frame, ther ombned belef n a hypothess H an be omputed usng Dempster s rule of ombnaton: m (B)m (C) 1 2 m (H) 1 2 = B C=H 1 m (B)m (C) 1 2 B C= Ths rule s appled reursvely untl the evdene from all n soures s aggregated. The output of the D-S fuson module s the followng nterval of belef: Belef(H) = m n (H) Plausblty(H) = 1 - Belef(~H) Belef Interval = [Bel(H), Pl(H)]. Then, the fnal predton hypothess havng the largest amount of support wth the smallest unertanty (.e., the dfferene Pl(H)-Bel(H)) s hosen. 4.5 Fuzzy Integral Fuson The fuzzy evdental reasonng sheme vews the outputs of multple networks or experts as ndependent soures of "objetve" or "observed" evdene, whh s ombned wth an evaluaton of the "relevane" or "mportane" of that evdene wth respet to eah hypothess. The ombnaton of both types of nformaton s aomplshed usng a fuzzy aggregaton operator alled the fuzzy ntegral. The fuzzy ntegral s a

6 nonlnear funton defned wth respet to a fuzzy measure, whh s a generalzaton of a probablty measure, that replaes the addtvty property (.e. P(A) + P(~A) = 1) wth a weaker monotonty ondton. The "relevane" of an nformaton soure s aptured by ths fuzzy measure or densty, whh may be subjetvely assgned by a human, or estmated from the tranng data. The fuzzy ntegral operator ntegrates the outputs of the neural network experts wth respet to ths aggregated relevane funton to ompute the possblty expetaton between the pooled evdene and t's ombned relevane. As shown n Fgure 4, the fuzzy ntegraton or possblty expetaton may be nterpreted as searhng for the maxmal agreement between the atual evdene, and ts aggregated relevane. Algorthm: For all lasses or hypotheses { 1) Sort lassfer evdene: h x ) h ( x )... h ( x ( 1 2 n { x1, x2,... x} A = 2) Fnd lambda parameter: n 1+ = (1 + g ) = ( 1, + ) = 1 3) Compute aggregated relevane: 1 g ( A1) = g g ( A ) = g + g ( A 1) + g g ( A 1 ) 4) Compute possblty expetaton: e = h ( x) g x n = MAX = 1 ), { MIN[ h ( x ), g ( A )]} } Compute fnal lassfaton deson: # Classes =1 ( e ) Class = MAX. Separate aggregaton networks are needed for fusng nformaton regardng eah hypothess. The fnal predton lassfaton or hypothess deson s taken to be the one returnng the largest fuzzy ntegral value as shown n Fgure 4. Fgure 4. Fuzzy ntegraton. 5. Expermental Results The data used to evaluate our system onssted of the losng pres of the U.S. Dollar versus the German Mark urreny exhange rate over a four-year perod. The pres (P t ) were normalzed to ompute the daly return (R t ) usng the followng formula: R t = P t P t 1 P 100% t 1 A plot of the omputed daly returns s shown n Fgure 5. Ths normalzed data was then dvded nto three sets wth the frst 500 samples used to tran the neural networks, and the remanng samples dvded nto two test sets. We hose to ombne fve neural networks eah traned n the same manner, although due to random weght ntalzaton, eah network started at a dfferent pont n the error surfae. Table 1 presents the predton ht rate results for the neural networks traned usng the squared-error funton and the oddness symmetry hnt. The performane on the seond test set s lower due to the fat that the tranng data s not representatve of the urrent market status, but nstead s out-of-date. Table 2 presents the predton ht rate results for the

7 Daly Returns USDM Day Bayesan a po steror probabltes, whh an easly be onverted to saled lkelhoods, and then ombned for hgher level deson makng. An ntellgent nformaton fuson sheme was used to ombne the predtons of the ndvdual lassfers, suh that the auray and relablty of the fnal predton was mproved. We obtaned nearly a 15% nrease n performane over the predtons of the ndvdual neural lassfers. In the future, we ntend to nvestgate the generaton of omplementary foreasters through the use of tme-frequeny transformatons. Fgure 5. U.S. Dollar vs. German Mark exhange rates. neural networks traned usng the ross-entropy ost funton and the oddness symmetry hnt. The results are lower beause ths ost funton tends to predt only large trends n the market at the expense of smaller varatons. Havng traned our neural-based foreasters, the goal s to ombne the outputs from the ndvdual networks to obtan an overall predton of the market trend. Fve dfferent fuson methodologes were mplemented and tested. Table 3 & 4 present the predton ht rates on the two dfferent test sets obtaned by ombnng the fve neural networks traned usng the MSE error and the oddness symmetry hnt. By examnng the results, we see that there s a lear beneft n usng evdental reasonng methods for fusng the ndvdual network predtons. We obtaned nearly a 15% nrease n performane over the predtons of the ndvdual neural lassfers 6. Conlusons We ntrodued a multple redundant neuralbased lassfer system for fnanal market analyss. Eah lassfer was traned n an dental manner, however random weght ntalzaton provded some network dversty. The objetve funtons used to optmze the network weghts produed estmates of the Referenes [1] Tan, C., and Wttg, G., A Study of the Parameters of a Bak-propagaton Stok Pre Predton Model, n Pro. ANNES, [2] Abu-Mostafa, Y.S., Fnanal Market Applatons of Learnng From Hnts, Neural Networks n the Captal Markets, Wley, [3] Jaobs, R., and Jordan, M., Nowlan, S., and Hnton, G., Adaptve Mxtures of Loal Experts, Neural Computaton, 3:79-87, [4] Xu, L., Krzyzak, A., and Suen, C., Methods of Combnng Multple Classfers and Ther Applatons to Handwrtng Reognton, IEEE Trans. Syst. Man Cybern., 22: , [5] Benedktsson, J., and Swan, P., Consensus Theoret Classfaton Methods, IEEE Trans. Syst. Man Cybern., 22: , [6] Rhard, M., and Lppmann, R., Neural Network Classfers Estmate Bayesan A Posteror Probabltes, Neural Computaton, 3: , [7] Chauvn, Y., and Rumelhart, D.E., Bakpropagaton: Theory, Arhtetures, and Applatons, Erlbaum, [8] Masters, T., Sgnal and Image Proessng wth Neural Networks: A C++ Sourebook, pp.38-55,wley, [9] Abd, M., and Gonzalez, R., Data Fuson n Robots and Mahne Intellgene, Aadem Press, [10] Deboek, G.J. (edtor), Tradng on the Edge: Neural, Genet, and Fuzzy Systems for Chaot Fnanal Markets, Wley, 1994.

8 Table-1 Performane of networks traned usng the MSE and oddness symmetry hnt. % IN Samples Total Samples = 500 (1 500 days) % OUT Samples Total Samples = 253 ( days) % OUT Samples Total Samples = 250 ( days) Classfers Corret Error Corret Error Corret Error Table-2 Performane of networks traned usng the Cross-Entropy error & oddness hnt. % IN Samples Total Samples = 500 (1 500 days) % OUT Samples Total Samples = 253 ( days) % OUT Samples Total Samples = 250 ( days) Classfers Corret Error Corret Error Corret Error Table-3 Performane ombnng fve networks traned usng MSE error and oddness symmetry hnt over the frst test set onsstng of ( days). Majorty Vote Arthmet Mean Bayesan Reasonng Dempster- Shafer Fuzzy Integral Corret Error Table-4 Performane ombnng fve networks traned usng MSE error and oddness symmetry hnt over the seond test set onsstng of ( days). Majorty Vote Arthmet Mean Bayesan Reasonng Dempster- Shafer Fuzzy Integral Corret Error