Training Neural Networks Beyond the Euclidean Distance, Multi-Objective Neural Networks using Evolutionary Training.

Transcription

1 Training Neural Neworks Beyond he Euclidean Disance, 1 Muli-Objecive Neural Neworks using Evoluionary Training. Absrac When commied o forecasing or classificaion asks Neural Neworks (NNs) are yically rained wih resec o Euclidean disance minimisaion / likelihood maximisaion. This is commonly irresecive of any oher end user references. In a number of siuaions, mos noably ime series forecasing, users may have oher objecives han solely Euclidean minimisaion. Users may, for examle, desire model redicions be consisen in heir residual error, be accurae in heir direcional change redicions, or may refer oher error measures han jus Euclidean disance. In his aer a mehodology is devised for raining Muli Layer Percerons (MLPs) wih comeing error objecives using an Evoluionary Sraegy (). Exerimenal evidence is resened, using he Sana Fe comeiion daa, which suors he effeciveness of his raining mehodology. Idenical neworks are rained wih differen and comeing error measures. Differen soluions are achieved, and hese soluions are shown in general o reflec he error measure references as defined during raining. Areas for fuure research and model exensions are also discussed. Index Terms Neural Neworks, Evoluionary Sraegies, Mulile Objecives, Time Series Forecasing. I. INTRODUCTION The use of NNs in he ime series forecasing domain is now well esablished, here are already review aers on his maer [2], as well as mehodology sudies [8, 10]. The main aribue which searaes NN ime series modelling from radiional Economeric mehods, and he reason raciioners mos ofen sie for heir use, is heir abiliy o generae non-linear relaionshis beween ime series inu variables wih lile or no a riori knowledge of he form ha his non-lineariy migh ake. This is oosed o he rigid srucural form of mos oher ime series forecasing mehods (e.g. linear regression models, Exonenial Smoohing models, (G)ARCH, and ARIMA). Aar from his imoran difference, he underlying aroach o ime series forecasing iself has remained relaively unchanged during is rogression from exlici regression modelling o he non-linear generalisaion aroach of NNs. Boh of hese aroaches are fundamenally based on he conce ha he mos accurae forecas, if no he acual realised (arge) value, is ha wih he smalles Euclidean disance from he acual. When measuring redicor erformance however, raciioners use a whole range of differen error

2 2 measures [1]. These error measures on he whole end o reflec he references of oenial end users of he forecas model, as oosed o jus he Euclidean measure. Recen aroaches o ime series forecasing using NNs have inroduced limied augmenaions o he radiional gradien descen algorihm in order o enalise aricular misclassificaions more heavily, e.g. [11]. They do no however resen a mehodology ha allows he user o sae heir own forecas references in erms of he many error measures commonly in use, and rain he NN model accordingly. This aer aims o solve his roblem by develoing mehods for raining neural neworks wih mulile and comeing objecives, hrough he framework of raining. In his aer Neural Neworks will be devised ha can incororae such objecives as error consisency and Direcion Success, which will be described in secion II. The model iself, and he necessary accomanying algorihms for objecive weighing and raining comleion, will be resened in secion III. Exerimenal resuls on ublicly available ime series daa will be resened in secions IV - VI using objecives of Mean Absolue Percenage Error (MAPE) minimisaion, Roo Mean Squared Error (RMSE) minimisaion and he maximisaion of he Percenage Direcion Success error measure. The aer concludes wih a discussion of he salien resuls (secion VII) and heoreical exensions o he model and fuure work (secion VIII). II. JUSTIFICATION Backroagaion (BP) and relaed algorihms are commonly used in sudies, as i is generally assumed ha error minimisaion / likelihood maximisaion is he only goal for an NN when modelling a relaionshi beween inu and ouu vecors. The underlying assumion of his aer, however, is ha his is no always necessarily he only objecive. This is because end users of ime series forecass commonly have oher aims han jus Euclidean disance minimised forecass. Examles of some of hese differen aims will be discussed in his secion. A oular error measure for ime series forecasing, due o is uni-free naure, is MAPE [1]. If a raciioner has a reference for his measure of accuracy i may also make sense ha heir model is rained wih i in mind. (Due o heir differing calculaion mehods, minimising he Euclidean error of a model does no mean ha he MAPE error of a model is minimised). The calculaion of MAPE is show in Eqs. 1 and 2, where APE is he Absolue Percenage Error a ime. APE y y = ˆ (1) y

3 MAPE APE = = 1 (2) 3 In Eqs. 1 and 2 ŷ is he rediced deenden variable a ime se, y is he acual (arge) value a ime and is he number of raining aerns in he daa se. The advanage of MAPE over he Euclidean error measuremen of RMSE is ha MAPE calculaes he error as a ercenage of he acual value. Therefore he error measure does no conain he bias ha he Euclidean does owards errors ha occur when he ime series value is near an absolue exreme of is range. In a MAPE oimised model an error of 0.5 on a arge of 5 is ranked he same as an error of 5 on a arge of 50 or an error of 50 on a arge of 500. In a Euclidean rained model an error of 50 on a arge of 500 is given an error value en imes higher han an error of 5 on a arge of 50 and one-hundred imes higher han an error of 0.5 on a arge of 5. Therefore a rade-off beween he wo measures is forced when raining, wih curren raining mehods by heir very naure favouring RMSE. Value e 1 e 2 Predicion : model A Acual Predicion : model B Time Fig. 1 Correc direcion change redicion versus Euclidean minimised model. The end user may wish o oimise learning wih resec o error measures such as he correc direcional change redicion [1, 3]. Consider redicing he consumer demand for a aricular good. Jus imoran as he acual level of demand rediced in he nex ime se, is wheher he demand a he nex ime se is higher or lower

4 4 han he resen level. The same can also be rue of many hysical and hysiological sysems. For examle, in he medical field changes in direcion of ime series (hear rae, blood ressure) can be as imoran as he acual level (see Fig. 1). If i is acceed ha any model forecas will conain some degree of error, he raciioner may be willing o sacrifice some of he Euclidean accuracy for a greaer degree of confidence associaed wih he rediced direcional change of he model forecas. This is achievable by calculaing a comosie error value, as described in Secion III, and using a raining rocess based on his comosie error erm. In Fig. 1, even hough he Euclidean error of model B (e 2) is smaller han ha of model A (e 1), model A correcly redics he direcional movemen of he series (wih resec o he realised revious ime se), and as such may be he user-referred model. In addiion, he end user migh no be solely concerned wih he accuracy of he forecas, bu also wih he roeries of he residual error. They may desire he model s error be consisen (having low volailiy), as oosed o one ha has boh eriods of very low error and also occasional eriods of very high forecas error. In oher words, end users may ofen refer a model ha has overall higher average Euclidean error han anoher, rovided i is less likely o roduce insances of very large redicion error. This reference may be derived from he coss associaed wih large forecas errors. Take for examle he siuaion of forecasing a manufacuring firm s invenory level. A cerain degree of forecas error may be absorbed by he comany, bu soradic large differences beween he rediced and acual could leave he firm wih a large surlus of maerial (and associaed sorage coss) - or worse, lacking he maerials i needs o saisfy demand. Given hese differen and comeing objecives, a NN model framework shall be discussed where learning can be oimised wih resec o differen error references. III. THE MULTI-OBJECTIVE NEURAL NETWORK MODEL In his secion he raining model for NNs will be inroduced and an exension roosed for encomassing mulile error measures. A mehod of error smoohing is also inroduced as an aem o solve he curren soing roblem associaed wih (and GA) rained NNs. The use of Evoluionary and Geneic aroaches o Neural Nework raining has received increasing aenion in recen years [4, 5, 6, 7, 9, 12, 14]. Indeed he abiliy of hese aroaches o faciliae NN raining beyond he Euclidean objecive was highlighed by Poro e al [9], bu aarenly aken no furher. Oher limied aroaches o muli-objecive raining do aear in he lieraure, bu in he form of simulaneously choosing he nework oology as well as Euclidean raining e.g. [7]

5 5 Poulaion of NNs a Eoch (generaion) e Selecion rouine Poulaion of NNs a Eoch (generaion) e+1 (offsring of seleced members of generaion e ) Diagram 1, generic oulaion selecion. The roosed for use in his muli-objecive learning model is ha used in [4, 9, 12] (in he laer wo i is referred o as Evoluionary Programming) and is shown in Eq. 3. Here he weigh sace of a nework is erurbed by some values drawn from a known disribuion each eoch (generaion). + = w + γ Θ w n i, j, k 1 n, i, j, k, (3) where w n,i,j,k is he weigh beween he i h and j h nodes of he of he n h nework in he oulaion a he k h eoch of raining, Θ is a number drawn a random from a Normal (0,1) Guassian disribuion and γ is some fixed (or weigh deendan) mulilier [4, 9, 12, 14]. Tyically a given nework s abiliy o reroduce ino he nex eoch s (generaion s) oulaion is deermined by is relaive average Euclidean error, eiher in relaion o he raining se, or he validaion se. Usually he oulaion of neworks is ranked a each eoch (generaion), wih he fies Neworks having a higher robabiliy of offsring in he nex generaion (see Diagram 1). In his muli-objecive case, a Nework s finess f n is defined by he error meric relaed o he user defined objecives, A weighed summaion is used in his aer as shown in Eq 4. Here he finess of a aricular nework is a linear comosie of k differen error erms, where he number k and ye of error measures incororaed are decided by he user. f n α1 ε1, n + α 2 ε 2, n + + α k ε k, n = (4) Before f n is calculaed, he various differen errors of he n h nework (ε 1,n o ε k,n in Eq. 4) are ransformed o lie in he range 0 and 1. The coefficien weighings hemselves (α 1 o α k) are user defined, being deenden on he user s references for aricular error measures (for insance, if hey have a larger reference for error measure ε 1 in comarison o ε 2, hey will se he α 1 coefficien larger han he α 2 coefficien). More comlex dynamic weighing algorihms for his urose are also discussed in secion VI. Of concern when raining Neural Neworks wih any raining algorihm are he soing crieria. In relaion o BP here are a number of mehods currenly in use. The more robus mehod incororae hree daa ses, one o rain on, one on whose error he nework makes is decision o so raining and he final being he es se (uon

6 6 which he nework's relaive success is judged). A common roblem wih Evoluionary and Geneic aroaches o raining NNs is ha heir error curves are no as smooh as hose of gradien descen algorihms (see Fig. 2) and as such he decision as o when o hal nework raining is more comlicaed. Indeed, in some ublished aers raining is soed a an a oseri seleced error level, which is se afer coninuous runs by he user o find he oimum (e.g. [4]), or afer an arbirary se error level is reached on he raining se [9]. This makes meaningful comarisons beween he erformance of differenly rained neworks difficul (given ha no only he raining mehod, bu also he soing rouine may be differen). A soluion o his roblem is roosed in his aer. By using a Finie Imulse Resonse Filer (also known as Single Exonenial Smoohing in he economic discilines) o ransform he realised nework error er eoch, a smoohed erm Eq. 5). ~ f n, e is generaed a eoch e (see ~ ~ = η + η (5) f n, e f n, e 1 (1 ) f n, e The degree of smoohing is deenden on he value of η; he higher he value of η he greaer he degree of smoohing (an η value of 0 would resul in ~ f n, e equalling f n,e). Fig 3. shows an examle of smoohed Nework raining and validaion error which is resulan from alying Eq. 5 on he errors in Fig. 2. Figures 2 and 3 were generaed by raining a sandard Euclidean NN on he Sana Fe ime series A (see secion IV for daa definiion). Figure 2. Sandard f, Training and Validaion. Figure 3. f-ilda, Training and Validaion As can be seen in Figs. 2 and 3, i is much easier o ascerain when a laeau is reached, or he when he validaion error sars o rise in he ransformaion ~ f n as oosed o fn. This ermis idenical soing regimes o be imlemened on gradien descen and Evoluionary rained neworks based uon he validaion global minimum, as oosed o he curren mehod of soing rained neworks a a fixed error level.

7 7 IV. NETWORK TOPOLOGY AND LEARNING RUL A. Nework oograhy In order o es he racicaliy and he effec of he muli-objecive NN raining a simle se of exerimens were devised. As, o he bes of he auhors knowledge, his is he firs aer o deal wih his secific roblem, i is inended o kee nework comlexiy o a minimum. This is in order o reduce he ossibiliy of exogenous facors affecing he resuls. y -1 y -2 ŷ Figure 4: Nework oograhy The chosen nework oology is idenical o ha used in [4] (see Fig. 4) where was also used in he ime series forecasing domain. This consiss of a single layered MLP wih four hidden nodes, wo inu nodes and one ouu node. The inu vecor being he as wo realisaions of he ime series, y -1 and y -2 where he nex realisaion is y. The funcional relaionshi herefore being as described in Eq. 6. ˆ = 1 2 y f ( y, y ) (6) B. Daa used Daa se Descriion Observaions A Laser generaed daa 1000 B1, Physiological daa, saced by 0.5 second inervals. The firs se is he each B2, B3 hear rae, he second is he ches volume (resiraion force), and he hird is he blood oxygen concenraion (measured by ear oximery). C Tickwise bids for he exchange rae from Swiss francs o US dollars; recorded by a currency rading grou from Augus 7, 1990 o Aril 18, 1991 D1, Comuer generaed series D2 E Asrohysical daa. A se of measuremens of he ligh curve (ime variaion of he inensiy) of he variable whie dwarf sar PG during March 1989, 10 second inervals Table Sana Fe Comeiion daa

8 8 The ime series daa ses used in his aer are drawn from he Sana Fe comeiion ses of 1991, and consis of 8 diverse ime series ses (see Table 1). These ime series were chosen due o he breadh and diverse naure of he ime series generaion rocesses. C. Learning rules and soing crieria The core learning rouine is ha resened in Eq. 3, he oulaion selecion rouine will follow he mehodology of [6]. 1) oulaion Selecion Algorihm (a) Iniiae 100 Neworks wih a randomised weigh vecors (beween 0.1 and 0.1). (b) Perurb weighs by. (c) Rank by finess. (d) Selec base neworks for nex generaion. (e) Go o (b) or so if soing crieria me. The erurbaion in (b) is as reviously described in Eq. (3). In (c) he oulaion of neworks are ranked in descending order by relaive finess according o Eq. (4). In (d) he fies 20 neworks are hen relicaed wice and insered ino he nex oulaion, he nex 60% are relicaed once and insered. The boom 20% is discarded from he oulaion. This is hen reeaed unil he crieria for soing is me. Eliism is also imlemened, such ha one of he wo relicas of he fies individual NN assed ino he nex generaion does no have is weighs erurbed. The erurbaion is erformed wih a fixed γ value of 0.02, as used in [4]. In he mulile objecive varian of Direcion success (MODS) was calculaed using he Eqs. 7 and 8, based on he firs Sobolov Norm. MODS = = S (7) = y ˆ y (8) Where S =1 if boh acual and rediced changes ( y and ˆ y ) are of he same sign, and S =-1 if acual and rediced changes are of differen signs. is as reviously defined in Eq. (2). As can be seen he maximum value his can ake is and he minimum -. This can be ransformed o a ercenage value using Eq. (9)

9 9 MODS + = 2 Percenage MODS 100 (9) This comares o he sandard calculaion of he Percenage Direcion Success error [1], as described in Eq. 10. S = 1 DS = 100 (10) Here S =1 if boh acual and rediced changes ( y and ˆ y ) are of he same sign, and S =0 oherwise. Eqs. 7 and 8 are used as exerimenaion showed ha he sandard formula in Eq. 9 caused sagnaion in learning, if used raining as he only error erm. I was found ha mos neworks rained using his equaion failed o achieve significanly above 50% direcion success on raining, validaion or es ses. The cause of his was he rained neworks always rediced significanly higher (or lower) han he acual, comleely ouside he range of he raining daa. Equaions 7 and 8 are used as hey do no enirely divorce he conce of direcion success from some disance measure. This forces he redicion o be in he general range of he ime series, while he direcion success drives he exac fiing. The re-weighing of he Direcion Success error of he models o lie beween 0 and 1 was relaively easy, due o he known maximum and minimum levels i could reach. In he case of MAPE and RMSE here was no a riori knowledge wih regards o he maximum errors (in heory he maximum of boh is and minimum 0); herefore a differen aroach had o be aken o he re-weighing of he errors before inserion ino Eq. 4. The average Euclidean error (RMSE) of he 100 iniialised neworks of he model was aken as a roxy for maximum RMSE error (and he average MAPE of he 100 iniialised neworks for he maximum MAPE calculaion). These values where hen used for re-weighing he errors of he neworks beween 0 and 1. The η value in he Finie Imulse Resonse Filer was se o and BP neworks were rained wih a learning rae of 0.05 and a momenum of 0.5.

10 V. EXPERIMENT OUTLINE 10 EXPERIMENT DISCRIPTION AIM Exerimen 1 Train five differen nework yes (wo radiional Euclidean based models, hree muli-objecive) on each of he Sana Fe original ime series. Exerimen 2 Exerimen 3 Train five differen nework yes (wo radiional Euclidean based models, hree muli-objecive) on each of he Sana Fe firs difference ime series. Train sixy differen muli-objecive neworks wih differen reference weighings on a single ime series Comare he error roeries of he fied models wih heir learning secificaions. Comare he error roeries of he fied models wih heir learning secificaions. To generae esimaed erao froniers, and highligh real rade-off choices ha are made in he choice of leaning mehod. Exerimenal ouline. Five neworks were rained on each of he daa ses (and heir firs difference ransformaions) according o he soing rule as defined below, wih a consecuive daa ariion of 80/10/10 (he firs 80% used as he raining se he nex 10% as he validaion se and he remaining 10% as he unseen es se). The firs differences were included as well as he originals, as hese ime series should (heoreically) be more difficul o redic, as a high degree of emoral correlaion is removed from he series by he ransformaion. I was herefore of ineres o see wheher muli-objecive raining aroaches would imrove erformance by he same degree on boh grous of ime series, or wheher i seemed beer suied o a aricular ye. Firs difference daa does however cause roblems for cerain error measures, secifically MAPE, due o he zero mean revering naure of he series. As can be seen in Eqs. 1 and 2, any near zero values in y will lead o exremely large values of APE (and herefore MAPE) even wih small differences in acual and rediced. The skewness RMSE exeriences by design on large error values is magnified in MAPE on near zero values. Therefore he MAPE resuls, wih resec o he difference series, should be reaed wih exreme cauion hey are only included for comleeness. The firs se of neworks were rained using he sandard BP algorihm, and he second using he sandard Euclidean aroach. The remaining hree ses of neworks were rained using he muli-objecive mehodology. The firs wih an equally weighed Euclidean / Direcion Success (E/D) finess funcion, he second wih an equally weighed Euclidean / MAPE (E/M) finess funcion and he hird wih an equally weighed MAPE / Direcion Success (M/D) finess funcion. I was decided o rain sandard NNs as well as he muli-objecive varians as a number of sudies [4, 9, 14] reored beer resuls when using he mehod as oosed o BP. By comaring he resuls wih sandard rained NNs, as well as BP rained NNs, i is hoed any imrovemens in erformance (wih resec o cerain error measures) can be judged as due o he augmened learning regime, as oosed o using he mehodology o rain NNs er say.

11 11 Afer raining, he fies NN of he oulaion (wih resec o is erformance on he validaion se) was used on he es se. Neural Nework raining was soed when he combined finess error on he validaion and raining se had fallen by less han % of heir value 5 eochs ago. The inclusion of he validaion error revens overfiing, however by summing he wo errors (as oosed o sricly using he validaion error on is own), a sligh rade-off is ermied beween he errors while ushing hrough local minima [13]. Commonly validaion error flucuaes while sill exhibiing a downwards rend, which ofen causes sub-oimal early soing when used as soing crieria. In addiion, his Soing was no acive unil 1000 Eochs had been reached. This was imlemened because during he early sages of raining he BP neworks ushed hrough exansive local minima which caused boh he no only he validaion error o rise, bu also he raining error o rise emorarily, his also would have caused sub-oimal early soing in he sandard regime. VI. RULTS A Exerimenal resuls Error BP euc E/D E/M D/M R rmse Dir.% gmrae BRW Table 2: Sana Fe series, average error values from Exerimen 1. Error BP euc E/D E/M D/M R rmse Dir.% gmrae BRW Table 3: Sana Fe difference Series, average error values from Exerimen 2. Geomeric Mean Relaive Absolue Error values in ables 2 and 3 are Tables 2 and 3 show he average erformance of all five models over he wo daa ses afer erforming Exerimens 1 and 2 as described in secion V. The error measures chosen are some of hose recommended by Armsrong and Colloy [1], he calculaion of which can be found in secion VII (aendix). Average MAPE values are no included in Tables 2 and 3 as he diverse range of hese values over he daa ses makes an average comarison meaningless. The similar ranges of he daa however allow average values of he (non unifree) RMSE and oher errors o be meaningfully comued.

12 Error RMSE Dir. MAPE Euclidean Model E/D E/M D/M BP Euc BP Euc BP Euc Table 4: Percenage of models wih beer error measures over Sana Fe series. Error RMSE Dir. MAPE Euclidean Model E/ D E/M D/M BP Euc BP Euc BP Euc Table 5: Percenage of models wih beer error measures over Sana Fe difference series. Error X Y Euclidean M N Model P A e Q B f P C g Q D h Table 6: Exlanaory able for he inference from ables 4 and 5, on ime series se Z. Tables 4 and 5 show he ercenage of ime series where he various muli-objecive NNs erform beer han he Euclidean models on he differen error measures. (Insances of equal erformance are removed before calculaion), Table 6 aids heir roer inerreaion. Here model M erformed beer han Euclidean model P on a ercenage of ime series from se Z on error erm X. Likewise model N erformed beer han Euclidean model Q on h ercenage of ime series from se Z on error erm Y. Those values shaded in dark grey are hose, ceris aribus (all hings remaining equal), execed o be 50% or above, due o he model raining aroach, if he iniial assumions hold. (N.B. MAPE values in able 5 are in ialics as no conclusions can be meaningfully made abou hem). VII. DISCUSSION A. General resuls As iniially hoed, he resuls from Exerimen 1 and 2 show a clear rade-off beween he Direcion Success error measure and he Euclidean error measure (RMSE), as can be seen in Tables 2, 3, 4 and 5. The resuls of raining wih MAPE do no however seem o suor he rade-off assumions. The naure of he MAPE error erm iself and resuls from Exerimen 3 aid he formaion of wo ossible causes for he resuls as found in Exerimens 1 and 2. These are discussed a he end of his secion (ar F) and secion VIII.

13 13 B. Exerimen 1, (refer o Tables 2 and 4). 1) Training using Direcion success and RMSE: When comared o he muli-objecive E/D models, he BP rained models had a lower RMSE on over 75% of he original ime series. However he muli-objecive models in reurn had higher direcion success han he BP models on all (100%) of he of he original ime series. The comarison beween he wo models (E/D and ure Euclidean) also suor hese iniial resuls, alhough he rade-off beween Euclidean disance minimisaion and direcion success accuracy is less ronounced han in he BP case. Here he Euclidean models had he lower RMSE on over 85% of he ime series, whereas he muli-objecive Eur/Dir models had he higher direcion success on wo hirds (67%) of hem. E/D rained NNs erform on average 2.75% beer han BP bu only 0.4% beer on average han Euclidean on Direcion Success error measure. 2) Training using Direcion success and MAPE: The resuls wih resec o Direcion Success of he D/M rained models also suor he resuls found from he E/D models, alhough he resuls are slighly less ronounced. The D/M models erform absoluely beer on 57% of he daa ses when comared wih BP, bu only 50% when comared wih Euclidean. This no wihsanding, he average Direcion Success of he D/M models (as shown in ables 2) is almos exacly he same as he average E/D models. The lower absolue erformance on he Direcion Success measure may be symomaic of roblems encounered when raining wih MAPE han Direcion Success iself. This will be discussed in secion VIII. As can be seen in Table 4, he D/M models erformed markedly less well on his measure comared o BP and Euclidean rained models (erforming beer on only 37.5% and 12.5% of he ime series resecively). 3) Training using RMSE and MAPE: The models rained on MAPE and RMSE erformed beer on MAPE han he models rained on Direcion Success and MAPE, he ossible reasons for which will be discussed in secion VIII. Average MAPE values are meaningless given he diverse range of he MAPE values over he daa ses (as menioned in secion VI), however, as can be seen in Table 4, he E/M models had lower MAPE on he es ses of 71.4% of he ime series comared o he Euclidean models, bu only on half on he ime series when comared o BP.

14 C. Exerimen 2, (refer o Tables 3 and 5). 14 1) Training using Direcion success and RMSE: When comared o he muli-objecive E/D models, he BP and Euclidean rained models boh had a lower RMSE on over 85% of he differenced ime series. However he muli-objecive models in reurn had higher direcion success han he BP models on 62.5% of he of he original ime series, and was beer han he Euclidean models on 57.1% of he differenced ime series. E/D rained NNs had, on average, 2.8% higher Direcion Success comared o BP models and were 4.7% higher comared Euclidean models. 2) Training using Direcion Success and MAPE: Due o he roblems in using MAPE in his ime series range i is difficul o draw conclusions, however he Direcion Success values are in line wih hose achieved using he E/D raining aroach. 3) Training using RMSE and MAPE: These resuls are resened urely for comleeness no inferences can be drawn due o he reasons saed in secion VI. I was hyohesised ha he relaively simlisic weighing mehod used o calculae he weighings of he errors when raining migh no ensure ha he resulan model was on or near is erao fronier when is raining was soed. Therefore, afer comiling he resuls of raining over he 16 daa ses, he exerimenaion was aken one se furher by he comosiion of examle erao froniers of he differen error measures. This was achieved using he firs difference of ime series A (see Fig. 5) as he raining se (wih idenical ariion and raining mehods as described reviously). The creaion of hese hels underline he real rade-off beween differen error roeries ha is ossible using he mehodology as described reviously, and also highlighs any sub-oimal raining ha he neworks may exerience.

15 D. Exerimen Series Value Time Se Fig. 5 Sana Fe Comeiion series 'A', firs difference. Fig. 6 shows he rade-off lo beween Direcion Success and Euclidean disance minimisaion on he series A difference daa se in he form of a erao curve (or fronier). This was creaed by raining 21 E/D models wih differen error weighings (and herefore user references). This was achieved by slowly increasing he α 1 coefficien of he direcion error measure (in Eq. 4) from zero o one (by 0.1 inervals) wih he α 2 coefficien of Euclidean disance fixed a one. This was followed by decreasing he α 2 coefficien of Euclidean disance o zero in he same manner wih he direcion error measure fixed. The lo herefore sars wih a urely Euclidean error driven model (marked wih a ) and ends wih a urely Direcion success rained model (marked wih a ο ). A highlighs he equally weighed model. 6 4 RMSE Direcion Success (%) 2 65 Fig 6. RMSE/Direcion Success Pareo Fronier. The exac cos of he rade-off a any oin on he fronier is deermined by he gradien of he fronier a ha oin. I is of ineres o noe ha he exreme of Euclidean only raining is no on he fronier, herefore highlighing he ooruniy o beer boh error erms a once by simly including direcion success meric by a small weighing in he learning regime.

16 MAPE RMSE Fig 7. MAPE / RMSE Pareo Fronier. Fig. 7 shows a rade-off lo beween MAPE and Euclidean disance minimisaion on he series A difference daa se. This was creaed in a similar fashion o Fig. 5. The lo again sars wih a urely Euclidean error driven model (marked wih a ) on he righ of he lo, and ends wih a urely MAPE rained model (marked wih a ο ) on he lef. A highlighs he equally weighed model. Once again he urely Euclidean model is seen o lie off he erao fronier, wih he same imlicaion as in figure 5, and he ure MAPE model, alhough on he fronier, is seen no o minimise he MAPE value MAPE Direcion Success (%) Fig 8. Direcion Success / MAPE Pareo Fronier. Fig. 8 shows a rade-off lo beween Direcion Success maximisaion and MAPE minimisaion. The lo again sars wih a urely MAPE driven model (marked wih a ) and ends wih a urely Direcion success rained model (marked wih a ο ). Wih highlighing he equally weighed model. In his case he Purely direcion success model is seen o lie away from he erao fronier.

17 E. Euclidean, Direcion Success and MAPE MAPE 10 5 MAPE RMSE Direcion Success (%) RMSE Direcion Success (%) Fig. 9 3-D Error scaer lo. Figure 10 Imlied 3-Dimensional Perao Fronier. Figs 9 and 10. are generaed using he hree error measures for he NNs used in he generaion of Figs. 5, 6 and 7. The hree dimensional fronier in Fig. 10 is ha imlied by hese errors. Figure 10 shows he wide range of error combinaions ossible whils sill on he series erao error surface (as defined by he nework oology used), F. Variable error dominance roblems wih he comosie error value. Afer he resuls of using he D/M model in Exerimen 2 where received, where he resuls were very similar o ha of he E/D model, i was realised ha he error values may no be changing in a uniform manner over ime, leading o differen errors exhibiing dominance in he comosie error value as raining rogressed, and skewing he final model away from he iniial user weighings. Figures suor his conclusion, he unbroken lines on each lo showing he error value and he dashed lines showing ha error as a roorion of he comosie error. The los where aken during he raining of an equally weighed E/D nework (on he A difference series daa se). RMSE Error / roorion of comosie Direcion Success / roorion of comosie Eochs Eochs Figure 11 RMSE error each eoch in E/D model wih Figure 12 Direcion Success each eoch in E/D model relaive comosie weighing (scaled beween 0 and 1). wih relaive comosie weighing (scaled beween 0 and 1).

18 18 As can be clearly disinguished in Figs 11 and 12, alhough boh errors erms decrease wih raining, he relaive conribuion of hese erms o he comosie error value does no remain consan. Due o he fixed weighing in Eq. 4, if he nework exeriences a eriod of raid rogress in one error measure, he resul is a comosie error dominaed by he oher measure. The raid rogress iself may be due o a number of facors; iniial calculaion of he maximum error for adjusing he range o beween 0 and 1 may be an incorrec roxy for he maximum, or he ineracion beween he errors may be unequal (i is lausible o have a model wih 100% direcion success bu which does exerience Euclidean error). An exension o he curren model o comensae for his effec could be he alicaion of a regular re-weighing of he errors beween 0 and 1 (in effec re-calculaing he maximum). VIII. CONCLUSION The resuls from exerimenaion suor he iniial remise of he aer, namely ha i is ossible for a user, if hey so desire, o re-deermine he error aribues of heir NN forecas model, which can be done hrough he raining rocess described. Exerimens 1 and 2 show ha NNs rained wih he Direcion Success error measure alongside oher error erms are much more likely o erform beer on his error measure when comared wih neworks rained in he radiional fashion. Resuls on NNs rained wih MAPE alongside oher errors were inconclusive, however heoreical reasons for hese resuls have been highlighed along wih ossible soluions In addiion, i has been demonsraed ha wihin a fixed (and fairly simle) nework oograhy, an enire range of ossible error rade-off combinaions is ossible (as highlighed by he hree-dimensional erao fronier of Fig. 10), jusifying he relevance and iniial assumions of muli-objecive NN raining. Furhermore he aroach demonsraed is no resriced o he error measures emloyed in his aer; here are a large number of error erms regularly used in ime series forecasing (15 menioned in [1] alone), and any oher forecas references a user may have is easily incororaed in he nework comosie error. Much furher work needs o be accomlished in his area however. aroaches o raining sill ake significanly longer han BP raining (aroximae 50 imes longer wih a oulaion of 100 neworks). In addiion, he mulile error weighing used in his aer is relaively simlisic (being a fixed summaion), lending only a degree of robabiliy o he final forecas roeries, and does no ensure he resulan model will lie on or near is erao fronier (as shown in Figs 6, 7 and 8). One exension o he model has been discussed a he end of he revious secion. A more radical exension o his aroach would be o rain a nework such ha i could move across he erao fronier (once i was reached). This would allow he user o acively infer he exac error roeries of heir end model (insead of simly saing cerain references).

19 IX. APPENDIX 19 Roo Mean Square Error RMSE ( yˆ y ) 1 = = Beer han Random Walk (BRW) BRW j 1 = = 100 Where j 1 if ŷ y < y 1 = 0 oherwise. y Geomeric Mean Relaive Absolue Error (GMRAE) RAE = yˆ y 1 y y GMRAE = = 1 RAE 1

20 X. REFERENC 20 [1] Adya, M. and Colloy, F., How Effecive are Neural Neworks a Forecasing and Predicion? A Review and Evaluion, Journal of Forecasing, Vol. 17, , [2] Armsrong, J.S. and Colloy, F., Error measures for generalizing abou forecasing mehods: Emirical comarisons, Inernaional Journal of Forecasing, Vol., 69-80, [3] Brownsone, D. Using ercenage accuracy o measure neural nework redicions in Sock Marke movemens, Neurocomuing, Vol. 10, No. 3, , [4] Greenwoood, G.W., Training Mulile-Layer Percerons o Recognise Aracors, IEEE Transacions on Evoluionary Comuaion, Vol. 1, No. 4, , [5] Janson, D.J. and Frenzel, J.F., Alicaion of Geneic Algorihms o he raining of Higher Order Neural Neworks, Journal of Sysems Engineering, Vol. 2, No. 4, 1992 [6] Janson, D.J. and Frenzel, J.F., Training Produc Uni Neural Neworks wih Geneic Algorihms, IEEE Exer, Ocober, 26-33,1993. [7] Merelo, J.J., Paon, M., Canas, A, Prieo, A and Moran, F., Oimizaion of a comeiive learning neural nework by Geneic Algorihms, Lecure Noes In Comuer Scince, Vol. 686, , [8] Moody, J., Forecasing he Economy wih Neural Nes: A Survey of Challenges and Soluions, Neural Neworks: Tricks of he Trade,(Orr, G,B., and Mueller, K-R, eds.,) Berlin: Sringer, , [9] Poro, V.W., Fogel, D.B. and Fogel, L.J., Alernaive Neural Nework Training Mehods, IEEE Exer, June, 16-21,1995. [10] Refenes, A-P.N., Burgess, A.N. and Benz, Y. Neural Neworks in Financial Engineering: A Sudy in Mehodology, IEEE Transacions on Neural Neworks, Vol. 8, No. 6, ,1997. [11] Saad, E.W., Prokhorov, D.V. and Wunsch,D.C., Comariive Sudy of Sock Trend Predicion Using Time Delay, Recurren and Probabilisic Neural Neworks, IEEE Transacions on Neural Neworks, Vol. 9, No.6, , [12] Saravanan, N. and Fogel, D.B., Evolving Neural Conrol Sysems, IEEE Exer, June, 23-27, [13] Singh, S. and Fieldsend, J.E., "Paern Maching and Neural Neworks based Hybrid Forecasing Sysem", Acceed Proceedings of ICAPR'2001 (Rio, Brazil), Forhcoming. [14] Yao, X., Evolving arificial neural neworks, Proceedings of he IEEE,Vol 89, No. 9, , 1999.