Which Methodology is Better for Combining Linear and Nonlinear Models for Time Series Forecasting?

Transcription

1 Journal of Indusrial and Sysems Engineering Vol. 4, No. 4, pp Winer 2011 Which Mehodology is Beer for Combining Linear and Nonlinear Models for Time Series Forecasing? Mehdi Khashei 1, Mehdi Bijari 2 1,2 Deparmen of Indusrial and Sysems Engineering, Isfahan Universiy of Technology, Isfahan, Iran 1 khashei@in.iu.ac.ir ABSTRACT Boh heoreical and empirical findings have suggesed ha combining differen models can be an effecive way o improve he predicive performance of each individual model. I is especially occurred when he models in he ensemble are quie differen. Hybrid echniques ha decompose a ime series ino is linear and nonlinear componens are one of he mos imporan kinds of he hybrid models for ime series forecasing. Several researches in he lieraure have been shown ha hese models can ouperform single models. In his paper, he predicive capabiliies of hree differen models in which he auoregressive inegraed moving average (ARIMA) as linear model is combined o he mulilayer percepron (MLP) as nonlinear model, are compared ogeher for ime series forecasing. These models are including he Zhang s hybrid ANNs/ARIMA, arificial neural nework (p,d,q), and generalized hybrid ANNs/ARIMA models. The empirical resuls wih hree well-known real daa ses indicae ha all of hese mehodologies can be effecive ways o improve forecasing accuracy achieved by eiher of componens used separaely. However, he generalized hybrid ANNs/ARIMA model is more accurae and performs significanly beer han oher aforemenioned models. Keywords: Arificial Neural Neworks (ANNs), Auo-Regressive Inegraed Moving Average (ARIMA), Time series forecasing, Hybrid linear/nonlinear models. 1. INTRODUCTION Applying quaniaive models for forecasing and assising invesmen decision making has become more indispensable in many areas. Time series forecasing is one of he mos imporan ypes of quaniaive models in which pas observaions of he same variable are colleced and analyzed o develop a model describing he underlying relaionship (Aryal & Yao-Wu, 2003). This modeling approach is paricularly useful when lile knowledge is available on he underlying daa generaing process or when here is no saisfacory explanaory model ha relaes he predicion variable o oher explanaory variables (Zhang, 2003). Forecasing procedures include differen echniques and models. Moving averages echniques, random walks and rend models, exponenial smoohing, sae space modeling, mulivariae mehods, vecor auoregressive models, coinegraed and causal Corresponding Auhor ISSN: , Copyrigh 2011 JISE. All righs reserved.

2 266 Khashei and Bijari models, mehods based on neural, fuzzy neworks or daa mining and rule-based echniques are ypical models used in ime series forecasing (Ragulskis & Lukoseviciue, 2009). Auo-regressive inegraed moving average (ARIMA) models are one of he mos imporan and widely used linear ime series models. The populariy of he ARIMA model is due o is saisical properies as well as he well-known Box Jenkins mehodology (Box & Jenkins, 1976) in he model building process. In addiion, various exponenial smoohing models can be implemened by ARIMA models. Alhough ARIMA models are quie flexible in ha hey can represen several differen ypes of ime series and also have he advanages of accurae forecasing over a shor period of ime and ease of implemenaion, heir major limiaion is he pre-assumed linear form of he model. ARIMA models assume ha fuure values of a ime series have a linear relaionship wih curren and pas values as well as wih whie noise, so approximaions by ARIMA models may no be adequae for complex nonlinear real-world problems. However, many researchers have argued ha real world sysems are ofen nonlinear (Zhang e al., 1998). These evidences have encouraged academic researchers and business praciioners in order o develop more predicable forecasing models han linear models (Khashei & Bijari, 2011). Several classes of parameric and nonparameric nonlinear models have been proposed in he lieraure in order o overcome he resricion of he linear models and o accoun nonlinear paerns observed in real problems. Among parameric models, he bilinear model (Granger & Anderson, 1978), he hreshold auoregressive (TAR) model (Tong & Lim, 1980), he auoregressive condiional heeroscedasic (ARCH) model (Engle, 1982) and generalized auoregressive condiional heeroscedasic (GARCH) model (Bollerslev, 1986), chaoic dynamics (Hsieh, 1991), and self-exciing hreshold auoregressive (Chappel e al., 1996) receive he mos aenion. While hese models may be good for a paricular siuaion, hey perform poorly for oher applicaions. The reason is ha he above-menioned models are developed for specific nonlinear paerns and are no capable of modeling oher ypes of nonlineariy in ime series (Khashei & Bijari, 2011). A number of nonparameric forecasing models such as mulivariae neares-neighbor mehods have also been proposed for ime series forecasing. However, he resuls of some researchers invesigaed in ime series forecasing sugges ha hese nonparameric models canno significanly improve forecass accuracy upon he oher ime series models (Mizrach, 1992). Arificial neural neworks (ANNs) are anoher ype of nonparameric nonlinear models, which have been proposed and examined for ime series forecasing. Given he advanages of neural neworks (Panda & Narasimhan, 2007), i is no surprising ha his mehodology has araced overwhelming aenion in ime series forecasing. Arificial neural neworks have been found o be a viable conender o various radiional ime series models (Chen e al., 2005; Giordano e al., 2007; Jain & Kumar, 2007). Lapedes and Farber (1987) repor he firs aemp o model nonlinear ime series wih arificial neural neworks. De Groo and Wurz (1991) presen a deailed analysis of univariae ime series forecasing using feedforward neural neworks for wo benchmark nonlinear ime series. Chakrabory e al. (1992) conduc an empirical sudy on mulivariae ime series forecasing wih arificial neural neworks. Poli and Jones (1994) propose a sochasic neural nework model based on Kalman filer for nonlinear ime series predicion. Corell e al. (1995) address he issue of nework srucure for forecasing real world ime series. Berardi and Zhang (2003) invesigae he bias and variance issue in he ime series forecasing conex. In addiion, several large forecasing compeiions (Balkin & Ord, 2000; Weigend & Gershenfeld, 1993) sugges ha neural neworks can be a very useful addiion o he ime series forecasing oolbox. Sanos e al. (2007) invesigae he hypohesis ha he nonlinear mahemaical models of mulilayer percepron and he radial basis funcion neural neworks are able o provide a more accurae ou-of-sample forecas han he

3 Which Mehodology is beer for Combining Linear radiional linear models. Their resuls indicae ha ANNs perform beer han heir linear models (Khashei & Bijari, 2011). Alhough arificial neural neworks have he advanages of accurae forecasing, heir performance in some specific siuaions is inconsisen. In he lieraure, several papers are devoed o comparing ANNs wih he radiional mehods. Despie he numerous sudies, which have shown ANNs are significanly beer han he convenional linear models and heir forecas considerably and consisenly more accuraely, some oher sudies have repored inconsisen resuls. Foser e al. (1992) find ha ANNs are significanly inferior o linear regression and a simple average of exponenial smoohing mehods. Brace e al. (1991) also find ha he performance of ANNs is no as good as many oher saisical mehods commonly used in he load forecasing. Denon (1995) wih generaed daa for several differen experimenal condiions shows ha under ideal condiions, wih all regression assumpions, here is lile difference in he predicabiliy beween ANNs and linear regression, and only under less ideal condiions such as ouliers, mulicollineariy, and model misspecificaion, ANNs perform beer. Hann and Seurer (1996) make comparisons beween he neural neworks and he linear model in exchange rae forecasing. They repor ha if monhly daa are used, neural neworks do no show much improvemen over linear models. Taskaya and Casey (2005) compare he performance of linear models wih neural neworks. Their resuls show ha linear auoregressive models can ouperform neural neworks in some cases (Khashei & Bijari, 2011). Mos oher researchers also make comparisons beween ANNs and he corresponding radiional mehods in heir paricular applicaions. Fishwick (1989) repors ha he performance of ANNs is worse han ha of he simple linear regression. Tang e al. (1991), and Tang and Fishwick (1993) ry o answer he quesion: under wha condiions ANN forecasers can perform beer han he linear ime series forecasing mehods such as Box- Jenkins models. Some researchers believe ha in some specific siuaions where ANNs perform worse han linear saisical models, he reason may simply be ha he daa is linear wihou much disurbance, herefore; canno be expeced ha ANNs o do beer han linear models for linear relaionships (Zhang e al., 1998). However, for any reason, using ANNs o model linear problems have yielded mixed resuls and hence; i is no wise o apply ANNs blindly o any ype of daa. One of he major developmens in neural neworks over he las decade is he model combining or ensemble modeling. The basic idea of his muli-model approach is he use of each componen model s unique capabiliy o beer capure differen paerns in he daa. Boh heoreical and empirical findings have suggesed ha combining differen models can be an effecive way o improve he predicive performance of each individual model, especially when he models in he ensemble are quie differen (Zhang, 2007). In addiion, since i is difficul o compleely know he characerisics of he daa in a real problem, hybrid mehodology ha has boh linear and nonlinear modeling capabiliies can be a good sraegy for pracical use. Alhough a majoriy of he neural ensemble lieraure is focused on paern classificaion problems, a number of combining schemes have been proposed for ime series forecasing problems (Zou e al., 2007). The lieraure of hybrid models for ime series forecasing has dramaically expanded since he early work of Reid (1968), and Baes and Granger (1969). Clemen (1989) provided a comprehensive review and annoaed bibliography in his area. Wedding and Cios (1996) described a combining mehodology using radial basis funcion neworks (RBF) and he Box Jenkins ARIMA models. Tsaih e al. (1998) presened a hybrid arificial inelligence inegraing he rule-based sysems echnique and he neural neworks echnique o predic accuraely he direcion of daily price changes in S&P 500 sock index fuures. Pelikan e al. (1992), and Ginzburg and Horn (1994)

4 268 Khashei and Bijari proposed o combine several feedforward neural neworks in order o improve ime series forecasing accuracy. Luxhoj e al. (1996) presened a hybrid economeric and ANN approach for sales forecasing. Goh e al. (2003) used an ensemble of boosed Elman neworks for predicing drug dissoluion profiles. Voor e al. inroduced a hybrid mehod called KARIMA using a Kohonen self-organizing map and auoregressive inegraed moving average mehod for shor-erm predicion (1996). Medeiros and Veiga (1989) consider a hybrid ime series forecasing sysem wih neural neworks used o conrol he ime-varying parameers of a smooh ransiion auoregressive model. Armano e al. (2005) presened a new hybrid approach ha inegraed arificial neural nework wih geneic algorihms (GAs) o sock marke forecas. In recen years, more hybrid forecasing models have been proposed, using auoregressive inegraed moving average and arificial neural neworks and applied o ime series forecasing wih good predicion performance. Pai and Lin (2005) proposed a hybrid mehodology o exploi he unique srengh of ARIMA models and Suppor Vecor Machines (SVMs) for sock prices forecasing. Chen and Wang (2007) consruced a combinaion model incorporaing seasonal auoregressive inegraed moving average (SARIMA) model and SVMs for seasonal ime series forecasing. Zhou and Hu (2008) proposed a hybrid modeling and forecasing approach based on Grey and Box Jenkins auoregressive moving average (ARMA) models. Khashei e al. (2009) presened a hybrid ARIMA and arificial inelligence approaches o financial markes predicion. Yu e al. (2005) proposed a novel nonlinear ensemble forecasing model inegraing generalized linear auo regression (GLAR) wih arificial neural neworks in order o obain accurae predicion in foreign exchange marke. Kim and Shin (2007) invesigaed he effeciveness of a hybrid approach based on he arificial neural neworks for ime series properies, such as he adapive ime delay neural neworks (ATNNs) and he ime delay neural neworks (TDNNs), wih he geneic algorihms in deecing emporal paerns for sock marke predicion asks. Tseng e al. (2002) proposed using a hybrid model called SARIMABP ha combines he seasonal auoregressive inegraed moving average (SARIMA) model and he back-propagaion neural nework model o predic seasonal ime series daa. Khashei e al. (2008) based on he basic conceps of arificial neural neworks, proposed a new hybrid model in order o overcome he daa limiaion of neural neworks and yield more accurae forecasing model, especially in incomplee daa siuaions. Hybrid echniques ha decompose a ime series ino is linear and nonlinear form are one of he mos popular hybrid models, which have recenly been shown o be successful for single models. The linear ARIMA and he nonlinear mulilayer perceprons are joinly used in hese hybrid models in order o capure differen forms of relaionship in he ime series daa. The moivaion of hese hybrid models come from he following perspecives. Firs, i is ofen difficul in pracice o deermine wheher a ime series under sudy is generaed from a linear or nonlinear underlying process; hus, he problem of model selecion can be eased by combining linear ARIMA and nonlinear ANN models. Second, real-world ime series are rarely pure linear or nonlinear and ofen conain boh linear and nonlinear paerns, which neiher ARIMA nor ANN models alone can be adequae for modeling in such cases; hence he problem of modeling he combined linear and nonlinear auocorrelaion srucures in ime series can be solved by combining linear ARIMA and nonlinear ANN models. Third, i is almos universally agreed in he forecasing lieraure ha no single model is he bes in every siuaion, due o he fac ha a real-world problem is ofen complex in naure and any single model may no be able o capure differen paerns equally well. Therefore, he chance in order o capure differen paerns in he daa can be increased by combining differen models (Zhang, 2003). In his paper, hree differen mehodologies ha have been proposed in order o combine he auoregressive inegraed moving average (ARIMA) as linear model and mulilayer percepron

5 Which Mehodology is beer for Combining Linear (MLP) as nonlinear model are presened. Moreover, he predicive capabiliies of he consruced models based on hese mehodologies for ime series forecasing Zhang s hybrid ANNs/ARIMA (Zhang, 2003), arificial neural nework (p,d,q) (Khashei & Bijari, 2010), and generalized hybrid ANNs/ARIMA are compared ogeher and also heir componens, using hree well-known real daa ses. The daa ses are including he Wolf s sunspo daa, he Canadian lynx daa, and he Briish pound agains he Unied Saes dollar exchange rae daa. The res of he paper is organized as follows. In he nex secion, he basic conceps and modeling approaches of he auoregressive inegraed moving average (ARIMA) models, arificial neural neworks (ANNs), and he abovemenioned hybrid models are briefly inroduced. Descripion of used daa ses is presened in secion 3. Empirical resuls of above-menioned hybrid models for ime series forecasing from hree real daa ses are repored in Secion 4. Secion 5 conains he concluding remarks. 2. THE AUTOREGRESSVE INTEGRATED MOVING AVERAGE, ARTIFICIAL NEURAL NETWORKS, AND HYBRID ANNs/ARIMA MODELS In his secion, he basic conceps and modeling approaches of he auoregressive inegraed moving average (ARIMA), arificial neural neworks (ANNs), and hybrid arificial neural neworks and auoregressive inegraed moving average models for ime series forecasing are briefly reviewed The auoregressive inegraed moving average (ARIMA) models For more han half a cenury, auoregressive inegraed moving average (ARIMA) models have dominaed many areas of ime series forecasing. In an auoregressive inegraed moving average (p,d,q) model, he fuure value of a variable is assumed o be a linear funcion of several pas observaions and random errors. Tha is, he underlying process ha generaes he ime series wih he mean has he form (Khashei & Bijari, 2010). where, d B y B a (1) y and a are he acual value and random error a ime period, respecively; p i q B 1 j B, i 1 i B 1 j 1 jb and ( j 1,2,..., q ) are model parameers, 1 B j are polynomials in B of degree p and q, (i 1,2,...,p ), B is he backward shif operaor, p and q are inegers and ofen referred o as orders of he model, and d is an ineger and ofen referred o as order of differencing. Random errors, a, are assumed o be independenly and idenically 2 disribued wih a mean of zero and a consan variance of (Khashei & Bijari, 2010). Based on he earlier work of Yule (1926) and Wold (1938), Box and Jenkins (1976) developed a pracical approach o building ARIMA models, which has he fundamenal impac on he ime series analysis and forecasing applicaions. The Box Jenkins mehodology includes hree ieraive seps of model idenificaion, parameer esimaion, and diagnosic checking. The basic idea of model idenificaion is ha if a ime series is generaed from an auoregressive inegraed moving average process, i should have some heoreical auocorrelaion properies. By maching he empirical auocorrelaion paerns wih he heoreical ones, i is ofen possible o idenify one or several poenial models for he given ime series. Box and Jenkins (1976) proposed o use he auocorrelaion funcion (ACF) and he parial auocorrelaion funcion (PACF) of he sample daa as he basic ools o idenify he order of he auoregressive inegraed moving average model. Some oher order selecion mehods have been proposed based on validiy crieria, he informaion- i

6 270 Khashei and Bijari heoreic approaches such as he Akaike s informaion crierion (AIC) (Shibaa, 1976) and he minimum descripion lengh (MDL) (Jones, 1975; Hurvich & Tsai, 1989; Ljung, 1987). In addiion, in recen years differen approaches based on inelligen paradigms, such as neural neworks (Hwang, 2001), geneic algorihms (Minerva & Poli, 2001; Ong e al., 2005) or fuzzy sysem (Haseyama & Kiajima, 2001) have been proposed o improve he accuracy of order selecion of ARIMA models (Khashei & Bijari, 2010). In he idenificaion sep, daa ransformaion is ofen required o make he ime series saionary. Saionariy is a necessary condiion in building an auoregressive inegraed moving average model used for forecasing. A saionary ime series is characerized by saisical characerisics such as he mean and he auocorrelaion srucure being consan over ime. When he observed ime series presens rend and heeroscedasiciy, differencing and power ransformaion are applied o he daa o remove he rend and o sabilize he variance before an auoregressive inegraed moving average model can be fied. Once a enaive model is idenified, esimaion of he model parameers is sraighforward. The parameers are esimaed such ha an overall measure of errors is minimized. This can be accomplished using a nonlinear opimizaion procedure. The las sep in model building is he diagnosic checking of model adequacy. This is basically o check if he model assumpions abou he errors, a, are saisfied (Khashei & Bijari, 2010). Several diagnosic saisics and plos of he residuals can be used o examine he goodness of fi of he enaively enerained model o he hisorical daa. If he model is no adequae, a new enaive model should be idenified, which will again be followed by he seps of parameer esimaion and model verificaion. Diagnosic informaion may help sugges alernaive model(s). This hree-sep model building process is ypically repeaed several imes unil a saisfacory model is finally seleced. The final seleced model can hen be used for predicion purposes (Khashei & Bijari, 2010) The arificial neural neworks (ANNs) models Recenly, compuaional inelligence sysems and among hem arificial neural neworks (ANNs), which in fac are model free dynamics, has been used widely for approximaion funcions and forecasing. One of he mos significan advanages of he ANN models over oher classes of nonlinear models is ha ANNs are universal approximaors ha can approximae a large class of funcions wih a high degree of accuracy (Zhang e al., 1998). Their power comes from he parallel processing of he informaion from he daa. No prior assumpion of he model form is required in he model building process. Insead, he nework model is largely deermined by he characerisics of he daa. Single hidden layer feed forward nework is he mos widely used model form for ime series modeling and forecasing. The model is characerized by a nework of hree layers of simple processing unis conneced by acyclic links (Figure 1). The relaionship beween he oupu ( y ) and he inpus ( y 1,..., y p ) has he following mahemaical represenaion (Khashei & Bijari, 2010). y q p w0 w j g( w0, j wi, j y i ) j 1 i 1 (2), where, w i, j i 0,1,2,..., p, j 1,2,..., q and w j j 0,1,2,..., q are model parameers ofen called connecion weighs; p is he number of inpu nodes; and q is he number of hidden nodes. Acivaion funcions can ake several forms. The ype of acivaion funcion is indicaed by he siuaion of he neuron wihin he nework. In he majoriy of cases inpu layer neurons do no have an acivaion funcion,

7 Which Mehodology is beer for Combining Linear as heir role is o ransfer he inpus o he hidden layer. The mos widely used acivaion funcion for he oupu layer is he linear funcion as non-linear acivaion funcion may inroduce disorion o he predicaed oupu. The logisic funcion is ofen used as he hidden layer ransfer funcion ha are shown in Eq. 3. Oher acivaion funcions can also be used such as linear and quadraic, each wih a variey of modeling applicaions (Khashei & Bijari, 2010). Sig 1 1 exp( x ) x. (3) Hence, he ANN model of (2), in fac, performs a nonlinear funcional mapping from pas observaions o he fuure value y, i.e., y f y,..., y,w, 1 p (4) where, w is a vecor of all parameers and f(.) is a funcion deermined by he nework srucure and connecion weighs. Thus, he neural nework is equivalen o a nonlinear auoregressive model. The simple nework given by (2) is surprisingly powerful in ha i is able o approximae he arbirary funcion as he number of hidden nodes when q is sufficienly large. In pracice, simple nework srucure ha has a small number of hidden nodes ofen works well in ou-of-sample forecasing. This may be due o he overfiing effec ypically found in he neural nework modeling process. An overfied model has a good fi o he sample used for model building bu has poor generalizabiliy o daa ou of he sample (Khashei & Bijari, 2010). The choice of q is daa-dependen and here is no sysemaic rule in deciding his parameer. In addiion o choosing an appropriae number of hidden nodes, anoher imporan ask of ANN modeling of a ime series is he selecion of he number of lagged observaions, p, and he dimension of he inpu vecor. This is perhaps he mos imporan parameer o be esimaed in an ANN model because i plays a major role in deermining he (nonlinear) auocorrelaion srucure of he ime series (Khashei & Bijari, 2010). Figure 1 Archiecure of a neural nework in he general form (N (p-q-1).) Alhough many differen approaches exis in order o find he opimal archiecure of a neural nework, hese mehods are usually quie complex in naure and are difficul o implemen. Furhermore, none of hese mehods can guaranee he opimal soluion for all real forecasing problems. To dae, here is no simple clear-cu mehod for deerminaion of hese parameers and he usual procedure is o es numerous neworks wih varying numbers of inpu and hidden unis, esimae generalizaion error for each, and selec he nework wih he lowes generalizaion error (Khashei & Bijari, 2010).

8 272 Khashei and Bijari 2.3. Hybrid arificial neural neworks and auoregressive inegraed moving average models Boh ARIMA and ANN models have achieved successes in heir own linear or nonlinear domains. However, none of hem is a universal model ha is suiable for all circumsances. The approximaion of ARIMA models o complex nonlinear problems may no be adequae. On he oher hand, using ANNs o model linear problems have yielded mixed resuls. Hence, i is no wise o apply ANNs blindly o any ype of daa. Since i is difficul o compleely know he characerisics of he daa in a real problem, hybrid mehodology ha has boh linear and nonlinear modeling capabiliies can be a good sraegy for pracical use. By combining differen models, differen aspecs of he underlying paerns may be capured (Zhang, 2003) Zhang s hybrid ANNs/ARIMA model Some researchers in hybrid linear and nonlinear models believe ha i may be reasonable o consider a ime series o be composed of a linear auocorrelaion srucure and a nonlinear componen (Zhang, 2003). Tha is, y N L (5) where L denoes he linear componen and N denoes he nonlinear componen. These wo componens have o be esimaed from he daa. Firs, we le ARIMA o model he linear componen, and hen he residuals from he linear model will conain only he nonlinear relaionship (Zhang, 2003). Le e denoe he residual a ime from he linear model, hen e y Lˆ (6) where Lˆ is he forecas value for ime from he esimaed relaionship (1). By modeling residuals using ANNs, nonlinear relaionships can be discovered (Zhang, 2003). Wih n inpu nodes, he ANN model for he residuals will be: e f ( e 1 (7),..., e n ) where f is a nonlinear funcion deermined by he neural nework and e is he random error. Noe ha if he model f is no an appropriae one, he error erm is no necessarily random (Zhang, 2003). Therefore, he correc model idenificaion is criical. Denoe he forecas from (7) as Nˆ, he combined forecas will be yˆ Lˆ Nˆ (8) The hybrid model explois he unique feaure and srengh of ARIMA model as well as ANN model in deermining differen paerns. Thus, i could be advanageous o model linear and nonlinear paerns separaely by using differen models and hen combine he forecass o improve he overall modeling and forecasing performance (Zhang, 2003) Advanages and disadvanages The performance of he Zhang s hybrid model is ofen saisfacory han each componen model used in isolaion (Taskaya & Ahmad, 2005). In addiion, i can be generally guaraneed ha he

9 Which Mehodology is beer for Combining Linear performance of he Zhang s hybrid model will no be worse han ARIMA model. However, despie he all advanages menioned for Zhang s hybrid model, i has some assumpions ha will degenerae is performance if he opposie siuaions occur. These assumpions are as follows: 1- This model supposes ha he linear and nonlinear paerns of a ime series can be separaely modeled by differen models and hen he forecass can be combined ogeher and his may degrade performance, if i is no rue. 2- This model supposes ha he relaionship beween he linear and nonlinear componens is addiive and his may underesimae he relaionship beween he componens and degrade performance, if here is no any addiive associaion beween he linear and nonlinear elemens and he relaionship is differen (Taskaya & Casey, 2005). 3- This model supposes ha he residuals from he linear model will conain only he nonlinear relaionship. However, one may no guaranee ha he residuals of he linear componen may comprise valid nonlinear paerns (Taskaya & Casey, 2005). In addiion, as menioned previously, i canno be generally guaraneed ha he performance of he Zhang s hybrid model will no be worse han ANN model An arificial neural nework (p, d, q) model Alhough radiional hybrid linear and nonlinear models such as Zhang s hybrid model have recenly been shown o be successful for single models, perhaps he danger in using hese hybrid models is ha here are some assumpions considered in consrucing process of hese hybrid models ha will degenerae heir performance if he opposie siuaions occur. Therefore, hey may be inadequae in some specific siuaions. For example, in hese models are assumed ha he exising linear and nonlinear paerns in a ime series can be separaely modeled or he residuals from he linear model conain only he nonlinear relaionship or he relaionship beween he linear and nonlinear componens is addiive. Therefore, hese assumpions may underesimae he relaionship beween he componens and degrade performance, if he opposie siuaion occurs, for example, if he exising linear and nonlinear paerns in a ime series canno be separaely modeled or he residuals of he linear componen don comprise valid nonlinear paerns or is no any addiive associaion beween he linear and nonlinear elemens and he relaionship is differen (for example muliplicaive). In addiion, as menioned previously, i canno be generally guaraneed ha he performance of hese hybrid models will be beer han boh componen models (Khashei & Bijari, 2010). Arificial neural nework (p,d,q) model is proposed in order o overcome he above-menioned limiaions of he radiional hybrid linear and nonlinear models such as Zhang s hybrid model. This model also is a hybrid linear and nonlinear model ha combines an auoregressive inegraed moving average (ARIMA) as linear model wih a mulilayer percepron as nonlinear model using a new mehodology in order o yield more accurae resuls. In he arificial neural nework (p,d,q) model such as in he Box Jenkins mehodology in linear modeling, he fuure value of a ime series is considered as nonlinear funcion of several pas observaions and random errors as follows (Khashei & Bijari, 2010). z,z,..., z, e,e,..., e y f (9) 1 2 m 1 2 n

10 274 Khashei and Bijari d where f is a nonlinear funcion deermined by he neural nework, z 1 B y, e is he residual of he ARIMA model a ime and m and n are inegers. So, in he firs sage, an auoregressive inegraed moving average model is used in order o generae he residuals ( e ). In second sage, a neural nework is used in order o model he nonlinear and linear relaionships exising in residuals and original observaions. Thus, z Q p p q w0 w j g( w0,j wi,j z i wi,j e p i ) j 1 i 1 i p 1 (10) where, w i, j i 0,1,2,..., p q, j 1,2,...,Q and w j j 0,1,2,...,Q are connecion weighs; p, q, Q are inegers, which are deermined in design process of final neural nework (Khashei & Bijari, 2010). I mus be noed ha any se of above menioned variables e i i 1,..., n or z i i 1,..., m may be deleed in design process of final neural nework. This maybe relaed o he underlying daa generaing process and he exising linear and nonlinear srucures in daa. For example, if daa only consis of pure nonlinear srucure, hen he residuals will only conain he nonlinear relaionship. For he reason ha auoregressive inegraed moving average is a linear model and does no able o model nonlinear relaionship; herefore, he se of residuals e i i 1,..., n variables maybe deleed agains oher of hose variables (Khashei & Bijari, 2010) Advanages and disadvanages I can be seen ha in he arificial neural nework (p,d,q) model in conras of he radiional hybrid models such as Zhang s hybrid model, no assumpion is required in consrucing process. In he arificial neural nework (p,d,q) model is no needed o be assumed ha he exising linear and nonlinear paerns in a ime series can be separaely modeled and hey modeled simulaneously; or he residuals from he linear model only conain he nonlinear relaionship. In addiion, in his model, no prior assumpion is considered for he relaionship beween he linear and nonlinear componens and i will be generally esimaed as funcion by neural nework. In addiional, i can be generally guaraneed ha he performance of he arificial neural nework (p,d,q) model will no be worse han eiher of he componens auoregressive inegraed moving average (ARIMA) and arificial neural neworks (ANNs) used separaely. However, despie he all advanages menioned for he arificial neural nework (p,d,q) model, i canno be generally guaraneed ha he performance of his model will be beer han he Zhang s hybrid model The generalized hybrid ANNs/ARIMA model In order o yield a more general and more accurae hybrid linear and nonlinear model han he arificial neural nework (p,d,q) model, generalized hybrid ANNs/ARIMA model has been proposed. The generalized hybrid ANNs/ARIMA model such as he arificial neural nework (p,d,q) has no above-menioned assumpion of he radiional hybrid ARIMA and ANNs models. In his model, a ime series is also considered as funcion of a linear and a nonlinear componen. Thus, y f ( L,N ), (11),

11 Which Mehodology is beer for Combining Linear where L denoes he linear componen and he main aim is linear modeling; herefore, an auoregressive inegraed moving average (ARIMA) model is used o model he linear componen. The residuals from he firs sage will conain he N denoes he nonlinear componen. In he firs sage, nonlinear relaionship ha linear model dose no able o model i, and maybe linear relaionship (Taskaya & Ahmad, 2005). Thus he L will be as follows. L p q ˆ i z i j j e L e, i 1 (12) j 1 d Lˆ is he forecas value for ime from he esimaed relaionship (1), where z 1 B y, and e is he residual a ime from he linear model. The forecased values and residuals of linear modeling are he resuls of firs sage ha are used in nex sage. In addiion, he linear paerns are magnified by ARIMA model in order o apply in second sage. In second sage, he main aim is nonlinear modeling; herefore, a mulilayer percepron is used in order o simulaneously model he nonlinear and probable linear relaionships ha may be remained in residuals of linear modeling and also he nonlinear and linear relaionships in he original daa. Thus, N N f e,...,e, (13) 1 n 2 f z,...,z, (14) 1 m 1 2 N f N,N, (15) 2 where f 1, f, and f are he nonlinear funcions deermined by he neural nework. n and m are inegers and are ofen referred o as orders of he model. Thus, he combined forecas will be as follows: y 1 2 f ( N,Lˆ,N ) f ( e,...,e,lˆ,z,...,z ) (16) 1 n 1 1 m 1 where f are he nonlinear funcions deermined by he neural nework. n 1 n and m 1 m are inegers deermined in design process of final neural nework. I mus be noed ha similar o he ANN (p,d,q) model, any aforemenioned variable e i i 1,..., n, Lˆ, and z j j 1,..., m or se of hem e i i 1,..., n or z i i 1,..., m may be deleed in design process of final neural nework. However in he generalized hybrid ANNs/ARIMA model, in opposie of he arificial neural nework (p,d,q), he linear componen ( Lˆ ) and original daa are simulaneously applied in order o model he linear srucures. As previously menioned, in building he auoregressive inegraed moving average as well as arificial neural nework models, subjecive judgmen of he model order as well as he model adequacy is ofen needed. I is possible ha subopimal models will be used in he hybrid model. For example, he curren pracice of Box Jenkins mehodology focuses on he low order auocorrelaion. A model is considered adequae if low order auocorrelaions are no significan even hough significan auocorrelaions of higher order sill exis. This subopimaliy may no

12 276 Khashei and Bijari affec he usefulness of he hybrid model. Granger (1989) has poined ou ha for a hybrid model o produce superior forecass, he componen model should be subopimal. In general, i has been observed ha i is more effecive o combine individual forecass ha are based on differen informaion ses (Granger, 1989) Advanages and disadvanages Alhough i can be guaraneed ha, he performance of he generalized hybrid ANNs/ARIMA model will no be worse han he arificial neural nework (p,d,q) model and also eiher of he componens, and a more general and more accurae model can be obained using he abovemenioned mehodology, here are no enough reasons ha we can sure ha he performance of he generalized hybrid ANNs/ARIMA will be also beer han Zhang s hybrid model. 3. DATA SETS Since we canno generally demonsrae ha which one of he above-menioned mehodologies is beer for consrucing a more appropriae and more effecive hybrid model for ime series forecasing, in his secion, hree well-known real daa ses including he Wolf s sunspo daa, he Canadian lynx daa, and he Briish pound/us dollar exchange rae daa are considered in order o compare he predicive capabiliies of he menioned hybrid models in pracice. These ime series come from differen areas and have differen saisical characerisics. They have been widely sudied in he saisical as well as he neural nework lieraure (Khashei & Bijari, 2010). Boh linear and nonlinear models have been applied o hese daa ses, alhough more or less nonlineariies have been found in hese series. Only he one-sep-ahead forecasing is considered. Two performance indicaors including MAE (mean absolue error) and MSE (mean squared error), which are compued from he following equaions, are employed in order o measure forecasing performance of he hybrid models. MAE MSE 1 N 1 N N i 1 N e i e i i 1 2 (17) (18) 3.1. The Wolf s sunspo daa The sunspo series is record of he annual aciviy of spos visible on he face of he sun and he number of groups ino which hey cluser. The sunspo daa, which is considered in his invesigaion, conains he annual number of sunspos from 1700 o 1987, giving a oal of 288 observaions. The sudy of sunspo aciviy has pracical imporance o geophysiciss, environmen scieniss, and climaologiss. The daa series is regarded as nonlinear and non-gaussian and is ofen used o evaluae he effeciveness of nonlinear models (Ghiassi & Saidane, 2005). The plo of his ime series (Figure 2) also suggess ha here is a cyclical paern wih a mean cycle of abou 11 years. The sunspo daa has been exensively sudied wih a vas variey of linear and nonlinear ime series models including ARIMA and ANNs. To assess he forecasing performance of proposed model, he sunspo daa se is divided ino wo samples of raining and esing. The raining daa se, 221 observaions ( ), is exclusively used in order o formulae he model and hen he es sample, he las 67 observaions ( ), is used in order o evaluae he performance of he esablished model.

13 Which Mehodology is beer for Combining Linear The Canadian lynx series Figure 2 Annual Wolf s sunspo ime series from 1700 o 1987 The lynx series, which is considered in his invesigaion, conains he number of lynx rapped per year in he Mackenzie River disric of Norhern Canada. The daa se are ploed in Figure 3, which shows a periodiciy of approximaely 10 years (Sone, 2007). The daa se has 114 observaions, corresponding o he period of I has also been exensively analyzed in he ime series lieraure wih a focus on he nonlinear modeling (Tang & Ghosal, 2007; Cornillon e al., 2008) see Wong and Li (2000) for a survey. Following oher sudies (Zhang, 2003), he logarihms (o he base 10) of he daa are used in he analysis. The raining daa se, 100 observaions ( ), is exclusively used in order o formulae he model and hen he es sample, he las 14 observaions ( ), is used in order o evaluae he performance of he esablished model Figure 3 Annual Canadian lynx ime series from 1821 o The exchange rae (Briish pound /US dollar) The las daa se ha is considered in his invesigaion is he exchange rae beween Briish pound and Unied Saes dollar. Predicing exchange rae is an imporan ye difficul ask in inernaional finance. Various linear and nonlinear heoreical models have been developed bu few are more successful in ou-of-sample forecasing han a simple random walk model. Recen applicaions of neural neworks in his area have yielded mixed resuls. The daa used in his paper conain he weekly observaions from 1980 o 1993, giving 731 daa poins in he ime series. The ime series plo is given in Figure 4, which shows numerous changing urning poins in he series. In his paper following Meese and Rogoff (1983) and Zhang (2003) and Khashei and Bijari (2010), he naural logarihmic ransformed daa is used in he modeling and forecasing analysis. The raining daa se,

14 Khashei and Bijari firs 13 years ( ), is exclusively used in order o formulae he model and hen he es sample, he las year (1993), is used in order o evaluae he performance of he esablished model Figure 4 Weekly Briish pound agains he Unied Saes dollar exchange rae series from 1980 o RESULTS In his secion, he predicive capabiliies of he hybrid models including Zhang s hybrid ANNs/ARIMA, arificial neural nework (p,d,q), and generalized hybrid ANNs/ARIMA are compared ogeher and also compared wih eiher of heir componens arificial neural neworks and auoregressive inegraed moving average using hree above-menioned daa ses The Wolf s sunspo daa forecass In he Wolf s sunspo daa forecas case, according o he Akaike s informaion crierion (AIC), we find ha a subse auoregressive model of order nine (AR (9)) is he mos parsimonious among all ARIMA models which are also found adequae judged by he residual analysis. Many researchers such as Subba Rao and Gabr (1984), Hipel and McLeod (1994), Zhang (2003), and Khashei and Bijari (2010) have also used his model. The neural nework model used is composed of four inpus, four hidden and one oupu neurons (in abbreviaed form, N (4-4-1) ), as also employed by De Groo and Wurz (1991), Corell e al. (1995), Zhang (2003), and Khashei and Bijari (2010). Two forecas horizons of 35 and 67 periods are used in order o assess he forecasing performance of he hybrid models and heir componens. The forecasing resuls of above-menioned models for he sunspo daa are summarized in Table 1. Table 1 Comparison of he performance of he hybrid models and heir componens for sunspo daa se forecasing Model 35 poins ahead 67 poins ahead MAE MSE MAE MSE Auo-Regressive Inegraed Moving Average (ARIMA) Arificial Neural Neworks (ANNs) Zhang s hybrid model Arificial Neural Nework (p,d,q) Generalized hybrid ANNs/ARIMA Resuls show ha while applying neural neworks alone can improve he forecasing accuracy over he ARIMA model in he 35-period horizon, he performance of ANNs is geing worse as ime horizon exends o 67 periods. This may sugges ha neiher he neural nework nor he ARIMA model capures all of he paerns in he daa and combining wo models ogeher can be an effecive

15 Which Mehodology is beer for Combining Linear way in order o overcome his limiaion. However, he resuls of he Zhang s hybrid model show ha; alhough, he overall forecasing errors of Zhang s hybrid model have been reduced in comparison wih ARIMA and ANN, his model may also give worse predicions han eiher of hose, in some specific siuaions. These resuls may be occurred due o he assumpions, which are considered in consrucing process of he hybrid model by Zhang (2003). The obained resuls of he arificial neural nework (p,d,q) model (Khashei and Bijari, 2010) confirm his hypohesis ha hese assumpions will degenerae he performance of Zhang s hybrid model if he opposie siuaions occur. The arificial neural nework (p,d,q) model has yielded more accurae resuls han Zhang s hybrid model and also boh ARIMA and ANN models used separaely across wo differen ime horizons and wih boh error measures. However, obained resuls show ha more accurae resuls can be obained using he generalized hybrid ANNs/ARIMA model. This model has yielded more accurae resuls han he arificial neural nework (p,d,q), Zhang s hybrid model and also boh componens used in isolaion across wo differen ime horizons and wih boh error measures, excep for MSE of he arificial neural nework (p,d,q) model in he 35-period horizon The Canadian lynx series forecass In a similar fashion, we fi a subse auoregressive model of order welve (AR (12)) o Canadian lynx daa, according o he Akaike s informaion crierion (AIC). This is a parsimonious model also used by Subba Rao and Gabr (1984) and Zhang (2003), and Khashei and Bijari (2010). In addiion, a neural nework, which is composed of seven inpus, five hidden and one oupu neurons (N (7-5-1) ), has been designed o Canadian lynx daa se forecas, as also employed by Zhang (2003), and Khashei and Bijari (2010). The overall forecasing resuls of he above-menioned models for he las 14 years are summarized in Table 2. Table 2 Comparison of he performance of he hybrid models and heir componens for Canadian lynx daa se forecasing Model MAE MSE Auo-Regressive Inegraed Moving Average (ARIMA) Arificial Neural Neworks (ANNs) Zhang s hybrid model Arificial Neural Nework (p,d,q) Generalized hybrid ANNs/ARIMA Numerical resuls show ha he used neural nework gives slighly beer forecass han he ARIMA model and he Zhang s hybrid model, significanly ouperform he boh of hem. However, according o he previous case, he obained resuls of he arificial neural nework (p,d,q) model are beer han Zhang s hybrid model and he obained resuls of he generalized hybrid ANNs/ARIMA model are beer han he arificial neural nework (p,d,q) model in boh error measures The exchange rae (Briish pound /US dollar) forecass Wih he exchange rae daa se and according o he Akaike s informaion crierion (AIC), he bes linear ARIMA model is found o be he simple random walk model: y y 1. This is he same finding suggesed by many sudies in he exchange rae lieraure ha a simple random walk is he dominan linear model. They claim ha he evoluion of any exchange rae follows he heory of efficien marke hypohesis (EMH) (Timmermann & Granger, 2004). According o his hypohesis, he bes predicion value for omorrow s exchange rae is he curren value of he exchange rae and

16 280 Khashei and Bijari he acual exchange rae follows a random walk. A neural nework, which is composed of seven inpus, six hidden and one oupu neurons (N (7-6-1) ) is designed in order o model he nonlinear paerns, as also employed by ohers (Zhang, 2003; Khashei & Bijari, 2010). Three ime horizons of 1, 6 and 12 monhs are used in order o assess he forecasing performance of models. The forecasing resuls of above-menioned models for he exchange rae daa are summarized in Table 3. Table 3 Comparison of he performance of he proposed model wih hose of oher forecasing models (exchange rae daa) * Model 1 monh 6 monh 12 monh MAE MSE MAE MSE MAE MSE Auo-Regressive Inegraed Moving Average Arificial Neural Neworks (ANNs) Zhang s hybrid model Arificial Neural Nework (p,d,q) Generalized hybrid ANNs/ARIMA * Noe: All MSE values should be muliplied by In he exchange rae daa se forecasing, similar o he previous secion, he performance of he generalized hybrid ANNs/ARIMA model is beer han he arificial neural nework (p,d,q) model, he performance of he arificial neural nework (p,d,q) model is beer han Zhang s hybrid model, and he performance of he Zhang s hybrid model is beer han eiher of he componens across hree differen ime horizons and wih boh error measures. 5. CONCLUSIONS In his paper, he predicive capabiliies of hree differen hybrid linear and nonlinear models in which he auoregressive inegraed moving average (ARIMA) as linear model is combined o he mulilayer percepron (MLP) as nonlinear model are compared ogeher for ime series forecasing. These models include Zhang s hybrid ANNs/ARIMA, he arificial neural nework (p,d,q), and he generalized hybrid ANNs/ARIMA models. Some general resuls obained from comparing hese models ogeher are as follows: 1- I can be generally guaraneed ha he performance of he Zhang s hybrid model will no be worse han auoregressive inegraed moving average (ARIMA) model. 2- I canno be generally guaraneed ha he performance of he Zhang s hybrid model will no be worse han he mulilayer percepron (MLP) model. 3- I can be generally guaraneed ha he performance of he arificial neural nework (p,d,q) model will no be worse han eiher of he componens including auoregressive inegraed moving average (ARIMA) and mulilayer percepron (MLP) models. 4- I canno be generally guaraneed ha he performance of he arificial neural nework (p,d,q) model will no be worse han he Zhang s hybrid model. 5- I can be generally guaraneed ha he performance of he generalized hybrid ANNs/ARIMA model will no be worse han eiher of he componens including auoregressive inegraed moving average (ARIMA) and mulilayer percepron (MLP) models.

17 Which Mehodology is beer for Combining Linear I can be generally guaraneed ha he performance of he generalized hybrid ANNs/ARIMA model will no be worse han he arificial neural nework (p,d,q) model. 7- I canno be generally guaraneed ha he performance of he generalized hybrid ANNs/ARIMA model will no be worse han he Zhang s hybrid model. Since, i canno be generally demonsraed ha he obained resuls of which one of hese models is more accurae, he predicive capabiliies of he above-menioned hybrid models are pracically compared ogeher. Empirical resuls wih hree well-known real daa ses including he Wolf s sunspo daa, he Canadian lynx daa, and he Briish pound agains he Unied Saes dollar exchange rae daa, indicae ha while all of hese mehodologies can be an effecive way o improve forecasing accuracy achieved by eiher of componens used separaely, he generalized hybrid ANNs/ARIMA model is more accurae and perform beer han arificial neural nework (p,d,q) and Zhang s hybrid ANNs/ARIMA models. ACKNOWLEDGEMENTS The auhors wish o express heir graiude o, Seyed Reza Hejazi, assisan professor of indusrial engineering, Isfahan Universiy of Technology, who grealy helped us. REFERENCES [1] Armano G., Marchesi M., Murru A. (2005), A hybrid geneic-neural archiecure for sock indexes forecasing; Informaion Sciences 170; [2] Aryal D.R., Yao-Wu W. (2003), Neural nework Forecasing of he producion level of Chinese consrucion indusry; Journal of Comparaive Inernaional Managemen 6(2); [3] Balkin S.D., Ord J.K. (2000), Auomaic neural nework modeling for univariae ime series; Inernaional Journal of Forecasing 16; [4] Baes J.M., Granger W.J. (1969), The combinaion of forecass; Operaion Research 20; [5] Berardi V.L., Zhang G.P. (2003), An empirical invesigaion of bias and variance in ime series forecasing: modeling consideraions and error evaluaion; IEEE Transacions on Neural Neworks 14(3); [6] Bollerslev T. (1986), Generalized auoregressive condiional heeroscedasiciy; Journal of Economerics 31; [7] Box P., Jenkins G.M. (1976), Time Series Analysis: Forecasing and Conrol; Holden-day Inc, San Francisco, CA. [8] Brace M.C., Schmid J., Hadlin M. (1991), Comparison of he forecasing accuracy of neural neworks wih oher esablished echniques; Proceedings of he Firs Forum on Applicaion for weigh eliminaion, IEEE Transacions on Neural Neworks of Neural Neworks o Power Sysems; Seale, WA, [9] Chakrabory K., Mehrora K., Mohan C.K., Ranka S. (1992), Forecasing he behavior of mulivariae ime series using neural neworks; Neural Neworks 5; [10] Chappel D., Padmore J., Misry P., Ellis C. (1996), A hreshold model for he French franc/deuschmark exchange rae; Journal of Forecasing 15(3); [11] Chen K.Y., Wang C.H. (2007), A hybrid SARIMA and suppor vecor machines in forecasing he producion values of he machinery indusry in Taiwan; Exper Sysems wih Applicaions 32;