COMPARISON OF TWO LEARNING NETWORKS. Daniel Nikovski. Carnegie Mellon University. Pittsburgh, Pennsylvania 15213, USA

Transcription

1 COMPARISON OF TWO LEARNING NETWORKS FOR TIME SERIES PREDICTION Daniel Nikovski Scool of Computer Science Carneie Mellon Universit Pittsbur, Pennslvania 53, USA Medi Zaram Department of Computer Science Soutern Illinois Universit at Carbondale Carbondale, Illinois 690, USA ABSTRACT Hierarcical mitures of eperts (HME) [JJ94] and radial basis function (RBF) networks [PG89] are two arcitectures tat learn muc faster tan multilaer perceptrons. Teir faster learnin is due not to ierorder searc mecanisms, but to restrictin te potesis space of te learner b constrainin some of te laers of te network to use linear processin units. It can be conjectured tat since teir potesis space is restricted in te same manner, te approimation abilities of te two networks sould be similar, even tou teir computational mecanisms are different. An empirical verication of tis conjecture is presented, based on te task of predictin a nonlinear caotic time series enerated b an infrared laser. INTRODUCTION Te problem of predictin te continuation of a iven time series is fundamentall a macine learnin problem, since suc a continuation requires te etraction b te prediction metod of te rules tat overn te dnamics of te time series. Tis etraction as to be done from eamples of past values of te time series. Te use of past values of te time series for reconstructin te dnamics of te underlin dnamic sstem is justied b Takens' teorem [WG94]. If te dnamic sstem tat enerates te time series is linear, two suitable forms for representin te inferred knowlede are reression coecients and transfer functions (ARIMA models) [LS83]. However, wen te underlin dnamic sstem is nonlinear, oter representations are necessar. One possibilit is to use locall linear models tat store all past values of te time series and ten produce a prediction based on locall linear interpolation [GW94]. Wile ivin ood predictions, suc models are obviousl useless wen te time series is of innite duration. Te oter possibilit is to use learnin metods suc as multilaer perceptrons (MLP) or oter learnin networks tat concentrate te etracted knowlede in a small number of parameters, wic are adjusted b on-line learnin rules [GW94]. Te requirements to suc learnin networks are tat te sould be universal function approimators, ave ood eneralization power, be able to learn quickl from eamples, and be implementable in ardware. Te most popular learnin networks, MLP, possess all of te above features ecept fast learnin; converence of MLP for nontrivial problems is in te order of undreds of tousands of iterations and can be impractical for most real-time enineerin applications even if it is accelerated b ardware. Te slow learnin in MLP as motivated te development of alternative learnin arcitectures tat can learn faster. Two suc learnin models are radial basis functions [PG89] and ierarcical mitures of eperts [JJ94]. Te two models are muc faster tan MLP because a sinicant part of teir processin is linear, tou in a dierent wa. Bot metods ave

2 been used for time series prediction RBF b Casdali [Cas89] and HME b Waterouse and Robinson [WR95]. Te approimation power and converence time of eac of te two metods ave been compared wit MLP so far [Cas89, JJ94], but on dierent test problems. It is tus dicult to see wic of te two metods is better for time series prediction or an oter nonlinear identication task, and it is of practical interest to compare tem on a sinle test problem. Section describes te cosen problem and te testin procedure. Te arcitecture and performance on te test problem of RBF and HME are presented in sections 3 and 4 respectivel. Section 5 provides a statistical comparison of te two metods, and section 6 discusses te results and concludes. DESCRIPTION OF THE PROBLEM Bencmarkin of nonlinear time series prediction metods requires test problems tat are representative of te problems commonl encountered in practice. Suc nonlinear time series ave been provided b te oranizers of te Santa Fe Institute competition in time series prediction [GW94]. We used as test problem one of tese time series, consistin of output of a CH3 far-infrared laser. Tis nonlinear caotic time series as been predicted successfull wit a locall linear model b Sauer [Sau94] and wit a timedela neural network b Wan [Wan94]. Wan used an embeddin dimension of 8; tis value for te dimensionalit of te embeddin space was adopted in our eperiments too, wic means tat our networks ad 8 inputs eac. Te oriinal data were measured as inteers in te rane from 0 to 55, wic introduces quantization error of about 0:%. Based on te provided time series data, one trainin and nine testin data sets were prepared. Te trainin set ad 99 eamples, and te testin sets ad 000 eamples eac. RBF and HME used te trainin set to estimate teir parameters. Te rst of te testin sets was used for cross-validation - i.e., te estimated model tat produced best out-of-sample error on te rst testin set was assumed to be te optimal one for te respective arcitecture. Te out-of-sample error of tis model was tested on te remainin eit testin sets in order to compare te two learnin metods. All errors reported in te paper are normalized root mean squared errors (NRMSE). For te purpose of comparison wit linear prediction metods, a recursive least squares (RLS) autoreressive model was tested too [LS83]. C f 3 K FIGURE. RBF network PREDICTION WITH RADIAL BASIS FUNCTIONS Radial Basis Functions (RBF) ave been known in approimation teor as a powerful and computationall ecient metod for interpolation and approimation of functions [PG89]. Wit te development of te connectionist approac to computation, it was noticed tat te computation performed b tese functions can be carried out b parallel and distributed processin elements similar to multilaer perceptrons [PG89]. An RBF network consists of a number of units wose output is a nonlinear function of te unit's input (Fi. ). Te output of te network is iven b K = f() =! (k, C k) = were is te vector input to te network,! are te weits tat sould be estimated, C are vectors called knots, and kkis te L norm on te space of input vectors. In our eperiments te knots C were cosen to be a subset of te trainin eamples; oter possible approaces include various unsupervised clusterin scemes or supervised adjustment of te knots. Te function belons to te class of radial basis functions [PG89]. For our eperiments, te linear function (r) = r was used. Te estimation of te weits! is a eneralized linear squares problem and is solvable in time O(K 3 ) in te number of knots K. A plot of te in- and outof-sample NRMSE versus te number of knots is iven in Fi. for te problem of laser output prediction. Te in-sample error decreases raduall to zero, as epected. Te out-of-sample error follows closel, but n

3 0.8 in-sample out-of-sample Gatin 0.6 NRMSE knots FIGURE. NRMSE for RBF approimation attens to 0.0 wen te number of knots approaces te number of trainin eamples. Some overttin occurs; te lowest out-of-sample error is attained for 990 knots, less te number of trainin eamples. Consequentl, te optimal model from cross-validation wit te rst testin set is te model wit 990 knots. Tis model was tested on te remainin 8 testin sets, resultin in averae out-of-sample error E RBF = 0:0. PREDICTION WITH HIERARCHICAL MITURES OF EPERTS Te HME arcitecture is a eneral nonlinear function approimation model introduced b Jordan and Jacobs [JJ94]. In particular, it can be used for nonlinear reression in time series prediction. Te arcitecture consists of a tree-like ierarc of atin nodes and a set of eperts at te leaves of te tree (Fi. 3). If HME is to be used for nonlinear reression, eac of te eperts forms a linear prediction ^ ij of te true output of te network for a iven input vector : ^ ij = u T ij Te purpose of te atin nodes is to assin weits to te predictions ^ ij of te individual eperts - tese weits sould be proportional to te precision wit wic an epert approimates te function at a particular location. For tis purpose, te weits depend on te input too. For te case of binar trees, sown in Fi. 3, two values and are output b te ate at te root of te tree, wit teir sum equal to : Gatin Epert Epert Epert FIGURE 3. HME arcitecture i = v T i Epert Gatin Here te vector v i parametrizes te atin epert. Te predicted output ^ is ten a weited averae of te predictions at te lower level ^ i : ^ = i^ i i= Te weits of te atin node at te root of te tree perform a partitionin of te input space into two parts, in eac of wic one of te weits is close to, wile te oter weit is close to 0. In tis wa a "soft split\ is formed, wose steepness and orientation are determined b te vector v, v. Te reater te lent of tis vector, te sarper te split is. Te splittin continues recursivel on te lower levels of te tree. Te predictions at te lower level are formed in a similar wa, tis time usin directl te predictions of te eperts: ^ i = jji^ ij j= Jordan and Jacobs [JJ94] derived learnin rules for te eperts and te ates on te basis of a probabilit model and a maimum-likeliood approac. Te probabilit distribution of te output of a particular linear epert is iven b P (j; u ij )= p e, (,^ ij ) i = ei Pk e k, i =;, k =; B dierentiatin te likeliood tat te observed output is enerated b te HME model wit a partic-

4 NRMSE in-sample out-of-sample te laser test problem, te number of levels was varied and te in-sample and out-of-sample errors were plotted, similarl to te testin wit RBF networks. Te results are sown in Fi. 4. Te tree was binar and te learnin rate was set to 0:0. For eac model, 0000 iterations were performed. Te least out-of-sample error was attained for a seven-level network (8 eperts). Tis network was used for comparison on te remainin 8 testin sets. Notice tat tis was not te network wit smallest in-sample error levels FIGURE 4. NRMSE of HME approimation ular set of parameters, te followin on-line radient ascent learnin rules result: u ij = i jji (, ^ ij ) v i = ( i, i ) v ij = i ( jji, jji ) were is a learnin rate and te quantities i, jji and ij are posterior probabilities dened as follows: i = ipj jjip ij (j;) Pi ipj jjip ij (j;) jji = ij = jjip ij (j;) Pj jjip ij (j;) i jji P ij (j;) Pi ipj jjip ij (j;) Jordan and Jacobs ave also derived anoter set of learnin rules, based on te Epectation- Maimization (EM) developed in statistics [JJ94]. Te EM learnin rules learn faster tan te ones based on te maimum-likeliood approac. In te HME arcitecture, te number of estimated parameters and ence te acieved eneralization depend on te number of levels in te ierarc. For COMPARISON OF THE TWO LEARNING MODELS Te relative performance of RLS and te best RBF and HME models is sown in Table. Wile te superiorit of HME and RBF over RLS is obvious, it is not clear ow te RBF and HME networks compare wit eac oter. To tis end a paired two-tailed t-test was performed on te 8 testin sets to verif te potesis tat one of te two networks is superior. Te results are sown in Table. Table sows tat tere is no statistical sinicance to te potesis tat RBF and HME ave dierent performance on te learnin task at and. CONCLUSIONS Te eperiments presented in te previous sections demonstrate tat te HME and RBF arcitectures possess comparable approimation abilities on a nonlinear identication task. Wile it mit be dicult to sow analticall suc similarit in te eneral case, te presented empirical results suest tat te two learnin networks consider similar potesis spaces (or, in oter words, impose similar smootness constraints on teir approimations.) Tis observation can be used to infer epectations about te performance on a iven task of one of te networks, knowin te approimation error of te oter. One application mit be to train fast an RBF network and if Table. NRMSE of RLS, RBF, and HME on te laser output time series. RLS RBF HME In-sample Out-of-sample ( E) Table. Comparison of RBF, HME, and RLS: t-statistic and attained sinicance level p. t p RBF vs. HME : RBF vs. RLS < 0, HME vs. RLS < 0,

5 te performance is satisfactor, proceed wit te more time-consumin trainin of a HME network, wose approimation is piecewise linear and because of tat can be more readil used for prediction and control applications. REFERENCES [Cas89] Casdali, M. (989). Nonlinear prediction of caotic time series. Psica D, vol. 35, [GW94] Gersenfeld, N.A., and A.S. Weiend (994). Te future of time series: learnin and understandin. In Weiend, A.S, and N.A. Gersenfeld (Eds.) (994) Time Series Prediction. Readin, MA: Addison- Wesle, -70. [JJ94] [LS83] Jordan, M.I., and Jacobs, R.A. (994). Hierarcical mitures of eperts and te EM aloritm. Neural Computation, vol. 6, 8-4. Ljun, L., and T. Soderstrom (983). Teor and Practice of Recursive Identication. Cambride, MA: MIT Press. [PG89] Poio, T., and F. Girosi (989). A Teor of s for Approimation and Learnin. AI Memo 40, MIT AI Lab. [Sau94] Sauer, T. (994). Time series prediction b usin dela coordinate embeddin. In Weiend, A.S, and N.A. Gersenfeld (Eds.) (994) Time Series Prediction. Readin, MA: Addison-Wesle, [Wan94] Wan, E.A.(994). Time series prediction b usin a connectionist network wit internal dela lines. In Weiend, A.S, and N.A. Gersenfeld (Eds.) (994) Time Series Prediction. Readin, MA: Addison- Wesle, [WR95] Waterouse, S.R., and A.J. Robinson (995). Nonlinear prediction of acoustic vectors usin ierarcical mitures of eperts, in G. Tesauro, D.S. Touretzk, and T.K. Leen, (Eds.), Neural Information Processin Sstems 7, Cambride, MA: MIT Press. [WG94] Weiend, A.S, and N.A. Gersenfeld (Eds.) (994) Time Series Prediction. Readin, MA: Addison-Wesle.