One step ahead prediction using Fuzzy Boolean Neural Networks 1

Transcription

1 One ste ahead rediction using Fuzzy Boolean eural etworks 1 José A. B. Tomé IESC-ID, IST Rua Alves Redol, Lisboa jose.tome@inesc-id.t João Paulo Carvalho IESC-ID, IST Rua Alves Redol, Lisboa joao.carvalho@inesc-id.t Abstract Time series rediction is a roblem with a wide range of alications, including energy systems lanning, currency forecasting, stock exchange oerations or traffic rediction. Accordingly, a number of different rediction aroaches have been roosed such as linear models, Feedforward eural network models, Recurrent eural networks or Fuzzy eural Models. In this aer one resents a rediction model based on fuzzy rules that relate ast data values with the next unknown value to be estimated. A Fuzzy Boolean eural etwork has been used for that urose and the laser data set of the Santa Fe contest has been used for illustration urose. The results turned to be encouraging when comared with other ublished methods. Keywords: Time Series Prediction, Fuzzy eural ets. 1 Introduction Many ractical rediction roblems, such as stock exchange, traffic or water stream flow forecast, are satisfied with a one ste ahead rediction. That is, in this tye of roblems one aims to estimate a time series value v t given ast values v t-1, v t-2,, v t-k, their sloes or other simle functions of these values. Traditionally, most techniques to model this tye of time series assume linear relationshis among the variables, as AR models based on the Box and Jenkins method [1], [5]. on-linear models have also been used, mostly based on neural network architectures both feed forward [3] and recurrent [2], but also using fuzzy concets [4]. In this aer one introduces a new rediction model based on Fuzzy Boolean eural etworks (FB), which have been reviously been resented as ets caable of learning qualitative rules and of reasoning using those rules [10], [11]. The idea behind this aroach is that one-ste ahead rediction is nothing more that reasoning about the next value from assed values, assuming the variable behaviour can be described by a set of qualitative rules. In order to test such a conjecture one has chosen a well-known and studied roblem, which is the estimation of a time series obtained from laser data, the so called data set A from the Santa Fe contest on revision [14]. Section II resents the FB s architecture, with details about its learning and fuzzy reasoning caabilities. Section III addresses the alication details of FB s to time series rediction and a study on the effect of various net arameters on the net erformance for laser data examle. Section IV concludes and discusses some issues that deserve further attention, such as the ossibility to understand the et results, through the fuzzy rules results. 1 This work is artially suorted by the FCT - Portuguese Foundation for Science and Technology under roject POSI/SRI/47188/

2 2 Fuzzy Boolean ets Fuzzy Boolean ets [10] are insired on natural neural systems, in which neurons are of the tye on/off and where activity of concets/variables is given by the rate of activated (firing) neurons on the corresonding neural areas. In such nets every neuron of a consequent variable or concet has the caability to memorize an individual and very simle (binary) vision of each one of the ossible qualitative rules. In this model fuzziness is an inherent emerging roerty that can be derived from the network behaviour. A very interesting roerty of the model concerns its robustness, in the sense that it is immune to individual neuron or connection errors, which is not the case of other models, such as the classic artificial neural nets. The architecture of the model is anchored in cards/areas of neurons. Meshes of weightless connections between antecedent neuron oututs and consequent neuron inuts link antecedent to consequent areas. Individual connections are random and neurons are binary, both in which concerns inuts and oututs. euron s internal unitary memories are rovided, however, with a third state in addition to the classical 1 and 0, with the objective of identifying not taught situations. To each area a concet or variable is associated and the value of that concet or variable, when stimulated, is given by the activation ratio of that area (that is, the relation between activated -outut 1 - neurons and the total number of neurons). Basically, each consequent neuron samles the antecedent saces using for each antecedent a limited number (say m) of their neuron s oututs as its own inuts, being this number, m, much smaller then the number of neurons er area. This means that for rules with - antecedent/1-consequent each neuron has.m inuts. The oerations carried out by each neuron are just the combinatorial count of the number of activated inuts from every antecedent. They are erformed with the classic Boolean oerations (AD, OR), using as oerands the inuts coming from antecedents and the Boolean internal state variables that are established during a teaching hase. It has been roved [10] that, from these micro oerations (that is, on individual neurons using local and limited information), it emerges a global or macro qualitative reasoning caability on the concets, which can be exressed on the form of rules of tye: IF Antecedent1 is A1 AD Antecedent2 is A2 AD.THE Consequent is Ci, In the exression, Antecedent1, Antecedent2,.., Consequent are variables or concets and A1, A2;, Ci are linguistic terms (or fuzzy sets) of these variables (such as, small, high, etc.). The roof [10] is achieved through the interretation made to the relationshi between antecedent and consequent activation ratios. The consequent activation ratio is given by: L 1 L 1... ( k1= 0 k= 0 i = 1 m.i qi.(1 i) m qi.r(q1,...,q)) qi qi Ski (1) In this exression L is the number of subsets of counts (each one a set of consecutive integers {0,1,, m}), Ski are those sets, qi reresents every ossible number of activated inuts on antecedent i belonging to the set indexed by ki, i are the antecedent activation ratios of area i and r(q1,.,qn) are activation ratios of the internal binary memories (measured through every neuron), which have been established during the teaching hase. It is suosed that each neuron has one element of binary memory (with additional caability of memorizing if it has or not been taught) er set of counts on the antecedent inuts, (each set reresented by the set of q1 Sk 1 activated inuts on antecedent1,..., and qn Skn activated inuts on antecedent n). Taking the inner sum above to the outside and considering each count individually rather than on sets Ski, the following exression is obtained: m m Π.. i1= 0 i= 0 j= 1 m ( ) ij m ij j ) j.(1.r(i1,.., i) ij (2) This equation reresents at the macroscoic/network level an emergent behaviour, which can be viewed as a fuzzy qualitative reasoning that has been obtained from the microscoic neural oerations defined above. In order to establish this fuzzy reasoning, the algebraic roduct and the bounded sum are interreted resectively as the t-norm and t- conorm fuzzy oerations. To this urose the equations above should be interreted as follows [10]: 501

3 Inut variables, the activation ratios j, are fuzzified through binomial membershi functions of the form m ij m ij j.(1 j) ij. The evaluation of the exression for a given j reresents the membershi degree of j in that fuzzy set. The roduct of the terms, the Π, reresents the j= 1 fuzzy intersection of the antecedents (i=1,), by definition of the above t-norm. Considering the consequent fuzzy sets as singletons (amlitude "1") at the consequent UD values (i1,..,i), it follows that the equations reresent the defuzzification by the Centre of Area method [10]. During the learning hase the network is submitted to a collection of exeriments that will set or reset the individual neuron s binary memories and will establish the r(i1,...,i) robabilities in the above exression (1). For each exeriment, a different inut configuration (defined by the inut areas secific samles) is resented to each and every one of the consequent neurons. This configuration addresses one and only one the internal binary memories of each individual neuron. Udating of each binary memory value deends on its selection (or not) and on the logic value of the consequent neuron. This may be considered a Hebbian tye of learning [6] if re and ost-synatic activities are given by the activation ratios: j for antecedent area j and out for the consequent area. For each neuron, the m+1 different counts are the meaningful arameters to take into account for re synatic activity of one antecedent. Thus, in a given exeriment, the correlation between osterior synase activity ( out) and re synatic activity -the robability of activating a given decoder outut d(i 1,..i ), corresonding to i 1 activated inuts on antecedent area 1,, i activated inuts on area, - can be reresented by the robability of activating the different binary unitary memories. In ractical terms, for each teaching exeriment and for each consequent neuron, the state of binary memory ff( i1,..., i ) is decided by, and only by, the Boolean values of decoder outut d(i 1,..., i ) and of the outut neuron state considered. Considering then the r(k1,...,k),in exression (2), as the synatic strengths, one may have different learning tyes, deending on how they are udated. Here, one is considering the interesting case when non-selected binary memories maintain their state and selected binary memories take the value of consequent neuron, which corresonds to a kind of Grossberg based learning. It corresonds to the following udating equation (where indexes are not reresented and is used in lace of r, for simlicity) and where P a is the robability of activating, in the exeriment, the decoder outut associated with and out the robability of one consequent neuron to be activated: (t+1)-(t) = P a. ( out - (t)). (3) It can be roved [11] that the network converges to any taught rule. More secifically, it may be concluded that with a set of coherent exeriments -teaching the same rule- it will aroach the taught value P out roortionally to the distance between the resent value of and P out itself, that is, with aroaching zero decreasing stes. Moreover, it has been roved [12] that the et is caable of learning a set of different rules without cross-influence between different rules, and that the number of distinct rules that the system can effectively distinguish (in terms of different consequent terms) increases with the square root of the number m. This is an interesting result, since it should be verified in animal brains, that is, the number of synases coming from a given concet area is exected, according this simle model, to be at least of the order of the square of the different linguistic terms on the consequent. Finally, it has also been roved that this model is a Universal Aroximator [11], since it theoretically imlements a Parzen Window estimator [9]. This means that these networks are caable to imlement (aroximate) any ossible multi-inut single-outut function of the tye: [0,1] n [0,1]. These results give the theoretical background to establish the caability of these simle binary networks to erform reasoning and effective learning based on real exeriments. The model can also be viewed as a natural alication of the aradigm of comuting with words. Since the nets resent toologic similarities with natural systems and resent also some of their imortant roerties, it may be hyothesized that it may constitute a simle model for natural reasoning and learning. 502

4 Also, as the emergent reasoning may be interreted as fuzzy reasoning, it may also be hyothesized that natural reasoning is intrinsically fuzzy. 3 Laser Data Prediction The time series roblem here treated is the wellknown laser data of the Santa Fe contest [14]. This is data is aroximately a Lorenz model of a two level system [7] and is considered an examle of a stationary hysical system, which behaviour and equations are well established. During the learning hase values v t-1, v t-2,, v t-k of the variable on the data set are rovided as antecedents and v t is also rovided as the consequent. The FB is a black box forecast system that eventually acquires the governing qualitative rules of the time series. In the alication hase, when values v t -1, v t -2,, v t -k are given as inuts it is suosed that v t is redicted by the FB. In order to use the Boolean fuzzy net it is necessary to comly the continuous values of the antecedents with the binary inuts exected. Since the binary nature of antecedent inuts result from the fact that they come from an area of neurons where the resective variable value is the activation ratio, one has to define the interval limits for each variable and to find the ratio between the value of each continuous observation and the interval. Then a random assignation of 1 s and 0 s is made to every antecedent neuron in order to obtain the desired antecedent ratio. A Monte Carlo tye method can be used for this urose. That is, for each antecedent A k one gets: Oi j = Rand(Vi j /(Max-Min)) ; where Oi j is the outut from neuron i in exeriment j, Vi j is the actual continuous value of the variable k in the same exeriment and Rand(x) is a function that returns 1 with robability x or 0 otherwise. The theoretical conditions for convergence [12] state that the number of inuts er consequent neuron and er antecedent should be of the order of the square of consequent linguistic terms that must be differentiated. These conditions are achieved when the number of exeriments (observations in this case) goes to infinity. The theoretical results obtained also roved that the results of the Fuzzy Boolean ets are equivalent to those achieved with non arametric estimation by Parzen windows [9], which imlies that every data exeriment is suosed to belong to a distribution with a given robability density. This same conjecture is made for roving the relation between the number of inuts er consequent neuron and the granularity. Thus one assumes that there exists a hidden set of qualitative rules relating a set of consecutive data observations (or simle functions of these observations, such as moving averages) to the next unknown value and this set of rules comlies with the above assumtions. The behaviour of the Boolean Fuzzy et has been studied [13], in terms of the quadratic error obtained when the number of used neurons, of inuts er consequent neuron and er antecedent (ma) and of different rules on the consequent (granularity) are used as arameters. Also the effect of the number of different teaching exeriments has been observed. In that study, exerimental values for antecedent and consequent have been artificially generated from normal distributed variables, with known average and variances. It has been observed that the error diminishes with the number ma until a level of minimum is achieved, deending on the number of neurons. This otimum value for ma, in those exeriments, is of the order of 50 for 100 neurons and of 70 for 1000 neurons, when the value of three is used for consequent granularity. Further increasing of the ma from that oint on has the effect of increasing or decreasing the error on an oscillation form. This minimum error, however, deends on the number of consequent neurons and for a granularity of 3 on the consequent (say: Low, Medium and High ) and for a number of the order of 100 exeriments this minimum is , for 100 neurons and , both for 1000 and 5000 neurons, being the error a value between these two for 500 neurons. These values may be used as a clue for the work of this aer but, due to the different nature of the roblem, a more secific investigation has been erformed. In order to evaluate the net erformance four different indicators have been calculated as follows. In this context d is considered the redicted value of index and y the actual data set value to be redicted. Also, the values of the data set have been standardized for the interval [0,1]. 2 Mean squared error, MSE= 1/ 2 (d y) 503

5 ormalized Mean Squared error, 2 ( y d) MSE= 2 ( y y ) 20 rules! A similar result is obtained here with 3 antecedents and 9 rules. Table 1 describes the obtained values for two of the tests. Mean Percentage Error, MRPE= 1/ d y.100% Percentage of Correct (u/down) Direction 1 Prediction, PCD= f *100% With f =1 if sign(d - y -1 )=sign(y - y -1 ); f =0 otherwise. FB Outut Real Data Different teaching and testing sets have been used, namely art of the extra data set (about 9000 oints) as teaching set and the main data set (about 1000 oints) as the test set or 800 oints of the main data set as teaching set and the other 200 oints for test set, with similar results. A set of different values for the arameters have been tried, namely: R(number of consequent rules) has been changed from 3 to 9, A(the number of antecedents) on the range from 3 to 5, (number of consequent neurons) has been tried with values 500 and The number of inuts er antecedent and er consequent neuron, ma, was used according the theoretical conditions stated above, that is, at least of the order of the square of the number of rules. The actual values were from 24 for 3 rules to 161 for 9 rules. Figure 1 dislays 200 oints both of real and redicted data, of the 1000 used for test. These values comare otimistically with those already obtained by other authors. In articular one of the best reorted results, using Hidden Markov Exerts [15], accounts for MSE values (the other indicators are not available) from 0.01 (using 25 Exerts) to (using 10 Exerts). It should, however, be stressed that that work has used 10 antecedents (comaring with 3 and 9 of this work) and that the so-called Exerts have a similar role to Qualitative Rules. Thus, the value of (10 antecedents and 20 Exerts) for MSE should be comared to a FB result using 10 antecedents and Figure 1: Real and Predicted data Table 1: Prediction MSE, MSE, MRPE and PCD MSE MSE MRPE PCD R=3;A=5; =500;m A= % 88% R=9;A=3; =500;m A= % 90.5% This suggests that the interolation caability of these nets are very well adated to this tye of time series forecast and that the number of ast values/antecedents necessary to a good rediction, as long as enough rules are used, is not so high as indicated in other works,. 4 Conclusions It has been shown that FB s are a good otion for one ste ahead rediction for certain time series roblems, articularly in non trivial and stationary roblems, such as the chaotic ulsations of low dimensionality of the used laser data set. The obtained error rates, both for mean square error and mean relative ercentage error, are quite good when comared with other reorted results. Also the high ercentage of correct direction rediction (in terms 504

6 of the derivative sign of the series variable) encourages the study the FB behavior with other time series tye where this sign revision is of most imortance, such as stock exchange or currency roblems. Finally, although the main objective of this aer was the forecast of the value of a variable given its assed values, an interesting question is the interretation of the redictions. That is, the issue of exlaining the rediction results by qualitative rules can be very useful for understanding the underlying laws governing the time series behavior. Since the FB model learns rules this question is automatically answered by listing those rules. Moreover for the test hase it is ossible to list also the activation ratio of each rule, instead of exhibiting only the final defuzzified value. References [1] G.Box,G.Jenkins and G.C.Reinsel, Time Series Analysis, Forecasting and Control, Oakland, California: Holden Day, [2] P.Chang and J. Hu, Otimal nonlinear adative rediction and modelling of MPEG video in ATM networks using ielined recurrent neural networks, IEEE J.Select.Areas Commun., vol 15, Aug [3] S.Chong,S.Li and J.Ghosh, Predictive dynamic bandwidth allocation for efficient transort of real time VBR over ATM, IEEE J.Select. Areas Commun.,vol13, 12-33, Jan [4] A.Geva, on-stationary time series rediction using fuzzy clustering, Proceedings of the 18 th International Conference of the orth American Fuzzy Information Processing Society, , [5] S.Haykin, Adative Filter Theory. Uer Saddle River,J: Prentice Hall, [6] Hebb, D. The Organization of Behaviour: A eurosychological Theory. John Wiley &Sons. (1949). [7] U. Huebner,.B. Abraham and C.Weiss Dimensions and Entroies of chaotic intensity ulsations in a single-mode far infraredh3 laser, Phys. Rev. A40, 6354(1989) [8] Lin, Chin-Ten and Lee,C.S. A euro-fuzzy Synergism to Intelligent Systems.ew Jersey : Prentice Hall. (1996). [9] Parzen, E.(1962). "On Estimation of a robability density function and mode" Ann. Math. Stat., 33, [10] Tomé,J.A."eural Activation ratio based Fuzzy Reasoning."Proc. IEEE World Congress on Comutational Inteligence, Anchorage,May 1998, [11] Tomé,J.A."Counting Boolean etworks are Universal Aroximators" Proc. of 1998 Conference of AFIPS, Florida, August 1998, [12] Tomé,J.A. and Carvalho, J.P. "Rule Caacity in Fuzzy Boolean etworks. Proc. of AFIPS- FLIT 2002 International Conference, ew Orleans, IEEE2002, [13] Tomé, J.A. Error convergence on Fuzzy Boolean ets. Internal Reort, IESC [14] A. Weigeng and.a. Gershenfeld, Time Series Prediction: Forecasting the Future and Understandibg the Past. Reading,MA: Addison-Wesley, 1999 [15] X.Wang, P. Whigham,Da Deng,M. Purvis, Time-Line Hidden markov Exerts for Time Series Prediction, eural Information Processing Letters and Reviews, Vol. 3,2, May