Evolution of Neural Networks in a Multi-Agent World

ZESZYTY NAUKOWE UNIWERSYTETU JAGIELLOŃSKIEGO MMCCXLII PRACE INFORMATYCZNE Z. 10 2000 Evolution of Neural Networks in a Multi-Agent World Marek Kisiel-Dorohinicki Magdalena Klapper-Rybicka Institute of Computer Science University of Mining and Metallurgy Kraków, Poland e-mail: {doroh,mklapper}@agh.edu.pl Abstract. The concept of decentralised evolutionary computation realised as evolutionary multi- system (EMAS) is described in the paper. Also based evolutionary approach to neural network architecture optimisation is presented. Then the problem of time-series prediction and a general idea of evolutionary neural multi- predicting system is introduced. Selected design issues together with preliminary simulation results conclude the work. Keywords: decentralised evolutionary computation, evolutionary multi- systems, evolutionary neural networks, time-series prediction. 1. Introduction Artificial neural networks (ANN) is a computational technique arising from an interest in modeling basic mechanisms of biological neural structures. Probably the main advantage of using neural networks is their ability to learn from examples and generalise the aquired knowledge to new cases in such a way that any explicit problem-dependent knowledge is not needed (eg. [11]). That is why this technique is dedicated to approximation problems, like classification (pattern recognition), prediction, and control problems. Even though the use of neural networks replaces the necessity to specify the deterministic way of solving a problem, one still needs to define the network architecture suitable for that (kind of) problem. It usually involves a

38 lot of experiments with different network structures, which is time-consuming and does not have to result in the best solution. To avoid this situation, some mechanisms to automatically find the (nearly) best neural network architecture were proposed (eg. [18]). Evolutionary algorithms (EA) or evolutionary computation (EC) is another heuristic problem-solving approach, which is based on models of organic evolution [3]. It has been successfully used in various computationally difficult search and optimisation problems. Recently evolutionary algorithms have also been used to support search for optimal neural network architecture [5, 15]. Yet until, now the question of how to build a proper evolutionary algorithm for any given problem remains open. We have not only to decide on the representation of individuals, selection/reproduction mechanisms and genetic operators, but also have to set various parameters. One of the reasons for that is the fact, that the theory of evolutionary computation is still weak [4]. There are also many important differences between the observed mechanisms of evolution and its model employed by EA. What surely limits the possibilities of EA s to model natural evolution is the centralisation of the evolution process. What we propose is to model evolution in a decentralised environment, such as multi- system. Thus defined evolutionary multi- system (EMAS) may support searching for the correct architecture of a neural network. The resulting evolutionaryneural system may be applied to the problem of time-series prediction. 2. Evolutionary (Multi-)Agent Systems Recently, technology has gained much interest both in industry and academic research, owing to the growing need for decentralised intelligent software systems. Among lots of issues connected with multi- systems (MAS) one can also find evolutionary computation supporting systems. In most such cases an evolutionary algorithm is used by an (Fig. 1a), to aid the realisation of some of its tasks, e.g. connected with learning or reasoning [8, 12]. However, it seems that interesting results may be achieved when applying some model of evolution in MAS at a population level (Fig. 1b), i.e. among s. In this case genetic operators together with the selection/reproduction mechanisms search for a (near) optimal configuration of s in the population. Such systems form a new class of auto-adaptive multi- systems (evolutionary multi- systems), where organic-like evolution helps to accomplish population-level goals [6, 7].

39 Evolutionary multi- systems may also be considered a new computational method based on model of decentralised evolution. Of course, these two approaches can be used at the same time, which may be considered an experiment with meta-evolution (cf. [3]). MAS evolutionary algorithm a) evolutionary algorithm evolutionary algorithm EMAS process of evolution b) Fig. 1. Evolutionary algorithms supporting particular s in MAS (a) and evolutionary multi- system (b) 2.1. Classical evolutionary computation Various techniques of evolutionary computation were developed [4], including genetic algorithms (GA), evolution strategies (ES) and evolutionary programming (EP). Also many variants of these canonical forms of EC, as well as hybrid methods joining not only different evolutionary algorithms, but also distinct heuristics or problem-dependent mechanisms are in use. While different forms of evolutionary computation model evolution of organisms at a specific level [2] (GA gene level, ES individual level, EP species level), they all employ a similar scheme. They operate on a given number of data structures (population) and repeat the same cycle of processing (generation) consisting of a selection of parents and generation of offspring using mutation and recombination operators. The differences between specific techniques concern mainly:

40 what type of structures are evolved (representation of individuals), how the structures are manipulated to produce offspring (mutation and/or recombination), how the structures are evaluated (selection/reproduction mechanism). As it was already suggested, the selection of a proper evolutionary algorithm for the given problem is not a trivial problem. One of the reasons for that is the weakness of the theory of evolutionary computation: We know that they work, but we do not know why [4]. One may find several mathematical models of how specific evolutionary algorithms work. Moreover, in particular cases it is even possible to prove the convergence of the algorithm to the desired solution. But there is no coherent theory explaining how the above-described scheme leads to satisfactory results in a general case. The only explanation for us still remains the fact that mechanisms exploited by EC are based on, what we think is, natural evolution. But the model of evolution followed by most EAs (with noticeable exceptions) is much simplified and lacks many important features observed in organic evolution [4]: dynamically changing environmental conditions, many criteria in consideration, neither global knowledge nor generational synchronisation assumed, co-evolution of species, evolving genotype-fenotype mapping, etc. What is more, since the mechanisms of natural evolution are not clear to us yet, surely our models are hardly perfect. The idea of decentralised evolutionary computation, realised as evolutionary multi- system introduced above, may help to avoid some of the above-mentioned shortcomings of the model of evolution employed in classical EC. 2.2. The model of -based evolutionary computation A general idea of how evolutionary multi- system should work has been presented in this section. The population of s remains in a common environment. The environment may have a spatial structure and contain some resources. The space and the resources may represent some aspects of the problem to be solved or aid the process of solving. The s may sense the environment (both

41 space structure and resources) and act over it. The states of the s or the results of s actions should constitute a basis for the formulation of a solution (in the simplest cases the state of the or its position in the space may represent a solution or the may return a solution to the environment). The set of parameters describing the basic behaviour of the is encoded in its genotype, and is inherited from its parent(s). Besides the may possess some knowledge acquired during its life (model of the environment), which is not inherited. Both the inherited and aquired information determine the behaviour of the in the system (phenotype). Special resource(s) energetic profile, life energy play the role of a fitness function. The energy is gained and lost when the executes actions in the environment. An increase in energy is a reward for a good behaviour of the, decrease penalty for a bad behaviour. The best individuals are selected for reproduction based on the value of its life energy. Before generation of an offspring, the genetic material is (ex)changed by means of genetic operators (mutation and recombination). 2.3. Advantages and disadvantages of EMAS-based computation The presented model of decentralised evolutionary computation enables the following: continuous evolution process means that evolution (when genotypes are developed) takes place in the same time space as the life of particular s (when fenotypes are developed), life energy is a more flexible way to evaluate s fitness; besides more than one criterion can be reflected with it, explicitly defined living space allows for application of advanced search mechanisms such as territorial niches, evolution process may be supported by new operations, e.g. aggregation enables a group of s to join together and act as a single and in this way modify their own environment, escape is a strong qualitative change of the s environment, which may be done by means of migration, evolution centers accelerate specialisation and support aggregation.

42 Of course, since EMAS may be regarded as a step towards a more complete model of evolution (surely more complicated one), it has two serious disadvantages: it is computationally complex (much more than the generic EA), there is no theory (like for the generic evolutionary algorithm) entirely explaining the basics of its work. Yet, as the work on theory of EC and MAS goes on, in the future it would be possible to present also theoretical foundations of the new ideas. 3. Evolutionary Neural Networks Neural networks are able to model very complex functions and non-linear structures with large numbers of variables, which are troublesome for traditional computational methods. Yet, the application of a neural network requires defining its topology (number of layers, number and type of neurons in each layer, connection structure) and initial state (weights), as well as choosing suitable training method and its parameters. The problem is meaningful, especially that too low complexity of a network may cause that it would not be able to project the function underlying the problem. On the other hand, too high complexity of a network may result in overfitting to the data. Also (time-)complexity of the learning algorithm strongly depends on the complexity of the network, especially on the number of connections. 3.1. Optimisation of NN architecture How to select the most appropriate architecture of a neural network for a particular application? Unfortunately, this question cannot be answered arbitrarily. Of course, some general rules of designing a network may be used. However, tuning the network precisely to a particular problem requires testing of many cases. There are several methods of optimisation of a neural network topology. The constructive methods start from a small network with a limited number of connections. Then successively add new units, training the network after each addition. In opposite, pruning algorithms start from a large full-connected network and decrease the number of units. In fact, manual optimisation of a neural network takes a lot of effort. And that is why, automatic realisations of

43 the above-mentioned algorithms have been proposed [5, 15]. The evolutionary neural networks (ENN), in which the search for a desirable neural architecture is made by the evolutionary algorithm is an alternative [13, 18]. 3.2. Genotypic representation of NN The most important question related to the design of an evolutionary algorithm for NN optimisation is the decision on the genotypic representation of a neural network. When designing an NN genotypic representation, the correctness of the resulting network architecture must be assured (e.g. there must exist some connections between two succeeding layers). Another problem is that one network topology may be represented by different chromosomes, which makes evolution process less efficient (the so-called permutation problem). If the evolution of architectures is separated from weights training, the fitness evaluation of the NN is always noisy. Two kinds of representation may be distinguished in view of the amount of: information about network topology coded into a chromosome, direct encoding (strong specification scheme) a chromosome contains full information about network nodes and connections [16], indirect encoding (weak specification scheme) developmental rules for generating a network architecture are encoded in a chromosome [10, 14]. Direct encoding requires longer chromosomes, which may slow down the process of evolution. Indirect encoding suffers from the effect of noisy fitness evaluation [17]. 4. EMAS for time-series prediction Prediction (or forecasting) is the generation of information about the possible future development of some process from data about its past and present behaviour [9]. The prediction may concern static data (as the outcome of a decease) as well as time-series data (as stock market or gas consumption). The latter one will be of our interest. Time-series prediction performed on the basis of only one independent variable (univariate) consists in searching for some trends in the sequence

44 of values of that variable. Multivariate prediction should also respect the relationships between particular series. 4.1. Predicting MAS In the most general case a time-series predicting system may be considered as a black box with some input sequences and predictions of successive values of (some of) these sequences as an output. Some intelligent mechanism inside that box should be able to discover hidden regularities and relationships in and between the input sequences. On the assumption that the characteristics of the signal(s) may change in time, this mechanism should also be able to dynamically adapt to these changes ignoring different kinds of distortion and noise. 1 1 1 0 1 0 1 0 0 1 a 5 1 a 4 0 1 a 3 0 0 a 2 1 a 1 0 Fig. 2. Multi- predicting system When the signal to be predicted is much complicated (e.g. when different trends change over time or temporal variations in relationships between particular sequences are present0 an idea of a (multi-) predicting system may be introduced [7]. The predicting MAS (see Fig. 2) may be viewed as the box (as above) with a group of intelligent s inside. The subsequent elements of the input sequence(s) are supplied to their environment, where they become available for all s. Each itself may analyse the incoming data and give predictions of (a subset of) the next-to-come elements of input. Specialisation in function or time of particular s allow for obtaining better results by cooperation or competition in the common environment. On the basis of predictions of all s, a prediction of the whole system may be generated.

45 4.2. Evolution in predicting MAS The configuration of s in such-defined system (kind of specialisation or method of cooperation) is often difficult to specify. What is more, when dynamic changes of the characteristics of the signal are possible, the configuration of the s should reflect these changes, automatically adapting to the new characteristics. The mechanisms similar to the ones found in evolutionary computation may help to transform the whole population of s (by means of mutation and/or recombination) so that it best fits the current profile of the input signal (with proper selection/reproduction). In fact, each simply possesses some vector of parameters, which describes its behaviour in the system. This vector plays the role of an s genotype, and as such may be modified by genetic operators when inherited by its offspring. The evaluation of the s is based on the quality of prediction by means of the gained/lost life energy. Then the selection is made as good s are more likely to reproduce new s, while bad s die. This evolutionary development of predicting MAS meets the general idea of evolutionary multi- system (EMAS) introduced previously. 4.3. Prediction with neural networks Fig. 3. Predicting neural network

46 For each an artificial neural network may be used as a basic mechanism to trace signal regularities in a system for time-series prediction. Many types of neural networks may be applied for prediction problems: multilayer perceptron, radial basic function network, generalised regression neural network. The choice of an architecture is greatly determined by the particular problem. The scheme of a neural network used for the time-series prediction is presented in Fig. 3. Usually the next value of a series is predicted based on a fixed number of previous ones (an input register). The number of input neurons corresponds to the number of values the prediction is based on. A network may be supervisory trained, using the comparison between the predicted and real values. Then the input register is shifted left and the next value of input is appended. 5. Design issues The behaviour of an in the environment is determined by interaction of its energetic profile and prediction profile [7]. The main part of the prediction profile consists of a three-layer fullconnected MLP network (multilayer perceptron). The fixed number of previous values of the series the network needs to predict the next one is held in a sequence-parallel register (see section 4.3). The network is trained with the back-propagation algorithm, comparing the last-predicted value with the real value of a series taken from the environment. The adjustment of the network is described by the quality of prediction, which is estimated basing on the accuracy of the fixed number of last predictions. The only resource possessed by the in the energetic profile is energy. The gains energy for accurate prediction. At the same time each action results in a decrease in the s energy. On the other hand, to perform an action the has to reach an appropriate energetic level. Some probability parameters prevent the from performing the same action several times one by one. The system is based on the event-driven simulation, i.e. the acts in response to the events registered by itself or other s, or incoming data. Subsequent elements of the input series supplied to the environment form the main stages of activity of the system. Within each stage the following scheme is realised: 1. succeeding value of the input series is delivered to the environment, 2. each acts according to the pattern:

47 (a) gets actual set of values from the environment, (b) estimates the quality of its own prediction, (c) trains the network with back propagation algorithm, (d) decides which action to perform mutation, migration, aggregation, reproduction or death, (e) predicts the next value, (f) submits the predicted value together with the quality of prediction to the environment, 3. the environment generates output prediction, based on the values submitted by all s; it chooses the best (or the most frequent) value of s answers, or a value computed as their weighted average. 6. Results of simulation experiments 1 0,8 0,6 0,4 0,2 0 Fig. 4. Avarage prediction probability in a typical simulation Preliminary results of simulation experiments are presented in Fig. 4 and Fig.5. The average prediction probability of s is shown in the first plot. One may notice that after several steps of evolution the average probability of prediction of individual s is about 80% and from that point the answers of the whole system are about 100% correct. The second plot shows the percent all of deaths zero at the begining of the simulation, then a significant rise due to elimination of the worst individuals, and stabilisation at a level of 10% when predictions are satisfactory (meaning a low exchange rate of the population).

48 7. Conclusions The idea of EMAS an extension of classical evolutionary algorithms. Evolving neural networks in a multi- environment an example of how an system allows for simultanous development of s phenotypes and evolution of the whole population of s. Predicting neural networks in evolutionary multi- environment a new approach to the problem of time-series prediction. Preliminary experimental studies new ideas at work. 0,4 0,35 0,3 0,25 0,2 0,15 0,1 0,05 0 Fig. 5. Percent of deaths in a typical simulation 8. References [1] Proceedings of Second International Conference on Multi-Agent Systems, AAAI Press, 1996. [2] Angeline P.J., Genetic Programming s Continued Evolution, in: Angeline P.J., and Kinnear K.E., Jr. (eds.), Advances in Genetic Programming, MIT Press, 1996, pp. 1 20.

49 [3] Bäck T., Evolutionary Algorithms in Theory and Practice, Oxford University Press, 1996. [4] Bäck T., Hammel U., Schwefel H.P., Evolutionary Computation: Comments on the History and Current State, IEEE Transactions on Evolutionary Computation, (1), 1997, pp. 3 17. [5] Burgess N., A Constructive Algorithm that Converges for Realvalued Input Patterns, International Journal of Neural Systems, 1994, pp. 59 66. [6] Cetnarowicz K., Evolution in Multi-Agent World = Genetic Algorithms + Aggregation + Escape, in: Position Papers of MAAMAW 96, Technical Report 96-1, Vrije Universiteit Brussel, Artificial Intelligence Laboratory. [7] Cetnarowicz K., Kisiel-Dorohinicki M., Nawarecki E., The Application of Evolution Process in Multi-Agent World (MAW) to the Prediction System, in: Proc. of ICMAS 96, [1]. [8] Denzinger J., Fuchs M., Experiments in Learning Prototypical Situations for Variants of the Pursuit Game, in: Proc. of ICMAS 96 [1]. [9] Kasabov N.K., Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, The MIT Press, 1996. [10] Kitano H., Designing Neural Network Using Genetic Algorithm with Graph Generation System, Complex Systems, 1990, pp. 461 476. [11] Krose B., van der Smagt P., An Introduction to Neural Networks, University of Amsterdam, 1996. [12] Liu J., Qin H., Adaptation and Learning in Animated Creatures, in: Proc. of Autonomous Agents 97, ACM Press, 1997. [13] Mitchell M., An Introduction to Genetic Algorithms, MIT Press, 1998. [14] Nolfi S., Parisi D., The Handbook of Brain Theory and Neural Networks, MIT Press, 1994, Chapter: Genotypes for neural networks. [15] Reed R., Pruning Algorithms a Survey, IEEE Trans. on Neural Networks, 1993, pp. 740 747. [16] Whitley L.D., and Schaffer J.D. (eds.), IEEE Computer Society Press, 1992. [17] Yao X., Evolutionary Artificial Neural Networks, International Journal of Neural Systems, 1993, pp. 203 222.

50 [18] Yao X., Liu Y., in: Proceedings of the Fifth Annual Conference on Evolutionary Programming, MIT Press, 1996. Received March 18, 1999 Department of Computer Science Jagiellonian University