Markovian Agents: A New Quantitative Analytical Framework for Large-Scale Distributed Interacting Systems

Transcription

1 Markovian Agents: A New Quantitative Analytical Framework for Large-Scale Distributed Interacting Systems Andrea Bobbio, Dipartimento di Informatica, Università del Piemonte Orientale, Alessandria, Italy and Dario Bruneo, Dipartimento di Matematica, Università di Messina, Messina, Italy and Davide Cerotti, Dipartimento di Informatica, Università di Torino, Torino, Italy and Marco Gribaudo, Dipartimento di Elettronica e Informazione, Politecnico di Milano, Milano, Italy bobbio@mfn.unipmn.it, dbruneo@unime.it, davide.cerotti@di.unito.it, gribaudo@elet.polimi.it ABSTRACT A Markovian Agent Model (MAM) is a stochastic model that provides a flexible, powerful and scalable way for analyzing complex systems of distributed interacting objects. The constituting bricks of a MAM are the Markovian Agents (MA) represented by a finite state continuous time Markov chain (CTMC) whose infinitesimal generator is composed by a fixed component (the local behaviour) and an induced component that depends on the interaction with the other MAs. An additional innovative aspect is that the single MA keeps track of its position so that the overall MAM model is spatial dependent. MAMs are expressed with analytical formulas suited for numerical solution. Extensive applications in different domains have shown the effectiveness of the approach. In the present paper, we propose an example that illustrates how the MAM technique can cope with extremely large state spaces. Keywords Markovian Agents, Distributed Interacting Systems, Performance Evaluation, Wireless Sensor Networks, Swarm Intelligence. 1. INTRODUCTION Markovian Agents (MAs) are stochastic entities introduced with the aim of providing a flexible, powerful and scalable technique for modeling complex systems of distributed interacting objects, for which feasible analytical and numerical solution algorithms can be implemented. Each object has its own local behaviour that can be modified by the mutual interdependencies with the other objects. MAs are scattered over a geographical area and retain their spatial position so that the local behaviour may be related to the geographical position and the mutual interdependencies may depend on the relative distances and the transmittance characteristics of the interposed medium. MAs are modeled by a discrete-state continuous-time finite CTMC whose infinitesimal generator may be influenced by the interaction with other MAs. The interaction among agents is represented by a message passing model combined with a perception function. When resident in a state or during a transition, an MA is allowed to send messages that are perceived by the other MAs, according to a spatial dependent perception function, modifying their behaviour. Messages may model real physical messages (as in wireless sensor networks) or simply the mutual influences of a MA over the other ones. Further, MAs may belong to a single class or to different classes with different local behaviours and interaction capabilities, and messages may belong to different types where each type induces a different effect on the interaction mechanism. The perception function describes how a message of a given type emitted by a MA of a given class in a given position in the space is perceived by MA of a given class in a different position. By consequence of the interaction mechanism, the entries of the infinitesimal generator of each MA are determined by the superposition of local terms and interaction induced terms. In the previous literature, the modelling and analysis of large scale stochastic systems composed by interacting objects has been mainly faced by resorting to the superposition of interacting Markov chains, to asymptotic models or to fluid models. In the first case, the available techniques require the generation of the global state space, defined as the Cartesian product of the state spaces of the individual interacting objects. The explosion of the state space can be mitigated by exploiting symmetry properties, often included in the system definition, and producing the global transition rate matrix by means of tensor algebra operators applied to the local matrices [11]. Representative attempts in this direction define the interacting objects directly as Markov chains [8], [1], [10], or as finite state automata [22], [23] or as Petri nets [9], [21]. However, the compositional approaches, based on finite state objects, do

2 not account for interactions related to the relative position of the local objects. Recently, asymptotic models based on mean field theory have been proposed in the domain of performance evaluation [4], [2], [3], but they are not able to include spatial dependencies. Finally, fluid models [20], [17], [18] are able to capture the global behaviour of the system but loosing the capability of detailing the local behaviour. In our approach, the local objects are finite state MAs and their interaction is represented by a fluid model. In this way, we do not need to explore the product state space, but we account for the effect of the global dependencies on the individual infinitesimal generators and we solve stochastic equations defined on individual sub-models. Interactive Markovian Agents have been introduced in [15], [16] for single class MAs and extended to Multi-class Multi-message Markovian Agent Model in successive works [14], [5], [13], [7]. In [12], mobility properties for the MAs have been introduced. Several application examples have been described and analyzed across the above papers and validation through simulation [7] or measuring real physical systems [6] has been provided. The aim of this paper is to present the new modeling framework based on Markovian Agent Models (limited for simplicity to single-class MAs) and to show how the analytical model can be defined and implemented. A very large scale example has been selected to conclude the paper with the aim of illustrating the capabilities of the approach in dealing with extremely large state spaces. 2. MOTIVATION Often complex systems are composed by many interacting objects that have their own local behaviour that can be modified by the mutual influences exercised with the other objects. As a representative example of this class of systems, Figure 1 represents a Wireless Sensor Network where the nodes of the network are sensors that behave according to their own specification but interact with the other sensors through the exchange of messages (represented by the links in the figure). In real networks, some nodes may fail or may be switched off, thus modifying not only the topology of the network but also the mechanism of the interaction and the mutual dependencies. Furthermore, the interaction may be dependent on the position of the objects and on their mutual distances. Fig. 1. Example of a complex system of interacting objects Distributed systems with similar features, problems and difficulties may be found in many different technological areas and in many practical situations, like grids of computers, smart grids of power stations, and, in general, any system that can be represented in the form of a network where the nodes have an autonomous behaviour but, at the same time, are interdependent on the other nodes. The Markovian Agent Model was specifically studied to cope with the following needs: i Provide analytical models that can be solved by numerical techniques, thus avoiding the need of long and expensive simulation runs; ii Provide a flexible and scalable modeling framework for distributed systems of interacting objects; iii Provide a framework in which local properties can be coupled with global properties; iv Local and global properties and interactions may depend on the position of the objects in the space (space-sensitive models); v The solution algorithm self-adapts to variations in the system topology and in the interaction mechanisms. The constituent elements of a MAM are the MAs. MAs are represented by a finite state continuous time Markov chain (CTMC) whose infinitesimal generator is composed by two parts: a fixed component (the local behaviour) and an interaction induced component that depends on the interdependencies with the other MAs. MAs are scattered over a geographical area. A position dependent density function takes into account the density of MAs in each location and in each state of the CTMC characterizing the MA. The local properties of a MA may depend on its position and the mutual interdependencies may depend on the relative distances and the transmittance characteristics of the interposed media. The interaction among MAs is represented by a message passing model combined with a perception function. Messages may represent either real physical messages (as in wireless sensor networks) or, in general, the mutual influences exercised by an MA over the other MAs. The perception function rules the propagation of messages by taking into account the MA position in the space, the routing policy for the messages and the transmittance of the medium. 3. MARKOVIAN AGENT SPECIFICATION The structure of a single MA is represented in Figure 2. States i, j,..., k are the states of the CTMC representing the MA. The transitions among the states are of two possible types that are drawn differently: - Solid lines (like the transition from i to j or the self-loop from i) represent the fixed component of the infinitesimal generator and represent the local or autonomous behaviour of the object that is independent on the interaction with the other MAs (like, for instance, the time to failure distribution, or the reaction to an external stimulus). Note that we include in the representation also self-loop transitions that require a particular notation since are not visible in the infinitesimal generator of the CTMC [24].

3 - Dashed lines (like the transition from i to k or the transition into i) represent the transitions induced by the interaction with the other MAs. The way in which the rates of the induced transitions are computed is explained in the following section. Fig. 2. Schematic structure of a Markovian Agent During a local transition (or a self-loop) an MA can emit a message with an assigned probability, as represented by the dotted arrows in Figure 2 emerging from the solid transitions with a label denoting the corresponding message generation probability. Messages generated by an MA may be perceived by other MAs, according to a suitable perception function, and the interaction mechanism between emitted messages and perceived messages generates the induced transitions (dashed lines). A MAM is a collection of interacting MAs defined over a geographical space V. Given a position v inside V, we define ρ(v) the density of MAs in v. According to the definition of the density ρ(v), we can classify a MAM with the following taxonomy: - A MAM is static if ρ(v) does not depend on time, and dynamic if it does depend on time; - A MAM is discrete if the geographical area on which the MAs are deployed is discretized and ρ(v) is a discrete function of the space or it is continuous if ρ(v) is a continuous function of the space. p 0 (v) is the initial probability vector. Note that in the single-class MAM even if the structure of the CTMC associated to each MA is the same for all the objects, the values of the parameters may depend on position v and, therefore, may vary from MA to MA. From the above definitions, we can compute the total rate β j (v) at which messages are produced by an MA in state j, in position v: β j (v) = λ j (v) p jj (v) + k j q jk (v) p jk (v) (2) where the first term in the r.h.s is the contribution of the messages emitted during a self-loop from j and the second term is the contribution of messages emitted during a transition from j to any k ( j). The interdependencies among MAs are ruled by a perception function u(v, i, v, j ) that defines how messages generated from an MA in state j at position v, are received by an MA in state i at position v. The perception function is a structural part of the model and it contributes to quantify the interdependencies among MAs. The perception function defines how messages issued by an MA in a given spatial location and in a given state propagate in the space and how they are perceived by other MAs in different locations. The transition rates of the induced transitions are primarily determined by the structure of the perception function. A pictorial and intuitive representation of how the perception function u(v, i, v, j ) acts, is given in Figure 3. The MA in the bottom right portion of the figure in position v broadcasts a message of type m that propagates in the geographical area until reaches the MA in the top left portion of the Figure in position v. Upon acceptance of the message according to the acceptance matrix A(v), a new induced transition (represented by a dashed line) is generated in the model. Mathematical formulation For the sake of simplicity we provide here the formal definition of a MAM with a single class of MAs and a single type of messages [16]. The extension to multi-class multimessage MAMs has been described in [7]. An MA is formally specified by the following tuple, where v is the position of the MA in the space V: MA(v) = {Q(v), Λ(v), P(v), A(v), p 0 (v)} (1) where: Q(v) is the local component of the infinitesimal generator; Λ(v) is the array of the self-jump transition rates; P(v) is the probability of message generation; A(v) is the probability of message acceptance; Fig. 3. Message passing mechanism ruled by a perception function With the above definitions we are now in the position to compute the components of the infinitesimal generator of an MA that depend on the interaction with the other MAs and that constitute the original and innovative part of the approach.

4 We define γ ii (t, v) the total rate at which messages coming from the whole volume V are perceived by an MA in state i in location v. γ ii (t, v) = n u(v, i, v, j) β j (v ) ρ j (t, v )dv (3) V j=1 γ ii (t, v) in Equation (3) is computed by taking into account the total rate of messages emitted by all the MAs in state j and in a given position v (the term β j (v )) times the density of MAs in v (the term ρ j (t, v )) times the perception function (the term u(v, i, v, j)) summed over all the possible states j of each MA and integrated over the whole space V. From an MA in position v and in state i an induced transition to state k (drawn in dashed line) is generated with rate γ ii (t, v) a ik (v) where a ik (v) is the appropriate entry of the acceptance matrix A(v). We then group the rates γ ii (t, v) in the diagonal matrix Γ(t, v) (see Equation (4)); Γ(t, v) = [ γ ii (t, v) ] (4) combining the local component Q(v) with the induced component Γ(t, v), and taking into account the message acceptance matrix A(v), we finally obtain the complete form of the infinitesimal generator K(t, v) of the MA as defined in Equation (5). K(t, v) = Q(v) + Γ(t, v) [A(v) I], (5) Once the complete generator for any MA is computed from (5) (and this is the most intensive computational part of the solution algorithm [7]) we can solve for the density ρ(t, v) of MAs in V the standard Chapman-Kolmogorov (second Equation in 6) under initial condition (first Equation in 6). ρ(0, v) = ρ(v)π 0 (v) (6) dρ(t, v) = ρ(t, v)k(t, v). dt Note that each equation in (6) has the dimension of the CTMC of a single MA. In this way we have decomposed a problem defined over the product state space of all the MAs into several subproblems, one for each MA, having decoupled the interaction by means of Equation (5). Solution of each Equation in (6) is obtained by resorting to standard numerical techniques for differential equations, and provides the basic time-dependent measures to evaluate more complex performance indices associated to the system 4. APPLICATIONS To show the flexibility of the MAM framework, several applications in different domains have been recently discussed also in cooperation with different research groups that were attracted by the capabilities offered by the model. The original and motivating application was related to Wireless Sensor Networks [15], [16]; then we showed applications related to the propagation of seismic waves in inhomogeneous media [14] and, in collaboration with colleagues of the Universidad Politecnica de Valencia (Spain), propagation of fire in inhomogeneous terrains and wind fields [13]. Finally, in collaboration with the University of Messina (Italy) we applied MAMs to swarm intelligent protocols [5], [6], [7]. The illustrative final example, presented in the following section, refers to a different field and regards an application in which a MAM provides a solution to a very-large-scale multibody optimization problem, whose solution may be very time consuming or even unfeasible also by a simulative approach. Large-scale multi-body optimization The problem can be formulated in the following terms. The geographical space is in the form of a square grid of n n = n 2 cells, and we put one MA per cell. In the geographical space a number of randomly assigned cells are special aggregation points called sinks. Each cell of the grid must find the shortest (optimal) path to one of the sinks. The sinks may vary in time, in position and in quantity (some sinks may disappear, or new sinks may appear). The defined optimization problem has been solved by resorting to swarm intelligent concepts [19], [7], based on the exchange of pheromone messages. In analogy with the biological process of ant colonies, each node sends a message containing its pheromone level and updates its value based on the level of its neighboring nodes, creating a pheromone gradient toward the sinks. The search for the shortest path is driven by the pheromone gradient, and in particular the shortest path is obtained by following the steepest pheromone gradient toward a sink. Any change on the network condition is reflected by an update of the pheromone intensity distribution. The pheromone intensity is discretized in P levels from 0 to P 1, and the gradient construction is triggered by the sinks that emit messages with the highest pheromone level P 1 with rate λ. The MA representing the sink is reported in Figure 4a). In all the experiments we have assumed P = 20, i.e. we have discretized the pheromone intensity in P = 20 levels. Fig. 4. Structure of swarm intelligent MAs An MA modeling a cell node has P states representing the discretized pheromone levels (Figure 4b). When an MA node in state i receives a message encoding a pheromone level m, generates an induced transition (dashed lines) and jumps to a

5 state j that represents the pheromone level that is the mean between its current level i and the one encoded in the received message m. More formally, the dashed transitions in Figure 4b are labelled with label M(i, j) defined as [7]: M(i, j) = {m [0 P 1] : round((m + i)/2) = j} i, j [0 P 1] : j > i. In other words, an MA in state i jumps to the state j that represents the pheromone level equal to the mean between the current level i and the level m encoded in the perceived message. Then the MA emits its pheromone message with the actual level intensity with rate λ (mechanism represented by the self-loops). Furthermore, the pheromone evaporates with rate µ (local solid transitions) allowing the system to forget old information. Hence, the actual pheromone level in any cell at any time is a stochastic balance between the emission and evaporation rates. (7) Fig. 6. sinks The grid with 10, 000 cells and 50 randomly scattered messages spread into the grid according to the described MAM model, until the pheromone intensity gradient reaches a steady condition, that depends on the balance between the emission rate λ and the evaporation rate µ, as represented in the bottom part of Figure 7. The steady gradient intensity in Figure 7 provides the required optimal solution. Fig. 5. The effect of the transmission range t r The pheromone propagation algorithm and the formation of the pheromone gradient depend on the pheromone emission rate λ, the pheromone evaporation rate µ and the transmission range t r. The transmission range t r takes into account that the energy of the messages emitted by the MAs is limited so that a message emitted in a given position can be perceived only by the MAs located inside a circle of radius t r as depicted in Figure 5. In all the experiments the transmission range covers only the 4 first neighboring cells (Figure 5). The effect of a limited transmission range is reflected in the structure of the perception function whose definition is given by: u(v, i, v, j ) = { 0 dist(v, v ) > t r 1 dist(v, v ) t r, Equation (8) represents the fact that a message emitted in location v can be perceived by an MA in location v only if the distance between the two locations is less that t r. In the following optimization experiment we have assumed a rectangular grid with n = 100 hence with = 10, 000 cells, and we have randomly scattered 50 sinks in the grid. The grid is represented in Figure 6, where the sinks are drawn as black spots. Since each cell is represented by an MA with P = 20 states, the product state space of the overall system has N = 20 10,000 states! At time t = 0 the sinks start emitting their pheromone message with the highest intensity level while all the other nodes have a pheromone level equal to 0. This situation is represented in the upper part of Figure 7. Then, the pheromone (8) Fig. 7. The pheromone gradient at t = 0 (upper part) and in the final steady condition (bottom part) Each cell, following the steepest gradient line in Figure 7, reaches the closest sink with the minimum number of hops. It is noteworthy to observe that the optimal solution is searched among N = 20 10,000 states, and that all the N states are in principle reachable depending on the position of the sinks. Furthermore, the algorithm is self-adapting to any topological modification like a variation in the position or in the number of sinks. The steady gradient configuration is reached in few minutes on a standard laptop, since we have strongly limited the

6 interaction term by means of the perception function defined in Equation 8. Nevertheless, an optimal solution is obtained with any configuration and number of sinks. 5. CONCLUSIONS The analytical MAM model has been presented to show how it can provide a flexible framework to model complex stochastic systems made of many interacting parts that have a local behaviour that can be modified by the global interaction. Moreover, the model is a location sensitive model, in the sense that the geographical position of the objects and their mutual distances are part of the model. This spatially dependent characteristic is reflected both in the structure of the model (for instance MAs of different classes) and in the values of the model parameters. The analytical MAM model is solved by resorting to numerical techniques. The illustrative example shows how a MAM model can search for an optimal solution in a huge state space for which any other techniques (both analytical and simulative) will fail. REFERENCES [1] F. Ball, R.K. Milne, I.D. Tame, and G.F. Yeo. Superposition of interacting aggregated continuous-time Markov chains. Advances in Applied Probability, 29:56 91, [2] M. Benaim and J.Y. Le Boudec. A Class Of Mean Field Interaction Models for Computer and Communication Systems. Performance Evaluation, 65(11-12): , [3] A. Bobbio, M. Gribaudo, and M. Telek. Analysis of large scale interacting systems by mean field method. In QEST 08, IEEE Computer Society, pages , [4] J.Y. Le Boudec, D. McDonald, and J. Mundinger. A generic mean field convergence result for systems of interacting objects. In 4th International Conference on Quantitative Evaluation of Systems - QEST2007, Edinburgh, [5] D. Bruneo, M. Scarpa, A. Bobbio, D. Cerotti, and M. Gribaudo. Analytical modeling of swarm intelligence in wireless sensor networks. In Fourth International Conference on Performance Evaluation Methodologies and Tools (Valuetools 2009), [6] D. Bruneo, M. Scarpa, A. Bobbio, D. Cerotti, and M. Gribaudo. Adaptive swarm intelligence routing algorithms for WSN in a changing environment. In IEEE Sensors 2010 Conference, pages , [7] D. Bruneo, M. Scarpa, A. Bobbio, D. Cerotti, and M. Gribaudo. Markovian agent modeling swarm intelligence algorithms in wireless sensor networks. Performance Evaluation, In Press, Corrected Proof:, [8] P. Buchholz. Hierarchical Markovian models -symmetries and aggregation. Performance Evaluation, 22:93 110, [9] P. Buchholz. Hierarchical structuring of superposed GSPNs. IEEE Transactions Software Engineering, 25: , [10] P. Buchholz and T. Dayar. Comparison of multilevel methods for Kronecker based Markovian representations. Computing, 73: , [11] P. Buchholz and P. Kemper. Kronecker based matrix representations for large Markov chains. In M. Siegle B. Haverkort, H. Hermanns, editor, Validation of Stochastic Systems, pages Springer Verlag - LNCS, Vol 2925, [12] D. Cerotti, M. Gribaudo, and A. Bobbio. Presenting dynamic markovian agents with a road tunnel application. In MASCOTS09, pages IEEE-CS, [13] D. Cerotti, M. Gribaudo, A. Bobbio, C.T. Calafate, and P.Manzoni. A Markovian agent model for fire propagation in outdoor environments. In Computer Performance Engineering (EPEW2010), pages Springer Verlag - LNCS, Vol 6342, [14] A. Bobbio D. Cerotti, M. Gribaudo. Disaster propagation in inhomogeneous media via markovian agents. In Critical Information Infrastructure Security, pages Springer Verlag - LNCS, Vol 5508, [15] M. Gribaudo and A. Bobbio. Performability analysis of a sensor network by interacting markovian agents. In Proceedings 8-th International Workshop on Performability Modeling of Computer and Communication Systems (PMCCS-8), [16] M. Gribaudo, D. Cerotti, and A. Bobbio. Analysis of on-off policies in sensor networks using interacting Markovian agents. In 4-th Int Workshop on Sensor Networks and Systems for Pervasive Computing - PerSens 2008, pages , [17] M. Gribaudo, C.-F. Chiasserini, R. Gaeta, M. Garetto, D. Manini, and M. Sereno. A spatial fluid-based framework to analyze largescale wireless sensor networks. In IEEE International Conference on Dependable Systems and Networks, DSN2002, [18] J. Hillston. Fluid flow approximation of PEPA models. In 2nd International Conference on Quantitative Evaluation of Systems - QEST, pages 33 43, [19] M.G. Hinchey, R. Sterritt, and C. Rouff. Swarms and swarm intelligence. IEEE Computer, pages , April [20] J.M. Kelif and E. Altman. Downlink fluid model of CDMA networks. In IEEE 61th Vehicular Technology Conference (VTC 2005), [21] P. Kemper. Transient analysis of superposed GSPNs. IEEE Transactions on Software Engineering, 25: , [22] B.D. Plateau and K. Atif. Stochastic automata network for modeling parallel systems. IEEE Transactions on Software Engineering, 17: , [23] B.D. Plateau and J.M. Fourneau. A methodology for solving Markov models of parallel systems. Journal of Parallel and Distributed Computing, 12: , [24] K. Trivedi. Probability & Statistics with Reliability, Queueing & Computer Science applications. Wiley, II Edition, 2001.