Simulation of Markovian models using Bootstrap method

Transcription

1 Simulation of Markovian models using Bootstrap method Ricardo M Czekster, Paulo Fernandes, Afonso Sales, Dione Taschetto and Thais Webber Faculdade de Informática Pontifícia Universidade Católica do Rio Grande do Sul AvIpiranga,6681,Prédio PortoAlegre RS Brasil {ricardoczekster, paulofernandes, afonsosales, dionetaschetto, thaiswebber}@pucrsbr Keywords: Markovian models, Discrete event simulation, Bootstrap method Abstract Simulation is an interesting alternative to solve Markovian models However, when compared to analytical and numerical solutions it suffers from a lack of precision in the results due to the very nature of simulation, which is the choice of samples through pseudorandom generation This paper proposes a different way to simulate Markovian models by using a Bootstrap-based statistical method to minimize the effect of sample choices The effectiveness of the proposed method, called Bootstrap simulation, is compared to the numerical solution results for a set of examples described using Stochastic Automata Networks modeling formalism 1 INTRODUCTION Modeling formalisms are usually employed to describe real systems, capturing their behavior An example of a wellknown modeling formalism used for such purpose is Markov Chains(MC) [20] MC models use simple primitives such as states and labeled transitions to exemplify system s evolution and operational semantics Besides Computer Science abundant applications, Markov modeling examples are present in bioinformatics, economics, engineering and chemistry, to nameafewdomains[20] However,themajorlimitationofMCisthestate spaceexplosion problem, observed when the amount of possible configurations is massively huge, rendering solution intractable and easily depleting computational resources [20] In order to mitigate this problem, structured formalisms are proposed such as Stochastic Automata Networks (SAN) [9], Generalized Stochastic Petri Nets [6] or Stochastic Process Algebras (for example, Extended Markovian Process Algebra [3] or Performance Evaluation Process Algebra [12]), among several other approaches Every structured formalism has a correspondent underlying MC and, in a particular way, SAN presents an inner representation based on tensor operations, producing a highly memory efficient fashion to map its underlying transition system SAN also enables complex modeling due to sophisticated primitives (such as events) to define the firing of transitions, locally or synchronously, through constant or functional rates(rates dependent on the other component states) Beyond model mapping, its numerical solution plays an equally important role when examining complex systems There are distinct ways to solve any given model, eg, using numerical methods or simulation Numerical methods often rely on iterative mathematical techniques such as the Power Method [20], Arnoldi [1] or GMRES [16] to calculate measures of interest On the other hand, simulation techniques consider the model events, firing transitions according to pseudorandom numbers generation Simulation precision is strongly related to the number of samples that are produced Normally, a simulation study comprehends large amounts of time in order to obtain significant performance indices, usuallyin functionofthemodelstatespace size Although numerical solution produces reliable results, it is bounded by the number of states of the model Current numerical methods are directed to research on the acceleration of iterative methods[9] However, methods based on Perfect Samplings for SAN have been used with satisfactory results, generating unbiased samples [10] Another alternative is to use pure traditional simulation methods for Markovian models[11] Such methods are based on simulating sets of trajectories and counting the amount of visits for each state, where every step produces a sample(saved for further analysis) When using traditional simulation methods one must be concerned about result quality, ie, precision Since every simulation trajectory only approximates the numerical solution, a thoroughly data inspection must be conducted, verifying results and searching for imprecise results Succinctly, traditional simulation starts from an arbitrarily chosen initial state and randomly walks on a predefined state space during a fixed quantity of steps(trajectory length) Ideal length for a trajectory can be estimated using confidence intervals, among other approaches Since every visited state generates a sample, dividing the number of times a state has been visited by the trajectory length gives an approximation of the mean permanence probability for each state Bootstrap methods were firstly proposed by Efron [8] to perform estimations applied to different areas (for instance, machine learning algorithms[2]) The main idea is to reduce the noise that were observedfrom the samples In the context of simulation, this noise corresponds to the noticed error in relation to the numerical solution The method consists on resamplings, where each sample maps not only one state (the common approach in the traditional simulation) but a set of

2 states In comparison, these sets of results, in average, are betterthantheaverageofasinglesampling Our aim in this work is to employ Bootstrap methods as a valid alternative to the solution of Markovian models by controlling the precision when producing large sets of samples We focus our attention on proposing a method, called Bootstrap simulation, compared against a traditional simulation in order to measure the corresponding error of both methods The remainder of this paper is presented as follows We show basic modeling concepts based on Markov Chains and SAN, followed by principles of simulation in Section 2 We discuss Bootstrap method and how it can be applied in the context of Markovian simulation in Section 3 We present the main results and comparisons with traditional simulation in Section 4 Finally, the conclusion draws some final considerations and future works 2 MARKOVIAN MODEL SIMULATION Markovian modeling are commonly used to describe complex system interactions using mathematical methods for solving the linear system of equations The results produce quantitative performance indices used to subject models for analysis and interpretation These performance indices normally signalize bottlenecks and configurations where the system will degradeor malfunction [7, 4, 18] A common modeling formalism is Markov Chains[20], used for system modeling through simple primitives such as a set of states S and transitions among states The transitions can be modeled by stochastic processes that vary according to a Continuous Time Markov Chains(CTMC) or Discrete Time Markov Chains (DTMC) mapping, respectively, exponential or geometric probability distributions A DTMC is represented by a probability matrix P, where every row i and column j indicate the probability to change from state i to j, where the matrix dimension corresponds to the total number of model states S, ie, the cardinality of S given by S The matrix is mapped to a linear system of equations for solution purposes, ie, the determination of the permanence probability of every state, solving πp = π For CTMC, instead of a probability matrix, a transition matrix, called infinitesimal generator Q [20], represents the MC In an infinitesimal generator all non-diagonal elements i, j represent the occurrence rate of a transition from state i to state j,andthediagonalelementsaresetwithnegativevaluestolet thesumofeveryrowequaltozeroinordertosolveactmc it is common to discretize (or uniformize) the infinitesimal generator Q to its analogous DTMC version[20], obtaining a probability matrix P When a MC model has a huge set of states S, usually, a larger amount of memory is needed to store the transition system Q and, consequently, P as well This problem, known as state space explosion, is always on evidence in Markovian models and more compositional approaches have been defined throughout the years to mitigate the hazardous effects present on massively sized models Among other structured formalisms, Stochastic Automata Networks(SAN)[15] are based on MC and use a high-level description language to build modular systems in a memory efficient manner due mainly to a tensor representation of Q instead of the full transition matrix storage [9] The main idea behind SAN is to model any reality with several subsystems, or automata, acting independently and, occasionally, synchronizing activities Each automaton has a set of states with local transitions and/or synchronizing events having constant or functional rates associated To compose the model state space, one must combine all local states from all automata, forming the model global states, ie, the Product State Space (PSS) However, when performing this combination, depending on initial states and the model transitions, only a possible set of states can be valid (reached), defining thus the Reachable State Space(RSS) of the model In order to inspect the performance indices for the models, Vector-Descriptor Product methods are usually used to multiply the probability vector π by the tensor representation of Q, named the Markovian Descriptor The operation is currently bounded by the available memory needed to entirely store the probability vector For example, in a 4GB machine, the maximumnumberforthepssisset to 65millionofstates Recent efforts are based on solution approximations, particularly based on simulation of the underlying MC as a valid option to handle massive state spaces [13, 19] Simulation methods, among many other advantages, are extremely useful to easily manipulate the components of Markovian models and use the probability vector to estimate reliable performance indices for huge state spaces Obviously, the number of collected samples should be large enough for a comprehensive statistical analysis, eg, the chosen amount should estimate according to the RSS size and the overall variance of the model in order to deliver the desired confidence interval Due to the fact that simulation only produces samples belonging to the reachable set RSS, simulation approaches can potentially produce interesting results within memory bounds In SAN, a probability vector of size RSS stores the number of occurrences of each reachable state within S = {,,,s r } and a stochastic matrix P maps transition probabilities P ij for any given row i and column j The simulation method needs a pseudorandom generation function U uniformly distributed in (01)[11] Every transition between two states occurs according to a transition function given by φ(s i,u),dictatinghowtherandomwalkingtakesplaceforthe set of valid states defined by the SAN model We are interested in defining an initial state and randomly visit the model states, generating a valid trajectory The transition function φ is generically described by the following expression:

3 φ(s i,u) = s j foru [0, P i0 ) foru [P i0, P i0 +P i1 ) s r foru [ j 1 P il, l=0 j l=0 [ ] r 1 foru P ir, 1 l=0 P il ) 3 BOOTSTRAP SIMULATION Bootstrap method is relevant in statistics to derive estimations about standard deviations for complex distributions In essence, this method works with previously unknown data pertaining an infinite sized population Λ and discovers the distribution in random samplings Λ of size n (extracted from this population, ie, Λ Λ) [14] Intuitively, the n samples obtained from the observations of the infinite population Λ, each one with probability 1 n, are used to model the unknown population as real as possible ThesetofvaluesofanewsampleK,alsohavingsizeequal to n ( K = Λ ) is exclusively obtained from the first sample Λ without generating new values for the population Λ, as showed by Figure 1 The main feature of the Bootstrap method is to sample with replacement, ie, the values that constitutes the sample could be repeated during the process To exemplify this method, lets consider the problem of finding the average height of the whole world population Since it is impractical to measure every individual within the available population(set Λ), only a subset of this population should be taken (subset Λ) Using Λ, a quantity of z resamplings must be performed, corresponding to the number of bootstraps as demonstrated by Figure 1, where K i = {x K i x Λ} and i [1z] Each bootstrap K i has a set of n values obtained from Λ through pseudorandom samplings that could repeat or not the values (ie, x 1 K i,x 2 K i x 1 = x 2 ) The mean x K, or the average population height, is calculated using the averages x K1, x K2,, x Kz ofeverybootstrap Λ K 1 K 2 K z x Λ x K Figure 1 Bootstrap method illustration Λ x K1 x K2 x Kz (1) The objective of Bootstrap method is to generate more precisevaluesfortheaverage x K thantheaverage x Λ Moreover, as the number of bootstraps is increased, it reduces the effects of bias samples(noise), generally associated with pseudorandom generators The adaptation of the method to be used in Markovian simulation concerns the transition function φ As mentioned before, a uniformly distributed sample is generated between 0 and 1 and the transition function φ(s i,u) is applied to determine the next state to be visited, forming a simulation trajectory that maps the Λ sample Figure 2 illustrates the Bootstrap simulation schema, where each visited state produces pseudorandom samplings that definethestatesaccountedineverybootstrapkthenumberof trials is equal to the amount of present values in Λ, which is equivalenttothetrajectorylength(n)inthisfigure,t b (where b [1z]) represents n samplings for each bootstrap K Thus, n z samplings take place for every trajectory step, comparing each pseudorandom value against an arbitrarily chosen constant value α between 0 and n 1 If the pseudorandom generated value is equal to α, the state is accounted in the correspondent bootstrap K This procedure is repeated for all bootstraps until a predefined trajectory length n is reached Figure 2 also shows that represents this execution When the trajectory is finished, the probability vector π contains the permanence probabilities of model states However, it is computationally too expensive to randomly generate n samples for every trajectory step Since the number of times each step will actually be randomly sampled is verylikelyto bemuchsmallerthann,wepropose,insteadof generating n samples for each bootstrap, only execute n trials, where n ntoevaluatetheimpactofreducingthisnumber of trials, fromn to n, we calculated the probabilityof a same value to be sampled more than n times duringn trials If this probability is negligible, it means that the number of samples nissufficienttocomputesuchavalue,weobservethat: Probability of a given number to appear at themost n times = n i=0 [ (1 ) i ( ) ] n 1 n i (2) n n Hence, simplifying (2) and considering its complement, ie, the probability of having more than n becomes: Probability of a given number appear more than n times [ n =1 i=0 (n 1) n i ] Forexample,assuming n =15andn=10 4,theprobability to obtain a same value more than 15 times with n samples is inferiorto10 16 Algorithm 1 presents the application of the Bootstrap in the context of Markovian simulation In this algorithm, variables and vectors used in the method are initialized between lines n n (3)

4 Initial state States U =008 K 1 x 1 x 2 x z π 0 π 1 π 2 π x 1[0] + x 2[0] + + x z[0] z x 1[1] + x 2[1] + + x z[1] z x 1[2] + x 2[2] + + x z[2] z U =087 where: Figure 2 Bootstrap simulation schema K 2 and U =032 U =006 U =056 K z P = = T U 1(0 n 1) U n(0 n 1) T z U 1(0 n 1) U n(0 n 1) Time Transition Matrix 1 and 4 The samples countedin the bootstrapsfor all trajectories of length n are shown from line 6 to line 16 Between line7 and 25, it is calculated the permanence probabilities ofstates foreach bootstrapforthe rest part ofthe algorithm (line6-31), the average probabilities of every state in vector π are adjusted according the information contained in the bootstraps n 4 NUMERICAL ANALYSIS The numerical analysis of Bootstrap simulation method is conducted using three SAN models with different characteristics [17, 7, 4], such as: Alternate Service Patterns (ASP), First Available Server (FAS) and Resource Sharing(RS) ASP model describes an Open Queueing Network [20] with servers that map different service patterns It has four queues represented by four automata, where its PSS is equal to (K +1) 4 P, K is the queues capacity and P the number of service patterns All states of this model are reachable FAS model indicates the availability of N servers, where every server is composed of a two state automaton, representing the two possible server conditions: available or busy In the model,tasks are firstly assigned to the first server, case it is available If the server is busy, the task must be sent to the second server and so on The first available server processes the task All states in this model are reachable and its PSS is equalto2 N Theclassical RS modelmapsrsharedresourcesto P processes Each process is represented by an automaton with two states: idle or busy The number of available resources is represented by an automaton that counts the number of resources being used The PSS for this model is equal to 2 P (R+1) and, due to the nature of the model transitions, not all states are reachable The parameters for these models adopted in this paper experiments are the following: ASP model - every queue with capacity two (K = 2) and two service patterns (P = 2); FAS model - with nine servers (N = 9); and RS model - with 10 processes(p =10)andfiveresources(R =5) Algorithm 1 Bootstrap simulation method 1: α U(0 n 1) { constant value α initialization with a pseudorandom value chosen between 0 and n 1 } 2: π 0 { initialization ofthe probability vector π } 3: K 0 { initialization ofall zbootstraps K } 4: s c { sets c as initial state } 5: { walk on a trajectory of length n } 6: fort =1tondo 7: s d φ(s c,u(01)) { finds destination state s d from s c according tou(01) } 8: for b =1tozdo 9: for c =1to ndo 10: if (U(0 n 1) == α) then 11: K b [s d ] K b [s d ]+1 {countsink b [s d ]everytimethesample equals to α } 12: end if 13: end for 14: end for 15: s c s d { currentstate s c is updated to s d } 16: endfor 17: for b =1tozdo 18: ω 0 19: for i =1to RSS do 20: ω ω+k b [i] { calculates in ω the total sum of accumulated values in K b } 21: end for 22: for i =1to RSS do 23: x b [i] K b[i] ω { calculates the probability ofi-th state in K b } 24: end for 25: endfor 26: for i =1to RSS do 27: for b =1tozdo 28: π[i] π[i]+ x b [i] 29: end for 30: π[i] π[i] z {calculates the average probability from the bootstraps} 31: endfor

5 Relative error Relative error Relative error (a) ASP model Trajectory length(n) Mean(Trad) Mean(Boot) Maximum(Trad) Maximum(Boot) (b) FAS model Trajectory length(n) Mean(Trad) Mean(Boot) Maximum(Trad) Maximum(Boot) (c)rs model Trajectory length(n) Mean(Trad) Mean(Boot) Maximum(Trad) Maximum(Boot) Figure 3 Traditional and Bootstrap simulation results The results obtained from traditional simulation are compared to the Bootstrap simulation, where the mean relative error is used to compare the approximate solution to the numerical solution produced by the SAN models solver[5] The main simulation results are presented in Figure 3 showing 95% confidence intervals for 50 executions, where the x-axis corresponds to the trajectory length and the y-axis indicates the relative error The dotted lines represent the maximum relative error and the bars indicate the mean relative error ASP model - Figure 3 (a) - presents the best precision results for the Bootstrap simulation, since the mean relative error is reduced for the set of tested trajectory lengths For the last tested trajectory (10 9 ), the mean relative error i0061 in traditional simulation and plummets to when the Bootstrap simulation is employed Comparing the results for FAS model- Figure3(b)-we also observedasignificant reduction in the mean relative error for Bootstrap when the trajectorylengthexceeds10 6 RSmodel-Figure3(c)-presents similar results for Bootstrap and traditional simulation (for instance,whenthetrajectorylengthisequalto10 9,themean relative error in traditional simulation i0010, whereas in Bootstrap the error i0011) However, even not presenting significant precision gains, the mean relative error is very low In simulation, the quantity of samples is a direct indication of the solution approximation quality, ie, the precision In the context of this paper, the measure of quality used is the simulation mean relative error that decreases, sometimes dramatically, as the trajectory length is increased Figure 3 shows that using the Bootstrap simulation is possible to generate samples that greatly approximate the numerical solution, when compared to the traditional simulation approaches, consideringtrajectorylengthsgreaterthan10 7 Fortheworst case(rsmodel)presentedinthiswork,weobtainedthesame precision of traditional simulation, which is a clear indication that our proposed method could be applied to greater models without significantly impairing precision 5 CONCLUSION The main contribution of this paper is the proposition of the Bootstrap method usage adapted for a Markovian simulation context Research on such algorithmic adaptation is extremely important when analytical modeling fails to produce relevant performance indices Simulation, for these cases, emerges as a valid alternative to deliver analysis of huge models, allowing result interpretations and helping decision making process as a whole Our aim is to present the Bootstrap simulation and inspect the mean relative error when compared to traditional simulation Our results produce evidence that Bootstrap simulation method has the potential to reduce the observed noise related to sampling from large state spaces This fact was detectedfortrajectorylengthsgreaterthan10 6 fortheavailable models However, more experiments must be done to validate such findings and report the classes of models that are more suitable for Bootstrap utilization In regards of the traditional simulation, trajectory length plays a relevant role in terms of the associated error We observed that small trajectory lengths induce high error levels, which is interesting only for executions requiring few precision Future works are directed to the following studies:(i) models classification - determination of classes of models more suitable for Bootstrap simulation; (ii) method parallelization and distribution - we shall take advantage from the fact that every sample is independent, so the computation can be con-

6 ducted separately for each model, improving the overall quality of samples for further analysis; and (iii) rare events - we plan to study the impact of Bootstrap in concealing (or not) rare events and its influence on precision of results The work in progress herein described clearly indicates that further research efforts must be done to increase the understanding in regard to state-based systems simulation applied with different statistical methods However, the preliminary results reveal that Bootstrap simulation is a worthy subject of research ACKNOWLEDGMENTS This work is funded by Petrobras ( ) Afonso Sales receives grants from CAPES-Brazil(02388/09-0) Paulo Fernandes is also funded by CNPq-Brazil (307272/2007-9) The authors thank Prof João Batista de Oliveira for the helpful discussion on this paper subject REFERENCES [1] W E Arnoldi The principle of minimized iterations in the solution of the matrix eigenvalue problem Quarterly of Applied Mathematics, 9:17 29, 1951 [2] E Bauer and R Kohavi An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants Machine Learning, 36(1-2): , 1999 [3] MBernardoandRGorrieriAtutorialonEMPA:atheory of concurrent processes with nondeterminism, priorities, probabilities and time Theoretical Computer Science, 202(1-2):1 54, 1998 [4] C Bertolini, L Brenner, P Fernandes, A Sales, and A F Zorzo Structured Stochastic Modeling of Fault-Tolerant Systems In International Symposium on Modelling, Analysis and Simulation on Computer and Telecommunication Systems(MASCOTS 04), pages , Volendam, The Netherlands, October 2004 [5] L Brenner, P Fernandes, B Plateau, and I Sbeity PEPS Stochastic Automata Networks Software Tool In International Conference on the Quantitative Evaluation of Systems (QEST 07), page63 164, Edinburgh, UK, 2007 IEEE Press [6] S Donatelli Superposed stochastic automata: a class of stochastic Petri nets with parallel solution and distributed state space Performance Evaluation (PEVA), 18(1):21 36, 1993 [7] F L Dotti, P Fernandes, A Sales, and O M Santos Modular Analytical Performance Models for Ad Hoc Wireless Networks In International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt 05), page64 173, Trentino, Italy, April 2005 [8] B Efron Bootstrap Methods: Another Look at the Jackknife The Annals of Statistics, 7(1):1 26, 1979 [9] P Fernandes, B Plateau, and W J Stewart Efficient descriptor-vector multiplication in Stochastic Automata Networks Journal of the ACM(JACM), 45(3): , 1998 [10] P Fernandes, J M Vincent, and T Webber Perfect Simulation of Stochastic Automata Networks In International Conference on Analytical and Stochastic Modelling Techniques and Applications (ASMTA 08), volume 5055 of LNCS, page49 263, 2008 [11] O Häggström Finite Markov Chains and Algorithmic Applications Cambridge University Press, 2002 [12] J Hillston A compositional approach to performance modelling Cambridge University Press, USA, 1996 [13] A M Law and D W Kelton Simulation Modeling and Analysis McGraw-Hill Higher Education, 2000 [14] B F J Manly Randomization, Bootstrap and Monte Carlo Methods in Biology Chapman& Hall/CRC, second edition, 1997 [15] B Plateau On the stochastic structure of parallelism and synchronization models for distributed algorithms In Proceedings of the 1985 ACM SIGMETRICS Conference on Measurements and Modeling of Computer Systems, page47 154, Austin, Texas, 1985 ACM Press [16] Y Saad and M H Schultz GMRES: A Generalized Minimal RESidual Algorithm for Solving Nonsymmetric Linear Systems SIAM Journal on Scientific and Statistical Computing, 7(3): , 1986 [17] A Sales and B Plateau Reachable state space generation for structured models which use functional transitions In International Conference on the Quantitative Evaluation of Systems (QEST 09), page69 278, Budapest, Hungary, September 2009 [18] K Sayre Improved techniques for software testing based on Markov chain usage models PhD thesis, University of Tennessee, Knoxville, USA, 1999 [19] R E Shannon Introduction to the art and science of simulation In Conference on Winter Simulation (WSC 98), pages 7 14, Los Alamitos, USA, 1998 [20] W J Stewart Probability, Markov Chains, Queues, and Simulation Princeton University Press, USA, 2009