Acknowledgements I wish to thank in a special way Prof. Salvatore Nicosia and Dr. Paolo Valigi whose help and advices have been crucial for this work.

Transcription

1 Universita degli Studi di Roma \Tor Vergata" Modeling and Control of Discrete Event Dynamic Systems (Modellazione e Controllo di Sistemi Dinamici a Eventi Discreti) Francesco Martinelli Tesi sottomessa per il conseguimento del titolo di Dottore di Ricerca in \Informatica ed Ingegneria dell'automazione" X Ciclo del corso di dottorato ( ) Docente guida (advisor): Prof. Salvatore Nicosia Coordinatore (coordinator): Prof. Giuseppe Iazeolla Roma, Febbraio 1998

2 Acknowledgements I wish to thank in a special way Prof. Salvatore Nicosia and Dr. Paolo Valigi whose help and advices have been crucial for this work. A special thank goes also to Prof. Michael Caramanis, Prof. Christos Cassandras and Prof. James Perkins from Boston University (Boston, USA), who gave me the opportunity of joining an international and very active research group.

3 Contents 1 Introduction 3 2 Discrete Event Dynamic Systems: modelling and control Discrete Event Dynamic System models State automata model Petri Net model Dioid Algebra Continuous ow model Control problem formulation and related issues The state reconstruction algorithm Review of some Perturbation Analysis Techniques The state reconstruction algorithm Notation and queueing network dynamics The state reconstruction problem A concurrent implementation Estimate analysis Control of Discrete Event Systems based on the path reconstruction approach Review of some control approaches Stochastic Comparison Algorithm for Non-Stationary Discrete Event Systems Optimization Problem Formulation The stochastic comparison algorithm for non stationary DEDS Analytical results for the deterministic case The general case A modied version of the Stochastic Comparison Algorithm for Non- Stationary Discrete Event Systems Some simulation results Some implementation problems Conclusions 107 1

4 2 Symbols and abbreviations ASA C-SRA CVDS DEDS FPA GA GSMP IPA JIT LR PA SA SC SCA SCANS SCANS' SPA SRA TWA J(; t) L(; t; t) Optimize Augmented System Analysis Concurrent SRA Continuous-Variable Dynamic Systems Discrete Event Dynamic Systems Finite PA Genetic Algorithm Generalized Semi-Markov Process Innitesimal PA Just in Time Likely-hood Ratio Perturbation Analysis Simulated Annealing Standard Clock Stochastic Comparison Algorithm SCA for Non-Stationary DEDS SCANS improved Smoothed PA State Reconstruction Algorithm Time Warping Algorithm A particular resource allocation or parameter vector value An optimal allocation The search space, i.e. the space of all possible The set of optimal allocations Performance function (or index) at time t for allocation Estimate of J(; t) observing the system for t Minimize

5 Chapter 1 Introduction System and control theory deals with the problem of modeling and controlling systems. Natural systems, such as, for example, mechanical or thermodynamical systems, are characterized by continuous variables: the state of such a kind of systems is a vector whose entries are real numbers. Their evolution is described by means of dierential equations and is time-driven, that is, the current value of the state directly depends on the time, which can be thought of as an independent variable. Once the equations of a model for this kind of systems have been derived, it is possible in many cases, based on these equations, to nd an analytical expression for a control law aimed at obtaining some desired results. Very similar to these systems, there is a class of systems which are always characterized by a real state vector, but their evolution, yet time-driven, is described by means of dierence equations: in this case the state of the system changes only at discrete time instants. Many results available for time-continuous systems can be adapted to t in this case. All of these systems, discrete or continuous time dependent, are time-driven and their state can be given as a vector with real entries. They will be denoted with the term of Continuous-Variable Dynamic Systems (CVDS)([1]). Many man-made systems can not be successfully described by models used for CVDS. It is the set of articial systems usually characterized by some entities which can provide a particular desired service (and will be referred to as servers) and some other entities which compete one another and wait for this service (and will 3

6 Introduction 4 be called customers). This situation is really general: even if in this thesis the attention will be devoted to manufacturing systems ([2]), the same concepts apply to communication and computer networks, or many general queueing-systems. If manufacturing systems are considered, a customer is typically a part which waits for an operation on a queue while a server is a machine which performs a given set of operations on parts. In the communication system case the customer can be a piece of information which could require a particular channel or another device, which must be considered as a server. There is a common feature in all these man-made systems: the state takes value on a discrete set. As an example, in the manufacturing case, the state typically comprises the number of customers in the queues and the state of the machines (which, again, is a discrete if not boolean variable). Moreover the evolution is eventdriven rather than time-driven: the value of the state changes when something, that is an event, happens. The time an event is observed is usually a random variable and the state of the system as a function of the time is a stochastic process. Discrete nature of the state and event-dependence make this kind of systems very dierent from CVDS. The term Discrete Event Dynamic Systems (DEDS) comprises these major characteristics. Classical models like dierential or dierence equations are not useful to model DEDS. A mathematical model is a set of equations which can be used to reproduce the behavior of a system. In Chapter 2 a review of some mathematical models introduced to describe DEDS will be presented. The main dierence between these models and the ones used for CVDS is that rarely they can be directly used to derive analytical expressions for metrics dened on the DEDS or for the control law which solves a particular control problem, how often is done in CVDS. A dierent procedure is required for DEDS. A DEDS is usually a man-made system which has been realized for some specic purpose. The purpose of a DEDS and, in particular, of a manufacturing system, is that of providing some service achieving some objectives ([3, 2]): at a high level these objectives can be those of making the most money, or guaranteeing high quality products, or staying in

7 Introduction 5 business as long as possible. At a lower level, i.e. to achieve the previous results, the objectives will be some performance measure like the number of part released by the system in the time unit (throughput), the time each customer has to wait for a service, the time a machine is in service, the cost and the gain corresponding to the dierent operations, the percentage of imperfect products, and so on. Therefore, it appears natural to formulate a control problem for a manufacturing system as the optimization of a performance function dened to capture some interesting feature of the system. This performance function depends on the allocation of some resources and/or on the denition of some policies or protocols: the dynamic choice of an allocation and/or the denition of a policy in order to optimize the considered performance function is then the objective of a control scheme. From a more general point of view, the performance function can be seen as dependent on a vector of parameters, where each possible value for this vector can represent a particular resource allocation, a particular policy or a mix of them. In this perspective the objective of a control scheme is the dynamic choice of the value of the parameter vector to optimize a performance function. This is the problem addressed in this thesis, and it will be formally stated in Chapter 2, together with the main diculties which arise to solve it. In many cases a closed expression of the performance index as a function of the parameter vector is not directly available through mathematical models of the system and many approaches, like Perturbation Analysis ([4, 5, 6]), Rapid Learning ([7]), Sample Path Analysis ([8]) and State Reconstruction ([9, 10]) are aimed to compute an estimate of the performance function for a particular parameter setting. The estimate is computed using data recorded during the observation of the system, which can evolve with a value of the parameter vector dierent from the one for which the estimate is sought. This will be the main subject of Chapter 3 which contains one of the contributions of this thesis, the State Reconstruction Algorithm. This algorithm presents many advantages with respect to other well known techniques: it can be applied to any DEDS considering every possible value for the parameter vector; it provides an exact estimate of the performance function and

8 Introduction 6 does not require any knowledge on future events of the DEDS unlike many classical Perturbation Analysis techniques. Chapter 4 deals with the control problem: how to use this kind of estimate approaches to design control algorithms. The basic idea is quite simple: once an estimate of the performance function for many and possibly all the values of the parameter vector is available, the objective is that of selecting the value of the parameter vector corresponding to the (supposed) best performance. Such a conceptually simple problem presents many complications. Among them: (i) the set of all possible values of the parameter vector, called the search space, can be very large, and an estimate of the performance function for all possible values of the parameter vector could become computationally unfeasible; (ii) the performance function is known only through estimates; (iii) the system can be time-varying, which means a good choice for the parameter vector can become quite bad in the future. All of these aspects require a careful study for designing an acceptable control scheme. A comparison between dierent proposals is oered in Chapter 4. Among them, a novel algorithm, the Stochastic Comparison Algorithm for Non- Stationary DEDS (SCANS) ([11, 12, 13]), and a modied version of it are presented in Chapter 4 and constitute the more signicant contribution of this thesis. SCANS has been derived modifying one of the more recent stochastic optimization approaches in this research area, the Stochastic Comparison Algorithm (SCA) proposed by Gong et al. ([14, 15]) and described also in Chapter 4. The change has been performed trying to maintain the interesting characteristics of the original algorithm but with the objective of controlling non stationary DEDS. Analytical results as well as simulation experiments assess the eectiveness of the proposed scheme, which can be applied to any DEDS and benets from advantages of order statistics with respect to cardinal estimates ([16, 17, 18]). The thesis ends with a concluding chapter, Chapter 5, which summarizes on the results presented and reports some indications for future work.

9 Chapter 2 Discrete Event Dynamic Systems: modelling and control This chapter focuses on two subjects: how to model a Discrete Event Dynamic System and how a control problem can be formulated. In particular, the control problem considered in this thesis will be formally stated. The rst section is devoted to the description of the major models which have been derived for studying DEDS [1, 19]. It must be remarked that a mathematical model of a system (a general system, also a Continuous-Variable Dynamic System) is a tool which allows to analyze and identify the behavior of the system. In that a model is not the real system but can be seen as another system, completely known and simple enough, which can be used to reproduce the behavior of the considered system with sucient precision in terms of the objectives for which the model has been created. A model is created to analyze and identify a system but the identication process is usually nothing but the rst step in the solution of a control problem. As a matter of fact, for CVDS, the models can often be directly used to analytically compute a control law which solves the considered control problem. As it will be more apparent in the following, the most part of models derived to describe a Discrete Event Dynamic System is usually very complex and much more suitable for simulation purposes rather than analytical studies. This means that these models constitute a good tool to reproduce the behavior of a DEDS and its evolution, but they are not very useful to derive an analytical expression of the state 7

10 Chapter 2 - Modeling and control 8 as a function of the time. This is not only due to the stochastic nature of events characterizing a DEDS but it depends also on the particular structure of such a kind of systems. The particular structure of a DEDS reects also on the way a control problem for this kind of systems can be formulated. Section 2 deals with this problem. A control problem, as said in the Introduction, can be often transformed in the dynamic optimization of a performance function dened on the system. In a manufacturing system, this objective function can depend on the particular policy adopted to process arriving parts or on the allocation of some resources. These can be considered as two dierent problems but also as two steps in the design of a manufacturing system: a long term control is applied to allocate resources, then a policy must be dened given the allocation. But resource allocation can also be seen as a short term control problem especially when the resource allocation can be accomplished very quickly. In the Introduction, a parameter vector which can comprise both the resource allocation and the type of policy implemented has been mentioned, and the problem is the dynamic choice of the value of the parameter vector to optimize a performance function dened on the manufacturing system. The discussion of this problem is reported in Section Discrete Event Dynamic System models The Discrete Event Dynamic Systems considered in this thesis are manufacturing systems, where some parts wait on queues to be serviced by some machines. The state of such a kind of systems comprises the number of parts waiting in all the queues contained in the system as well as the state of the machines, which can be idle, blocked, down or servicing a part. The state changes when an event happens. For manufacturing systems an event can be a service completion, a part arrival, a machine failure or repair, and so on. A sequence of events drives the state of the system from a value to another value. If each event is associated with a symbol of a language, a sequence of events can be seen as a word in this language ([20, 21]).

11 Chapter 2 - Modeling and control 9 So, it is natural to try to describe a DEDS by means of a state automaton, because a state automaton is a machine characterized by a set of states and the transition between states is driven by the words of the language accepted by the automaton. State Automata formalism is the basic model considered in this thesis. Other models can be used and will be briey sketched in this Section. These models are dierent algebras to describe DEDS ([22]) and a common feature is that they are well dened both in an untimed and in a timed scenario, as it will be more apparent in the following State automata model In the DEDS framework, a state automaton can be formally given as a ve-tuple (E; X; (x); f; x 0 ), where E is the set comprising all the events dened on the DEDS, X is the discrete set of all the states the DEDS can assume, (x) E, dened for all x 2 X, is the set of all the events which can happen when the state of the DEDS is x, f is a transition function which maps a state x 2 X and an event e 2 E into a new state x 0 2 X. It means that if the DEDS is in x and e happens, the new state will be x 0. Finally x 0 simply denotes the initial state of the DEDS. In this description the time can be omitted: once a sequence of events is available, it is possible to feed the DEDS using this sequence and nd, by the iterative application of the transition function f, the state sequence of the DEDS corresponding to this event sequence. It is possible to give a graphic representation of the state automaton modeling the DEDS. In this representation every possible state of the DEDS x 2 X is associated with a node and, for any couple of nodes associated with states x and x 0, there is an edge from x to x 0 labelled by e 2 E if f(x; e) = x 0. The structure described right now is an untimed model of the DEDS. To consider time in such a description it is enough to associate each event with the time it happens. Consider event e 2 E and associate with this event a clock sequence t e := ft e;1 ; t e;2 ; : : :g comprising all the lifetimes of event e. This means that when the DEDS leaves state x and enters state x 0 where event e is enabled (i.e. e 2 (x 0 ) but e 62 (x)) for the n-th time, if it is not disabled it will happen after a time given by t e;n. It is disabled

12 Chapter 2 - Modeling and control 10 Figure 2.1: The simple queueing system considered in the example. if the state changes (due to another event which happens before) and event e is no more feasible in the new state. In this case its lifetime t e;n is lost. When the clock sequences are not deterministic but are associated with some stochastic distribution (that is lifetimes are not deterministic numbers but are random variables) and the state transition does not depend on a deterministic function but there is some state transition probability p(x 0 ; x; e), the state automaton models a Generalized Semi- Markov Process (GSMP), which is a very general class of stochastic DEDS [23]. The markovian aspect is the fact that the system behaves like a Markov Chain at state transitions. It is not a Markov process however because the times between state transitions may obey to general stochastic probability laws and the evolution is characterized by the clock mechanism sketched above. The GSMP formalism will be extensively used in this thesis being the model used to analyze the manufacturing systems considered in the following. When all the lifetimes are exponentially distributed the DEDS is a Markov Chain. In this case analytical results can be derived to determine closed form expressions for many metrics dened on the system, like the time a machine is down in the average and so on ([3]). To conclude this description an example will be presented. Consider the system depicted in Figure 2.1 comprising a machine failure-prone which can go down only while servicing a part according to some stochastic process. If this happens, the part is lost and the interrupted service resumes with a new part when the machine is repaired and a part is available. This machine provides service to parts arriving according again with some random process. The parts wait on a queue comprising only two buer slots. When the queue is full part arriving are dropped while the machine cannot work if the queue is empty. The stochastic processes characterizing

13 Chapter 2 - Modeling and control 11 a a a (0,u) c (1,u) c (2,u) c (3,u) r f r f r f a (0,d) (1,d) (2,d) a a a Figure 2.2: The state automaton corresponding to the considered queueing system. failures, repairs, part arrivals and service durations are not interesting here (they characterize the clock sequences of the corresponding events). The untimed state automaton corresponding to this system is depicted in Figure 2.2. In this automaton E = fa; c; f; rg, where a=part arrival, c = service completion, f = machine failure and r = machine repair. The state (n; s) comprises the number of parts in the system (queue and machine) (n = 0; 1; 2; 3) and the state of the machine (s = u; d, where u=up and d=down). The machine can be considered in service if it is not down and n > 0: in other words no blocking mechanism is possible. So the state s = u comprises both the state in service if n > 0 and idle if n = Petri Net model Petri Nets ([24, 25, 26, 27]) are a very general tool for DEDS modeling. Indeed, it turns out that any DEDS modeled by means of state automata (which are a really general tool to model DEDS) can always be represented using a Petri Net ([1]). But the use of this kind of models is not simply an equivalent way to describe a DEDS. Rather, it can be used as a complementary tool. As a matter of fact, in many cases, a Petri Net model turns out to be more natural for a particular DEDS than a State Automaton and vice versa. It is important to gure out which model ts to the particular application. In Petri Nets, events are associated with transitions. A transition is enabled (that is an event can happen) if some conditions are met:

14 Chapter 2 - Modeling and control 12 in the Petri Net formalism a transition is enabled if the places before it contain a sucient number of tokens. So, a Petri Net comprises a given number of places lled by tokens. Any disposition of tokens in the places corresponds to a particular state of the Petri Net and hence to a particular state of the DEDS associated with this Petri Net. Formally, a Petri Net can be given as a four-tuple (P; T; A; w), where P is a set of places, T is a set of transitions, A is a set of arcs between places and transitions and between transitions and places, and w is a weight associated with each arc in the net. Each transition t i is associated with a set I(t i ) of input places and a set O(t i ) of output places. Places are lled by tokens. A transition t i is enabled and can re when every place p j 2 I(t i ) contains a number of tokens not smaller than the weight w ji associated with the arc from p j to t i. The transition res immediately in untimed Petri Nets. In timed Petri Nets this happens after some time, which is the time associated with the transition t i. When the transition res, w ji tokens disappear from all the places p j 2 I(t i ) and w ik tokens are added to all places p k 2 O(t i ). So, tokens present in a place mean that some condition is met. To better understand how Petri Nets can be used to model a DEDS the previous example will be considered and the corresponding untimed Petri Net is reported in Figure 2.3 where weights w are all equal to Dioid Algebra An appealing way to model DEDS is that of dioid or max-plus algebra [28, 29, 30]. The basic idea of this approach is the crucial observation that a sum between lifetimes of events and a minimum operation performed to pick up the next triggering event are the only two operations required for an analytical description of DEDS. Max-plus algebra allows to derive a formalism which can be used to describe DEDS of a particular class by means of a state model equation like that used for linear CVDS: x(k + 1) = Ax(k) + Bu(k) (2.1)

15 Chapter 2 - Modeling and control 13 Part arrives Queue empty Lost parts Part accepted Idle Queue Start service Part in service Service completed Fail Machine broken Repair Figure 2.3: The Petri Net corresponding to the considered queueing system. where x is the state of the system and u the control. When used for CVDS the product and the sum which appear in (2.1) are the usual operations dened in the standard algebra. It is possible to dene these two operations in a dierent way (in particular introducing an addition operator between two real numbers as the maximum between them and a multiplication operator as the usual sum of the two numbers) in such a way that a similar equation can be written for some DEDS, with the new meaning assigned to the operations sum and product. This approach allows to apply many concepts which hold for the analysis and the control of CVDS to DEDS. These concepts include behavior of the system at steady state, maximum eigenvalue assignment and so on ([31, 32, 33, 34]). The major limits of such an approach is on the topology of the queueing network which can be considered by means of a max-plus algebra (in particular no routing nodes can be considered) and the fact that the model can easily be obtained only for deterministic DEDS, i.e. when events happen according to deterministic clock sequences.

16 Chapter 2 - Modeling and control Continuous ow model In some cases, to solve particular control problems, the discrete ow of parts in a manufacturing system can be approximated by a continuous ow and the system can be described as a CVDS, making the analysis and the solution of a control problem analytically feasible ([35, 36, 37, 38, 39, 40, 41, 42]). This means that the state of the manufacturing system is characterized by real entries and its change does not depend on events any more. Rather, parts arrive in the queue with a given (deterministic or stochastic) rate and are serviced by machines with some other rate. In some more complex scenarios, machines can fail according to a given stochastic process ([3]). 2.2 Control problem formulation and related issues When a DEDS is modeled by means of a Markov Chain, or by the equations of maxplus algebra or when it can be reduced to a continuous system having considered parts as a continuous ow, it is possible to derive closed-form expressions for many characteristics as a function of its parameters. In the Markov Chain framework for example it is possible [3] in many cases to derive an expression for many interesting characteristics of the system, like the average length of a queue, the average service time of a machine and so on. When the transition probabilities of the Markov Chain can be modied by an external input the corresponding model will be addressed to by the term of Controlled Markov Chain and this external input can be used to drive the system in such a way to optimize some performance function. The modication of the transition probability is then the eect of a control policy dened on the queueing system. When the policy depends only on the current state it is called stationary. In these cases the solution can be obtained by means of Dynamic Programming solving the Hamilton Jacobi Bellman (HJB) optimality equation. This equation can be solved numerically and/or can be used to prove some properties which the optimal solution satises [43, 44, 45].

17 Chapter 2 - Modeling and control 15 The problem of nding a control policy to optimize a performance function is addressed in many works which use a continuous ow model of the DEDS. As an important example consider a scheduling problem: for a reliable (i.e. without failures), exible (i.e. with negligible set-up times and costs) machine M which can provide service to n dierent part types i = 1; 2; : : : ; n. Suppose part type i can be processed at maximum rate i and let d i be the deterministic and constant demand rate for part type i supposing an innite reservoir of raw material is available for machine M. Let x i denote the quantity of part type i in the system, with x i > 0 representing an inventory x i of part type i and x i < 0 a backlog of x i. If u i is the production rate at time t for part type i, it is possible to write: dx i dt = u i d i i = 1; 2; : : : ; n (2.2) which shows how a DEDS can be reduced to a CVDS introducing a continuous ow approximation and can be solved given u i and the initial state x i (0) 8 i. A capacity bound must be considered for machine M: X Hence, for the system to be stable, it must be i X i u i i 1: (2.3) d i i < 1; (2.4) which guarantees that machine M has capacity enough to meet the demand. Suppose a backlog/inventory cost is associated with each part type i: c i (x i ) = c i;+ x + i + c i; x i ; (2.5) where x + = maxfx; 0g, x = maxf x; 0g and c i;+, c i; are two positive constants. Let c(x) = nx i=1 c i (x i ) (2.6) be the complete cost function. Given an initial state x(0) = x 0, the control problem is that of nding an expression for the production rates u i (control variables) to

18 Chapter 2 - Modeling and control 16 minimize J(x 0 ) = Z 1 0 c[x(t)]dt: (2.7) If the system is stable (eq. (2.4)), the solution is u i (t) = 0 if x i (t) > 0 and work on part types with x i (t) < 0 having the maximum c i; i index. If many part types achieve this maximum, the total capacity of the machine is splitted among them in an arbitrary way: it is enough that machine M works at full capacity (i.e. eq. (2.3) satised with equality) until the backlog is cleared. This is the well known c-rule. Observe that u i (t) = 0 if x i (t) > 0. This is the \Just in Time" (JIT) policy: never work to accumulate a stock of products. It is possible here because the system is stable and machine M is reliable. If M was failure prone, with stochastic and exponentially distributed failure times, it would have been necessary to introduce a hedging point h i > 0 for each part type i and produce until x i = h i rather than x i = 0 to cope with breakdown periods of machine M. The c-rule described above can be proved through classical analysis arguments like Pontryagin's maximum principle but also by means of interchange arguments ([46]). The c-rule, however, is just a particular case of the general result about myopic policies which applies when the cost function has a more general expression than the piecewise linear function given in (2.5) [47]. In the more general case also, the state-feedback nature of this type of control (the value assigned to u(t) = (u 1 (t); u 2 (t); : : : ; u n (t)) at time t depends on the state x(t) = (x 1 (t); x 2 (t); : : : ; x n (t))) is much more evident. In many situations however, it is of interest to consider the discrete nature of a DEDS. In this thesis, in particular, the problem of dynamic resource allocation for general DEDS modelled using the GSMP formalism discussed in section 2.1 is addressed. The control problem is then formulated as follows. A performance function J is dened on the DEDS to capture some interesting feature of it. This function depends on a parameter vector which can comprise general characteristics of the DEDS. In the following this vector will be usually assumed to denote a particular resource allocation for the DEDS. To stress such a dependence, the performance

19 Chapter 2 - Modeling and control 17 θ(t) DEDS J(θ(t),t) Figure 2.4: The DEDS considered in the control problem. function will be denoted by J(). Usually, dealing with DEDS, a given number of resources is available and must be allocated into the system. For this reason, a particular value for the parameter vector will be often referred to as an allocation. The set of all possible allocations will be denoted by. As it will be remarked in Chapter 4, a DEDS is usually non-stationary: the performance function J() is not a time invariant function of. For this reason another entry will be considered in the performance function: J(; t) denotes the value of the performance function when the system is allocated and at time t. The control problem can be formulated as an optimization problem, i.e. nd (t) = arg min J(; t) (2.8) 2 Once the control problem has been formulated in this way, it is possible to see the performance function J(; t) as an output of the DEDS, an output which must be controlled, and the particular allocation (t) chosen for the system at time t as a control input. The system can be represented by the block diagram shown in Figure 2.4. In this perspective, classical considerations can be reported. For example, a closed loop control scheme with a reference signal r(t) can be considered for this system and it is reported in Figure 2.5. The reference signal r(t) is in this case just a command which asks the control to minimize the output J(; t) but it could be a dierent signal, since also a regulation problem can be addressed at this point. Anyway, the above structure presents many dierences with the corresponding structure for CVDS. The measures taken on the DEDS are used to compute an estimate of J(; t) for many and possibly all 2 to derive the allocation (t) which is the best with

20 Chapter 2 - Modeling and control 18 r(t) Controller θ(t) DEDS J(θ(t),t) Measures Figure 2.5: The block diagram comprising the DEDS and the controller. respect to the criterion assigned by r(t). Moreover, all of the signals have a dierent meaning with respect to the CVDS case: a control action is performed at the end of an interval (control epoch) where the measurements are taken on the DEDS. So, the input (t) and the output J(; t) of the DEDS are not dened for all t but only at a control epoch. This will be more apparent in the following. Nevertheless the main idea of a dynamic control scheme is still true: in many situations the DEDS is non-stationary; this means that an allocation which is optimal now can be no more optimal in the future. The control scheme must be able to change allocation to react to this change. Moreover, the search of the optimal through the estimation procedure can be very dicult, especially if the search space is very large. So, the second step in the design of a control scheme is the denition of a strategy which must be applied to explore the set. The estimation problem for the performance function and the design of a search strategy will be addressed in Chapter 3 and 4 respectively.

21 Chapter 3 The state reconstruction algorithm In the previous chapter an introduction to dierent models of Discrete Event Dynamic Systems and the control problem have been outlined. The control of discrete event dynamic systems, as it was pointed out in the previous chapter, was based on the dynamic choice of an allocation 2 in such a way to optimize an objective function J(; t). The problem is that a closed form expression for J(; t) as a function of is usually not available and its value must be estimated in some way. To control a DEDS the simplest way would be to start with some allocation 0, observe the system for some time, then change the allocation to a dierent value 1 and see if the observed performance has improved with this change. This can be repeated with many dierent allocations to choose the one which gave the best observed performance. This is not a clever way to proceed. If the system under control is a real system this would mean to operate it for a long time under many and probably bad allocations, with the result that the time required to get a good performance can be very large. If the system is simulated, it is necessary to produce a simulation run for every allocation to test. Also in this case the time required to get a good performance is very large. These approaches are often referred to as brute force techniques. Observe also that simulating a real system could be impossible if the stochastic process characterizing the system is not known and no trick is used (it is 19

22 Chapter 3 - State Reconstruction Algorithm 20 not possible to generate random variables for the simulation). The idea to solve this problem is estimating the behavior of the system for different allocations without actually change allocation but only using data collected during the evolution of the system under allocation 0. This allows to save computation time and makes possible the analysis of unknown systems, whose simulation can not be actuated directly. The techniques which try to estimate the evolution of the system for an allocation dierent from the current one, only using data collected during the evolution under allocation 0 are the subject of the so called Perturbation Analysis (PA), which is a very important eld of research (see [4, 5, 1, 6] for a review). There are many proposals in the Perturbation Analysis area, which address different kinds of problems. The algorithm described in this chapter, the state reconstruction algorithm, can be seen as a part of this research area. As it will be more apparent in the following, it is a way very general for evaluating the behavior of a discrete event dynamic system when some of its parameters are modied. Although, historically, the main motivation for the introduction of Perturbation Analysis techniques was the attempt to reduce simulation computational eort, the perturbation analysis turns out to be very useful to implement on-line control schemes as it will be claried in the next chapter. As already remarked, there are many proposals in the framework of Perturbation Analysis. These proposals will be sketched in Section 3.1 as an introduction to the State Reconstruction Algorithm, which is one of the contributions of this thesis and will be presented in Section Review of some Perturbation Analysis Techniques Perturbation Analysis deals with the problem of estimating the \structure" of the objective function J(; t) as a function of allocation only using data collected during the evolution of the system with a particular allocation, which will be often

23 Chapter 3 - State Reconstruction Algorithm 21 referred to as the nominal allocation, and will be denoted by 0. The term \structure" of objective function denotes the dependence of the objective function on the allocation. The question that PA tries to answer to is then: \what happens on the objective function if some change is performed on the allocation?". To x ideas consider the case the allocation denotes the value of a single parameter, with 2. Hence must be regarded as an interval or a set of possible values for. 1 Dierent techniques then apply if the parameter is a continuous parameter or if it can take values only on a discrete set. In the rst case, i.e. when a continuous parameter is considered, there are two dierent kinds of estimate: the estimate of the sensitivity of J(; t) on changes in, i.e. the derivative of the objective function with respect to the parameter or the change of the objective function due to a nite variation of the parameter. In the second case, of course, only the second kind of estimate can be considered. An estimate of the sensitivity of J(; t) to changes in can be solved using an Innitesimal Perturbation Analysis (IPA) approach ([48, 49, 50, 51, 52, 53, 54, 55]) or the so called Likelyhood Ratio (LR) approach ([56, 57, 58]), which appears much simpler than the IPA approach but it gives an estimate with larger variance ([1, 5]). IPA is a well established body of research and there are many works on this subject. The application of an IPA estimator is usually easy, even if the analysis necessary to derive it could be really involved ([1]). Nevertheless, IPA is not always feasible in all the problems. First of all, even if the objective function depends on a continuous parameter, this dependence could be non-continuous: Smoothed Perturbation Analysis (SPA) ([59, 60, 61]) tries to overcome this problem, as well as Discontinuous Perturbation Analysis ([62]). If the parameter under consideration takes values on a discrete set, it is obvious that IPA can not be applied, and a dierent approach is required. 1 If a resource allocation problem is considered, is a discrete set. But, from a more general point of view, can be also a real value, for example when it represents some threshold or a hedging point.

24 Chapter 3 - State Reconstruction Algorithm 22 One more point to emphasize is that, even if IPA can be applied to a problem, it always produces local estimates, that is, it gives information only with respect to innitesimal variations of the nominal parameters, in a neighbour of the current nominal value. Remember that controlling a DEDS means changing a parameter in such a way to improve the performance of the system. In this perspective, the IPA estimator, which gives an estimate of the derivative of the objective function with respect to the parameter to adjust, suggests the direction and the magnitude of the change to apply. More formally, the control strategy consists in the update of in the direction suggested by this sensitivity estimate according to 0 = [@J=@] IP A where is a small number, [@J=@] IP A is the sensitivity estimate, is the current parameter value and 0 is the new parameter value, the one after the update. It is clear that this strategy may lead to a local minimum of J(; t), unless some assumptions are made on the performance function. When the objective function depends on a discrete parameter, a dierent PA approach, as already pointed out, must be used for the study of the structure of the objective function. Dierent techniques ([5, 4, 48, 49, 63, 64, 65]) have been proposed to evaluate the change of the objective function due to a generally small but nite change of a considered parameter. These approaches are referred to as Finite Perturbation Analysis (FPA) techniques. The classic and modied FPA techniques, i.e. the ones presented in the works just cited, present some drawbacks. First of all, only small changes in the parameter value can be considered, in order to avoid a too deep modication of the original path, where path denotes the ordered sequence of events. In the control problem, as it will be clearer in the following, larger perturbations should be considered. To understand if there is a change in the order of events (due to the parameter change), also future events must be considered: this makes this approach applicable only o line. The reconstruction of the behavior of the system with the modied parameter is approximate since some hypothesis are taken in order to simplify the computation.

25 Chapter 3 - State Reconstruction Algorithm 23 These problems make FPA approach sometimes not suitable for the application of control algorithms. A dierent approach is the one used to solve the so called constructability problem, i.e. the problem of determining the evolution of the DEDS under a perturbed parameter. Suppose a DEDS is characterized by some parameter and suppose the current value of this parameter is 0. Under this parameter value the DEDS gives a particular performance. Consider any other value ^ for. What would have been the performance of the system if the value of the parameter was ^ instead of 0? The answer should be possible for any parameter value (unlike the FPA techniques discussed above), using only information about the past of the system, and giving exact reconstructions. It is clear that only an estimate can be provided of the performance function, since only a particular and nite realization is considered. The term exact here refers to the reconstruction realized: the sequence produced by the real system with that realization (and that value of parameter ) and the sequence given by the reconstruction algorithm are the same. Observe that all FPA techniques can be also applied to a continuous parameter, and also to provide a sensitivity estimate when IPA approach fails. Anyway, for the control algorithms presented in the next chapter, this is not the case. The solution of the constructability problem for many and possibly all values of the considered parameter is addressed by the term Rapid Learning ([7]): providing a performance estimate for many dierent parameter values allows to rapidly learn the structure of the performance function J(; t) as a function of, making easy the choice of a \good" value (or allocation). One possible solution to this problem is the Standard Clock (SC) algorithm ([66]). This algorithm can be applied only to DEDS whose events are characterized by lifetimes exponentially distributed. An articial sequence is generated to test the DEDS with a given parameter value. The drawbacks of this approach are mainly due to the fact it only applies to DEDS with lifetimes exponentially distributed. Moreover a sequence of random variables must be generated. A dierent approach is the Augmented System Analysis (ASA) proposed in [67,

26 Chapter 3 - State Reconstruction Algorithm 24 68]. The idea is to reconstruct the evolution of the system under a perturbed parameter only using data collected observing the evolution of the real system in the nominal parameter value. So, unlike SC, a sequence of random variables is not generated for this analysis and a real sequence is now exploited. In a rst version this algorithm could be applied to DEDS with events all generated by Poisson processes (i.e. like in the SC approach). A successive extension allowed to consider DEDS with at most one event generated by a non-poisson process. Two conditions must be satised by the DEDS in order to reconstruct its evolution under a dierent parameter using data collected in the real path ([69]). These conditions are the so called observability and constructability conditions. The observability condition states that the set of feasible events in the current reconstructed state must be contained in the set of feasible events in the current nominal state. This condition implies that all the feasible events in the reconstructed path can be compared with the corresponding events in the nominal path in order to determine the time they happen. The constructability condition comprises the observability but, in addition, requires that the time probability distribution of the feasible events in the reconstructed and nominal paths are the same. Due to the memoryless property of exponential lifetimes, it turns out that for DEDS with Poisson events, the constructability condition coincides with the observability condition. In the ASA approach, applied to DEDS with Poisson events, only the observability condition must be assured, in order to make the reconstruction possible. When the observability condition is not satised the event matching algorithm is applied to make the reconstruction possible. The idea is to wait for the next state in the nominal path which guarantees the observability condition. If some events have lifetimes non exponentially distributed the idea is that of saving these lifetimes when the reconstruction is suspended and resume the reconstruction process when the observability condition is met and ages of non exponential events (that is events with non exponentially distributed lifetimes) match with the corresponding saved ages. This is the idea of the Age Matching Algorithm. From the analysis of this algorithm it turns out that ASA, through event matching and age matching algorithms, can

27 Chapter 3 - State Reconstruction Algorithm 25 be applied to DEDS with Poisson events and possibly one non-poisson event. The main drawback of this approach is that a long time could be wasted for the real system to enter a state where the observability condition (and the age matching condition, in the non exponential case) is satised. In the meanwhile, many lifetimes could be recorded and used when the reconstruction resumes. This idea is the basis of the Time Warping Algorithm (TWA) for the sample path reconstruction of general DEDS (see [8]). This approach is very near to the one presented in the next section. The main dierence is that TWA is applied for an on-line reconstruction of the evolution of the system, while the State Reconstruction Algorithm (SRA) described below reconstructs the evolution of the system periodically at the end of a given interval of time, where a control action is performed based on the reconstruction. For this reason the time where the reconstruction is carried out is referred to as control epoch (see below). This aspect makes it more suitable for the control applications presented in the next chapter. Concurrent estimation is studied also in [70]. Concluding this introductive section, it is possible to say that the SRA algorithm (and TWA as well) adds to the already mentioned advantages typical of all constructability techniques (i.e. analysis of any perturbation using only past history and providing an exact reconstruction) also the possibility to be applied to any DEDS with general and unknown lifetimes distribution. Some perturbations can not be considered anyway. A change in the topology of the DEDS cannot be considered, as well as any perturbation which adds some new events in the perturbed system ([8]). Some \dicult" perturbations can be allowed using some trick. Consider for example a perturbation which modies the lifetime distribution of a particular event. Since the SRA and TWA approaches work recording and using the lifetimes observed in the nominal evolution of the system, such a perturbation would seem unfeasible. Anyway, if the value of the cumulative distribution function corresponding to the considered lifetime is stored in place of the lifetime itself, this value can be used to evaluate the lifetime corresponding to that realization but considering the perturbed distribution. This example is depicted in [8] and in the next section.

28 Chapter 3 - State Reconstruction Algorithm The state reconstruction algorithm This section summarizes some results presented in ([71, 9, 72, 10, 73, 74]) about a novel algorithm designed to reconstruct the evolution of a general DEDS only using data collected in the real evolution of the system itself. In [9, 10, 74] the algorithm is presented in detail. The application to control problems is especially presented in [72] while [71, 73] contain some points about a parallel implementation of the algorithm Notation and queueing network dynamics The class of Discrete Event Dynamic Systems which will be considered and for which a control scheme will be provided, is depicted in this section. It is the class of systems that can be modeled by means of queueing networks with: (a) general service time at each node, (b) general routing policy at each node, (c) general scheduling policy at each node, (d) nite buer capacity, (e) multi-class, (f) non-preemptive service at each node, (g) innite arrival rate sources, (h) innite capacity sinks. Each node comprises a server and several input queues or buers; in particular, it is assumed that each server has as much input queues as the number of classes it can provide service to, and each class is associated with a single dedicated input queue. The dynamics of such a class of Discrete Event Systems can be modeled by means of Generalized Semi-Markov Processes (see Section 2.1). The event of type \completion of service of a customer of a specic class on a node" is sucient to fully describe the evolution of the system considered here. Observe that this class of DEDS is really general: any system where some servers provide service to dierent customers waiting in some queue is in the considered class. Then the state reconstruction algorithm presented in this section can apply to many dierent situations: production lines, computing systems, communication networks and so on. The following three subsections are intended as introductory to the issue of dy-