Perturbation Analysis and Optimization of Stochastic Flow Networks

Transcription

1 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. XX, NO. Y, MMM Perturbation Analysis an Optimization of Stochastic Flow Networks Gang Sun, Christos G. Cassanras, Yorai Wari, Christos G. Panayiotou, an George Riley Abstract We consier a Stochastic Flui Moel (SFM) of a network consisting of several single-class noes in tanem an perform perturbation analysis for the noe queue contents an associate event times with respect to a threshol parameter at the first noe. We then erive Infinitesimal Perturbation Analysis (IPA) erivative estimators for loss an buffer occupancy performance metrics with respect to this parameter an show that these estimators are unbiase. We also show that the estimators epen only on ata irectly observable from a sample path of the actual unerlying iscrete event system, without any knowlege of the stochastic characteristics of the ranom processes involve. This reners them computable in on-line environments an easily implementable for network management an optimization. This is illustrate by combining the IPA estimators with stanar graient base stochastic optimization methos an proviing simulation examples. Keywors Infinitesimal Perturbation Analysis, Stochastic Flui Moels, Non-linear Optimization. I. Introuction Stochastic Flui Moels (SFM) have recently been aopte as an alternative moeling paraigm to queueing networks for telecommunication applications, as well as other complex iscrete event systems. Introuce in [1] an then in [2] for the purpose of analysis, flui moels have also been consiere for simulation an control [3],[4],[5],[6],[7],[8],[9]. Using this moeling framework, a new approach for network congestion management has been propose, base on Infinitesimal Perturbation Analysis (IPA) [10],[11],[12],[13]. The cornerstone of this approach is the on-line estimation of graients (sensitivities) of certain congestion-relate performance measures (e.g., loss rates, average buffer levels) as functions of various controllable parameters. These graient estimates are use in conjunction with stanar stochastic approximation algorithms to optimize the parameter settings. As operating conitions change, the graient estimates change, therefore, this approach aims at continuously seeking to optimize The work of G. Sun an C.G. Cassanras was supporte in part by the National Science Founation uner Grants ACI , by AFOSR uner contract F , an by ARO uner grant DAAD The work of Y. Wari was supporte in part by the National Science Founation uner grant DMI an by DARPA uner contract F G. Sun an C.G. Cassanras are with the Dept. of Manufacturing Engineering, Boston University, MA gsun@bu.eu, cgc@bu.eu. Y. Wari is with the School of Electrical Engineering, Georgia Institute of Technology, Atlanta, GA. wari@ee.gatech.eu C.G. Panayiotou is with the Dept. of Electrical an Computer Engineering, University of Cyprus, Nicosia, Cyprus. christosp@ucy.ac.cy. G. Riley is with the School of Electr. an Computer Engin., Georgia Institute of Technology, Atlanta, GA. riley@ece.gatech.eu a generally time-varying performance metric. All work to ate has been limite to a single noe SFM. In this paper, we exten the approach to networks of noes connecte in tanem an, in the process, stuy how a buffer level perturbation in one noe in a network can propagate to other noes an how local congestion control may affect the rest of a network. To ate, many implementations of network control mechanisms have relie on ajusting traffic parameters (e.g., inflow rates) by monitoring an measuring certain performance measures (e.g., average buffer levels, elay jitter, an loss rates). Arguably, control algorithms that rely on both performance measures an their graients with respect to controllable parameters will perform better. In fact, some erivative-base congestion control algorithms have been propose in [14],[15]. Our approach is centere aroun the on-line estimation of such erivatives an it relies on the use of IPA. IPA has been evelope in the general setting of Discrete Event Dynamic Systems (DEDS), an queueing moels in particular. However, in the setting of queueing networks, IPA cannot usually provie unbiase graient estimators outsie the realm of simple moels with a single customer class, infinite buffers, an stateinepenent routing [16],[17]. These limitations exclue many telecommunication application features such as ifferentiate services, packet loss ue to buffer capacity limitations, an virtual-path routing. However, in the context of SFMs, as oppose to queueing systems, recent work [10] has shown that IPA graient estimators for important performance metrics are enowe with the following crucial properties: (i) They are unbiase, (ii) They are nonparametric, i.e., they are computable by expressions that are inepenent of the probability laws of the unerlying traffic processes, an (iii) They are extremely simple an easy to implement. The first property implies that the IPA graient estimators can be truste in performance preiction; the secon implies that the IPA estimators can be compute from fiel measurements instea of merely simulation environments; an the thir property points to the possibility of real-time computation. The use of IPA in single-noe SFMs has been stuie in [18],[10],[11],[12]. In [10], a SFM was aopte for a single traffic class network noe in which threshol-base buffer control is exercise. For the problem of etermining a threshol that minimizes a weighte sum of loss volume an buffer content, it was shown that IPA yiels remarkably simple nonparametric sensitivity estimators for this performance metric with respect to a threshol parameter, which, in aition, are unbiase uner very weak structural assumptions on the efining traffic processes. More-

2 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. XX, NO. Y, MMM over, a solution of the performance optimization problem base on the IPA-base approach outline above recovers or gives close approximations to the solution of the associate queueing moel. Extensions of the results erive in [10],[11] to general networks have ha to procee in two irections: the incorporation of multiple traffic classes an the analysis of general topology networks. The former irection has been pursue in [12],[19], where results analogous to those in [10] were obtaine. The latter irection is pursue in the present paper, whose primary focus is on tanem networks; some early results for the two-noe case may be foun in [20]. The main contributions are as follows. First, we consier a SFM consisting of M 2 single-class noes in tanem an evelop IPA erivative estimators of loss an buffer occupancy performance metrics with respect to a threshol parameter at the first noe. We show that these estimators are unbiase an iscuss their applicability to general-topology networks. Despite the inevitable burensome notation necessary to erive an analyze the estimators, their implementation is simple an rests on two intuitively appealing perturbation propagation rules : (i) A queue content perturbation at noe m propagates ownstream whenever the buffer at m becomes empty, an (ii) A perturbation at noe m is eliminate after its buffer becomes either empty or full. Finally, we emonstrate the use of the IPA estimators for network performance optimization purposes through simulation experiments. The rest of the paper is organize as follows. Section 2 presents the stochastic flow moeling framework for a network of noes in tanem. In Section 3 we carry out perturbation analysis of the network with respect to a threshol parameter at noe 1 an erive explicit IPA estimators for loss an queue content metrics. We also prove the unbiaseness of these estimators. Section 4 presents some simulation results illustrating the use of the estimators in network performance optimization. Section 5 conclues the paper an outlines relate ongoing work. II. Tanem Network SFM an Preliminary Results Consier a tanem network viewe as a Stochastic Flui Moel (SFM) as shown in Fig. 1 with M noes inexe by m = 1,..., M. The outflow of noe m is the inflow to noe m + 1, an we assume there is no feeback in the system. In the context of communication network applications, this implies that we limit ourselves here to network settings operating with protocols such as the User Datagram Protocol (UDP), but not the Transmission Control Protocol (TCP); the inclusion of feeback information that affects the incoming flow is a separate problem we aress elsewhere (see [21]) an it has not yet been incorporate in this multinoe analysis. Let b m enote the buffer size of noe m, m = 1,..., M, where b m > 0. At the first noe, we consier the buffer size as a controllable parameter; equivalently, we view it as a threshol enote by θ = b 1 which is ajustable for the purpose of congestion control. We will assume that the real-value parameter θ is confine to a close an boune (compact) interval Θ. The inflow rate of each noe m = 2,..., M is enote by α m (θ; t), to inicate the fact that it generally epens on θ, whereas α 1 (t) is an external process inepenent of θ. The processing rate of noe m = 1,..., M at time t is enote by β m (t) an is inepenent of θ. The buffer level is enote by x m (θ; t), the outflow rate is enote by δ m (θ; t) an the overflow rate is enote by γ m (θ; t). The external processes {α 1 (t)} an {β m (t)}, m = 1,..., M, which are inepenent of θ, can have a very general form for the purpose of our analysis; in particular, they nee not be statistically inepenent. We are intereste in stuying sample paths of this SFM over a time interval [0, T ] for a given fixe 0 < T <. The ynamics of the buffer level x m (θ; t), m = 1,..., M, are escribe by the following one-sie ifferential equation: x m (θ; t) t + = 0, if x m (θ; t) = 0 an α m (θ; t) β m (t) 0, 0, if x m (θ; t) = b m an α m (θ; t) β m (t) 0, α m (θ; t) β m (t), otherwise. (1) where, to maintain uniformity in the notation, it is unerstoo that α 1 (θ; t) = α 1 (t). With this convention in min, the outflow rate from noe m = 1,..., M 1 is the inflow rate to the ownstream noe m + 1, so that for all m = 2,..., M we have α m (θ; t) = { βm 1 (t), if x m 1 (θ; t) > 0 α m 1 (θ; t), if x m 1 (θ; t) = 0. (2) Finally, the overflow rate γ m (θ; t) at noe m ue to a full buffer is efine by α m (θ; t) β m (t), if x m (θ; t) = b m an γ m (θ; t) = α m (θ; t) β m (t) 0, 0, otherwise. (3) For convenience, we efine A m (θ; t) := α m (θ; t) β m (t). (4) We stress again that in this SFM the flow rates {α 1 (t)} an {β m (t)}, m = 1,..., M, are treate as stochastic processes representing the ranom instantaneous rates of the arriving traffic an of the noe processing rates. This is why in consiering a typical sample path of the SFM (as in Fig. 2) the buffer content is shown not as piecewise linear (which correspons to fixe flow rates over specific intervals), but only as piecewise analytic. α1(t) γ1(t) θ β1(t) α2(t) γ2(t) b2 Fig. 1. β2(t) System Moel αm(t) γm(t) bm βm(t)

3 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. XX, NO. Y, MMM Let us now take a closer look at (2) which escribes the only connection between noe m an its upstream noes. The value of α m (θ; t), m > 1, is given by either β m 1 (t), which is inepenent of θ, or by α m 1 (θ; t). In turn, the value of α m 1 (θ; t) is given by either β m 2 (t) or by α m 2 (θ; t). Proceeing recursively, we fin that the value of α 2 (θ; t) is either β 1 (t) or α 1 (t) which are both inepenent of θ. Thus, the value of α m (θ; t) is ultimately given by one of the processes {α 1 (t)} an {β i (t), i = 1,..., m} which are all inepenent of θ; the way in which α m (θ; t) switches among them epens on θ through the states x i (θ; t), i = 1,..., m 1 an the points in time when this switching occurs efines the switchover points iscusse in the sequel. Focusing on noe m, the inflow process {α m (θ; t)} an the service process {β m (t)} are referre as efining processes of noe m, since they efine the local ynamics at that noe. The buffer level {x m (θ; t)}, outflow process {δ m (θ; t)} an overflow process {γ m (θ; t)} are referre as erive processes, since they can be erive from the efining processes via (1)-(3). Viewing the network as a iscrete event system, the SFM ynamics are epenent on a number of events. For the purpose of our analysis, we efine an event of noe m = 1,..., M to be one of the following: e 1 - A jump (iscontinuity) in either α m (θ; t) or β m (t). e 2 - A time instant when A m (θ; t) becomes 0 with no iscontinuity in A m (θ; t) at t. e 3 - A time instant when the buffer level x m (θ; t) becomes full or empty. Two types of sample performance metrics will be consiere throughout this paper, both over the time interval [0, T ]. The loss volume at noe m = 1,..., M, enote by L m (θ; T ), is efine by L m (θ; T ) = T 0 γ m (θ; t)t, (5) an the work at noe m = 1,..., M, enote by Q m (θ; T ), is efine by Q m (θ; T ) = T 0 x m (θ; t)t. (6) IPA provies the erivatives (graient) of the sample performance functions with respect to various control parameters. In our case, we concentrate on the erivatives L m(θ; T ) an Q m(θ; T ), where we shall use the prime notation to enote a erivative with respect to θ throughout the rest of the paper. Consiering a typical sample path of the buffer level x m (θ; t) in this SFM, as shown in Fig. 2, we observe that it can be ecompose into Bounary Perios (BP) an Non-Bounary Perios (NBP). A BP is one uring which x m (θ; t) = 0 or x m (θ; t) = b m, whereas a NBP is one uring which 0 < x m (θ; t) < b m. A BP is further categorize as either an Empty Perio (EP) uring which x m (θ; t) = 0 or as a Full Perio (FP) uring which x m (θ; t) = b m. Since the function x m (θ; t) is generally continuous in t for a fixe θ, we will consier EPs an FPs to be close intervals an NBPs to be open intervals in the relative topology inuce by [0, T ]. Let B m,n = [τ m,n (θ), σ m,n (θ)] enote the nth BP, n = 1,..., N m, where N m is the total (ranom) number of BPs in [0, T ]. Note that the start of B m,n, τ m,n (θ), is an e 3 event of noe m. For notational economy, we will omit θ in τ m,n (θ) an σ m,n (θ) in what follows, but will keep in min that τ m,n an σ m,n are generally functions of θ. Next, observe that NBPs an BPs appear alternately throughout [0, T ] an let B m,n = (σ m,n 1, τ m,n ) enote the NBP that precees B m,n. For convenience, we shall set σ m,0 = 0 an σ m,nm = T. Depening on the value of x m (θ; t) at the starting an ening points of a NBP B m,n = (σ m,n 1, τ m,n ), we can efine four types of NBPs ( E stans for Empty an F stans for Full ): 1. (E, E): x m (θ; σ m,n 1 ) = 0 an x m (θ; τ m,n ) = (E, F ): x m (θ; σ m,n 1 ) = 0 an x m (θ; τ m,n ) = b m. 3. (F, E): x m (θ; σ m,n 1 ) = b m an x m (θ; τ m,n ) = (F, F ): x m (θ; σ m,n 1 ) = b m an x m (θ; τ m,n ) = b m. In the example shown in Fig. 2, the BPs B m,n 1 = [τ m,n 1, σ m,n 1 ], an B m,n = [τ m,n, σ m,n ] are both FPs, whereas B m,n+1 = [τ m,n+1, σ m,n+1 ] is an EP. The NBP B m,n 1 = (σ m,n 2, τ m,n 1 ) is of type (E, F ), B m,n = (σ m,n 1, τ m,n ) is of type (F, F ), B m,n+1 = (σ m,n, τ m,n+1 ) is of type (F, E), an B m,n+2 = (σ m,n+1, τ m,n+2 ) is of type (E, E). b m σ m,n-2 τ m,n-1 σ m,n-1 τ m,n σ m,n τ m,n+1 σ m,n+1 τ m,n+2 Fig. 2. Typical Sample Path of Noe m The switchover points of α m (θ; t) for m > 1, as seen in (2), occur as follows: (i) Just before an EP of noe m 1 starts, we have α m (θ; t) = β m 1 (t). When the EP starts, the output of m 1 switches from β m 1 (t) to α m 1 (θ; t). (ii) When the EP of noe m 1 ens, the output of m 1 switches once again from α m 1 (θ; t) to β m 1 (t). (iii) The thir instance is less obvious. During the EP at noe m 1, it is possible that an EP at noe m 2 starts, in which case α m 1 (θ; t) switches from β m 2 (t) to α m 2 (θ; t). When this happens, the output of m 1 switches from α m 1 (t) to α m 2 (θ; t), therefore, α m (θ; t) = α m 1 (t) = α m 2 (t). Clearly, it is possible that a sequence of j such events occurs so that α m (θ; t) = α m 1 (t) =... = α m j (t), where j = 1,..., m 1. In this case, all noes m j,..., m 1 are empty an m inherits all switchovers experience by these upstream noes as each one starts an EP. For switchover points of α m (θ; t) uner case (ii) above, we next prove that they are locally inepenent of θ.

4 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. XX, NO. Y, MMM Lemma II.1: Let σ m 1, m > 1, be a switchover point of α m (θ; t) with α m (θ; σm 1 ) = α m 1(θ; σm 1 ) an α m (θ; σ m 1 + ) = β m 1(σ m 1 + ). Then, σ m 1 is locally inepenent of θ. It immeiately follows from Lemma II.1 that the en of an EP is inepenent of θ. Moreover, for m > 2, uring an EP of noe m 1 we can see in (2) that α m (θ; t) = α m 1 (θ; t), which implies that if a switchover occurs at α m 1 (θ; t), this switchover will be inherite by α m (θ; t), as well as the θ-epenence of it. This iscussion motivates our efinition of an active switchover point, which is generally a function of θ an is enote by s m,i (θ), m > 2, i = 1, 2,...: Definition 1. A switchover point of α m (θ; t) is terme active, if: 1. s m,i (θ) is the time when an EP at noe m 1 starts; or 2. s m,i (θ) is the time when α m 1 (θ; t) experiences an active switchover within an EP of noe m 1. In Fig. 2, assuming m > 2, the points τ m,n+1 an τ m,n+2 both start EPs an are, therefore, active switchover points of α m+1 (θ; t). In aition, any point in [τ m,n+1, σ m,n+1 ] is potentially an active switchover point of α m+1 (θ; t) if it happens to be an active switchover point of α m (θ; t). An active switchover point s m,i (θ) at noe m may belong to a BP B m,n or to a NBP B m,n. We efine the following inex sets that will help ifferentiating between ifferent types of active switchover points epening on the type of interval they belong to: Ψ m,n := {i : s m,i B m,n } (7) Ψ o m,n := {i : s m,i (τ m,n, σ m,n )} (8) Ψ m,n := { } i : s m,i B m,n (9) Note that B m,n = [τ m,n, σ m,n ], so we ifferentiate between open an close intervals that efine BPs in efining the sets Ψ m,n an Ψ o m,n. As we will see, of particular interest are active switchover points that coincie with the en of a FP, so we efine the set of all BP inices that inclue such a point, Φ m, as well as Γ m Φ m, a subset that inclues those FPs that are followe by a NBP of type (F, E): Φ m := {n : σ m,n is an active switchover point, n = 1,..., N m } (10) Γ m := {n : n Φ m an B m,n+1 is of type (F, E)}. (11) III. Infinitesimal Perturbation Analysis (IPA) Our objective is to estimate the erivatives of the performance metrics E[L m (θ; t)] an E[Q m (θ; t)], where L m (θ; t) an Q m (θ; t) were efine in (5) an (6), through the sample erivative L m(θ; T ) an Q m(θ; T ), which is commonly referre to as the Infinitesimal Perturbation Analysis (IPA) estimators; comprehensive iscussions of IPA an its applications can be foun in [16],[17]. The IPA erivative-estimation technique computes the sample erivative L T (θ) of some performance metric L T (θ) along an observe sample path ω. An IPA-base estimate L T (θ) of a performance metric erivative E[L T (θ)]/ θ is unbiase if E[L T (θ)]/ θ = E[L T (θ)]. Unbiaseness is the principal conition for making the application of IPA useful in practice, since it enables the use of the sample (IPA) erivative in control an optimization methos that employ stochastic graient-base techniques. The case of a single noe where we are intereste in L 1(θ; T ) an Q 1(θ; T ) has been stuie in [10], so here we aress the inter-noe effects an stuy the resulting IPA estimators L m(θ; T ) an Q m(θ; T ) for m > 1. Due to the tanem topology an the absence of feeback between noes, the inter-noe effects have only one irection: from upstream to ownstream. Therefore, our analysis is base on the impact of the threshol parameter at the first noe on performance metrics at the ownstream noes. Since we are concerne with the sample erivatives L m(θ; T ) an Q m(θ; T ) we have to ientify conitions uner which they exist. As we will see, these erivatives epen on the erivatives of the active switchover points, i.e., specific event times, with respect to θ. Excluing the possibility of the simultaneous occurrence of two events (e 1, e 2, or e 3 as efine earlier), the only situation preventing the existence of these erivatives involves some t such that A m (θ; t) = α m (θ; t) β m (t) = 0; in such cases, the one-sie erivatives exist an can be obtaine through a finite ifference analysis (as in [10]). However, to keep the analysis simple, we focus only on the ifferentiable case by proceeing uner the following technical conitions: Assumption 1. a. W.p.1, the functions α 1 (t), an β m (t), m = 1,..., M are piecewise analytic in the interval [0, T ]. b. For every θ Θ, w.p.1 no two events of a certain noe m occur at the same time. c. W.p.1, no two processes {α 1 (t)}, {β m (t), m = 1,..., M} have ientical values uring any open subinterval of [0, T ]. All three parts of Assumption 1 are mil technical conitions. Regaring part c, note that α m (θ; t), through (2), ultimately epens on one or more of the processes {α 1 (t)}, {β i (t)}, i = 1,..., m, therefore the requirement A m (θ; t) 0 is reflecte by the general statement uner c. Recall that a switchover point of α m (θ; t) is the time it switches among {α 1 (t)} an {β i (t)}, i = 1,..., m. It is possible that a switchover may not cause a jump (iscontinuity) in α m (θ; t); for example, at t = s, α m (θ; t) switches from α m 1 (θ; t) to β m 1 (t) while α m 1 (θ; s) = β m 1 (s) an such a switchover is not qualifie as a noe m event (e 1, e 2, or e 3 as efine earlier). The following lemma is a consequence of Assumption 1 an shows that for an active switchover point, α m (θ; t) must experience a jump. Recall that an active switchover point s m,i (θ) is generally a function of θ, but, for the sake of notational simplicity, we shall simply write s m,i. Lemma III.1: If an active switchover point of α m (θ; t) occurs at t = s m,i, then w.p. 1 it is an e 1 event of noe m.

5 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. XX, NO. Y, MMM A. Queue Content Derivatives We shall procee by etermining the erivative x m(θ; t) of a buffer level in the SFM with respect to the controllable parameter θ an will show that it epens exclusively on the way that θ affects the switchover points of α m (θ; t) that were terme active in Definition 1. Focusing on active switchover points s m,i, i = 1, 2,... we efine the following two quantities for m > 1 that turn out to be crucial in our analysis: an, for n Φ m : ψ m,i := [α m (θ; s + m,i ) α m(θ; s m,i )]s m,i, (12) φ m,n := [α m (θ; σ + m,n) β m (σ m,n )]σ m,n. (13) Let us now consier the erivative x m(θ; t) of a buffer level in the network with respect to the controllable parameter θ. The case m = 1 was consiere in [10], so we shall focus on cases with m > 1. The following establishes the connection between x m(θ; t) an the two crucial quantities efine above. Note that 1 [ ] is the usual inicator function. Lemma III.2: If m = 1, for n = 1,..., N 1 { x 1 1(θ; t) = 0 If m > 1, then for n = 1,..., N m if t B 1,n or t B 1,n+1, x 1 (θ; σ 1,n ) = θ otherwise (14) x m(θ; t) = { 0 if t Bm,n K m,n(t) ψ m,k 1 [n Φ m ] φ m,n if t B m,n+1 (15) where K m,n (t) is the number of active switchover points in the interval (σ m,n, t) B m,n+1. It is now clear from (15) that ψ m,k an φ m,n are crucial quantities associate with noe m. In the next two lemmas, we show that they provie the means to connect x m(θ; t) to x m 1(θ; t) an hence she light into the way in which buffer level perturbations propagate across noes. Lemma III.3: For m > 1, let s m,i be an active switchover point of α m (θ; t). If it is the start of an EP at noe m 1, then ψ m,i = x m 1(θ; s m,i ) (16) Otherwise, if s m,i occurs uring an EP of noe m 1, then for some j such that s m,i = s m 1,j. Next for m > 1, we efine: ψ m,i = ψ m 1,j (17) R m,n (θ) := α m(θ; σ + m,n) β m (σ m,n ) α m (θ; σ + m,n) α m (θ; σ m,n) (18) By efinition, σ m,n is the en of a BP at noe m. We will make use of R m,n (θ) when n Φ m, i.e., when σ m,n happens to be an active switchover point. If this is the case, then it follows from Lemma III.1 an Assumption 1(b) that β m (t) is continuous at t = σ m,n. Note that this quantity involves the processing rate information β m (σ m,n ) (typically known, otherwise measurable) at t = σ m,n, an the values of the incoming traffic rates before an after a BP ens at t = σ m,n. Using this efinition, the next lemma allows us to obtain a simple relationship between the two crucial quantities ψ m,i an φ m,n. Lemma III.4: Let n Φ m an σ m,n = s m,i for some active switchover point of α m (θ; t). Then, φ m,n = R m,n (θ) ψ m,i (19) where 0 < R m,n (θ) 1. (20) Combining Lemmas III.2-III.4 we obtain the following: Theorem III.1: For m > 1 an n = 1,..., N m : x m(θ; t) = 0 if t B m,n Km,n (t) x m i (θ; s m,k ) + 1 [n Φ m ] R m,n (θ)x m i (θ; σ m,n) if t B m,n+1 (21) where i := min {j : x m j(θ; s m,k ) > 0} (22) j=1,...,m 1 an K m,n (t) is the number of active switchover points in the interval (σ m,n, t) B m,n+1. Taking a closer look at (21) we get significant insight regaring the process through which changes in the buffer level of one noe affect the buffer levels of ownstream noes. Let us view x m(θ; t) as a perturbation in x m (θ; t). For simplicity, let us initially ignore the case where n Φ m an assume i = 1. Thus, we have x m(θ; t) = Km,n (t) x m 1(θ; s m,k ) if t B m,n+1. We can see that noe m 1 only affects noe m at time s m,k when an EP at noe m 1 starts (recalling Definition 1). In simple terms: whenever noe m 1 becomes empty, it propagates ownstream to m its current perturbation. These perturbations accumulate at m over all K m,n (t) active switchover points containe in a NBP B m,n+1. For example, in Fig. 3, s m,i+1 is a point where an EP ens at noe m 1 while noe m is in a NBP; at that time we get x m(θ; t) = x m 1(θ; s m,i+1 ). Moreover, when the NBP ens at τ m,n+1, the value of x m(θ; τm,n+1 ) is in turn propagate ownstream to m + 1, before setting x m(θ; τ + m,n+1 ) = 0 at the start of the ensuing EP at m. Any cumulative perturbation at m is eliminate by the presence of any BP, i.e., when t B m,n as inicate by (21). For example, in Fig. 3, s m,i is a point where an EP ens at noe m 1 while noe m is in a FP; therefore, it has no effect on x m (θ; t), i.e., x m(θ; t) = 0. The conclusion is that in orer for a noe to have a chance to propagate a

6 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. XX, NO. Y, MMM b m-1 s m,i-1 s m,i s m,i+1 concentrate on the sample erivatives of the two performance metrics we have ientifie, L m (θ; T ) an Q m (θ; T ) efine in (5) an (6). The case of L 1 (θ; T ) an Q 1 (θ; T ) was consiere in [10], so we will focus on m > 1 in what follows. b m σ m,n-1 τ m,n σ m,n τ m,n+1 Fig. 3. A sample path example with two ajacent noes an three active switchover points for α m(θ; t) perturbation ownstream, it must become empty before it becomes full. In view of this fact, we can argue that control at the ege of a tanem network is generally expecte to have a limite impact on noes that are several hops away, since propagating perturbations requires the combination of several events: a perturbation to be present an to be propagate at the start of an EP before it is eliminate by a FP; moreover this has to be true for a sequence of noes. The probability of such a joint event is likely to be small as the number of hops increases. This provies an analytical substantiation to the conjecture that congestion in a network cannot be easily regulate through control exercise several hops away, unless the intermeiate noes experience frequent EPs proviing the opportunity for perturbation propagation events. Let us now look at the two aspects that were ignore in the iscussion above. First, suppose that i > 1. This means that an EP occurs not just at noe m 1, but also noes m 2,..., m i, all at the same time. Thus, instea of propagating a perturbation from m 1 to m, the propagation now takes place from m i to m. Secon, let us consier the case where n Φ m in (21). This allows an EP that starts at m 1 to cause the en of a FP at noe m. When this occurs, only a fraction, given by R m,n (θ), of the perturbation at m 1 is propagate to noe m. For example, in Fig. 3, the point s m,i coincies with σ m,n an it therefore contributes another term scale by R m,n as seen in (21). Finally, note that the iscussion above is inepenent of the way in which the controllable parameter affects the buffer content at m = 2 an subsequently all ownstream noes through (21). In the particular case we are consiering, however, we can see from (14) that the erivatives at noe 1 are always given by 1. Thus, the entire perturbation analysis process here reuces to counting EP events at all noes that cause propagations through (21). The only exception is for those events that en an EP at some m 1 an at the same time a FP at m; in this case, the erivative at noe m is affecte by some amount epenent on R m,n (θ) (0, 1].Up to this point, we have characterize the mechanism through which x m(θ; t) can be evaluate recursively for all m = 1,..., M, making use of the quantities ψ m (s m,i ) an φ m (s m,i ). In the next two sections, we B. The IPA Derivative L m(θ; T ) Our objective here is to estimate the erivative of the expecte loss volume E[L m (θ; T )] at noe m = 2,..., M through the sample erivative L m(θ; T ). Let us efine Ϝ m to be the set of all inices of BPs that happen to be FPs at noe m over [0, T ], i.e., Ϝ m := {n : x m (θ; t) = b m for all t B m,n, n = 1,..., N m }. Observing that only FPs at noe m will experience loss, we have an L m (θ; T ) = L m(θ; T ) = σm,n τ m,n γ m (θ; t)t, σm,n γ m (θ; t)t. (23) θ τ m,n By Lemma III.1 an Assumption 1(b), τ m,n cannot be an active switchover point, since at τ m,n a noe m event of type e 3 must occur. Therefore, for any n Ϝ m, active switchover points can occur either in the open FP interval (τ m,n, σ m,n ) or they may coincie with the en of the FP at time σ m,n. To establish an expression for L m(θ; T ) in terms of observable sample path ata we nee three preliminary results, state below as Lemmas III.5-III.7. Since we focus on noe m, we rop the subscript m for notational convenience in presenting these results. Lemma III.5: For n Ϝ, σn [ ] γ(θ; t)t = A(θ; σn )σ n A(θ; τ n )τ n ψ k θ τ n k Ψ o n (24) Lemma III.6: For n Ϝ, A(θ; τ n )τ n(θ) = ψ k + A(θ; σ n 1 + )σ n 1 (25) k Ψ n The next result concerns the en point σ n of a FP. Lemma III.7: For n Ϝ, { [A(θ; σ n + ) A(θ; σn )]σ ψi, if n Φ with σ n = n = s i 0, if n / Φ (26) We can now obtain the IPA erivative L m(θ; T ), using once again the subscript m. We will also introuce the set Ω m,n = Ψ m,n Ψ m,n (27)

7 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. XX, NO. Y, MMM which, recalling (7) an (9), inclues the inices i of all active switchover points in the BP B m,n = [τ m,n (θ), σ m,n (θ)] an the NBP that precees it B m,n = (σ m,n 1, τ m,n ). Theorem III.2: The loss volume IPA erivative, L m(θ; T ), m = 2,..., M, has the following form: L m(θ; T ) = i Ω m,n ψ m,i + φ m,n (28) where ψ m,i an φ m,n are given by (16)-(17) an (19). In simple terms, to obtain L m(θ; T ) we accumulate terms ψ m,i over all active switchover points s m,i for each interval (σ m,n 1, σ m,n ], n = 1, 2,... However, the result contributes to L m(θ; T ) only if σ m,n ens a FP. The secon term of (28) moifies the accumulation process as follows: Occasionally, σ m,n is followe by a NBP (σ m,n, τ m,n+1 ) of type (F, E), i.e., the buffer at noe m becomes empty. When this event takes place, the contribution ψ m,i for s m,i = σ m,n is moifie by aing φ m,n to it. In the example shown in Fig. 3, there are two active switchover points in the interval (σ m,n 1, σ m,n ] at s m,i 1 an at s m,i. These contribute terms ψ m,i 1 an ψ m,i to L m(θ; T ) since the BP that ens at σ m,n is a FP. The secon one happens to coincie with the en of the FP, i.e., s m,i = σ m,n. Since the next NBP is of type (F, E), we have n Γ m an a term φ m,n is contribute to L m(θ; T ). In aition, the active switchover point at s m,i+1 oes not contribute to L m(θ; T ). The terms ψ m,i an φ m,n are given in Lemmas III.3 an III.4, where we can see that they epen on the erivatives x m 1(θ; s m,i ) propagate from the upstream noe m 1 through every EP event that occurs at m 1. These erivatives are in turn provie by (21) in Theorem III.1. We emphasize the fact that, as in earlier work for a single noe SFM [10], the IPA estimator oes not involve any knowlege of the stochastic processes characterizing arriving traffic or noe processing an allows for the possibility of correlations. The only information involve is the one require to calculate R m,n in (21), which, incientally, occurs only when the en of a FP happens to be an active switchover point; one can argue that uner certain loaing conitions such contributions (recall also that 0 < R m,n 1) are minimal an coul be ignore for the benefit of obtaining computationally efficient approximations; in this case, (28) becomes a simple counter, since the values of ψ m,i are originally given by 1 at noe 1, as seen in (14). This is further iscusse in Section 4. Theorem III.3: The IPA erivative, L m(θ; T ), m = 2,..., M, is unbiase, i.e., Proof: [ ] E L m(θ; T ) = E[L m(θ; T )] θ See Appenix II. C. The IPA Derivative Q m(θ; T ) Recall the efinition of Q m (θ; T ) in (6). By partioning [0, T ] into NBPs an BPs an recalling that N m was efine as the total number of BPs in [0, T ], we have Q m (θ; T ) = N m n=1 [ τm,n σ m,n 1 x m (θ; t)t + σm,n τ m,n x m (θ; t)t Upon taking erivatives with respect to θ an in view of the fact that x m (θ; t) is continuous in t, we obtain Q m(θ; T ) = N m τm,n n=1 N m n=1 N m σm,n n=1 N m n=1 σ m,n 1 x m(θ; t)t { } x m (θ; τ m,n )τ m,n x m (θ; σ m,n 1 )σ m,n 1 τ m,n x m(θ; t)t { } x m (θ; σ m,n )σ m,n x m (θ; τ m,n )τ m,n After taking into account the cancellation of several terms an in view of the fact that σ m,0 = σ m,n m = 0, this reuces to [ N m τm,n ] σm,n Q m(θ; T ) = x m(θ; t)t + x m(θ; t)t. n=1 σ m,n 1 τ m,n (29) We can now make use of the expression for x m(θ; t) erive in Lemma III.2 an Theorem III.1 to obtain the IPA estimator Q m(θ; T ) for m = 2,..., M. Theorem III.4: The workloa IPA erivative, Q m(θ; T ), m = 2,..., M, has the following form: Q m(θ; T ) = N m [τ m,n s m,i ]ψ m,i n=1 i Ψ m,n [τ m,n+1 σ m,n ]φ m,n (30) n Φ m where ψ m,i an φ m,n are given by (16)-(17) an (19). For a simple interpretation of the IPA estimator (30), note that, similar to the IPA estimator in (28), it involves accumulating terms ψ m,i over active switchover points s m,i. In this case, however, we are only intereste in s m,i containe in NBPs (σ m,n 1, τ m,n ), n = 1,..., N m. The accumulation is one at τ m,n with each such term scale by [τ m,n s m,i ] measuring the time elapse since the switchover point took place. The secon term in (30) as similar contributions mae at the en of a NBP of type (F, E) ue to active switchover points that coincie with the en of a FP at some time σ m,n. Theorem III.5: The IPA erivative, Q m(θ; T ), m = 2,..., M, is unbiase, i.e., Proof. [ ] E Q m(θ; T ) = E[Q m(θ; T )] θ See Appenix II. ]

8 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. XX, NO. Y, MMM IV. Experimental Network Optimization Results This section presents results of simulation experiments in which we optimize a weighte sum of loss an workloa in the two-queue tanem system shown in Fig. 4, as a function of the buffer limits (buffer sizes) at the two queues. All of the experiments were performe using the Georgia Tech Network Simulator (GTNetS) [22], moifie to inclue the requisite IPA erivative calculations. λs n1 Sources Backgroun Generator Fig. 4. λ b β 1 β2 θ 1 S1 θ 2 S2 Topology: Two-Stage Simulations n Sinks The approach we have taken here is to purposefully aopt a very practical engineering point of view in trying to integrate the analytical results of the previous section with a stochastic optimization methoology. We have mae several simplifications, our goal being to test the practical value of using IPA estimates to ynamically improve (in an acceptable time scale) network performance within an optimization framework. First of all, because of the simple form of the IPA estimators of the erivatives of loss (28) an work (30) for the SFM, all ata require for their evaluation can be irectly obtaine from a sample path of the actual queueing system, as was also one an explaine in etail in our earlier work [10]. In other wors, the form of the IPA estimators is obtaine by analyzing the system as a SFM, but the associate values are base on real ata. This provies a goo approximation of the performance erivative estimates of the queueing system (which, if obtaine irectly from the queueing system, woul be biase). Seconly, we implemente a stanar stochastic approximation technique (e.g., see [23]) in conjunction with the IPA erivatives obtaine in Section 3, but inclue some simple heuristics that are empirically known to accelerate convergence, at the expense of staying within the bouns of the usual technical conitions require to guarantee convergence. In aition, although all our analysis is base on the assumption that all observe sample paths start with all queues at the empty state, we have nonetheless applie the IPA estimates at the nth iteration of the optimization algorithm using the ening state of the (n 1)th iteration. The final simplifying step we have taken concerns the contribution of the term involving φ m,n in the IPA estimators (28) an (30). As alreay argue, base on (14), (16)- (17), an (19)-(20), each instance of this term is boune by [0, 1]. Moreover, the term is nonzero only when an active switchover point coincies with the en of a FP at noe 2, i.e., an EP starts at noe 1 causing a FP to en at noe 2. This is likely to occur only when the buffer limits are largely imbalance (that of noe 2 is too small), in which case the performance sensitivity with respect to the buffer limit of noe 2 is expecte to be large (hence, the buffer limit of noe 2 will be increase at the next algorithm iteration), making the contribution of a term boune by [0, 1] likely to be negligible. Since this argument is obviously not rigorous, we proceee by performing the optimization process twice: once with all these terms ignore, an once with the values of these terms, whenever they arise, set to their maximum value of 1. We foun the results numerically inistinguishable, substantiating this approximation. The significance of the approximation cannot be overemphasize: without the inclusion of the term involving φ m,n in the IPA estimators, these estimators are fully nonparameteric, i.e., they require only simple event counters an timers an no traffic rate information whatsoever, since R m,n in (18) is no longer involve. In the system of Fig. 4, intene to represent the operation of a communication network, the inflow process at the first queue consists of n 1 multiplexe on off ata sources generating bursty traffic. When in the on state, each source generates a continuous ata stream at the rate of α bits per secon. These ata streams are use to construct 554-byte UDP packets which are forware to the buffer at the first queue an thence across the rest of the network. The on times an off times are ii ranom variables sample from the exponential istribution with mean 0.1 secons. The channel transporting packets from the first queue to the secon queue has a capacity of β 1 bps. The inflow process to the secon queue consists of the outflow process from the first queue an of traffic from the backgroun generator. The backgron traffic consists of n 2 inepenent sources. Each one of these sources has the same statistical characteristics as the sources to the first queue. The outgoing channel from the secon queue has a capacity of β 2 bps. Note that the average bit rate from either one of the inepenent sources is α/2 bps, since the expecte urations of the off perios an the on perios are ientical. Therefore, the expecte bit rate of the aggregate flow to the first queue is (n 1 α/2) (554/512), where the latter term accounts for the insertion of the heaers. Consequently, the traffic intensity at the first queue, enote by ρ 1, is given by ρ 1 = n 1 α (31) β 1 Similarly, the traffic intensity of the secon queue is enote by ρ 2. In our simulation experiments we set n 1 = n 2 = 100, β 1 = 10 Mbps, an β 2 = 20 Mbps. Our simulation program was esigne to utilize the traffic intensities as simulation input, an we set ρ 1 = ρ 2 = The program then calculate α accoring to (31). Let θ = (θ 1, θ 2 ) enote the two-imensional parameter vector consisting of the buffer limits at the first an secon queue respectively. Recall that the loss volumes an workloas at the two queues are enote by L j (θ; T) an Q j (θ; T), j = 1, 2. Let us efine the cost function F (θ; T) as the following weighte sum of the average loss rate an

9 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. XX, NO. Y, MMM workloa rate. F (θ; T) = 1 T [L 1(θ; T) + 10Q 1 (θ; T) +L 2 (θ; T) + 20Q 2 (θ; T)]. (32) We set the value of T to T = 1. We seek to minimize this function using a stanar stochastic approximation technique (e.g., see [23]) in conjunction with the IPA erivatives obtaine in Section 3. The optimization algorithm iteratively computes a sequence of points, θ(i) = (θ(i) 1, θ(i) 2 ), i = 1, 2,...,. Its basic iteration step has the form: θ(i + 1) = θ(i) ζ(i)h(i), (33) where ζ(i) 0 is the ith stepsize (we aopte ζ(i) = 10/i 0.6 ), an h(i) is an estimate of the graient of F (θ(i); T) obtaine via IPA. As alreay pointe out, because of the simple form of the IPA estimators (28) an (30) for the SFM, all ata require for their evaluation can be irectly obtaine from a sample path of the actual queueing system. In aition, we use a simple heuristic to boun the isplacement θ(i + 1) θ(i) along each coorinate by moifying the vector h(i) = (h(i) 1, h(i) 2 ) as follows. We first compute the partial erivatives F (θ(i);t) F (θ(i);t) θ(i) j, j = 1, 2. If ζ(i) 5 then we set h(i) j = F (θ(i);t) θ(i) j h(i) j = 5sgn( F(θ(i);T) θ(i) j, an if ζ(i) θ(i) j F (θ(i);t) θ(i) j > 5 then we set )/ζ(i). The parameters θ(i) j (j = 1, 2) were consiere as real numbers, but the simulation runs were performe at the respective integer values closest to them. Recall that the simulation time horizon at each iteration point θ(i) was T = 1.0 secon. The simulation state at the en of each iteration was preserve, an use as the initial state for the simulation at the next iteration point, θ(i + 1). Likewise, we preserve the final state of the process of computing the IPA erivative, an use it as the initial state for the IPA erivative process at the next iteration. Note that only one ranom see is calle for each optimization experiment. Queue 1 Limit Fig. 5. Time Initial Limit 5 Initial Limit 40 θ(i) 1 Ajustments vs. Time We ran the optimization algorithm twice, with two ifferent initial parameters: first with θ(1) = (5, 5), an then Queue 2 Limit Fig. 6. Time Initial Limit 5 Initial Limit 40 θ(i) 2 Ajustments vs. Time with θ(1) = (40, 40). In either case we ran the algorithm for 100 iterations (i.e., 100 secons). For each experiment, we plotte the evolution of θ(i) 1 an θ(i) 2 as a function of i, an show the results in Figs. 5 an 6 respectively. Each of the figures shows one trajectory for the θ(1) = (5, 5) initial conition, an a secon one for the θ(1) = (40, 40) initial conition. The results inicate asymptotic convergence to approximately ˆθ = (15, 14) within approximately 20 secons. As alreay mentione, this optimization process was performe without the term involving φ m,n in the IPA estimators (28) an (30); it was then repeate with the inclusion of this term set to 1 (its upper boun) in all instances when it arises an the results obtaine corresponing to Figs. 5 an 6 were numerically inistinguishable Q2 Limit Q1 Limit Fig. 7. Cost Function F (θ 1, θ 2 ; T ) Finally, to a valiity to these results, we plotte the graph of F (θ 1, θ 2 ; T ) as shown in Fig. 7. Each point on the plot is the average of 10 separate simulation experiments with T = 100 secons, each with a ifferent see for the ranom number generators. However, each set of the 10 simulation experiments uses the same set of 10 ranom sees as all other sets of experiments. This graph clearly

10 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. XX, NO. Y, MMM corroborates the results obtaine by the optimization runs, i.e., it shows that ˆθ = (15, 14) is inee optimal. V. Conclusions an Future Work We have consiere in this paper a Stochastic Flow Moel (SFM) for a communication network of multiple noes in tanem. Our objective is to control threshol parameters at network noes so as to optimize performance capture by combining loss an workloa metrics. We have evelope IPA estimators for these metrics with respect to the threshol an shown them to be unbiase. The simplicity of the estimators erive an the fact they are not epenent on knowlege of the traffic arrival or service processes makes them attractive for on-line control an optimization. This work has extene results applicable to a single-noe, single-class SFM in [10], an the next step is to incorporate multiple traffic classes at various noes, along the lines of [12]. Our ongoing work is also investigating the use of this approach in general topology networks, which we believe to be possible. For example, the presence of cross-traffic at noe m in our SFM can be capture by varying the processing rate β m (t) at that noe. Finally, an very importantly, ongoing work is also consiering how to evelop IPA an relate control an optimization methos that inclue network feeback effects (i.e., allowing arriving traffic processes to epen on the buffer content in ifferent ways); some relate initial results are reporte in [21]. Appenix I Proof of Lemma II.1 Recalling (2), we have x m 1 (θ; σm 1 ) = 0 an x m 1 (θ; σ m 1 + ) > 0, which implies that at σ m 1 there is a change of sign in A m 1 (θ; t) = α m 1 (θ; t) β m 1 (t) from non-positive to positive. For m = 2, since α 1 (t) an β 1 (t) are inepenent of θ, the time of the sign change of A 1 (t) is inepenent of θ too, an it follows that σ 1 is locally inepenent of θ. For m > 2, there are two ways in which a sign change in A m 1 (σ m 1 ) can take place: continuously or as a result of a jump in either α m 1 (θ; t) or β m 1 (t) at t = σ m 1. Let us consier each of these two cases next. If no iscontinuity occurs at t = σ m 1, then by (2), we have either α m 1 (θ; t) = α m 2 (θ; t) or α m 1 (θ; t) = β m 2 (t). In the latter case, A m 1 (θ; σ m 1 ) = β m 2 (σ m 1 ) β m 1 (σ m 1 ) is clearly inepenent of θ. In the former case, we have A m 1 (θ; σ m 1 ) = α m 2 (θ; σ m 1 ) β m 1 (σ m 1 ) where, once again, either α m 2 (θ; t) = α m 3 (θ; t) or α m 2 (θ; t) = β m 3 (t) at t = σ m 1. In the latter case, A m 1 (θ; σ m 1 ) = β m 3 (σ m 1 ) β m 1 (σ m 1 ) is inepenent of θ. In the former case, we have A m 1 (θ; σ m 1 ) = α m 3 (θ; σ m 1 ) β m 1 (σ m 1 ) an the process repeats until we get A m 1 (θ; σ m 1 ) = α 1 (σ m 1 ) β m 1 (σ m 1 ) which is inepenent of θ. Thus, if the change in sign occurs continuously, we conclue that σ m 1 is inepenent of θ. This leaves only the possibility that the sign change occurs as a result of a jump in either α m 1 (θ; t) or β m 1 (t) at t = σ m 1. Note that α m 1 (θ; t) an β m 1 (t) may jump simultaneously at t = σ m 1, but only one of them ominates the sign change, i.e., the jump in the other one alone woul not have cause the sign change. The ominating jump in β m 1 (t) is obviously inepenent of θ. Therefore, the only possibility is that α m 1 (θ; t) experiences a ominating jump at t = σ m 1. Moreover, since α m 1 (θ; t) β m 1 (t) experiences a sign change from non-positive to positive, α m 1 (θ; t) must switch to a larger value at σ m 1, i.e., α m 1 (θ; σm 1 ) < α m 1(θ; σ m 1 + ). The jump of α m 1 (θ; t) has three possible ways of occurring: (i) switching from β m 2 (t) to α m 2 (θ; t), (ii) switching from α m 2 (θ; t) to β m 2 (t), or (iii) having α m 1 (θ; t) = α m 2 (θ; t) because x m 2 (θ; t) = 0, an inheriting a jump of α m 2 (θ; t) at that time. Case (i) is infeasible by the following argument: if σ m 1 is a switchover point of α m 1 (θ; t) from β m 2 (t) to α m 2 (θ; t), then the buffer at noe m 2 becomes empty at that time, which implies that β m 2 (σ m 1 ) α m 2 (θ; σ m 1 ) > 0; this contraicts the fact that α m 1 (θ; t) must switch to a larger value at σ m 1. If case (iii) applies, then α m 2 (θ; t) must switch to a larger value at σ m 1, an we repeat the same argument as the one use above for α m 1 (θ; t) until either case (ii) applies for some α m i (θ; t) with m i > 2 or we reach noe 2, in which case only case (ii) is possible. Thus, the proof reuces to consiering case (ii), i.e., showing that if σ m 1 is a switchover point of α m 1 (θ; t) with α m 1 (θ; σm 1 ) = α m 2(θ; σm 1 ) an α m 1 (θ; σ m 1 + ) = β m 2(σ m 1 + ) then, σ m 1 is locally inepenent of θ. Observe that this is precisely the statement of the lemma with m replace by m 1 in α m (θ; t) an β m 1 (t). Therefore, using the same argument as above, this process is repeate until the proof is reuce to showing that if σ m 1 is a switchover point of α 2 (θ; t) with α 2 (θ; σm 1 ) = α 1(σm 1 ) an α 2(θ; σ m 1 + ) = β 1(σ m 1 + ) then σ m 1 is locally inepenent of θ. This, however, was alreay establishe above base on the fact that α 1 (t) an β 1 (t) are both efining processes inepenent of θ. Proof of Lemma III.1 If s m,i is an active switchover point of α m (θ; t), it follows from (2) an Definition 1 that there are two possible cases: (i) an EP starts at noe m 1, or (ii) s m,i lies within an EP of noe m 1 an is an active switchover point of α m 1 (θ; t). In case (i), an event e 3 occurs at noe m 1. By Assumption 1(c), we can only have β m 1 (t) = α m 1 (θ; t) at a single time instant an by Assumption 1(b) that cannot coincie with another event at noe m 1. Therefore, α m (θ; t) must experience a jump from β m 1 (t) to α m 1 (θ; t) at t = s m,i, which is an e 1 event at noe m. In case (ii), s m,i is an active switchover point of α m 1 (θ; t), so either it starts an EP at noe m 2 or it