A Fair Comparison of Pull and Push Strategies in Large Distributed Networks

Transcription

1 1 A Fair Comparison of Pull an Push Strategies in Large Distribute Networks Wouter Minnebo an Benny Van Hout, Member, IEEE Abstract In this paper we compare the performance of the pull an push strategy in a large homogeneous istribute system. When a pull strategy is in use, lightly loae noes attempt to steal jobs from more highly loae noes, while uner the push strategy more highly loae noes look for lightly loae noes to process some of their jobs. Given the maximum allowe overall probe rate R an arrival rate λ, we provie close form solutions for the mean response time of a job for the push an pull strategy uner the infinite system moel. More specifically, we show that the push strategy outperforms the pull strategy for any probe rate R > 0 when λ < φ 1, where φ = (1 + 5)/ is the golen ratio. More generally, we show that the push strategy prevails if an only if 2λ < (R + 1) 2 + 4(R + 1) (R+1). We also show that uner the infinite system moel, a hybri pull an push strategy is always inferior to the pure pull or push strategy. The relation between the finite an infinite system moel is iscusse an simulation results that valiate the infinite system moel are provie. Inex Terms Performance analysis, Distribute computing, Processor scheuling I. INTRODUCTION Distribute networks typically consist of a set of noes interconnecte through a network, each equippe with a single server to process jobs. Jobs may enter the network via one or multiple central ispatchers (e.g., [1], [2], [3]) or via the processing noes themselves (e.g., [4], [5], [6], [7]). In the former case the ispatchers will istribute the jobs among the noes using some loa balancing algorithm. In the latter case, lightly loae noes may attempt to take/steal/pull jobs from more highly loae noes or highly loae noes may try to forwar/push some of their pening jobs to lightly loae noes. When the initiative is taken by the lightly loae noes only, we say that a pull strategy is use. If only the highly loae noes initiate the exchange of jobs, we say that a push strategy is use. Pull strategies are often calle work stealing schemes, while push strategies are sometimes calle work sharing solutions. To facilitate the exchange of jobs, some central information may be store. However, as the network size grows continuously upating this information becomes more challenging. Therefore, fully istribute networks o not rely on centralize information. Instea a noe that wishes to pull/push a job will transmit a probe to one (or multiple) other noes that are typically selecte at ranom. If a noe that receives such a probe is willing to take part in the exchange, it will sen This is an extene version of the paper entitle Pull versus Push Mechanism in Large Distribute Networks: Close Form Results publishe in the Proceeings of ITC-24. a positive reply an the job exchange can procee. Clearly, the overall rate at which probe messages are sent base on a strategy plays a pivotal role in its effectiveness. The performance of both push an pull strategies has been stuie by various authors. A comparison for a homogeneous istribute system with Poisson arrivals an exponential job lengths was presente in [4], [8]. The approach was base on a ecoupling assumption an relie on numerical methos to solve some nonlinear equation. Numerical examples showe that pull strategies achieve a lower mean response time uner high loas, while push strategies are superior uner low to meium loas. Similar observations were mae for heterogeneous systems in [9] again by relying on a ecoupling assumption. A similar approach to stuy the influence of task migrations in share-memory multi-processor systems was presente in [10]. In each of these papers, noes sen out probes as soon as the number of jobs rops below some threshol T 1 in case of the pull strategy, while for the push strategy probes are sent whenever the number of jobs in the queue is at least T upon arrival of a new job. When job transfer an signaling elays are assume negligible setting T = 1 is optimal [9], [5]. Another common feature in these papers is that the number of noes N is not a moel parameter, instea they provie a numerical approach for a system with N =, which we will call the infinite system moel. Although the insights provie by these comparisons are very valuable, the strength of the push an pull mechanism was only compare to some extent, mainly because the overall probe rate R of both strategies may be very ifferent (an epens on the loa λ < 1). This is especially true for the hybri pull/push strategy introuce in [5], where it has been shown to outperform both the push an pull strategy for all loas λ. However, as inicate in [5], such a hybri strategy results in a (far) higher probe rate R. Our aim in this paper is to compare the pull an push mechanism given that both generate the same overall probe rate R. Further, we also provie close form expressions for the main performance measures uner the infinite system moel. To this en we introuce a slightly ifferent pull an push strategy, uner which noes o not transmit probes at job completion or job arrival times. Instea ile noes will generate probes at some rate r uner the pull strategy, while uner the push strategy noes sen probes at some rate r whenever they have jobs waiting. A esirable property of both these strategies is that they can match any overall probe rate R uner any loa λ by setting r in the appropriate manner. More specifically, let R be the average number of probes sen by a noe per time unit, irrespective of its queue length. Clearly, the

2 2 overall probe rate R will be less than the rate r. By establishing a simple relationship between r an R, we will etermine r to match any preefine overall probe rate R. In fact, we show that if rate base pull/push strategy matches the overall probe rate of the traitional pull/push strategy, it also matches the queue length istribution an therefore the mean response time (uner the infinite system moel). Given some R > 0, we will show that uner the infinite system moel the push strategy outperforms the pull strategy for any (R + 1)2 + 4(R + 1) (R + 1) λ <, 2 in terms of the mean response time (as well as in the ecay rate of the queue length istribution). As R approaches zero, the right-han sie ecreases to φ 1, where φ = (1+ 5)/2 is the golen ratio, which inicates that the push strategy prevails for any R when λ < ( 1 + 5)/ As a sie result we show that the queue length istribution of the pull an push strategy is in fact ientical when they use the same rate r (instea of the same R). We also consier a hybri strategy where ile noes probe at rate r 1 an noes with pening jobs probe at rate r 2, where r = r 1 +r 2 is again etermine by matching R, leaving one egree of freeom. We will show that for any for λ an R the optimal policy exists in setting either r 1 or r 2 to zero. This implies that the hybri strategy is in fact never better when the overall probe rate R is not allowe to increase. The infinite system moel correspons to a istribute system with an infinite number of noes N. Using simulation results, we will show that the infinite system is quite accurate for both strategies an moerate to large size systems, e.g., for N 100 the relative error is typically below 1 percent. For smaller systems, e.g., N = 25, the infinite system moel results in higher relative errors, especially for the pull strategy uner high loas. The loas for which the push strategy outperforms the pull strategy are however still quite accurately preicte by the infinite system moel, even for small systems. The paper is structure as follows. In Section II we introuce the push, pull an hybri strategy consiere in this paper an iscuss its relation with many existing strategies stuie before. The infinite system moel is presente in Section III an close form results are erive for the queue length istribution an mean elay. Using these results we ientify the loas at which the push strategy outperforms the pull strategy an prove that a hybri strategy is always inferior. Simulation results that valiate the infinite system moel are presente in Section IV. In Section V we iscuss an prove the technical issues relate to showing that the infinite system moel is inee the proper limit process of the sequence of finite system moels, while in Section VI we show that rate-base strategies matching the probe rate of the traitional strategies also match the queue length istribution. Conclusions are rawn an future work is iscusse in Section VII. II. PULL AND PUSH STRATEGIES As in [4], [11], [9], [6] the moel consiere in this paper relies on the following assumptions: 1) The system consists of N noes, where each noe consist of a single server an an infinite buffer to store jobs. 2) Each noe is subject to its own local Poisson arrival process with rate λ. Jobs require an exponential processing time with mean 1 an are serve in a first-come-firstserve (FCFS) orer. 3) The time require to transfer probe messages an jobs between ifferent noes can be neglecte in comparison with the processing time (i.e., the transfer times are assume to be zero). The above assumptions also imply that all jobs can be successfully execute by any noe in the system an that there are no communication failures on the network that interconnects the noes. We consier the following three basic strategies: 1) Push: Whenever a noe has i 2 jobs in its queue, meaning i 1 jobs are waiting to be serve, the noe will generate probe messages at rate r. Thus, as long as the number of jobs in the queue remains above 1, probes are sent accoring to a Poisson process with rate r. Whenever the queue length i rops to 1, this process is interrupte an will remain interrupte as long as the queue length remains below 2. The noe that is probe is selecte at ranom an is only allowe to accept a job if it is ile. 2) Pull: Whenever a noe has i = 0 jobs in its queue, meaning the server is ile, the noe will generate probe messages at rate r. Thus, as long as the server remains ile, probes are sent accoring to a Poisson process with rate r. This process is interrupte whenever the server becomes busy. The probe noe is also selecte at ranom an the probe is successful if there are jobs waiting to be serve. 3) Hybri: This strategy combines the above two strategies. When queue length i equals 0 a noe generates probes at rate r 1, while for i 2 the probe rate is set equal to r 2. We will show that uner the infinite system moel (i.e., N = ) the push an pull strategy result in exactly the same queue length istribution when the same rate r is use. This is even true for the hybri strategy if we efine r = r 1 + r 2 (i.e., the queue length only epens on the sum of r 1 an r 2 ). However, when the same rate r is use by these ifferent strategies, the overall probe rate R will typically iffer. Hence, we aim at comparing these strategies when the rates r are set such that the overall probe rate matches some preefine R. The pull strategy consiere in this paper is in fact ientical to the pull strategy with repeate attempts consiere in [11, Section 2.4], except that our noes o not immeiately generate a probe message when the server becomes ile. Generating probes in that way woul automatically result in a high probe rate R when the loa λ is small an woul

3 3 no longer allow us to match any R > 0 by setting r in the appropriate manner. The traitional pull an push strategies consiere in [4], [9], [5] an iscusse in Section VI (for T = 1) are somewhat ifferent. The pull strategy tries to attract a job whenever a job completes an the resulting queue length is below T, while the push strategy tries to push arriving jobs that fin T or more jobs in the queue upon arrival. Further, instea of sening a single probe, both strategies repeately sen probes until either one gets a positive reply or a preefine maximum of L p probes is reache. The overall probe rate R clearly epens on T, L p an the loa λ, which makes it har to compare the pull an push strategies in a completely fair manner. When the time require to transfer probes an jobs between noes is neglecte (as in [4]), setting T = 1 ensures that exchange jobs can immeiately start (as in our setup). Assuming zero transfer time for probe messages is quite realistic as transferring jobs typically requires consierably more time than sening a probe. The moels in [9], [5] o take an exponentially istribute job transfer time into account (while still assuming zero transfer time for the probes). The results show that when increasing the transfer time, the elays also increase, while the performance ifferences between the pull an push strategies become less significant (but remain similar). Further, the setting T = 1 minimizes the mean response time when the job transfer times are sufficiently small (but also results in a higher overall probe rate R). The strategies consiere in [6], [7] are more aggressive pull an push strategies. In [6] a successful probe message results in exchanging half of the jobs that are waiting, while in [7] the number of probes sen uner the push strategy epens on the current queue length. Although the push strategy of [7] significantly reuces the mean response time an outperforms the pull strategy, its overall probe rate is also much higher. III. INFINITE SYSTEM MODEL In this section we present various analytical results in close form for the system with N = noes, terme the infinite system moel. The evolution of the infinite system moel will be efine by a set of orinary ifferential equations (ODEs) an is thus eterministic. The evolution of the finite system moels (for which N < ) on the other han will be capture by an N-imensional continuous time Markov chain (CTMC) an is therefore stochastic. To efine the infinite system moel we first consier a system with a finite number of noes N. Due to the assumptions on the arrival process, processing times an transfer times, it suffices to keep track of the N queue lengths in orer to obtain a CTMC. Further, as the system is homogeneous, it also suffices to keep track of the number of noes that have i jobs in their queue for all i 1. More precisely, we efine a CTMC {X (N) (t) = (X1 N (t), X2 N (t),...)} t 0, where X (N) i (t) {0,..., N} is the number of noes with at least i jobs in the queue at time t (the superscript N is use to inicate that we consier a finite system consisting of N noes). For any state x = (x 1, x 2,...) we clearly have that x i x i+1 for all i 1. Let us first inicate that the transition rates of this CTMC are ientical for the push, pull an hybri strategy provie that they use the same rate r. Transitions take place when either one of the following three events takes place: an arrival, a job completion, or a job exchange between an ile noe an a noe with at least two jobs. Let q (N) (x, y) be the transition rate between state x = (x 1, x 2,...) an state y = (y 1, y 2,...). If an arrival occurs in a queue with i 1 jobs, then x i will increase by one. Thus, ue to the arrivals we have q (N) (x, y) = λ(x i 1 x i ), for y = x + e i an i 1, where x 0 = 1 (as all the queues contain zero or more jobs) an e i is a vector with a 1 in position i an 0s elsewhere. Similarly, a job completion in a queue with i jobs reuces x i by one: q (N) (x, y) = (x i x i+1 ), for y = x e i an i 1. A job exchange between an ile noe an a noe with i jobs increases x 1 by one an ecreases x i by one; hence, y = x + e 1 e i. Uner the push strategy the rate of such exchanges equals the number of noes with exactly i jobs x i x i+1 times r times the probability that a probe message is successful 1, which equals (N x 1 )/N. Hence, for the push strategy we have q (N) (x, y) = r(1 x 1 /N)(x i x i+1 ), for y = x + e 1 e i an i 2. Uner the pull strategy this event takes place with a rate equal to the number of ile noes (N x 1 ) times r times the probability (x i x i+1 )/N that we select a noe with i jobs. The transition rate is therefore the same in both systems. For the hybri strategy these events occur at rate r 1 (1 x 1 /N)(x i x i+1 ) (ue to pull) plus r 2 (1 x 1 /N)(x i x i+1 ) (ue to the push), which results in the same overall rate. Using a coupling argument one can prove that this CTMC is positive recurrent for all λ < 1 (see Appenix C). Let π (N) = (π (N) 1, π (N) 2,...) be the unique stationary measure of the Markov chain {X (N) (t)} t 0, then the overall probe rate R (N) for the pull, push an hybri strategy equal r(1 π (N) 1 ), rπ (N) 2 an r 1 (1 π (N) 1 ) + r 2 π (N) 2, respectively. We will now efine the infinite system moel, the evolution of which is escribe by a set of ODEs, using the rates q (N) (x, y). As these rates are the same for the three strategies, they also result in the same set of ODEs. Let L = {e i, i 1} { e i, i 1} {e 1 e i, i 2} be the set of possible transitions (arrivals, job completions an job transfers). Next, we efine F (x) = l L lβ l (x), where β l (x) is efine such that β l (x/n) = q (N) (x, x+l)/n. Given the above expressions for q (N) (x, x+l)/n, this implies β ei (x) = λ(x i 1 x i ), β ei (x) = (x i x i+1 ), for all i 1 (with x 0 = 1 for i = 1) an β e1 e i (x) = r(1 x 1 )(x i x i+1 ), 1 We assume that the probe noe is selecte at ranom, in fact we even allow a noe to select itself with probability 1/N. Disallowing noes to select themselves results in the same limiting process.

4 4 for i 2. In other wors, F (x) = i 1 (e i β ei (x) e i β ei (x)) + (e 1 e i )β e1 e i (x), where x = (x 1, x 2,...), with x i [0, 1] an x i x i+1 for i 1. The set of ODEs escribing the evolution of the infinite system moel is now given by t x(t) = F (x(t)), where x i(t) represents the fraction of the number of noes with at least i jobs at time t in the infinite system. This set of ODEs can be written as an t x 1(t) = (λ + rx 2 (t))(1 x 1 (t)) (x 1 (t) x 2 (t)), (1) t x i(t) = λ(x i 1 (t) x i (t)) (1 + r(1 x 1 (t)))(x i (t) x i+1 (t)), (2) for i 2. In Section V we will iscuss the relation between this ynamical system an the finite system moels for large N. Let E = {(x 1, x 2,...) x i [0, 1], x i x i+1, i 1, j 1 x j < }. The next two theorems show that this set of ODEs is Lipschitz on E an it has a unique fixe point in E. Theorem 1. The function F is Lipschitz on E. Proof: F is Lipschitz provie that for all x, y E there exists an L > 0 such that F (x) F (y) L x y. By efinition of F (x) one fins F (x) F (y) 2(λ r) x y + 2r x 1 (x i x i+1 ) y 1 (y i y i+1 ). The above sum can be boune by (x 1 y 1 )(x i x i+1 ) + y 1 (x i x i+1 y i + y i+1 ), which is boune by 2 x y on E. Hence, F is Lipschitz by letting L = 2λ r. As E is a Banach space the Lipschitz conition of F suffices to guarantee that the set of ODEs tx(t) = F (x(t)), with x(0) E, has a unique solution 2 φ t (x(0)) [12, Section 1.1]. Theorem 2. The set of ODEs given by (1) an (2) has a unique fixe point π = (π 1, π 2,...) with i 1 π i <. Further, ( ) i 1 λ π i = λ. 1 + (1 λ)r Proof: Assume π is a fixe point with i 1 π i <, meaning F i (π) = 0 for i 1, where F (x) = 2 The solution φ t(x) belongs to the class of continuously ifferentiable functions as in the finite imensional case. (F 1 (x), F 2 (x),...). When i 1 π i <, we can write i 1 F i(π) using (2) as λ(π i 1 π i ) i 1 F i (π) = F 1 (π) + (1 + r(1 π 1 ))(π i π i+1 ) = F 1 (π) + λπ 1 (1 r(1 π 1 ))π 2 = λ π 1, where the last equality follows from (1). In other wors, i 1 F i(π) = 0 implies that π 1 = λ. Further, by efining η i = π i π i+1, the conition F i (π) = 0, for i 2, reaily implies that η i+1 = λη i /(1 + (1 π 1 )r) an therefore by inuction we fin the expression for π i, for i 2. If we take the set of ODEs in (1) an (2) an replace the first x 2 (t) by π 2 in (1) an x 1 (t) by λ in (2), then we en up with the Kolmogorov equations for a state epenent M/M/1 queue with λ 0 = λ + rπ 2, λ i = λ, for i 1, µ 1 = 1 an µ i = 1 + (1 λ)r, for i 2. The arrival process of such an M/M/1 queue is Poisson with rate λ i an the service is exponential with rate µ i whenever the queue length equals i. Hence, the fixe point π also correspons to the steay state of a state epenent M/M/1 queue (where π i is the probability that the queue contains at least i jobs). The set of ODEs in (1) an (2) escribes the transient behavior of the infinite system, while we are in fact intereste in its behavior as t goes to infinity. Thus, we are intereste in the limit of all the trajectories of this set of ODEs. In Appenix A we prove the following theorem: Theorem 3. All the trajectories of the set of ODEs given by (1) an (2), starting from x E converge towars the unique fixe point π. Due to the above theorem, we can now express the main performance measures of the pull, push an hybri strategy via Theorem 2, where the overall probe rate R for the pull, push an hybri strategy are efine as r(1 π 1 ), rπ 2 an r 1 (1 π 1 ) + r 2 π 2, respectively: Corollary 1. The mean response time D of a job uner the push, pull an hybri strategy equals D = 1 + λ (1 λ)(1 + r). Uner the hybri strategy the overall probe rate R can be expresse as R = (1 λ)r 1 + λ 2 r (1 λ)r, with r = r 1 + r 2. Setting (r 1, r 2 ) = (r, 0) an (0, r) results in the probe rate R of the pull an push strategy, respectively. Proof: The mean response time D can be expresse as i 1 π i/λ = 1 + λ/(1 + (1 λ)r λ) by Little s law. The overall probe rate uner the pull an push strategy equals r(1 π 1 ) an rπ 2, respectively. Uner the hybri strategy the overall probe rate equals r 1 (1 π 1 ) + r 2 π 2. Our interest lies in comparing the mean response time D of the three policies given λ an the overall allowe probe rate

5 5 R. Using the above result, we can easily set r such that the overall probe rate equals some preefine R. For the hybri policy this still leaves one egree of freeom as only the sum of r 1 +r 2 has been etermine. The above result also inicates that R converges to λ 2 /(1 λ) as r goes to infinity uner the push strategy (which is in contrast to the pull strategy where R also goes to infinity). This inicates that an overall probe rate R close to λ 2 /(1 λ) suffices to get a mean response time close to 1 uner the push strategy. We shoul however also note that this rate R becomes large as λ approaches one. Theorem 4. The mean response time D of a job uner the push strategy equals D push = λ (1 λ)(λ + R), for R < λ 2 /(1 λ) an D push = 1 for R λ 2 /(1 λ). Uner the pull strategy we get D pull = 1 + R 1 λ + R. Hence, given λ the push strategy outperforms the pull strategy if an only if (1 + R) > λ 2 /(1 λ) an given R the push is the best strategy if an only if (1 + R)2 + 4(1 + R) (1 + R) λ <. 2 Further, the push strategy outperforms the pull strategy for all λ < φ 1, where φ = (1 + 5)/2 is the golen ratio. Proof: The expressions for D push an D pull are reaily obtaine from Corollary 1 by plugging in the appropriate value for r in the expression for D. Requiring that D push = D pull results in a quaratic equation for R with roots in 0 an λ 2 /(1 λ) 1, which results in the conition for (1 + R) an λ. The last result is obtaine by noting that (1 + R)2 + 4(1 + R)/2 (1 + R)/2 is an increasing function in R an its limit for R going to zero equals 5/2 1/2. Looking at the expression for the mean elay in Corollary 1, we note that a strategy with a lower mean response time actually has a larger r value when matching R. By Theorem 2 we also know that the queue length istribution ecays geometrically with parameter λ/(1 +(1 λ)r). Hence, a smaller mean elay therefore also implies a faster ecay of the queue length istribution. In fact, in this case a smaller mean elay even implies that the queue length istribution becomes smaller in the usual stochastic orering sense [13]. We observe another funamental ifference between the push an pull strategy when the loa approaches 1. In this case the mean elay of the push strategy still goes to infinity as in the M/M/1 queue (the mean response time of which is 1/(1 λ)). For the pull strategy the mean elay approaches 1 + 1/R, hence remains finite. We shoul note that r oes go to infinity when λ approaches 1 uner the pull strategy (for any R > 0). Theorem 5. The mean elay uner the hybri strategy (r 1, r 2 ) with overall probe rate R is minimize by setting r 1 or r 2 Mean Delay (D push ) λ = 0.95 λ = 0.9 λ = 0.8 λ = Infinity Number of Noes (N) Figure 1. Mean elay an relative error of the push strategy in a finite vs. infinite system for R = 1 equal to zero. Hence, a pure pull or push strategy is always optimal. Proof: Let R 1 an R 2 = R R 1 be the overall probe rate generate by the pull an push operations, respectively. By Corollary 1, we have R 1 = (1 λ)r 1 an R 2 = λ 2 r 2 /(1+(1 λ)r), while we also note that D is minimize by maximizing r = r 1 + r 2. Hence, by letting R 2 = y an R 1 = R y, we wish to maximize g(y) = R y y(1 + R y) + 1 λ λ 2 (1 λ)y, for y [0, R] an R < λ 2 /(1 λ). For R λ 2 /(1 λ) the response time is minimize by setting r 1 = 0 as D push = 1. Some basic algebraic manipulations show that y g(y) = ((1 + R) λ2 1 λ ) ( λ λ 2 (1 λ)y ) 2, on y [0, R] with R < λ 2 /(1 λ). Depening on the sign of (1 + R) λ 2 /(1 λ) the erivative of g(y) is therefore positive or negative on the entire interval an the minimum is foun in y = 0 (i.e., r 2 = 0) or y = R (i.e., r 1 = 0). IV. MODEL VALIDATION In this section we valiate the infinite system moel by comparing the close form results of Theorem 4 with time consuming simulation results for systems with a finite number of noes N. The infinite an finite system moel only iffer in the system size. Hence, the rate r in the simulation experiments is inepenent of N an was etermine by λ an R using the expression for R in Corollary 1. Each simulate point in the figures represents the average value of 25 simulation runs. Each run has a length of 10 6 time units (where the service time is exponentially istribute with a mean of 1 time unit) an a warm-up perio of length 10 6 /3 time units. Figure 1 compares the mean elay in a finite system with N noes with the mean elay in the infinite system moel uner the push strategy with R = 1 for N = 25, 50,..., 1600 an

6 λ = λ = 0.7 Mean Delay (D pull ) λ = λ = λ = Infinity Number of Noes (N) Request Rate (R push ) λ = 0.8 λ = 0.9 λ = Infinity Number of Noes (N) Figure 2. Mean elay an relative error of the pull strategy in a finite vs. infinite system for R = 1. Figure 3. Overall request rate of the push strategy in a finite vs. infinite system for R = 1. λ = 0.7, 0.8, 0.9 an For each combination of N an λ we also show the relative error. The error clearly ecreases to zero as N goes to infinity. Further, even for a system with N = 100 noes we observe a relative error of 1% only. It may seem unexpecte that the relative error is nearly insensitive to the loa, as one might expect higher errors as λ increases. In fact, if r is kept fixe we woul observe an increase error. However, we are looking at the curves for R = 1, meaning r = 1/(λ 2 (1 λ)) ecreases with λ (see Corollary 1). As setting r = 0 gives exact results for any finite N, we can expect an improve accuracy for smaller r values (if λ remains fixe). Thus, in Figure 1 we see more or less the same relative errors because higher loas, which worsen the accuracy, correspon to lower r values, which improve the accuracy. Figure 2 epicts the same results as Figure 1, but for the pull strategy. Although we still see the convergence as N goes to infinity, the relative errors grow quickly with λ an an error of 9% is observe even for a system with N = 100 noes. Uner the pull strategy r = 1/(1 λ) for R = 1, which implies that larger λ values also correspon to larger r values. Therefore, the less accurate results for higher loas are not unexpecte. The overall request rate observe in the simulation experiments was typically within 0.1% of the targete R value, meaning the relation R = (1 λ)r seems highly accurate even for finite systems. This is not unexpecte as the fraction of ile noes shoul also match (1 λ) in the finite system. Figure 3 shows the observe overall request rate for the push strategy, which excees the targete value of R an ecreases as a function of N an λ. Hence, the relation R = λ 2 r/(1 + (1 λ)r) of Corollary 1 is not highly accurate for small system sizes. This can be explaine by noting that the infinite system moel is optimistic with respect to the queue length istribution for N finite an therefore also preicts a lower overall probe rate. In Figure 4 we compare the mean elay of the push an pull strategy in the infinite system moel (full lines) with a finite system consisting of N = 100 noes (crosses) for λ Mean Delay (D) push, R=0.5 push, R=1.0 pull, R=0.5 pull, R= Loa (λ) Figure 4. Mean elay of the push an pull strategy in a finite system with N = 100 noes (crosses) vs. infinite system moel (full lines) for R = 0.5 an R = 1. 0 an R = 0.5 an R = 1. The results inicate that the infinite system moel provies accurate results uner any loa λ, while the pull strategy becomes less accurate as the loa increases (which is in agreement with the results in Figures 1 an 2). Note, uner the push strategy setting λ < ( 5 1)/ implies that r can be chosen arbitrarily large such that the overall probe rate R remains below 1 (see Theorem 4). For r = the mean elay becomes 1 an there is little use in simulating the system for finite N. In Figure 5 we have zoome in on the intersection of the pull an push curves for R = 1 to inicate that the region where the push strategy outperforms the pull strategy is in perfect agreement with the infinite system moel. This can be unerstoo by noting that the r value use uring the simulation is etermine by the relation between R an r in Corollary 1. When λ = (1 + R) 2 + 4(1 + R)/2 (1+R)/2, we therefore make use of the same r value for the push

7 7 Mean Delay (D) push, R=1.0 pull, R= Loa (λ) Figure 5. Mean elay of the push an pull strategy in a finite system with N = 100 noes (crosses) vs. infinite system moel (full lines) for R = 1. an pull strategy. Hence, the evolution of the finite system moel with N noes is capture by the same Markov chain (X (N) (t)) t 0, meaning both strategies have the same queue length istribution an mean elay for all N. We shoul however keep in min that the observe overall probe rate tens to excee R uner the push strategy, especially for small systems. It is therefore fair to say that when N is small, the region where the push strategy outperforms the pull strategy is in fact overestimate by the infinite system moel. V. FINITE VERSUS INFINITE SYSTEM MODEL In this section we iscuss the relation between the set of ODEs in (1) an (2) an the sequence of Markov chains {X (N) (t)} t 0 as N tens to infinity. More specifically, we will ientify an prove the technical issues relate to formally showing that the steay state measures π (N) of {X (N) (t)} t 0 converge to the unique fixe point π. These issues arise from having an infinite imensional state space E. Replacing the infinite size buffer in each noe by a finite large buffer (such that the loss rate can be neglecte) woul result in a finite imensional (compact) space E an woul resolve most of the issues. This also explains why large finite buffers are often consiere as oppose to infinite buffers (see [6], [7]). We start by recalling the efinition of a ensity epenent family of Markov chains [14]. A set of Markov chains {X (N) (t)} t 0, with N 1, where E N = E {k/n, k Z m } is the state space of {X (N) (t)} t 0, is a family of ensity epenent Markov chains provie that the transition rates q (N) (x, y) between state x E N an y E N can be written as q (N) (x, y) = Nβ (y x)n (x), where β l is a function from E R m to R +. Let F (x) = l L lβ l(x), where L is the set of all possible transitions. Note, the set of CTMCs consiere in Section III matches this efinition, with L = {e i, i 1} { e i, i 1} {e 1 e i, i 2}, except that E is not a part of R m for some finite m. However, this efinition was extene to R in [15], where the following generalization of Kurtz s theorem was proven [15, Theorem 3.13]: Theorem 6 (Kurtz). Consier a family of ensity epenent CTMCs, with F Lipschitz. Let lim N X (N) (0) = x a.s. an let φ t ( x) be the unique solution to the initial value problem tx(t) = F (x(t)) with x(0) = x. Consier the path {φ t ( x), t T } for some fixe T 0 an assume that there exists a neighborhoo K aroun this path satisfying l sup β l (x) <, (3) x K then l L lim sup X (N) (t) φ t ( x) = 0 a.s. N t T In the finite imensional case, the set L is finite an therefore (3) is automatically met. For our system, conition (3) correspons to showing that there exists an environment K such that sup x K(x i x i+1 ) <. The following theorem is proven in Appenix B: Theorem 7. Given x E an T 0, there exists an environment K of {φ t ( x), t T } such that sup x K(x i x i+1 ) <. Hence, the set of ODEs given by (1) an (2) escribes the proper limit process of the finite systems over any finite time horizon [0, T ]. A natural question is whether this convergence extens to the stationary regime. Sufficient conitions for the finite imensional case can be foun in [16]. We will instea rely on a more general result in [17], which consiers a family of stochastic processes on some Polish space E, which inclues the set of infinite imensional, separable an complete spaces. As E = {(x 1, x 2,...) x i [0, 1], x i x i+1, i 1, j 1 x j < } is a subspace of the space l1 = {(x 1, x 2,...) j 1 x j < }, it is separable. E is clearly also complete an therefore Polish. Let π (N) = (π (N) 1, π (N) 2,...) be the unique stationary measure of the Markov chain {X (N) (t)} t 0. Given that we have a unique solution φ t (x) (which is continuous in t for all x) an that convergence over finite time intervals occurs, Corollary 1 of [17] can be rephrase as: Theorem 8 (Benaïm, Le Bouec). Given that φ t (x) is continuous in x for all t an that the sequence (π (N) ) N 1 is tight, we have lim lim X (N) (t) π = 0, N t in probability. The sequence (π (N) ) N 1 is tight if for every ɛ > 0 there exists some compact set K ɛ such that P{π N K ɛ } > 1 ɛ for all N. Note, if E is compact (as is often the case in finite imension), tightness is immeiate. In our case E is not compact an the following theorem is proven in Appenix C: Theorem 9. The sequence of measures (π (N) ) N 1 is tight. The continuity of φ t (x) in x for all t is guarantee by the uniqueness of the solution in finite imensions, but this

8 8 result oes not in general exten to Banach spaces of infinite imension [18]. However, for F Lipschitz, as in our case, the classical finite imensional results still hol an we may conclue that convergence of the steay state measures to the fixe point π occurs. VI. RATE-BASED VERSUS TRADITIONAL STRATEGIES The aim of this section is to show that the performance of the rate-base pull/push strategies coincies with the traitional pull/push strategies when the former match the overall probe rate of the latter. To this en, we introuce an infinite system moel for the following traitional pull an push strategy: 1) Traitional Push: A server starts sening probes whenever a job arrives in a queue with i 1 jobs, meaning i 1 jobs are waiting to be serve. The noes that are probe are selecte at ranom an a noe is only allowe to accept a job if it is ile. The server starts by probing a single noe. If the probe fails (because the selecte noe is not ile), the server sens another probe. This proceure is repeate until a probe is either successful or L p unsuccessful probes were sent. 2) Traitional Pull: A server starts sening probes whenever the server becomes ile. The noes that are probe are selecte at ranom an a noe is only allowe to transfer a job if its queue length excees one. Probes are sent one at a time until one is successful or L p unsuccessful probes were sent. Analytical moels to assess the performance of a class of pull an push strategies that inclue the above two strategies were presente in [4], [8]. These moels relie on a ecoupling assumption an the mean response time was expresse as the solution to a nonlinear equation that was solve numerically. In this section we present ODE moels for the traitional strategies similar to the ODE moel in Section III for the rate-base strategies an show that its unique fixe point can be expresse in close form. It is not har to verify that the nonlinear equation for the unique fixe point of the ODE correspons to the nonlinear equation in [4] for the pull strategy with T = 1 an the one in [8] for the push strategy with T = 1 an C = 0. A. Traitional push Let s i (t) enote the fraction of queues containing at least i jobs at time t an set s(t) = (s 1 (t), s 2 (t),...). The ynamics of the infinite system moel for the traitional push strategy is capture by the following set of ODEs: s 1 (t) = λ(1 s 1 (t)) + λs 1 (t)(1 s 1 (t) Lp ) t (s 1 (t) s 2 (t)) (4) s i (t) = λ(s i 1 (t) s i (t))s 1 (t) Lp (s i (t) s i+1 (t)) (5) t for i 2. The terms s i (t) s i+1 (t), for i 1, correspon to the service completion events (as the job urations are exponential with mean 1). The rate at which arrivals occur in a noe with exactly i 1 jobs is λ(s i 1 (t) s i (t)) an s 1 (t) Lp is the probability that L p probes are unsuccessful; hence queues of length i are create at rate λ(s i 1 (t) s i (t))s 1 (t) Lp, for i 2. Finally, queues of length 1 are create by new arrivals (at rate λ(1 s 1 (t))) or job transfers. The latter occur at rate λs 1 (t)(1 s 1 (t) Lp ), as λs 1 (t) is the rate at which probes are sent to the ile noes an (1 s 1 (t) Lp ) is the probability that one of the probes is successful. The set of ODEs given by (4) an (5) has a unique fixe point ˆπ = (ˆπ 1, ˆπ 2,...) with i 1 ˆπ i < given by ˆπ 1 = λ, ˆπ 2 = λ Lp+2, ˆπ i+1 = ˆπ i λ Lp+1 (ˆπ i 1 ˆπ i ), for i > 2, where the first equality follows from taking the sum of (4) an (5) for i 1. For the traitional push strategy, every busy noe will sen on average L p ˆπ 1 i = 1 λlp 1 λ i=1 probes at the task arrival rate λ, meaning that the overall probe rate ˆR equals ˆR = ˆπ 1 λ 1 λlp 1 λ. From the relationship R = r push π 2, we observe that a ratebase push strategy with r push = λˆπ 1 π 2 1 λ Lp 1 λ matches ˆR. By substituting the rate rpush in Theorem 2 we fin that uner the infinite system moel, the traitional an rate-base push strategy have the same fixe point. This inicates that the rate-base strategy matches the queue length istribution of the traitional variant, provie that we match the overall probe rate R. B. Traitional pull A similar set of ODEs escribes the evolution of the traitional pull strategy (for L p = 1 this correspons to the moel in [11]): s 1 (t) = λ(1 s 1 (t)) (s 1 (t) s 2 (t))(1 s 2 (t)) Lp t (6) s i (t) = λ(s i 1 (t) s i (t)) (s i (t) s i+1 (t)) t (7) (s i(t) s i+1 (t)) (s 1 (t) s 2 (t))(1 (1 s 2 (t)) Lp ), s 2 (t) for i 2, where si(t) t = λ(s i 1 (t) s i (t)) if s 2 (t) = 0 an i 2. The intuition is similar as for the push strategy, where we note that (s 1 (t) s 2 (t)) is the rate at which probes are generate, (1 (1 s 2 (t)) Lp ) is the probability that one of the probes is successful an (s i (t) s i+1 (t))/s 2 (t) is the probability that the accepting busy queue has length i, for i 2.

9 9 Probe Rate (R) Push Pull Loa (λ) L p = 3 L p = 2 L p = 1 L p = 3 L p = 2 L p = 1 Figure 6. Mean overall probe rate for the traitional pull an push strategy as a function of λ for L p = 1, 2 an 3. The system given by (6) an (7) also has a unique fixe point π = ( π 1, π 2,...) with i 1 π i <. As π 1 = λ, π 2 is foun as the unique positive real root of g(x) = λ(1 λ) (λ x)(1 x) Lp = 0, with x [0, λ]. The uniqueness follows by noting that g(x)/x = (1 x) Lp 1 (1 x+l p (λ x)) is strictly positive on [0, λ], while g(0) < 0 an g(λ) > 0 (as λ < 1). Finally, π i, for i > 2, is given by π i+1 = π i λ( π i 1 π i ) 1 + (λ π 2 )(1 (1 π 2 ) Lp )/ π 2. For the traitional pull strategy, every noe with exactly one job will sen on average L p (1 π 2 ) i = 1 (1 π 2) Lp π 2 i=1 probes each time the server becomes ile (as a probe fails with probability (1 π 2 )), meaning that the overall probe rate R for the traitional pull strategy equals R = ( π 1 π 2 ) 1 (1 π 2) Lp π 2. From the relationship R = (1 π 1 )r pull, we observe that a rate-base pull strategy with r pull = π 1 π 2 1 π 1 1 (1 π 2 ) Lp π 2 amounts to the same overall probe rate R. Substituting the rate r pull in Theorem 2 allows us to conclue that the ratebase pull strategy matches the queue length istribution of the traitional variant, provie that we match the overall probe rate R. The mean overall probe rate ˆR an R for the traitional push an pull strategy respectively is shown in Figure 6. As the probe rate of both strategies iffers significantly, it is sometimes har to compare these strategies in a fair manner. VII. CONCLUSIONS AND FUTURE WORK In this paper we compare the ability of the push an pull strategy to reuce the mean elay in a homogeneous istribute system given an overall probe rate R. We showe that the push strategy outperforms the pull strategy if an only if λ < (R + 1) 2 + 4(R + 1)/2 (R + 1)/2 in the infinite system moel an showe, by simulation, that this formula is also quite accurate for small finite systems, e.g., systems with N = 25 noes. We further emonstrate that a hybri strategy is always inferior to the pure push or pull strategy when the overall probe rate R is not allowe to increase. Some technical issues to formally prove the convergence of the steay state measures of the finite system moel to the infinite system moel were ientifie an proven. Finally, we showe that rate-base strategies matching the probe rate of traitional strategies, also match the queue length istribution. For future work we inten to stuy push strategies where the probe rate epens on the current queue length. The push strategy consiere in this paper generates probes at rate r whenever the queue length i excees 1. We believe we can further ecrease the mean elay by making r a function of i without increasing the overall probe rate R. Note, this is very much relate to the choice of the threshol T in [4], [9], [5], where T = 1 was argue to be optimal in case the probe an job exchange times are zero. However, for the strategies stuie in [4], [9], [5] smaller T values result in larger probe rates R. Another possible extension exists in consiering strategies where noes accept jobs whenever their queue length is small enough, instea of equal to zero as in the current paper. Moels in which each noe contains multiple servers instea of one may also be of interest, in this case a variety of pull strategies may be consiere. One coul also try to exten the current set of results to a heterogeneous moel (i.e., where noes can have ifferent job arrival an service rates), but obtaining close form results (or even the stability conition) may be quite challenging. REFERENCES [1] M. Mitzenmacher, The power of two choices in ranomize loa balancing, IEEE Trans. Parallel Distrib. Syst., vol. 12, pp , October [2] N. Vveenskaya, R. Dobrushin, an F. Karpelevich, Queueing system with selection of the shortest of two queues: an asymptotic approach, Problemy Pereachi Informatsii, vol. 32, pp , [3] Y. Lu, Q. Xie, G. Kliot, A. Geller, J. R. Larus, an A. Greenberg, Joinile-queue: A novel loa balancing algorithm for ynamically scalable web services, Perform. Eval., vol. 68, pp , [4] D. Eager, E. Lazowska, an J. Zahorjan, A comparison of receiverinitiate an sener-initiate aaptive loa sharing, Perform. Eval., vol. 6, no. 1, pp , [5] R. Mirchananey, D. Towsley, an J. Stankovic, Analysis of the effects of elays on loa sharing, IEEE Trans. Comput., vol. 38, no. 11, pp , [6] N. Gast an B. Gaujal, A mean fiel moel of work stealing in largescale systems, SIGMETRICS Perform. Eval. Rev., vol. 38, no. 1, pp , [7] B. Van Hout, Performance comparison of aggressive push an traitional pull strategies in large istribute systems, in Proceeings of QEST 2011, Aachen (Germany), IEEE Computer Society, SEP 2011, pp [8] D. Eager, E. Lazowska, an J. Zahorjan, Aaptive loa sharing in homogeneous istribute systems, Software Engineering, IEEE Transactions on, vol. SE-12, no. 5, pp , may 1986.

10 10 [9] R. Mirchananey, D. Towsley, an J. A. Stankovic, Aaptive loa sharing in heterogeneous istribute systems, J. Parallel Distrib. Comput., vol. 9, no. 4, pp , [10] M. Squillante an R. Nelson, Analysis of task migration in sharememory multiprocessor scheuling, ACM SIGMETRICS Perform. Eval. Rev., pp , [11] M. Mitzenmacher, Analyses of loa stealing moels base on families of ifferential equations, Theory of Computing Systems, vol. 34, pp , [12] K. Deimling, Orinary Differential Equations in Banach spaces. Lect. Notes in Math. 596, [13] M. Shake an J. G. Shanthikumar, Stochastic Orers an their Applications. Associate Press, [14] T. Kurtz, Approximation of population processes. Society for Inustrial an Applie Mathematics, [15] M. Mitzenmacher, The power of two choices in ranomize loa balancing, Ph.D. issertation, University of California, Berkeley, [16] M. Benaïm an J. Le Bouec, A class of mean fiel interaction moels for computer an communication systems, Performance Evaluation, vol. 65, no , pp , [17], On mean fiel convergence an stationary regime, CoRR, vol. abs/ , Nov [18] F. e Blasi an G. Pianigiani, Uniqueness for ifferential equations implies continuous epenence only in finite imension, Bulletin of the Lonon Mathematical Society, vol. 18, no. 4, pp , [19] J. Walker, Dynamical Systems an Evolution Equations. Theory an Applications. Plenum Press, New York, [20] P. Billingsley, Probability an Measure. John Wiley an Sons, APPENDIX A PROOF OF THEOREM 3 We start by proving the following Lemma: Lemma 1. Let x(t) be the unique solution of the ODEs given by (1) an (2) with x(0) E. The L 1 -istance to the unique fixe point i 1 x i(t) π i oes not increase as a function of t. Proof: Define ɛ i (t) = x i (t) π i, for i 1, such that Φ(t) = i 1 ɛ i(t) represents the L 1 -istance. As t x i(t) = t ɛ i(t) an π is a fixe point of (1) an (2), we fin an t ɛ 1(t) = λɛ 1 (t) + (1 + r(1 π 1 ))ɛ 2 (t) rɛ 1 (t)(ɛ 2 (t) + π 2 ) ɛ 1 (t), (8) t ɛ i(t) = λ(ɛ i 1 (t) ɛ i (t)) (1 + r(1 π 1 ))(ɛ i (t) ɛ i+1 (t)) +rɛ 1 (t)(ɛ i (t) ɛ i+1 (t) + π i π i+1 ), (9) for i 2. Assume for now that ɛ i (t) 0 for all i such that tφ(t) is properly efine as t Φ(t) = t ɛ i(t) t ɛ i(t). i:ɛ i(t)>0 i:ɛ i(t)<0 If ɛ i (t) has the same sign for all i, one fins that t Φ(t) = ɛ 1 (t) by summing (8) an (9). We will show that t Φ(t) ɛ 1(t), hols in general. Let I = {i 1, i 2,...}, with i 1 < i 2 <..., be the set of inices where ɛ i (t) changes sign, that is, ɛ i 1 (t) an ɛ i (t) have a ifferent sign if an only if i I. Assume ɛ 1 (t) < 0 an let I + = {i 1, i 3,...} an I = {i 2, i 4,...} such that i I + implies that ɛ i 1 (t) < 0 an ɛ i (t) > 0, while i I implies that ɛ i 1 (t) > 0 an ɛ i (t) < 0. When ɛ i 1 (t) an ɛ i (t) are both positive (for i 2), the terms λɛ i 1 (t), (1 + r(1 π 1 ))ɛ i (t) an rɛ 1 (t)(ɛ i (t) + π i ) in t ɛ i(t) are cancele by t ɛ i 1(t) when computing t Φ(t). However, if ɛ i 1 (t) < 0 an ɛ i (t) > 0, t ɛ i 1(t) is replace by t ɛ i 1(t) in tφ(t) an therefore contains these three terms twice. Hence, in general t Φ(t) = ɛ 1(t) + 2 (λɛ i 1 (t) (1 + r(1 π 1 ))ɛ i (t)) }{{} i I + <0 2 (λɛ i 1 (t) (1 + r(1 π 1 ))ɛ i (t)) }{{} i I >0 +2 rɛ 1 (t)(ɛ i (t) + π i ) 2 rɛ 1 (t)(ɛ i (t) + π i ). i I + i I This implies that t Φ(t) ɛ 1(t) provie that (ɛ i (t) + π i ) (ɛ i (t) + π i ) 0, i I + i I (10) which clearly hols as this expression is equal to (x i1 (t) x i2 (t)) + (x i3 (t) x i4 (t)) +... an x i (t) x j (t) for i < j. Hence, t Φ(t) ɛ 1(t) if ɛ 1 (t) < 0. A similar argument can be use for ɛ 1 (t) > 0. Finally, we consier the technical issue of efining t Φ(t) in case ɛ i (t) = 0 for some i an t = t 0. In this case the above proof remains unchange provie that we rely on the upper right-han erivative (as in [1, Theorem 3]), that is, if we efine t ɛ i(t 0 ) as t ɛ i(t 0 ) = lim t t + 0 ɛ i (t) t t 0. The above lemma shows that the L 1 -istance to the fixe point oes not increase along any trajectory x(t) in E, an only remains the same whenever x 1 (t) = π 1 (as ɛ 1 (t) = 0 in such a case). Lemma 2. The only trajectory x(t) of the ODE given by (1) an (2) with x(0) E for which x 1 (t) = π 1 for all t is given by x(t) = π for all t. Proof: If x 1 (t) = π 1 = λ for all t, then (1) implies that x 2 (t) = π 2. Similarly, for i 2, if x j (t) = π j for all j i an t, then (2) implies that x i+1 (t) = π i+1. We now recall La Salle s invariance principle for Banach spaces, where a (positively) invariant subset of K E of an ODE efine on E is such that x(t) K for all t provie that x(t) is the unique solution of the ODE with x(0) K. Theorem 10 ([19]). Let V (x) be a continuous real value function from E to R with t V (x) = lim sup 1 t 0 + t (V (x(t)) V (x)) 0, where x(t) is the unique solution of an ODE with x(0) = x. Let K = {x E tv (x) = 0} an let M be the largest

11 11 (positively) invariant subset of K. If x(t) is precompact (i.e., remains in a compact set) for x(0) E, then lim ist(x(t), M) = 0, t where ist(x, M) represents the Banach istance between the point x E an the set M E. We are now in a position to prove Theorem 3: Proof of Theorem 3: We rely on La Salle s invariance principle for Banach spaces by setting V (x) equal to the L 1 - istance to the fixe point, i.e., V (x) = i 1 x i π i. Lemma 1 implies that tv (x) 0, while Lemma 2 shows that M = {π} is a singleton. Hence, π is a global attractor provie that we can show that the trajectory x(t) remains in a compact set if x(0) E. Let m = i 1 x i(0) π i, then by Lemma 1 we know that x(t) remains in the set E m = {x E i 1 x i π i m}. This set is not compact in the Banach space E equippe with the L 1 -norm, but La Salle s invariance principle hols in any Banach space. If E is equippe with the weighte L 1 -norm x i i 1 2, the sets E i m are compact an global attraction follows from Theorem 10. APPENDIX B PROOF OF THEOREM 7 We start by proving the following lemma: Lemma 3. For any T > 0 an x(0) E, the unique solution x(t) = (x 1 (t), x 2 (t),...) to the initial value problem efine by (1) an (2) satisfies sup x i (t) exp(λt ) 1 + x i (0). (11) 0 t T Proof: By (2) we have t x i(t) λx i 1 (t) for i 2. Hence, for i 2 t t x i (t) = x i (0) + x i (u) x i (0) + λ x i 1 (u)u. u=0 u=0 Combining the above inequality with the fact that x 1 (t) 1, reaily allows us, by inuction on i, to establish the following inequality: i x i (t) x j (0) (λt)i j (i j)! + (λt)i 1 (i 1)!. j=2 Interchanging the orer of summation therefore yiels sup x i (t) x j (0) (λt ) i j 0 t T (i j)! + (λt ) i 1 (i 1)!, j 2 i j which implies (11). Theorem 7 reaily follows from (11) by efining K as K = {x E t, i 0 : x i x i (t)) < 2 i }, such that sup x i < ( ) sup x i (t) + 2 i x K 0 t T exp(λt )(1 + x i (0)) + 1 <. APPENDIX C PROOF OF THEOREM 9 Define the set F m E as F m = {x E i 1 x i m}. If we consier the metric space (E, ρ), where ρ is the weighte L 1 -norm x i i 1 2, then F i m is compact for any m > 0. Note, it suffices to prove tightness in this metric space as Prokhorov s theorem hols in any separable metric space [20]. To prove that the measures (π (N) ) N 1 are tight, we will show that for any ɛ > 0, setting m ɛ = 1 (1 λ)ɛ implies that P{πN F mɛ } > 1 ɛ for all N. We start by consiering a moifie system consisting of N noes in which we give preemptive priority to local jobs, that is, transferre jobs are interrupte whenever a local job arrives (an can be transferre to yet another noe). Let X (N) i,mo (t) {0,..., N} be the number of noes with at least i jobs in the queue at time t in the moifie system. Due to the exponential job size urations we have X (N) i,mo (t) = X(N) i (t), which implies that the moifie system consisting of N noes has the same stationary measure π (N), meaning it suffices to prove tightness for the moifie system. Using the moifie system consisting of N noes, we can now rely on a simple sample path argument to show that the length of queue i, for i = 1,..., N, in the moifie system is upper boune by one plus the queue length of the i-th queue in a system consisting of N inepenent M/M/1 queues. After all, in the moifie system service to the local jobs is never prevente by a transferre job, each queue contains at most one transferre job an some local jobs may even be transferre. As the stationary queue length istribution in an M/M/1-queue is geometric with a mean equal to λ/(1 λ), we may conclue that the mean queue length in the i-th noe of the moifie system is upper boune by 1/(1 λ), for i = 1,..., N. Let Y (N) i,mo (t) be the ranom variable representing the queue length of the i-th queue in the moifie system at time t an set Y (N) i,mo = lim t Y (N) i,mo (t), then 1 N Y (N) i,mo N (t) = 1 N X (N) i,mo N (t). i=1 i=1 Therefore, by the Markov inequality, { } P{π N 1 N F mɛ } = 1 P Y (N) i,mo N > m ɛ i=1 { } 1 N 1 E Y (N) i,mo /m ɛ N with m ɛ = 1 (1 λ)ɛ 1 for all N. i=1 1 (1 λ)m ɛ = 1 ɛ,

12 12 Wouter Minnebo receive his M.Sc. egree in Computer Science from the University of Antwerp (Belgium) in In October 2011, he joine the Performance Analysis of Telecommunication Systems research group as a PhD-stuent, at the Mathematics an Computer Science Department of the University of Antwerp. His main research interests inclue the performance analysis of loa balancing an sharing strategies in large istribute networks. Benny Van Hout (benny.vanhout@ua.ac.be) receive his M.Sc. egree in Mathematics an Computer Science, an a PhD in Science from the University of Antwerp (Belgium) in July 1997, an May 2001, respectively. From August 1997 until September 2001 he hel an Assistant position at the University of Antwerp. Starting from October 2001 onwars he has been a postoctoral fellow of the FWO-Flaners. In 2007, he became a professor at the Mathematics an Computer Science Department of the University of Antwerp, where he is a member of the PATS research group. His main research interest goes to the performance evaluation an stochastic moeling of computer systems an communication networks. He has publishe several papers, containing both theoretical an practical contributions, in a variety of international journals (e.g., IEEE JSAC, IEEE Trans. on Inf. Theory, IEEE Trans. on Comm., Performance Evaluation, Journal of Applie Probability, Stochastic Moels, Queueing Systems, etc.) an in conference proceeings (e.g., ACM Sigmetrics, Networking, Globecom, Opticomm, ITC, etc.).