On the Relative Value of Local Scheduling versus Routing in Parallel Server Systems

Transcription

1 On the Relative Value of Loal Sheduling versus Routing in Parallel Server Systems Rong Wu as.om 343 Thornall Street Edison, NJ, USA Douglas G. Down Department of Computing and Software MMaster University Hamilton, Ontario, Canada L8S 4K1 Abstrat We onsider a system with a dispather and several idential servers in parallel. Tas proessing times are nown upon arrival. We first study the impat of the loal sheduling poliy at a server. To this end, we study random routing followed by a priority sheme at eah server. Our numerial results show that the performane mean waiting time) of suh a poliy ould be signifiantly better than the best nown suggested poliies that use FCFS at eah server. We then propose to use multi-layered round robin routing, whih is shown to further improve system performane. Our analysis involves a ombination of omparing analyti models, heavy traffi asymptoti and numerial wor. 1. Introdution A system with a front-end dispather and several baend parallel servers an be used to model systems suh as lustered web servers. We study models in whih tas proessing times are nown upon arrival. Many sheduling poliies have been proposed and studied for suh a system we list several referenes below). The sheduling poliies that we onsider have two omponents. The first is the routing poliy, whih is used to determine how tass are assigned to servers. Examples are random routing, where an arrival is assigned to a server aording to a fixed set of probabilities, round robin RR) the ith arriving tas is assigned to the i mod )th server, where is the number of servers), and Join the Shortest Queue JSQ). Notie that different information about the system state may be required by a routing rule: round robin does not require any information from the servers, while JSQ requires the queue length information at eah server. The seond part is the loal sheduling poliy, whih shedules the order of proessing at a server. Examples are First Come First Served FCFS) and Shortest Remaining Proessing Time SRPT). Reent researh shows that high tas proessing time variane appears to be a feature in many aspets of omputing systems, suh as sizes of files transferred through the Web [2], I/O times [9], and sizes of FTP transfers in the Internet [8]. For a system with high tas size variability, Harhol-Balter and her o-authors in [6] proposed a size based poliy alled SITA-E. There is a size range assoiated with eah server and a tas is routed to the appropriate server by its size. Eah server uses FCFS loal sheduling to proess its assigned tass. They also showed that under highly variable tas proessing time distributions, SITA-E routing may signifiantly outperform a routing poliy that assigns arrivals to the queue with the smallest worload with FCFS loal sheduling). The advantage of SITA-E is that it redues the variane of tas proessing times at eah server and thus improves the overall system performane. Similar poliies suh as EquiLoad and AdaptLoad are suggested in the wor of Risa et al. [1, 11, 15]. In Zhang et al. [14], the authors onsidered a size based poliy for the situation where tass are autoorrelated. They proposed a poliy whih distributes tass suh that the load to eah server is proportional to the orrelation struture of the arrival proess. Their results showed that for the autoorrelated ase, not all servers are equally utilized, but this imbalane brings signifiant performane improvement. Currently, this group of poliies appears to demonstrate the best performane in the literature. They are all also relatively easy to implement. For a single server system, Shrage in [12] showed that for an arbitrary arrival proess and arbitrary proessing time distribution, SRPT minimizes the number of tass in system. Therefore if one wishes to loo at suh performane measures as mean waiting time, SRPT is a good hoie for the loal sheduling poliy. In [3], we did simulations for SITA-E and round robin routing followed by shortestremaining-proessing-time loal sheduling RR-SRPT) for proessing times that follow a bounded Pareto distribution. The results showed that the expeted queue length for SITA-

2 E an be signifiantly larger than that for RR-SRPT. In [4], we proposed the use of multi-layered round robin routing followed by SRPT loal sheduling RRK-SRPT) for a disrete tas proessing time distribution with K types of tass). By multi-layered round robin, we mean that the dispather uses independent round robin poliies for eah type of tas. We showed that RRK-SRPT performed better than RR-SRPT in heavy traffi RRK-SRPT is asymptotially optimal in heavy traffi). This provides some evidene that similar poliies may be good in general, but is not ompletely onvining. The main drawbas are that [3] is based on simulation only, while in [4], ontinuous distributions are not onsidered, whih in turn prevents a fair omparison with poliies onsidered elsewhere in the literature. There are two barriers to a further study of SRPT-based poliies. On the theoretial side, it is diffiult to analyze SRPT. For example, the expressions in Shrage and Miller [13] for an M/G/1 queue are quite ompliated, so when we move to the multiple server environment, where in partiular the arrival stream to an individual server may no longer be Poisson, exat analysis beomes quite diffiult. One possibility would be to loo at a diffusion approximation as in [4], but unfortunately no results exist for the diffusion approximation of a single queue under SRPT when the proessing time distribution is ontinuous. The other barrier is on the implementation side. In most appliations, SRPT is simply too diffiult to implement. However, in [5], Harhol-Balter et al. show that SRPT may be approximated by a priority sheme in a manner that is straightforward to implement. We provide more details of this sheme in Setion 3 below. It turns out that examining suh an approximate poliy allows us to set aside the diffiulties in the theoretial analysis of SRPT. In [7], Jaiswal provided a formula to alulate the mean number of eah type of tass for an M/G/1 preemptive priority queue, whih gives us a way to analyze the performane diretly. In addition, diffusion approximations for priority systems are well studied, see Reiman [10], amongst others. We are onerned in this paper with showing that there exists a routing poliy that, ombined with the SRPT approximation, yields better performane than if FCFS were used as the loal sheduling poliy in partiular, we mae omparisons with SITA-E). Thus, our ontribution is not so muh methodologial we apply existing results), but shows that a different viewpoint on sheduling systems of parallel servers an yield better performane. Rather than using routing to ompensate for ineffiienies in the loal sheduling poliy when FCFS is used in the fae of high variane), we suggest using routing to play to the strengths of a more effiient yet still implementable) loal sheduling poliy. If one has ontrol over the loal sheduling poliy whih is not always the ase), then it may be worthwhile to onsider suh poliies. It is worth noting that many operating systems have built in support for suh priority shemes. To emphasize this viewpoint we first study a system onsisting of naive i.e. random) routing of arrivals followed by priority sheduling at eah server. We give expressions for the performane of suh a system that are valid over all system loads and perform a numerial study. We then give asymptoti results that show superior performane even with naive routing. We then give a simple adjustment to the routing sheme that in turn gives a performane improvement. We hope that our results will in general persuade designers to first tale loal sheduling in suh systems if at all possible) and if time/resoures permit. then examine the routing issue. The organization of this paper is as follows. In Setion 2, we study the performane of preemptive priority sheduling KLP) on a single server. We provide an algorithm to alulate the utoff points for tas types. In Setion 3, we introdue the multiserver model and desribe SITA-E and our proposed poliy Random-KLP). In Setion 4, we ompare the performane of SITA-E and Random-KLP by both numerial and heavy traffi analysis. In Setion 5, we disuss the performane of RRK-KLP by simulation and heavy traffi analysis, demonstrating additional performane gains. Setion 6 provides some onluding remars. 2. K Level Preemptive Priority on a Single Server In this setion, we desribe the proposed poliy for a single server. We assume that arrivals follow a Poisson proess with rate λ 0. The proessing times are assumed to be independent and identially distributed i.i.d. ) and follow a density fx). We also assume that the proessing times are nown upon arrival. We have mentioned that the authors in [5] suggested to use preemptive priority sheduling to approximate SRPT. For the proposed poliy, we partition the support of the proessing time distribution into K intervals. A tas whose proessing time lies in interval given by [y 1, y ) is alled a type tas. Type i tass have preemptive priority over type j tass if i < j. The proportion of type tass is α = y y 1 fy)dy, and the onditional density funtion for the size of a tas given that it is type is f y) = fy)/α, y [y 1, y ). A server then uses FCFS within tass of the same type. There are K priority levels. Note that this differs slightly from the poliy in [5], as they appear to implement a mehanism that dynamially hanges a tas s priority. The performane of their poliy should atually be better than that desribed here but is diffiult to analyze. On the other hand, the proposed poliy is very straightforward to implement, if a priority sheduling poliy is available. We all suh a poliy K level preemptive priority KLP). 2

3 Jaiswal in [7] derived the formula for the mean queue length of type tass Q ) for an M/G/1 preemptive priority queue, where type tass have preemptive resume) priority over type tass if <. Let λ = α λ 0, µ = 1/ y y 1 yf y)dy), and ρ = λ /µ. For KLP, if i=1 ρ i < 1, from 7.42) of [7], we have Q = ρ 1 1 i=1 ρ + i λ i=1 λ ie[y 2 i ] 21 1 i=1 ρ i)1 i=1 ρ i), 1) where E[Yi 2] = y i y i 1 y 2 f i y)dy, and ρ = λ /µ is the load due to type tass. The total mean queue length over all types, Q 0 is Q 0 = + ρ 1 1 i=1 ρ i λ i=1 λ ie[yi 2] 21 1 i=1 ρ i)1 i=1 ρ i) ). 2) The expression for the mean total queue length is quite ompliated. One would lie to ompute the utoffs for tas type i using 2), however this is quite diffiult. In general, one should hoose as many intervals as possible. If, for implementation reasons, one wishes to derease the number of tas types, Proedure 1 below gives a means to use 2) to find a reasonable number of tas types. Proedure 1 For an M/G/1 system operated under KLP with arrival rate λ 0, if the tas proessing times follow a ontinuous distribution with density funtion fx), the utoff points an be alulated by the following steps. Step i: Partition the support of the proessing time distribution into N intervals we also all them N types) suh that ρ i = ρ j for all 1 i, j N and i j, where ρ i is the load for type i tass ρ i = λ i /µ i ). Calulate the total mean queue length Q N with the N intervals by using 2). Let Q min = Q N. Step ii: If N = 1 then stop; else derease the number of intervals by 1 by merging two adjaent intervals. Note that this involves N 1 ases. Calulate the mean queue length for eah of the N 1 ases with N 1 intervals and tae the minimum mean queue length Q N 1. When merging intervals and N 1), the load for the merged interval is ρ + ρ +1. Step iii: If Q N 1 Q min Q min > δ δ 0, is hosen by the user); then stop, return the N intervals. else go to the next step. Step iv: If Q N 1 < Q min then Q min = Q N 1. Let N = N 1, go ba to Step ii. We now study the performane numerially. Tables 1 and 2 provide the mean queue length for different utoff points K is the number of intervals) with different system loads using proessing times following a bounded Pareto distribution, varying the parameters to ahieve different worloads ρ). The bounded Pareto distribution has density f given by: fx) = { αp α 1 1 p 1/p 2) α x α 1 0 < p 1 x p 2 0 otherwise where 0 < α < 2 is the exponent of the power law, p 1 is the smallest possible observation, and p 2 is the largest possible observation. We let p 1 = 512 and p 2 = The reason we hoose these parameters is to eep onsistent with the study in [6]. Tables 1 and 2 show how, for various α, expression 2) varies when one starts with 20 intervals and uses Proedure 1 to redue the number of intervals. It is not hard to see that varying δ will yield different results. Further, note that if we hoose the tolerane as small as δ = 0.1, for the bounded Pareto distribution, we ould ahieve better mean queue length using 4-6 types. If we allow a higher tolerane, for moderate and light system loads, good performane an be ahieved using just two types. In the next setion, we apply KLP to multiple servers. We now move to showing how well parallel server systems eah implementing KLP) perform, under naive random) routing. Table 1. Mean Queue Length with ρ = 0.9 for a Single Server K α = 1.2 α = 1.5 α = Parallel Server Model In this setion, we introdue the system model for the proposed poliy, SITA-E, and related poliies. We assume 3

4 Table 2. Mean Queue Length with ρ = 0.5 for a Single Server K α = 1.2 α = 1.5 α = that arrivals follow a Poisson proess with rate λ. Upon arrival, a tas must be immediately assigned to one of idential servers. The proessing times are assumed to be i.i.d. and follow a density fx) whih satisfies 0 x 2 fx)dx <. Note that this disounts some heavy-tailed distributions, but ertainly allows for proessing times with high variane we onsider bounded Pareto distributions later). We let {u i, i 1} and {v i, i 1} be the interarrival and proessing time sequenes, µ be the proessing rate, and s the variane of the proessing times SITA-E and related poliies For SITA-E [6], an arrival is routed to server if its proessing time lies in the range [x 1, x ), where x 0 = 0 and x =. The intervals are hosen suh that x1 0 xfx)dx = x2 x 1 xfx)dx = = x 1 xfx)dx. The loal sheduling poliy at eah server is FCFS. Note that the hoie of intervals balanes the load at eah of the servers, while at the same time yielding a proessing time variane at eah server that is typially muh less than the overall proessing time variane. Simulation results in [6] show that this poliy appears to signifiantly outperform other poliies in partiular a routing poliy that assigns arrivals to the queue with the smallest worload, followed by FCFS loal sheduling). Other size based poliies in [1, 11, 15] are similar to SITA-E. If the tas proessing time distribution is fixed, the performane of these poliies is the same as SITA-E. Therefore, in this paper, we only ompare the proposed poliy with SITA-E. Notie that we do not onsider the situation when the tas proessing time distribution is not fixed. We refer the reader to [1, 11, 15] for poliies that reat to time varying worloads Random-KLP For the proposed poliy, we would lie to use KLP as the loal sheduling poliy. To fous on the relative importane of the loal sheduling poliy, we first onsider a naive routing poliy random routing). 4. SITA-E and Random-KLP - Performane Analysis In this setion, we ompare SITA-E and Random-KLP both numerially and asymptotially in a heavy traffi sense). Setion 4.1 ompares the performane by using exat analyti expressions. In Setion 4.4, we ompare the performane in heavy traffi Analyti models In this setion, we ompare the performane of SITA-E and Random-KLP by providing expressions for both the mean waiting time and slowdown. The formulas are of suffiient omplexity that a diret omparison is not possible, so we provide numerial results to support our findings. Setion 4.2 provides the numerial results for mean queue length. Setion 4.3 disusses the slowdown of eah tas type Mean queue length We are interested in minimizing the mean waiting time, but due to Little s Law, we an equivalently loo at the mean number of tass in the system. Let Q denote the mean queue length at server and Q represent the total mean queue length for SITA-E. Also, let β = x x 1 fx)dx be the proportion of arrivals that are assigned to server. Define λ = β λ and µ = 1/ x x 1 xfx)/β dx). We then have, using the Pollaze-Kinhin formula for eah server, Q = ρ + ρ2 + λ2 s ), 3) 21 ρ ) where ρ = λ /µ, and s = x x 1 x 2 fx)/β dx x x 1 xfx)/β dx) 2. In Setion 2, we provided the formula for total mean queue length of KLP at a single server. Let λ = α λ/ and µ = 1/ y y 1 yf y)dy). For Random-KLP, the total 4

5 mean queue length Q ) is thus Q = + ρ 1 1 i=1 ρ i λ i=1 λ i E[Y i 2] 21 1 i=1 ρ i )1 i=1 ρ i ) ), 4) where E[Yi 2] = y i y i 1 z 2 f i z)dz, and ρ = λ /µ, whih is the load due to type tass at a server. It is hard to ompare Q and Q sine the expressions 3) and 4) are quite ompliated. Thus, we use the expressions 3) and 4) to gain insight through numerial omparisons. Tables 3 and 4 provide some numerial results for SITA-E and Random-KLP using proessing times following a bounded Pareto distribution, varying the parameters to ahieve different worloads ρ). In the Poliy olumn in Tables 3 and 4 we add a symbol * to represent the ase when the utoff points are hosen to balane the loads for eah type of tass at eah server. This means we hoose a utoff point suh that λ i /µ i = λ j /µ j λ i represents the arrival rate for type i tass to a single server), where 1 i, j K. For the poliies without *, we use the utoff points alulated aording to Proedure 1 with a starting point of 20 intervals). Note that for Random-KLP, we get the results diretly from the single server formula. Table 3. Mean Queue Length with ρ = 0.9 Poliy α = 1.2 α = 1.5 α = 1.9 SITA-E Random-3LP Random-4LP Random-4LP* Random-7LP Random-9LP Random-13LP SITA-E Random-3LP Random-7LP Random-8LP* Random-13LP SITA-E Random-4LP Random-13LP Random-16LP* The numerial results for the mean queue length under both poliies show that, under bounded Pareto distributions, if we hoose the same tas intervals for Random- KLP and SITA-E, Random-KLP onsistently outperforms SITA-E for different system loads and tas proessing time Table 4. Mean Queue Length with ρ = 0.5 Poliy α = 1.2 α = 1.5 α = 1.9 SITA-E Random-2LP Random-4LP Random-4LP* Random-5LP SITA-E Random-4LP Random-5LP Random-8LP Random-8LP* SITA-E Random-5LP Random-8LP Random-13LP Random-16LP* variane. A further signifiant improvement an be made by using Proedure 1. While these numerial results are quite suggestive, they are not onlusive. We provide additional evidene in the form of heavy traffi asymptoti results in Setion Slowdown Slowdown is a performane metri that inorporates the notion of fairness. For any poliy, if E[Wx)] is the mean waiting time for a tas of size x, the slowdown for a tas of size x, Sx), is given by Sx) = E[Wx)]. 5) x For SITA-E, the mean waiting time for type 1 ) tass is E[Wx)] = 1/µ + ρ2 + λ2 s 2λ 1 ρ ), x 1 x < x. 6) For Random-KLP, the mean waiting time for type 1 K) tass is E[W x)] = ρ λ 1 1 i=1 ρ i )+ i=1 λ i E[X2 i ] 21 1 i=1 ρ i )1 i=1 ρ i ), 7) where x 1 x < x. Combining 5), 6), and 7), we an ompare the slowdown for eah type of tass under SITA-E and Random-KLP. Figure 1 gives the slowdown for tass under SITA-E and Random-4LP with 4 servers, where α = 1.2 and ρ = 0.9. The utoff points for Random-4LP are alulated aording to Proedure 1. 5

6 Slowdown SITA E Tas Size x Random 4LP Figure 1. Slowdown for Random-4LP unbalaned loads) and SITA-E The results show that for a bounded Pareto distribution, if we use the utoff points alulated by Proedure 1, the slowdown of Random-KLP for some tass may not be better, although the mean queue length is better. However, if we use the same types for SITA-E and Random-KLP, the slowdown for eah type of tass for Random-KLP appears to be better than that of SITA-E. Figure 2 illustrates suh a situation, where we hoose four types with balaned wor loads for α = 1.2 and ρ = 0.9. Slowdown Random 4LP Tas size x SITA E Figure 2. The Slowdown for SITA-E and Random-4LP balaned loads) Our numerial studies suggest that the performane for Random-KLP is better than that of SITA-E for bounded Pareto distributions if we use the same tas type intervals. To ahieve lower mean queue length, we ould use Proedure 1 with lower tolerane e.g. 0.1) and a larger number of starting points 20 in our examples) to partition tass. However, the tradeoff for this is that the slowdown for ertain types of tas for Random-KLP ould be worse than for SITA-E. The expressions we use here allow one to experiment to hoose a good performane tradeoff, aording to the designer s goals there is no need to resort to simulation) Performane analysis in heavy traffi In the previous setion, we provided evidene of the relative benefits of loal sheduling versus routing. In this setion, while we are unable to provide a proof that Random- KLP outperforms SITA-E over all loads, we are able to show better performane in an asymptoti sense, when the load on the system approahes one. We use the tool of heavy traffi limits to approximate the mean queue length for SITA-E and Random-KLP. To do this, assume that there is a sequene of systems indexed by n) suh that λn) λ, µn) µ, and sn) s, as n. Further assumptions on the sequene of systems will be made as needed below SITA-E We first onsider the performane of SITA-E. For eah server, let {u i, i 1}, {v i, i 1} represent the interarrival and proessing time sequenes, respetively. For the sequene of systems indexed by n), we have for 1, λ n) = E[u 1 n)]) 1 = β λn) λ = β λ, a n) = V aru 1 n)) = 1 β 2λ2 n) a = 1 β 2, λ2 µ n) = E[v1n)]) 1 1 = x x 1 xf n) x)dx 1 µ = x x 1 xf x)dx, s n) = V arv1n)) x ) 2 x = x 2 f n) x)dx xf n) x)dx x 1 x 1 x ) 2 x s = x 2 f x)dx xf x)dx. x 1 x 1 We also require sup n 1 E[u 1 n))2+ε ] < and sup n 1 E[v 1 n))2+ε ] <, for some ε > 0. 6

7 The load of type tass in the nth system is defined as ρ n) = λn)β /µ n), 1 K. We have ρ n) ρ = β λ/µ 8) for all. We assume that ρ n) < 1 for all and n, and ρ n) 1 as n for all. We define Q t) to be the number of tass at server n) at time t. We are interested in the proess ˆQ t) = n 1/2 Q n) nt). We assume that Qn) 0) = 0, 1. For eah server, we also need to define d n) = nρ n) 1). We assume d n) d as n, < d < 0. 9) We all 9) the heavy traffi ondition note that 9) implies our earlier assumption that ρ n) 1 as n ). We now have the following result, where RBMa, b) denotes a refleted Brownian motion with drift a and variane b. Also, = denotes wea onvergene in the metri spae D onsisting of all right ontinuous funtions with left limits. Theorem 1 For queue 1 ) in a system operating under SITA-E, under the heavy traffi onditions, ˆQ n) = µ RBM d, β λs + 1 ). 10) β λ Proof. This follows diretly from Theorem 3.4 of [10] Random-KLP We fous on one server and separate the K different types into different arrival streams for that server. For eah type, we use {ũ i, i 1}, {ṽ i, i 1} to represent the interarrival and proessing time sequenes at eah server), respetively. Without loss of generality, we have ũ 1 = j=1 u j and ṽ 1 = v 1, where {u i, i 1} and {vi, i 1} are the interarrival and proessing time sequenes for type tass to the system. If λ, a, µ, and s are, respetively, the arrival rate, interarrival time variane, servie rate, and proessing time variane for type tass at a single server, we then have 1 λ n) λ = E[ u j] = α λ/, a n) a = V ar j=1 u j) = j=1 2 α 2 λ2, 11) µ n) µ = E[v1 1 ]) 1 = y y 1 yf y)dy, s n) s = V arv 1) [ We also require sup n 1 E ] j=1 u j n))2+ε < and sup n 1 E[ṽ1 n))2+ε ] <, for some ε > 0. We define Q t) to be the number of type tass at a single server at time t. As in the previous setion, we ˆQ n) are interested in the proess t) = n 1/2 Q n) nt). We assume that Q n) 0) = 0, 1 K. We also define d n) = n i=1 ρ i ), n)/ 1 where ρ i n) = λn)α i /µ i n) denotes the load of type i tass on the system. We mae the following assumptions d n) as n, 1 K 1,12) d Kn) d as n, < d < 0. 13) From the definition of ρ n), we have the load of type tass on a single server ρ n)/ ρ / 14) as n, 0 ρ / < 1, 1 K 1. We all 12), 13), and 14) the heavy traffi onditions. We an now give a result for the heavy traffi limit under Random-KLP. Theorem 2 For a single queue operating under Random- KLP, under the heavy traffi onditions, ˆQ n) = 0, 1 K 1, 15) ˆQ n) K = µ KRBM d α λs, + α λ z )) z 1 yf y)dy) 2. 16) Proof. This follows diretly from Theorem 4.1 and Theorem 4.3 of [10] Comparison Now we an ompare SITA-E and Random-KLP in heavy traffi. We first need to ompute the mean of the heavy traffi limit for the total number of tass in the system. Let QRandom = Q Random, where Q Random is the mean of the RBM in 16). Also let QSITA-E = Q SITA-E, where Q SITA-E is the mean of the RBM in 10). The following theorem suggests that RRK-KLP should outperform SITA-E under high loads, for all proessing time distributions with finite variane. = ) 2 y y 2 f y)dy yf y)dy. y 1 y 1 y Theorem 3 For a ontinuous proessing time distribution with density funtion fx), where x2 fx)dx <, 7

8 in heavy traffi, if the proessing time distribution is partitioned for Random-KLP in an idential manner as for SITA-E whih in turn implies K = for Random-KLP), Q Random < Q SITA-E. Proof. First, we note that the fat that K = implies that the assumptions we made for both systems are in fat equivalent, so the omparison is fair. It also implies that x = y, d = d, µ = µ, a = a, s = s, and α = β, so in the remainder of the proof we will use the quantities on the left hand side of eah of the above equalities. First, for SITA-E, Q SITA-E = = K + 1 ) 1 1 µ α λ + α λs α λs µ. The last step is due to the heavy traffi onditions the load on server approahes 1). Now, for Random-KLP, Q Random = µ K < 1 α λ = K + 1 = Q SITA-E. µ 2 + α ) λs α λµ K µ µ K + 1 α λs µ α λs µ So, we have a result that suggests that Random-KLP outperforms SITA-E for heavily loaded systems. Note that this is for a partiular hoie of K, so if we were to find the optimal value of K and the assoiated intervals [y 1, y )), one would expet the performane to further improve. However, we will see that if we were to use the limiting proess to do suh an optimization, the problem is not well-posed. We emphasize that the main result of this setion provides a guarantee of better performane with respet to the limiting proesses. Of ourse, one suggestion would be to use the tehniques desribed in Setion 2. Using the asymptoti expressions, it is not possible to alulate the optimal utoff points, even for the ase of K = 2. Let T be the sole utoff point for K = 2. We annot use 16) to alulate the utoff point. The reason is that the expression for Q Random is dereasing as T is inreasing. As T goes to the maximum value of the tas proessing time, α 1 goes to 1 and µ 1 goes to µ. Hene the load for type 1 tass is λ/µ), whih equals 1. This ontradits the assumption that ρ 1 < 1. In other words, 15) will not hold for this ase. The fat that the optimal value of T in 16) leads to a situation whih violates the assumptions required for Theorem 2 leads to a logial inonsisteny that prevents us from using the limiting proesses to find the optimal utoff point. Suh problems also arise for larger values of K. 5. Multi-layered Round Robin Routing We have provided some guidelines for parameter hoies suh that Random-KLP performs better than SITA-E. Our results in [4] suggest that if we use multi-layered round robin routing tass of type follow a round robin poliy that is independent of arrivals of all other tas types), we may ahieve muh better performane. We use RRK to represent multi-layered round robin routing. In this setion, we disuss the performane of RRK-KLP. Setion 5.1 provides some simulation results for RRK-KLP. Setion 5.2 provides asymptoti results for RRK-KLP Simulation results Our simulation is based on a bounded Pareto distribution for the proessing times. Table 5 gives the 90% onfidene interval for the mean queue length with ρ = 0.9 and = 4 for RRK-KLP. We ran 30 repliations for eah ase, and eah repliation onsists of arrivals. The utoff points are hosen aording to Proedure 1, where we use the best utoff points the one providing the minimum mean queue length for Random-KLP). We also put the orresponding numerial results for Random-KLP in the table. Table 5. Mean Queue Length of RRK-KLP and Random-KLP Poliy α 1/λ Mean Queue Length RR13-13LP , 9.348) Random-13LP RR9-9LP , ) Random-9LP RR7-7LP , ) Random-7LP Heavy traffi As we did for Random-KLP in Setion 4.6, we fous on one server and separate the K different types into different arrival streams for that server. Define Q t) to be the number of type tass at a single server at time t. We are interested in the proess Q ˆ n) t) = n 1/2 Q n) nt). We assume that Q n) 0) = 0, = 1,...,K. Under the same onditions as Theorem 2 note that the interarrival time variane onverges to /α 2 λ2 ) see 11))), we have the following result. 8

9 Theorem 4 For a single queue operating under RRK- n) KLP, under the heavy traffi onditions, Q ˆ 0, = 1,...,K 1 and ˆ Q n) K = µ K RBM d, α λs )) + α λ z z 1 yf y)dy) ) Proof. This follows diretly from Theorem 4.1 and Theorem 4.3 of [10]. Let Q RRK = Q RRK, where Q RRK is the mean of the RBM in 17). The following theorem suggests that RRK- KLP outperforms both Random-KLP and SITA-E under high loads, for all proessing time distributions with finite variane. Theorem 5 For a ontinuous proessing time distribution with density funtion fx), where x2 fx)dx <, in heavy traffi, if the proessing time distribution is partitioned in an idential manner for RRK-KLP, Random- KLP, and SITA-E whih in turn implies K = for RRK- KLP and Random-KLP), Q RRK < Q Random < Q SITA-E. Proof. For RRK-KLP, Q RRK = µ K < µ K α λ 2 µ 2 α λ = Q Random. µ 2 + α ) λs + α ) λs Combining with Theorem 3, we immediately get that Q RRK < Q Random < Q SITA-E. 6. Conlusion We have shown that poliies that are simple to implement and fous on loal server sheduling yield better heavy traffi performane than polies whih use FCFS for loal sheduling. We have provided a detailed algorithm and guidelines for parameter hoies that guarantee a performane improvement, with formulas that allow easily omputed omparisons should a designer wish to hoose other parameter values. Considering fairness issues, our study also shows that good mean waiting time may not lead to good mean slowdown for all type of tass at a server. However, if we balane the load for eah type of tass, the proposed poliy performs mean waiting time and slowdown) better than poliies based on FCFS loal sheduling. Finally, it would be worthwhile to explore whether there is a means to diretly alulate the optimal utoff points number and loation). Referenes [1] G. Ciardo, A. Risa, and E. Smirni. equiload: a load balaning poliy for lustered web servers. Performane Evaluation, 46: , [2] M. E. Crovella and A. Bestavros. self-similarity in world wide web traffi: evidene and possible auses. IEEE/ACM Transations on Networing, 5: , [3] D. G. Down and R. Wu. sheduling distributed server systems with highly variable proessing times. In Proeedings of the 2003 International Symposium on Performane Evaluation of Computer and Teleommuniation Systems SPECTS 03), Montreal, [4] D. G. Down and R. Wu. multi-layered round robin routing for parallel servers. Queueing Systems, 53: , [5] M. Harhol-Balter, N. Bansal, B. Shroeder, and M. Agrawal. "implementation of SRPT sheduling in web servers". In Pro. 7th Annual Worshop on Job Sheduling Strategies for Parallel Proessing, pages 11 35, [6] M. Harhol-Balter, M. Crovella, and C. Murta. on hoosing a tas assignment poliy for a distributed server system. Journal of Parallel and Distributed Computing, 59: , [7] N. Jaiswal. Priority Queues. Aademi Press, [8] V. Paxson and S. Floyd. wide-area traffi: the failure of poisson modeling. IEEE/ACM Transations on Networing, pages , [9] D. L. Peterson and D. B. Adams. fratal patterns in dasd i/o traffi. In CMG Proeedings, [10] M. Reiman. some diffusion approximations with state spae ollapse. In Leture Notes in Controls and Information Sienes, volume 60, pages , [11] A. Risa, W. Sun, E. Smirni, and G. Ciardo. adaptload: effetive balaning in lustered web servers under transient load onditions. pages , [12] L. Shrage. a proof of the optimality of the shortest remaining proessing time disipline. Operations Researh, 16: , [13] L. Shrage and L. W. Miller. "the queue M/G/1 with the shortest remaining proessing time disipline". Operations Researh, 14: , [14] Q. Zhang, N. F. Mi, A. Risa, and E. Smirni. load unbalaning to improve performane under autoorrelated traffi. In Proeedings of the 26th International Conferene on Distributed Computing Systems ICDCS 06), [15] Q. Zhang, A. Risa, W. Sun, E. Smirni, and G. Ciardo. load balaning for lustered web servers. IEEE Transations on Parallel and Distributed Systems, 16: ,