Multi-server queueing systems with multiple priority classes

Transcription

1 Muti-server queueing systems with mutipe priority casses Mor Harcho-Bater Taayui Osogami Aan Scheer-Wof Adam Wierman Abstract We present the first near-exact anaysis of an M/PH/ queue with m > 2 preemptive-resume priority casses. Our anaysis introduces a new technique, which we refer to as Recursive Dimensionaity Reduction (RDR. The ey idea in RDR is that the m-dimensionay infinite Marov chain, representing the m cass state space, is recursivey reduced to a 1-dimensionay infinite Marov chain, that is easiy and quicy soved. RDR invoves no truncation and resuts in ony sma inaccuracy when compared with simuation, for a wide range of oads and variabiity in the job size distribution. Our anaytic resuts are then used to derive insights on how muti-server systems with prioritization compare with their singe server counterparts with respect to response time. Muti-server systems are aso compared with singe server systems with respect to the effect of different prioritization schemes smart prioritization (giving priority to the smaer jobs versus stupid prioritization (giving priority to the arger jobs. We aso study the effect of approximating m cass performance by coapsing the m casses into just two casses. Keywords: M/GI/, M/PH/, muti-server queue, priority queue, matrix anaytic methods, busy periods, muti-cass queue, preemptive priority. 1 Introduction Much of queueing theory is devoted to anayzing priority queues, where jobs (customers are abeed and served in accordance with a priority scheme: high-priority (H jobs preempt medium-priority (M jobs, which in turn preempt ow-priority (L jobs in the queue. Priority queueing comes up in many appications. Sometimes the priority of a job is determined by the job s owner via a Service Leve Agreement (SLA, whereby certain customers have chosen to pay more so as to get high-priority access to some high-demand Supported by NSF Career Grant CCR-13377, NSF Theory CCR , NSF ITR CCR , and IBM Corporation via Pittsburgh Digita Greenhouse Grant 23. Emai: harcho@cs.cmu.edu; Web: harcho; Address: Department of Computer Science, Carnegie Meon University, 5 Forbes Ave., Pittsburgh, PA, 15213; Phone: ; Fax: Emai: osogami@cs.cmu.edu; Address: Department of Computer Science, Carnegie Meon University, 5 Forbes Ave., Pittsburgh, PA 15213; Phone: Emai: awof@andrew.cmu.edu; Address: Tepper Schoo of Business, Carnegie Meon University, 5 Forbes Ave., Pittsburgh, PA, 15213; Phone: Emai: acw@cs.cmu.edu; Address: Department of Computer Science, Carnegie Meon University, 5 Forbes Ave., Pittsburgh, PA, 15213; Phone:

2 resource. Other times, the priority of a job is artificiay created, so as to maximize a company s profit or increase system utiization. For exampe, an onine store may choose to give high-priority to the requests of big spenders, so that those customers are ess iey to go esewhere, see [17]. Anayzing the mean response time (and higher moments of response time for different casses of jobs is ceary an important probem. 1 Whie this probem has been we understood in the case of a singe-server M/GI/1 queue since the 195 s [5], the probem becomes much more difficut when considered in the context of a muti-server M/GI/ system, and even for an M/M/ system when jobs have different competion rates. This is unfortunate since such muti-server systems are prevaent in many appications where prioritization is used, e.g., web server farms and super-computing centers. The reason that priority queueing is so difficut to anayze in a muti-server setting is that jobs of different priorities may be in service (at different servers at the same time, thus the Marov chain representation of the muti-cass, muti-server queue appears to require tracing the number of jobs of each cass. Hence one needs a Marov chain which is infinite in m dimensions, where m is the number of priority casses. Whie the anaysis of a 1-dimensionay infinite Marov chain is easy, the anaysis of an m-dimensionay infinite Marov chain (m > 1 is argey intractabe. Prior wor The number of papers anayzing muti-server priority queues is vast, however amost a are restricted to ony two priority casses. Of those restricted to two priority casses, a assume exponentia service times. The ony papers not restricted to two priority casses are coarse approximations based on assuming that the muti-server behavior is reated to that of a singe server system [2] or approximations based on aggregating the many priority casses into two casses [19, 22]. Two priority casses We start by describing the papers restricted to two priority casses and exponentiay-distributed service demands. Techniques for anayzing the M/M/ dua priority system can be organized into four types on which we eaborate beow: (i approximations via aggregation or truncation; (ii matrix anaytic methods; (iii generating function methods; (iv specia cases where the priority casses have the same mean. Uness otherwise mentioned, preemptive-resume priorities shoud be assumed. Neary a anaysis of dua priority M/M/ systems invoves the use of Marov chains, which with two priority casses grows infinitey in two dimensions (one dimension for each priority cass. In order to overcome this, researchers have simpified the chain in various ways. Kao and Narayanan truncate the chain by either imiting the number of high priority jobs [12], or the number of ow priority jobs [1]. Nishida 1 We wi use the term response time throughout the paper to denote the time from when a job arrives unti it is competed. We wi aso occasionay ta about the deay (wasted time, which we define as the job s response time minus its service requirement. 2

3 aggregates states, yieding an often rough approximation [22]. Kapadia, Kazmi and Mitche expicity mode a finite queue system [13]. Unfortunatey, aggregation or truncation is unabe, in genera, to capture the system performance as the traffic intensity grows arge. Athough, in theory, the matrix anaytic method can be used to directy anayze a 2D-infinite Marov chain (see for exampe [3], the matrix anaytic method is much simper and more computationay efficient when it is appied to a 1D-infinite Marov chain. Therefore, most papers that use the matrix anaytic method to anayze systems invoving 2D-infinite Marov chains first reduce the 2D-infinite chain to a 1D-infinite chain by, for exampe, truncating the state space by pacing an upper bound on the number of jobs [12, 1, 16, 21]. Mier [18] and Ngo and Lee [21] partition the state space into bocs and then super-bocs, according to the number of high priority customers in queue. This partitioning is quite compex and is uniey to be generaizabe to non-exponentia job sizes. In addition, [18] experiences numerica instabiity issues when ρ >.8. A third stream of research capitaizes on the exponentia job sizes by expicity writing out the baance equations and then finding roots via generating functions. In genera these yied compicated mathematica expressions susceptibe to numerica instabiities at higher oads. See King and Mitrani [19]; Gai, Hanter, and Tayor [8, 9]; Feng, Kowada, and Adachi [7]; and Kao and Wison [11]. Finay there are papers that consider the specia case where the mutipe priority casses a having the same mean. These incude Davis [6], Kea and Yechiai [14] (for non-preemptive priorities, and Buzen and Bondi [4]. The ony wor deaing with non-exponentia service times is contained in a pair of papers, not yet pubished, by Septcheno et. a. [25, 26]. Both papers consider a two-priority, muti-server system where within each priority there may be a number of different casses, each with its own different exponentia job size distribution. This is equivaent to assuming a hyper-exponentia job size distribution for each of the two priority casses. The probem is soved via a combination of generating functions and the matrix anaytic method. In theory, their technique may be generaizabe to PH distributions, though they evauate ony hyper-exponentia distributions due to the increased compexity necessary when using more genera PH distributions. More than two priority casses For the case of more than two priority casses, there are ony coarse approximations. The Bondi-Buzen (BB approximation [2] is beautifu in its simpicity and usabiity. Is is based on an intuitive observation that the improvement of priority scheduing over FCFS scheduing under servers is simiar to that for the case of one server with equa tota capacity: E[D M/GI//prio ] E[D M/GI//FCFS ] E[DM/GI/1/prio ] E[D M/GI/1/FCFS = scaing factor. (1 ] 3

4 Here E[D M/GI//prio ] is the overa mean deay under priority scheduing with servers of speed 1/, and E[D M/GI//FCFS ] is defined simiary for FCFS. This reation is exact when job sizes are exponentia with the same rate for a casses; however what happens when this is not the case has never been estabished. The other approximation (which we denote by MK-N which aows for more than two priority casses and exponentia job sizes is due to Mitrani and King [19], and aso used by Nishida [22] to extend the atter author s anaysis of two priority casses to m > 2 priority casses. The MK-N approximation anayzes the mean deay of the owest priority cass in an M/M/ queue with m 2 priority casses by aggregating a the higher priority casses. Thus, instead of aggregating a jobs into one cass, as BB does, MK-N aggregates into two casses. The job size distribution of the aggregated cass is then approximated with an exponentia distribution by matching the first moment of the distribution. Contributions of this paper In Section 2, we introduce a new anaytica approach that provides the first near-exact anaysis of the M/PH/ queue with m 2 preemptive-resume priority casses. Our approach, which we refer to as Recursive Dimensionaity Reduction (RDR, is very different from the prior approaches described above. RDR aows us to recursivey reduce the m-dimensionay infinite state space, created by tracing the m priority casses, to a 1-dimensionay infinite state space, which is anayticay tractabe. The dimensionaity reduction is done without any truncation; rather, we reduce dimensionaity by introducing busy period transitions within our Marov chain, for various types of busy periods created by different job casses. The ony approximation in the RDR method stems from the fact that we need to approximate these busy periods using Marovian (phase-type PH distributions. We find that matching three moments of these busy periods is usuay possibe using a 2-phase Coxian distribution, and provides sufficient accuracy, within one or two percent of simuation, for a our experiments (our experiments span oad ranging from ρ =.5 to ρ =.95 and job size variabiity ranging from C 2 = to C 2 = 128. The accuracy of the RDR method can be increased arbitrariy by better approximating the busy periods. In theory RDR can hande systems with any number of servers, any number of priority casses, and PH service times. In addition, RDR is quite efficient; for a the scenarios expored in this paper, the computation time under RDR is ess than a few seconds. However, the compexity of the RDR method does increase with both the number of servers and the number of casses m. Because RDR becomes ess practica under high m and, we deveop a much simper, but ony sighty ess accurate, approximation RDR-A (see Section 2.5. RDR-A simpifies cacuations by approximating an m priority system with a two priority system, which is then soved using RDR. In Section 3 we present resuts from both RDR and RDR-A for per-cass mean response time for an M/PH/ queue with mutipe priority casses, and aso discuss the computation of higher moments of response time. In Section 4, we use these resuts to obtain many interesting insights about priority queueing. First, in Section 4.1 we compare the performance of priority queueing in a muti-server system with servers 4

5 each of speed 1/ versus a singe server of speed 1. We find that the effect of priorities in a singe server system can be very different than in a muti-server system of equa capacity. (A non-surprising consequence of this resut is that the BB approximation, which reies on reating a muti-server system to a singe server system, can exhibit arge errors. Next, in Section 4.2, we study the effect of priority poicies that favor short jobs ( smart prioritization versus priority poicies that favor ong jobs ( stupid prioritization under systems with different numbers of servers. Understanding the effect of smart prioritization is important because many common scheduing poicies are designed to give priority to short jobs. Lasty, in Section 4.3, we as how effective cass aggregation (aggregating m > 2 priority casses into just 2 priority casses is as an approximation for deaing with systems having than two priority casses. We evauate severa types of cass aggregation incuding that proposed by the MK-N approximation and that used in RDR-A to show when cass aggregation serves as a reasonabe approximation. 2 RDR anaysis of M/PH/ with m priority casses In this section we describe the RDR technique. We divide our expanation into three parts. As an introduction, in Section 2.1, we dea ony with the simpest case of m = 2 priority casses and exponentia job sizes, which we sove using the techniques in [24]. We then move to the difficut case of m > 2 priority casses, but sti exponentia service times, in Section 2.2. Here the techniques from [24] do not appy, so we introduce Recursive Dimensionaity Reduction (RDR. The RDR approach uses the anaysis of the m 1 priority casses to anayze the m-th priority cass. This is a non-trivia procedure for m > 2 since it invoves evauating many compex passage times (busy periods in the chain representing the m 1 priority casses, as these passage times now form transitions within the chain representing m priority casses. Finay in Section 2.3, we show how RDR can be appied to the most genera case of m > 2 priority casses, with PH service times. A the anaysis up to through Section 2.3 deas with how to derive mean per-cass response times. In Section 2.4 we iustrate how the RDR method can be extended to obtain variance of response time for each cass. Finay, in Section 2.5, we introduce RDR-A, which is an approximation of RDR, aowing very fast (< 1 second evauation of high numbers of priority casses and servers, with sma (< 5% error. 2.1 Simpest case: Two priority casses, exponentia job sizes Consider the simpest case of two servers and two priority casses, high (H and ow (L, with exponentiay distributed sizes with rates µ H and µ L respectivey. Figure 1(a iustrates a Marov chain of this system, whose states trac the number of high priority and ow priority jobs; hence this chain grows infinitey in two dimensions. Observe that high priority jobs simpy see an M/M/2 queue, and thus their mean response time is we-nown. Low priority jobs, however, have access to either an M/M/2, M/M/1, or no server at a, 5

6 λl λl λl L,H 1L,H 2L,H 3L,H µ L 2µ L 2µ L µ λh λh H µ H λ µ H L λ µ H L L,1H 1L,1H 2L,1H 3L,1H µ L µ µ L L 2µ H 2µ H 2µ H 2µ H λl λl λl L,2H 1L,2H 2L,2H 3L,2H 2µ λh λh λh 2µ H 2µ H 2µ H H λh L,3H 1L,3H 2L,3H 3L,3H (a L,H 1L,H 2L,H 3L,H µ L 2µ L 2µ L µ λ H µ H µ H µ H H L,1H 1L,1H 2L,1H 3L,1H µ L µ L µ L B 2µH B 2µH B 2µH B 2µH L,2 H 1L,2 H 2L,2 H 3L,2 H (b L,H 1L,H 2L,H 3L,H µ L 2µ L 2µ L µ λ H µ H µ H µ H H L,1H µ 1L,1H L µ 2L,1H L µ L 3L,1H = B 2µH L,2 H 1L,2 H 2L,2 H 3L,2 H (c Figure 1: (a Marov chain for a system with two servers and two priority casses where a jobs have exponentia sizes. This Marov chain is infinite in two dimensions. Via the Dimensionaity Reduction technique, we arrive at the chain in (b, which uses busy period transitions, and is ony infinite in one dimension. In (b, the busy period is represented by a singe transition. In (c, the busy period is represented by a two phase PH distribution (with Coxian representation, yieding a one-dimensionay infinite Marov chain. depending on the number of high priority jobs. Thus their mean response time is more compicated, and this is where we focus our efforts. Figure 1(b iustrates the reduction of the 2D-infinite Marov chain to a 1D-infinite chain. The 1Dinfinite chain tracs the number of ow priority jobs exacty. For the high priority jobs, the 1D-infinite chain ony differentiates between zero, one, and two-or-more high priority jobs. As soon as there are two-or-more high priority jobs, a high priority busy period is started. During the high priority busy period, the system ony services high priority jobs, unti the number of high priority jobs drops to one. 2 The ength of time spent in this high priority busy period is exacty an M/M/1 busy period where the service rate is 2µ H. We denote the duration of this busy period by the transition abeed B 2µH. The busy period B 2µH is not exponentiay-distributed. Hence it is not cear how it shoud fit into a Marov mode. We use a PH distribution (specificay a Coxian distribution to match the first three 2 Throughout the paper a higher priority busy period is defined as the time from when the system has higher priority jobs unti there are ony 1 higher priority jobs. 6

7 H,L H,1L H,2L H,3L µ H µ L 2µ L 3µ L µ H µ H µ H 1H,L 1H,1L 1H,2L 1H,3L µ L 2µ L 2µ L 2µ λ H H 2µ λ H H 2µ λ H H 2µ H 2H,L 2H,1L 2H,2L 2H,3L µ L µ L µ L B 3µH B 3µH B 3µH B 3µH 3H,L 3H,1L 3H,2L 3H,3L Figure 2: This chain iustrates the case of two priority casses and three servers. The busy period transitions are repaced by a Coxian phase-type distribution matching three moments of the busy period duration, as shown in Figure 1. moments of the distribution of B 2µH. Parameters of the PH distribution, whose first three moments match those of B 2µH, are cacuated via the cosed form soutions provided in [23]. Figure 1(c iustrates the same 1D-infinite chain as in Figure 1(b, except that the busy period transition is now repaced by a two phase Coxian distribution. The imiting probabiities in this 1D-infinite chain can be anayzed using the matrix anaytic method, which yieds the mean response time for ow-priority jobs via Litte s aw. The ony inaccuracy in the above approach is that ony three moments of the high-priority busy period have been matched. We wi see ater that this suffices to obtain very high accuracy across a wide range of oad and job size distributions. Figure 2 shows the generaization to a three server system. We simpy add one row to the chain shown in Figure 1, and now differentiate between, 1, 2, or 3-or-more high priority jobs. This can be easiy extended to the case of > 3 servers. 2.2 Harder case: m priority casses, exponentia job sizes We now turn to the more difficut case of m > 2 priority casses. We iustrate this for the case of two servers and three priority casses: high-priority (H, medium-priority (M, and ow-priority (L. The mean response time for cass H jobs and that for cass M jobs are easy to compute. Cass H jobs simpy see an M/M/2 queue. Cass M jobs see the same system that the ow-priority jobs see in an M/M/2 queue having two priority casses. Repacing the L s by M s in the chain in Figure 1 yieds the mean response time for the M cass jobs. The anaysis of the cass L jobs is the difficut part. The obvious approach woud be to aggregate the H and M jobs into a singe cass, so that we have a 2-cass system (H-M versus L jobs. Then we coud appy the technique of the previous section, tracing exacty the number of ow-priority jobs and maintaining imited state information on the H-M cass. This is the approach that we foow in Section 2.5 in deriving our RDR-A approximation. However, this approach is imprecise because the duration of the busy periods in the H-M cass depends on whether the busy period was started by 2H jobs, 1H and 1M job, or 2M jobs 7

8 (u 1L,M,H ul,m,h (u1l,m,h µ H µ M λ M (u 1L,1M,H ul,1m,h (u1l,1m,h (u 1L,M,1H ul,m,1h (u1l,m,1h (u 1L,2M,H (u 1L,2M,H λ M p 2M,H λ M p 2M,M ul,2m,h ul,2m,h B 2 B 1 (u1l,2m,h (u1l,2m,h M,H 1M,H 2M,H 3M,H (u 1L,1M,1H (u 1L,1M,1H λ M p MH,H λ p ul,1m,1h H MH,H λ M p MH,M p MH,M ul,1m,1h B 4 B 3 B 6 (u1l,1m,1h (u1l,1m,1h 2 possibe states for end of H M busy periods M,1H 1M,1H 2M,1H 3M,1H M,2 H 1M,2 H 2M,2 H 3M,2 H (u 1L,M,2H (u 1L,M,2H p 2H,H p 2H,M ul,m,2h ul,m,2h B 5 (u1l,m,2h (u1l,m,2h 3 possibe states for start of H M busy periods Figure 3: (Left Portion of the 1D-infinite chain used to compute mean response time for ow-priority jobs in the case of three priority casses and two servers, and a exponentia service times. (Right Chain used to compute moments of the durations of the six busy period transitions. in service. By ignoring the priorities among H s and M s, we are ignoring the fact that some types of busy periods are more iey than others. Even given the information on who starts the busy period, this sti does not suffice to determine its duration, because the duration is aso affected by the prioritization within the aggregated H-M cass. Thus a precise response time anaysis of cass L requires maintaining more information. As before we want to exacty trac the number of cass L jobs. Given that there are two servers, we need to further differentiate between whether there are zero H and M jobs, one H or M job, or two or more H and M jobs. Whenever there are two or more H and M jobs, we are in an H-M busy period. For an M/M/2 with three priority casses, there are six types of busy periods possibe, depending on the state at the start the busy period (M,2H, (1M,1H, or (2M,H and the state in which the busy period ends (M,1H or (1M,H. We derive the busy period duration by conditioning on who starts and ends the busy period. Figure 3 (eft shows the eve of the 1D infinite chain in which the number of cass L jobs is u. In state (ul,vm,wh, v cass M jobs and w cass H jobs are in the system if v w < 2; otherwise, the state (ul,vm,wh denotes that we are in a H-M busy period that was started by v cass M jobs and w cass H jobs. Observe that there are six types of busy periods depicted, abeed B 1, B 2,..., B 6 ; the busy period is determined by the state in which it was started and the state in which it ends. Notice, for exampe, that both states in the fourth and fifth row are abeed (ul,2m,h, meaning that the busy period is started by two cass M jobs; but these two states differ in the cass of the job that is eft at the end of the H-M busy period: In state (ul,2m,h of the fourth row, the busy period ends eaving a cass H job, whereas in state of the 8

9 fifth row, the busy period ends eaving a cass M job. (Reca that the cass of job eft at the end of a busy period is probabiisticay determined at the beginning of the busy period and the duration of the busy period is conditioned on the cass of the job eft at the end. Here p 2M,H, for exampe, denotes the probabiity that the busy period started by two cass M jobs ends eaving one cass H job, whereas p MH,M denotes the probabiity that the busy period started by one cass M and one cass H job, ends eaving one cass M job. The remaining probabiities are defined simiary. It remains to derive the moments of the duration of busy periods, B 1, B 2,..., B 6, and probabiities p 2M,M, p 2M,H, p MH,M, p MH,H, p 2H,M, and p 2H,H in Figure 3(eft. The tric to deducing these quantities is to observe that the six busy periods correspond to passage times between two diagona (shaded eves in the chain shown in Figure 3(right, which is the 1D-infinite chain that we used to anayze the cass M performance. Note that the 3 states in the right shaded diagona eve correspond to the three possibe start states for busy periods, and the two states in the eft shaded diagona eve correspond to the two possibe end states for the busy periods. Thus, for exampe, busy period B 1 in Figure 3(eft corresponds to the first passage time from state (2M,H to state (1M,H in the chain in Figure 3(right. Liewise, probabiity p 2M,M corresponds to the probabiity that, in Figure 3(right, state (1M,H is the first state of the two possibe end states that is reached, given that the start state is (2M,H. Inter-eve passage times and ending probabiities within the chain in Figure 3(right can be cacuated using techniques deveoped by Neuts in [2]. We provide a precise description of this in Appendix A. Observe that these computations are greaty faciitated by the fact that our chains are infinite in ony one dimension. The extension of RDR to m > 3 casses is straightforward. For exampe, for the case of m = 4 casses, we proceed as in Figure 3, where we first create a chain that tracs exacty the number of jobs in cass 4, and creates busy periods for the aggregation of the three higher priority casses. Then to derive the busy periods for the three higher priority casses, we mae use of the existing chain for three casses shown in Figure 3(eft, and compute the appropriate passage times for that chain. For an M/M/ with m priority casses, there are ( m 2( m 3 1 possibe busy periods. That is, the number of different types of busy periods is poynomia in if m is constant (Θ( m, and it is poynomia in m if is constant (Θ(m ; however, it is exponentia in and m if neither nor m is constant. 3 3 Remar: We note that in practice the number of busy periods can be reduced further, so that an M/M/ with m priority casses has ( m busy periods of cass 1 to cass m 1 jobs. An advantage of this reduction is that the number of busy periods of cass 1 to cass m 1 jobs becomes independent of the type of PH distributions that is used to approximate the busy period of cass 1 to cass m 2 jobs. The tric to reducing the number of busy periods is iustrated by considering the exampe of the M/M/2 with three casses, shown in Figure 3. Here, by taing the mixture of the six busy periods, B 1, B 2,..., B 6, we can approximate the H-M busy period by four PH distributions. These four distributions of the H-M busy period are differentiated by the state from which we enter the H-M busy period (either (M,1H or (1M,H and by the state we return to after the H-M busy period (either (M,1H or (1M,H. We iustrate how using B 1 and B 3, we can obtain (the moments of the distribution of the conditiona H-M busy period when we enter the H-M busy period from (1M,H and return to (1M,H. When we are at state (1M,H, an arriva of an H job or an M job starts an H-M busy period. When the arriva is an H job (respectivey, an M job, the H-M busy period ends with an M job with probabiity p MH,M (respectivey, p 2M,M, and the conditiona duration of the H-M busy period is B 3 (respectivey, B 1. Since the arriva processes are Poisson, this conditiona H-M busy period, which ends with an M job, starts at state (1M,H with rate 9

10 Practicay speaing, the RDR approach is fast for a sma number of servers and a sma number of priority casses. In exampes we ran with an M/M/2 and 1 priority casses, the RDR agorithm yieded mean response times within tens of seconds. 2.3 Genera case: Anaysis of M/PH/ with m priority casses In this section, we expicity describe how RDR can be appied to anayze the case of PH job size distributions. We describe RDR for the case of two servers ( = 2 and two priority casses (m = 2, high (H and ow (L, where the cass H jobs have a particuar 2-phase PH job size distribution with Coxian representation, shown in Figure 4(a. 4 Generaization to higher s and higher m s is straightforward by appying the recursive agorithm introduced in Section 2.2. Anayzing the performance of cass H is trivia, since high-priority jobs simpy see the mean response time in an M/PH/2 queue, which can be anayzed via standard matrix anaytic methods. To anayze the cass L jobs, as before, we create a 1D-infinite Marov chain tracing the cass L jobs, and use busy period transitions to represent needed information regarding the cass H jobs. Observe that under the 2-phase Coxian job sizes distribution, we wi need four different types of busy periods for high priority jobs, depending on the phases of the two jobs starting the busy period (1 & 1, or, 1 & 2 and the phase of the job eft at the end of the busy period (1 or 2. To derive the durations of these busy periods, we observe that the busy periods correspond to passage times from shaded eve 3 to shaded eve 2 in the Marov chain shown in Figure 4(b. Figure 4(b describes the behavior of cass H jobs, where states trac the number of high priority jobs in the system and the phases of the jobs being processed. Namey, at state (H there are no high priority jobs in the system; at state (1H,i, there is one high priority job in phase i; at state (nh,i,j there are n high priority jobs in the system and the two jobs are being processed are in phase i and j, respectivey (jobs in the queue are a in phase 1. The first passage times in Figure 4 are computed again using techniques in [2]. Figure 4(c shows a eve of the chain that tracs the number of ow priority jobs, where the number of ow priority jobs is u. The ow priority job sizes are assumed to be exponentiay distributed, but this can be easiy generaized to PH distributions. In state (ul,h, no high priority jobs are in system. An arriva of a high priority job in state (ul,h triggers a transition to state (ul,1h,1. In state (il,1h,j, one high priority job in phase j is in the system for j = 1,2. An arriva of a high priority job in state (ul,1h,j triggers a transition to state (il,2 H,1,j for j = 1,2. In state (il,2 H,1,j, at east two high priority jobs are in the system, and the two jobs that started the busy period were in phase one and j, respectivey, for j = 1,2. The four types of busy periods are abeed as B 1, B 2, B 3, and B 4, and the duration of these busy periods is λ M p 2M,M p MH,M. Thus, the duration of this conditiona H-M busy period is B 3 with probabiity λ M p 2M,M λ M p 2M,M p MH,M and B 1 otherwise. The other three H-M busy periods can be anayzed anaogousy. 4 Under the Coxian job size distribution, a job starts in phase one where it is processed for a time exponentiay distributed with rate µ (1 H, and then either competes (with probabiity qh = 1 ph or moves to phase two (with probabiity ph. 1

11 H λh 2 possibe states for end of busy periods. µ (1 H q H (2 µ H p (1 µ H H µ (2 H q H =1 p H µ (1 H q H (a λ λ H H 1H,1 2H,1,1 3H,1,1 (1 q (1 2µ 2µ H H H q H µ (1 p (1 (1 H H (2 2µ p (2 µ H H µ 2µ H p H H H 1H,2 2H,1,2 3H,1,2 (2 2µ H 2 possibe states for start of busy periods. µ (1 H q H µ (1 H p H (2 2µ H 2H,2,2 (b µ (1 H p H 3H,2,2 (u 1L,H (u 1L,1H,1 (u 1L,1H,2 (u 1L,2 H,1,1 (u 1L,2 H,1,1 µ (2 H ul,h µ (1 H q H ul,1h,1 µ (1 H p H ul,1h,2 p (1,1,2 B 2 ul,2 H,1,1 λ p B 1 H (1,1,1 ul,2 H,1,1 (u1l,h (u1l,1h,1 (u1l,1h,2 (u1l,2 H,1,1 (u1l,2 H,1,1 p (1,2,2 B 4 (u 1L,2 H,1,2 ul,2 H,1,2 (u1l,2 H,1,2 B λ p 3 H (1,2,1 (u 1L,2 H,1,2 ul,2 H,1,2 (u1l,2 H,1,2 (c Figure 4: (a A 2-phase PH distribution with Coxian representation. (b Marov chain which wi be used to compute the high-priority job busy periods, in the case where high-priority job size have a PH distribution with Coxian representation shown in (a. (c Chain for a system with two servers and two priority casses where high priority jobs have Coxian service times. approximated by PH distributions by matching the first three moments of the busy period distribution (note that the busy period cannot start with two jobs in phase two. Finay, p (1,j, denotes the probabiity that a busy period started by two jobs in phases one and j, ends with a singe job in phase, for j = 1,2, and = 1, Computing variance of response time and higher moments Throughout our discussion of RDR thus far, we have been concerned with computing the mean per-cass response time. It turns out that computing higher moments of per-cass response time is not much more difficut. Before we present our approach, we mae two remars. First, observe that it is trivia to derive a moments of the steady-state per-cass number of jobs in the system, directy from the steady-state probabiities for the Marov chain, which we have aready computed. Unfortunatey, however, we cannot appy the beautifu generaization of Litte s Law to higher moments (see [27, 1] to immediatey get the per-cass higher moments of response time for free, since jobs do not necessariy eave our system in the order in which they arrive. Beow we setch our approach for computing per-cass variance in response time for the case of two servers, two priority casses (H and L, and exponentia service times. We wi refer to Figure 1(c during 11

12 our discussion. For cass H jobs, it is easy to compute the variance of their response time, via standard matrix anaytic methods, since they are obivious to cass L jobs. Thus we wi concentrate on cass L jobs. Consider the 1D-infinite Marov Chain shown in Figure 1(c that tracs the number of cass L jobs. Our approach thus far has been to compute the imiting probabiities, use those to derive the mean number of cass L jobs in the system, and then appy Litte s Law to yied mean response time for cass L jobs. Now, we instead use the imiting probabiities to condition on what a cass L arriva sees. Specificay, by PASTA (Poisson Arrivas See Time Averages a cass L arriva with probabiity π (il,jh wi see state (il,jh when it arrives, and wi cause the system state to change to ((i 1L,jH at that moment. To cacuate the variance in response time seen by this cass L arriva, we remove a the arcs from the Marov chain in Figure 1(c, so that there are no more cass L arrivas. This enabes us to view the response time for the cass L arriva as the first passage time of this modified chain from state ((i 1L,jH to the state where our cass L arriva departs. The ony compexity is in figuring out exacty in which state our cass L arriva departs, where our cass L arriva is the ast cass L job to enter the system. The fina cass L arriva may depart the modified Marov chain the first time it hits (1L,H or (1L,1H, depending on the sampe path the chain foows. We wi compute the passage time to go from state ((i 1L,jH to one of these states { (1L,H or (1L,1H }. It is important to observe that the first time we hit a state with 1L, the state we hit cannot be (1L,2 H, by virtue of the fact that the Marov chain doesn t have decreasing arcs in its bottom rows. If (1L,1H is the first state that we hit with 1L, then we now that we must have gotten there from (2L, 1H, which means that the singe L job remaining is in fact the ast arriva. (We re assuming preemption is done odest first to be preempted. Thus we need to now add on the passage time to go from (1L, 1H to (L, to get the fu response time for the arriva. If (1L,H is the first state that we hit with 1L, then we now that we got there from state (2L,H. In this case, the remaining 1L is equay iey to be the ast arriva or not. With probabiity haf, the ast arriva is aready gone, in which case we add nothing to the response time. With probabiity haf, this ast arriva remains, in which case we now add on the passage time to go from (1L,H to (L, to get the fu response time for the arriva. Observe that computing the above passage times is straightforward, since a the arcs have been removed. 2.5 Introducing RDR-A We have seen that the RDR method can become computationay intensive as the number of priority casses grows. This motivates us to introduce an approximation based on RDR caed RDR-A. RDR-A appies to m > 2 priority casses and PH job size distributions. The ey idea behind RDR-A is that the RDR computation is far simper when there are ony two priority casses: H and L. In RDR-A, under m priority casses, we simpy aggregate these casses into two priority 12

13 casses, where the m 1 higher priority casses become the new aggregate H cass and the m th priority cass becomes the L cass. We define the H cass to have a PH job size distribution that matches the first three moments of the aggregation of the m 1 higher priority casses. Observe that the RDR-A method is simiar to the MK-N approximation. The difference is that in MK- N, both the H and L casses are exponentiay-distributed. Thus under MK-N, the H cass ony matches the first moment of the aggregate m 1 casses, whereas under RDR-A three moments are matched. The reason that we are abe to match the first three moments, rather than just the first moment is that we have the RDR technique, which aows the anaysis of muti-server priority queues with PH job size distributions, as described in Section Resuts and Vaidation In this section we present resuts for per-cass mean response times in M/M/ and M/PH/ queues with m = 4 priority casses, derived using RDR and RDR-A, respectivey. To the best of our nowedge, these are the first such anaytica resuts in the iterature. We wi vaidate our resuts against simuation, and show that their reative error is sma. Furthermore the time required to generate our resuts is short, typicay ess than a second for each data point. Figure 5 (top row shows our resuts for per-cass mean response times in an M/M/2 queue (eft pot and an M/PH/2 queue (right pot, both as a function of oad ρ. The PH distribution used is a 2-phase PH distribution with squared coefficient of variation, C 2 = 8. A job casses have the same distribution, and the oad is distributed eveny between the four casses. The eft pot is derived using RDR and the right pot using RDR-A. Observe that the M/PH/2 queue (right pot resuts in higher mean response time than the M/M/2 queue (eft pot, as expected. In both cases the mean response time of the ower-priority casses dwarfs that of the higher-priority casses. Figure 5 (bottom row shows the reative per-cass error for our resuts, when compared with simuation. Throughout the paper we aways show error in deay (queueing time rather than response time (sojourn time, since the error in deay is proportionay greater. We define reative error as error = 1 (mean deay by RDR or RDR-A (mean deay by simuation (mean deay by simuation (%. We ony show the error for casses 2, 3, and 4, since our anaysis is virtuay exact for cass 1 (soved via matrix-anaytic methods. We see that the reative error in the mean deay of RDR and RDR-A compared to simuation is within 2% for a casses and typicay within 1%, for a ρ s. We wi see ater that this error increases ony sighty when we move to the case of priority casses with different means. Figure 6 (eft again uses RDR-A to cacuate per-cass mean response time in the M/PH/2 queue with four casses, but this time as a function of C 2, the squared coefficient of variation of the job size distribution. 13

14 mean response time cass 1 cass 2 cass 3 cass 4 mean response time cass 1 cass 2 cass 3 cass ρ ρ error (% error (% cass 2 cass 3 cass ρ 2 cass 2 cass 3 cass ρ (a M/M/2 with four casses (b M/PH/2 with four casses Figure 5: Top row shows per-cass mean response time for M/M/2 (eft and M/PH/2 (right with four priority casses. Left graph is derived using RDR and right graph is derived using RDR-A. Bottom row shows the error in our anayticay-derived mean deay reative to simuation resuts, for the corresponding graphs in the top row. (Again, a casses have the same job size distribution. As we see from the figure, the per-cass mean response time increases neary ineary with C 2. Figure 6 (right shows the reative error in mean deay when the resuts of the RDR-A anaysis in the eft pot are compared with simuation. Again the error is under 2%. We wi see ater that this error increases ony sighty when we move to the case of priority casses with different means. Finay, we note that in the above computations RDR is much more computationay efficient than simuation. Simuation requires tens of minutes to generate each figure, since the simuation is run 3 times, and in each run 1,, events are generated. By comparison our anaysis taes ony a few seconds for each figure. Further, if we try to reduce the number of events in the simuation to 1, events, to speed it up, we see five times as much variation in the simuation around our anaytica vaues. Thus, it is possibe that as we increase the number of events in simuation, the difference in our anaysis and the simuation may decrease even further. 14

15 mean response time cass 1 cass 2 cass 3 cass C 2 error (% cass 2 cass 3 cass C 2 Figure 6: (Left Per-cass mean response times for M/PH/2 with four priority casses, derived via RDR-A anaysis. (Right Reative error in anaysis of mean deay compared with simuation. 4 Comparisons and Insights In this section we appy RDR and RDR-A to answer fundamenta questions on prioritization in mutiserver systems. In Section 4.1 we compare the behavior of muti-server versus singe server systems under prioritization. In this context, we aso evauate the BB approximation, which approximates the effect of prioritization in a muti-server system by that in a singe server system. In Section 4.2 we evauate the effect of prioritization schemes which favor short jobs in muti-server systems. Finay, in Section 4.3 we study the effect of aggregating mutipe priority casses into just two casses, so as to significanty speed up the anaysis. In this context we aso evauate the MK-N approximation discussed earier. 4.1 Comparing muti-server versus singe server performance under prioritization In this section we compare systems with different numbers of servers. It is important to note that throughout these comparisons, we hod the tota system capacity fixed. That is, we compare a singe server of unit speed with a 2-server system, where each server has speed haf, with a 4-server system, where each server has speed one-fourth, etc. Figure 7 considers an M/PH/ system with two priority casses where is one (eft then two (midde then four (right, but the tota system capacity is hed fixed, and oad is fixed at ρ =.8. The ow-priority jobs are exponentiay-distributed. The high-priority jobs foow a Coxian distribution where the squared coefficient of variation for high priority jobs, CH 2, is varied. The means of the two casses are the same and the oad is spit eveny between the two casses. The pots show per-cass mean response time as a function of CH 2. A resuts are computed using RDR. The first thing to observe is that the response times in the case of one server appear very different from the response times in the case of two servers, or four servers. The effect of prioritization in a singe server system offers itte (quantitative insight into the effect of prioritization in a muti-server system, aside from 15

16 2 High Priority Low Priority 2 High Priority Low Priority 2 High Priority Low Priority E[T] 1 E[T] 1 E[T] C 2 H C 2 H C 2 H (1 server (2 servers (4 servers Figure 7: Contrasting per-cass mean response time under one server (eft, two servers (midde and four servers (right for an M/PH/ with two priority casses. Tota system capacity is fixed throughout, and ρ =.8. Resuts are obtained using RDR. the fact that in a cases the response times appear to be a neary inear function of C 2 H. Figure 7 aso iustrates some other interesting points. We see that as we increase the number of servers, under high CH 2, the performance of both high-priority and ow-priority jobs improves. By contrast, under ow CH 2, the performance can get worse as we increase the number of servers. To understand this phenomenon, observe that when CH 2 is high, short jobs can get stuc behind ong jobs, and increasing the number of servers can aow the short jobs to get a chance to serve. By contrast when CH 2 is ow, a jobs are simiar in size, so we don t get the benefit of aowing short jobs to jump ahead of ong jobs when there are more servers. However we do get the negative effect of increasing the number of servers, namey the underutiization of system resources when there are few jobs in the system, since each of the servers ony has speed 1/. The behavior under ow CH 2, where more servers ead to worse performance, is more prominent under ower oad ρ. Figure 7 aready impies that the effect of prioritization on mean response time in a muti-server system may be quite different from that in a singe server system. In Figure 8 we investigate this phenomena more cosey, by evauating when the BB approximation [2], which is based on this assumption of simiar behavior in singe and muti-server priority queues, is accurate. Looing at Figure 8, we see that the error in the BB approximation appears to increase for higher C 2 (right graph and for more casses. With four casses and two servers, the error is aready 1% when C 2 = 8 and higher for higher C 2. By contrast, for the same 4-cass case as shown in Figure 8, the error in RDR is aways < 2% independent of C 2 and the number of servers (we have omitted this graph. In the above graphs a casses were statisticay identica. In the case where the casses have different means, the error in BB can be much higher, whereas RDR-A is insensitive to this. 4.2 The effect of biasing toward short jobs in muti-server versus singe server systems Unti now, we have assumed that a job casses are statisticay equivaent. In this section and the next section, we remove this assumption. In this section we consider the effect of priority schemes which favor 16

17 4 3 2 cass 2 cass 3 cass cass 2 cass 3 cass 4 error (% 1 1 error (% number of servers number of servers (a C 2 = 8 (b C 2 = 25 Figure 8: Error in predicting mean deay using the BB approximation (compared with simuation for an M/PH/2 with four casses where C 2 = 8 (eft or C 2 = 25 (right and ρ =.8. short jobs in muti-server systems. Biasing towards short jobs is a common method for improving mean response time in any system. We use RDR to understand how the benefit of favoring short jobs in a singe server system compares to that for a muti-server system. Figure 9 considers a job size distribution comprised of an exponentia of mean 1, representing jobs which are short in expectation, and an exponentia of mean 1, representing jobs which are ong in expectation (where job sizes are measured in a singe-server system. The probabiity of each type of job is chosen to spit oad eveny between the short and ong jobs. The SMART scheduing poicy assigns high priority to the short jobs, and the STUPID scheduing poicy assigns high priority to the ong jobs (possiby due to economic reasons. Figure 9 shows the resuts for a (a one server, (b two server, and (c four server system. Looing at Figure 9, the SMART and STUPID poicies are the same when oad ρ is ow. At ow oad, the response time for both poicies converges to simpy the mean job size, which in these figures is 2 11 for the singe server system, for the 2-server system, and 11 for the 4-server system (reca that in a system with servers, each server runs at 1/th the speed. The most interesting observation is that more servers ead to ess differentiation between SMART and STUPID schemes. For exampe, at oad ρ =.6, there is a factor of five differentiation between SMART and STUPID with one server and ony a 25% difference between SMART and STUPID with four servers. The effect appears more prominent under ighter oad. This can be expained by recaing our earier observation that muti-server systems aow short jobs a chance to jump ahead of ong jobs, hence the negative effects of the STUPID scheme are mitigated. 17

18 5 4 SMART STUPID 5 4 SMART STUPID 5 4 SMART STUPID E[T] E[T] E[T] ρ ρ ρ (1 server (2 servers (4 servers Figure 9: Iustration of mean response time under SMART versus STUPID prioritization in a 2-cass system, where the casses are exponentiay-distributed with means one and ten respectivey, for the case of one server, two servers, and four servers. 1 RDR A MK N 1 RDR A MK N 5 5 error (% error (% C ρ Figure 1: Effect of aggregation. Graphs show error in mean deay of the 4th (owest priority cass in the MK-N and RDR-A approximations for an M/PH/2 with SMART prioritization. On the eft as a function of C 2 where ρ =.8, and on the right as a function of ρ where C 2 = 8. The casses a have a 2-phase Coxian distribution with squared coefficient of variation C 2 and means: 1, 2, 4, and How effective is cass aggregation: RDR-A In the eary 8 s Mitrani and King (ater foowed by Nishida in the eary 9 s proposed anayzing prioritization in a muti-server system via aggregation as foows: To obtain the mean response time of the m th cass, simpy aggregate casses 1 through m 1 into a singe high-priority cass, and et cass m represent the ow-priority cass then anayze the remaining two cass system. The above MK-N approximation required further approximating the singe aggregate cass by an exponentia job size distribution, since it was not nown how to anayze even a two cass muti-server system with non-exponentia job size distributions. Since RDR enabes the anaysis of muti-server priority queues with genera PH job size distributions, we can reappy the MK-N aggregation idea, but where now we are abe to capture the higher moments of the aggregated cass. We ca this approximation RDR-A, since it combines the use of RDR together with 18

19 aggregation. To understand the effect of aggregation, we consider a two server system with four priority casses. A the casses have a two phase PH distribution, with varying squared coefficient of variation (C 2. The casses differ however in their mean, having means 1, 2, 4, and 8, respectivey, and are prioritized according to the SMART scheme; casses with ower means have higher priority. (STUPID prioritization yieds simiar insights. Figure 1 examines the error in the mean deay of the 4th cass under RDR-A and under MK-N as a function of C 2 (eft and as a function of ρ (right. We see that the error in RDR-A is never more than 5% regardess of C 2 or ρ. By contrast, the error in MK-N is amost never ess than 5%, and gets worse under higher oad and C 2. What this tes us is that aggregation into two casses is a good method for approximating prioritization in muti-server systems where the number of casses is m > 2. However, the aggregation needs to be done carefuy the distribution of the aggregate cass must be modeed more cosey than can be captured by an exponentia distribution. Thus another benefit of RDR is reveaed; by aowing for PH job size distributions it enabes more accurate approximations of muti-cass systems via aggregation. 5 Concusion This paper introduces the RDR technique, providing the first near-exact anaysis of an M/PH/ queue with m 2 priority casses. The RDR agorithm is efficient (requiring ony a second or two for each data point in the paper and accurate (resuting in < 2% error for a cases that we studied. Furthermore, RDR appears to maintain its accuracy across a wide range of oads and job size variabiity (in this paper we studied oad ρ, ranging from.5 to.95 and studied squared coefficient of variation, C 2, ranging from to 128. Athough the RDR agorithm is efficient when the number of priority casses is sma, it becomes ess practica when the number of priority casses grows (e.g., for an M/M/2 with 1 priority casses, the running time can get as high as tens of seconds. Hence we aso introduce the RDR-A approximation, which wors by aggregating the m > 2 priority casses into ony two priority casses. The distribution of each aggregate cass is then captured by a PH distribution, and the resuting 2 cass system (with PH job sizes is soved using RDR. The RDR-A agorithm is extremey efficient (< 1 second for a data point, regardess of the number of casses, since its running time is that of the RDR agorithm for ony two casses. Furthermore, the RDR-A agorithm has high accuracy (< 5% error across a oads and C 2. We use our anaysis to obtain insights about priority queueing in muti-server systems. We start by comparing muti-server systems with singe server systems of equa capacity. We find that the effect of prioritization in muti-server systems cannot be predicted by considering a comparabe singe server system. The reason is that adding servers creates compex effects not present in a singe server. For exampe, mutipe servers provide a strong benefit in deaing with highy variabe job sizes, however they aso hinder performance under ighter oad. We aso compare muti-server with singe server systems, by evauating the 19