Heavy-traffic Delay Optimality in Pull-based Load Balancing Systems: Necessary and Sufficient Conditions

Transcription

1 Heavy-traffic Deay Optimaity in Pu-based Load Baancing Systems: Necessary and Sufficient Conditions XINGYU ZHOU, The Ohio State University JIAN TAN, The Ohio State University NSS SHROFF, The Ohio State University In this paper, we consider a oad baancing system under a genera pu-based poicy. In particuar, each arriva is randomy dispatched to one of the servers with queue ength beow a threshod; if none exists, this arriva is randomy dispatched to one of the entire set of servers. We are interested in the fundamenta reationship between the threshod and the deay performance of the system in heavy traffic. To this end, we first estabish the foowing necessary condition to guarantee heavy-traffic deay optimaity: the threshod wi grow to infinity as the exogenous arriva rate approaches the boundary of the capacity region (i.e., the oad intensity approaches one but the growth rate shoud be sower than a poynomia function of the mean number of tasks in the system. As a specia case of this resut, we directy show that the deay performance of the popuar pu-based poicy Join-Ide-Queue (JIQ ies stricty between that of any heavy-traffic deay optima poicy and that of random routing. We further show that a sufficient condition for heavy-traffic deay optimaity is that the threshod grows ogarithmicay with the mean number of tasks in the system. This resut directy resoves a generaized version of the conjecture by Key and Laws. CCS Concepts: Mathematics of computing Queueing theory; Networks Network performance modeing; Network performance anaysis; Additiona Key Words and Phrases: Heavy-traffic deay optimaity; Pu-based; Load baancing; Necessary and sufficient conditions ACM Reference format: Xingyu Zhou, Jian Tan, and Ness Shroff Heavy-traffic Deay Optimaity in Pu-based Load Baancing Systems: Necessary and Sufficient Conditions. Proc. ACM Meas. Ana. Comput. Syst. 2, 3, Artice 44 (January 209, 33 pages INTRODUCTION We consider a cassica oad baancing system that consists of a centra dispatcher and N servers, each associated with an infinite buffer queue and a service rate µ n. The exogenous tasks arrive with rate λ Σ, and upon arriva they must be immediatey dispatched to one of the queues. A key to the performance of such a system is the oad baancing poicy it uses since it directy determines which queue the arriving tasks shoud join. To design effective oad baancing poicies and hence provide good deay performance, it is imperative to deveop anaytica toos to evauate the system performance under different oad This work has been funded in part through ONR grant N and NSF grants CNS-7937, 77060, and Permission to make digita or hard copies of a or part of this work for persona or cassroom use is granted without fee provided that copies are not made or distributed for profit or commercia advantage and that copies bear this notice and the fu citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or repubish, to post on servers or to redistribute to ists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. 209 Association for Computing Machinery /209/-ART44 $ Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

2 44:2 X. Zhou et a. baancing poicies. Towards that goa, one important ine of research has focused on the so-caed heavy-traffic regime, where the exogenous arriva rate approaches the boundary of the capacity region, i.e., the heavy-traffic parameter ϵ = µ n λ Σ approaches zero. An attractive property of the heavy-traffic regime, as pointed out in [5, is that the important features of good contro poicies are often dispayed in the sharpest reief. It has been shown that we-known poicies such as Join-Shortest-Queue (JSQ and Power-of-d can achieve asymptoticay optima deay performance in the heavy-traffic regime [5, 7, 8, 9. Under these two poicies, an incoming task is assigned to a server with the shortest queue among d 2 servers (d = N for JSQ samped uniformy at random. However, due to the samping process, the amount of communication overhead is 2d per arriva (d for query and d for response, which is undesirabe for a arge vaue of d, especiay in the JSQ poicy when d = N. More importanty, since the dispatching decision can ony be made after coecting the queue ength feedback, there exists a non-zero dispatching deay, which contributes to an increase in the response time. To avoid these drawbacks, an aternative approach, often caed pubased oad baancing, has received significant recent attention. Instead of activey sending queries to servers and waiting for responses, the dispatcher under a pu-based oad baancing scheme passivey istens to the reports from the servers. In particuar, each server wi report its ID to the dispatcher when it satisfies a certain condition (e.g., its queue ength drops beow a threshod from above. Then, upon task arriva, the dispatcher checks its record. If it is not empty, the dispatcher randomy removes one ID and sends the arriva to the corresponding server; otherwise, it just randomy seects a queue to join. The cassica pu-based poicy is the Join-Ide-Queue (JIQ poicy investigated in [6, 22, under which the dispatcher maintains a record of IDs of the ide servers (i.e., the reporting threshod is one. JIQ has been shown to enjoy a ow message overhead (at most one per arriva, zero dispatching deay, and better deay performance than Power-of-d under medium oads. Nevertheess, under high oads, its deay performance degrades substantiay due to the ack of ide servers. This directy suggests that a varying reporting threshod with respect to the oad is necessary to guarantee good deay performance in heavy traffic. Motivated by this observation, in a recent work [30, the authors propose a specific way to update the reporting threshod in a pu-based poicy, which is proven to be heavy-traffic deay optima, whie sti enjoying many of the nice features of JIQ. In this paper, instead of focusing on another specific way of determining the reporting threshod, we step back and work towards answering the foowing fundamenta question: How woud different reporting threshods affect the (heavy traffic deay performance of a pu-based poicy? To address this question, we take a systematic approach and summarize the main contributions as foows. We first present a necessary condition on the reporting threshod for the deay optimaity of a pu-based poicy in heavy-traffic. In particuar, we show that to achieve heavy-traffic deay optimaity, the reporting threshod r shoud grow to infinity as the heavy-traffic parameter ϵ approaches zero, however, it cannot grow too fast (sower than a poynomia function: see Theorem 3.2. An important coroary of Theorem 3.2 is that the deay performance of the JIQ poicy (i.e., constant threshod r = in heavy traffic ies stricty between that of any heavy-traffic deay optima poicies (e.g., JSQ and that of random routing. This resut is somewhat counter-intuitive, since at first gance one may guess that JIQ woud degenerate to random routing in heavy traffic since there are hardy any ide servers in the system. However, it turns out that it is not true, and aows us to get a sharp characterization of the JIQ poicy in heavy traffic. We then estabish a sufficient condition on the reporting threshod for heavy-traffic deay optimaity of pu-based poicies. Specificay, we show that a ogarithmic growth rate of the reporting threshod with respect to the mean number of tasks in the system is sufficient to Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

3 44:3 guarantee the steady-state deay optimaity in heavy traffic (see Theorem 3.3. This resut directy resoves a conjecture by Key and Laws in [5. In particuar, the authors in [5 consider a two-server system with Poisson arrivas and exponentia service under a varying reporting threshod. They conjecture that as ong as the threshod is greater than a specified constant times the ogarithm of the mean number of tasks in the system, then asymptotic deay optimaity hods in heavy traffic. Thus, our resut not ony resoves the conjecture but generaizes it to any fixed finite number of servers with genera arriva and service distributions. It is aso worthing noting that the asymptotic deay optimaity achieved in our paper is in steady-state whie the resut in [5 hods ony for a finite time interva. The techniques introduced in this paper may be of independent interest for the anaysis of genera oad baancing poicies. More precisey, the key to estabishing heavy-traffic deay optimaity in this paper is a notion of state-space coapse, which is different from the statespace coapse resut often adopted in previous works. As a resut, it requires us to deveop a new Lyapunov function to conduct the drift anaysis. More importanty, due to this new type of state-space coapse, we have to devise a new approach to reate the state-space coapse resut to the fina heavy-traffic deay optimaity.. Reated Work The investigation of queueing deay in heavy traffic with dynamic routing dates back to [8, in which the authors considered a two-server system under the JSQ poicy, and they showed that the two separate servers under JSQ act as a pooed resource in heavy traffic via diffusion approximations. Since then, the methodoogy of diffusion approximations has been adopted in a number of works on parae queues [3, 5, 3, 4, 2, 26. For exampe, the author in [2 generaized the resuts in [8 to the case of renewa arrivas and genera service times. The functiona centra imit theorems for the JSQ poicy in a oad baancing system with mutipe servers was derived in [3. In [5, the Power-of-d poicy was shown to have the same diffusion imit as JSQ in the heavy-traffic imit. Many of the works based on the diffusion approximation method rey on showing that a scaed version of queue engths converges to a reguated Brownian motion. This resut typicay eads to a sampe-path optimaity in a finite time interva. However, showing the convergence to the steady-state distribution requires the additiona vaidation of the interchange of imits, which is often not taken (some exceptions incude [4, 9, in which the authors proved an interchange of imit argument for generaized Jackson networks with a fixed routing matrix. Motivated by this, the authors in [7 proposed a Lyapunov drift-based approach, which is abe to estabish steady-state heavy-traffic optimaity of the oad baancing poicy JSQ and scheduing poicy MaxWeight. One of the main features of this framework is that it is abe to avoid the interchange-of-imits issue by directy working on the stationary distribution. This approach has been utiized to show steadystate heavy-traffic deay optimaity of Power-of-d in [9. Moreover, based on this approach, it has been shown in [25 that a joint JSQ and MaxWeight poicy is heavy-traffic deay optima for MapReduce custers. As discussed in the introduction, whie JSQ and Power-of-d enjoy heavy-traffic deay optimaity, they both have non-zero dispatching deay, and a reativey high message overhead. Motivated by this, a pu-based design of oad baancing poicies has gained significant recent popuarity. The main feature of pu-based oad baancing is the introduction of oca memory at the dispatcher, which maintains a record of servers satisfying a pre-defined condition (e.g., its queue ength is beow a threshod in most cases. The dispatching decision is made purey based on the oca memory: if it is nonempty, randomy choosing a server in memory to join; otherwise, randomy choosing a server from a the servers. For instance, one iustrative exampe is the JIQ poicy proposed and studied in [6, 22, under which the oca memory maintains a the ide servers. As a resut, the Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

4 44:4 X. Zhou et a. arriva is aways dispatched to one of the ide servers if there are any; otherwise, it is dispatched randomy. It has been shown that JIQ has a ow message overhead (at most one per arriva, zero dispatching deay, and better performance compared to Power-of-2 in medium oads. Nevertheess, since ony the ide servers are stored in memory, when the oads become high, its performance degrades substantiay because the memory is empty and hence random routing is adopted most of the time. Therefore, this directy suggests that a varying threshod is necessary to guarantee good performance in heavy traffic for a pu-based poicy. To this end, in a recent work [30, the authors successfuy propose a pu-based poicy with a varying threshod, which is proven to be heavy-traffic deay optima in steady state whie keeping the nice features of JIQ. This naturay raises the question about the fundamenta reationship between the choice of the threshod and the deay performance, which is the main focus of this paper. In particuar, our work is mainy motivated by the semina paper [5, in which Key and Laws give a conjecture regarding the choice of the threshod that is abe to guarantee deay optimaity in heavy traffic. More precisey, they consider a two-server system with Poisson arrivas and exponentia service. The arriva is dispatched randomy, except when one queue is beow the threshod r and the other is above, in which case the arriva is dispatched to the shorter one. Note that this dynamic poicy can be exacty impemented by a pu-based oad baancing scheme with a threshod r. Key and Laws conjecture that as ong as the threshod r is greater than a specific constant times the ogarithm of the mean number of tasks in the system, then the sum queue engths process under this threshod poicy has the same diffusion imit as that under JSQ. Therefore, the ogarithmic growth rate resut in our sufficient conditions (see Theorem 3.3 not ony directy resoves the conjecture in [5, but generaizes it to systems with any fixed finite number of servers as we as genera arriva and service distributions. Moreover, the diffusion imit resut conjectured in [5 ony gives the optimaity in a finite time interva whie our heavy traffic optimaity resut obtained by Lyapunov drift-based approach is in steady state. It is aso worth noting that a ogarithmic growth in the threshod is not a coincidence, and has been found in a wide range of scenarios. For exampe, the authors in [23 consider an asymmetric threshod poicy for a two-server case. In that setting, ony one server has a threshod r (say server 2. The arrivas are aways dispatched to server uness the queue ength of server 2 is ess than the threshod, in which case the arriva is sent to server 2. One of the main contributions in [23 is that a ogarithmic growth rate of r is sufficient to guarantee that this threshod poicy achieves the same diffusion imit as that under JSQ in heavy traffic. This resut can be seen as a first attempt to resove the conjecture in [5 with a simper mode. In particuar, since there is ony one threshod in [23, the network can be characterized by a one-dimensiona refected Brownian motion in heavy traffic. In contrast, the imit process in [5 is a two-dimensiona Brownian motion, which is harder to rigorousy prove optimaity. Besides dynamic routing, a ogarithmic growth rate of the threshod aso criticay affects the performance of scheduing poicies in [2, 4. Both authors considered a system of two parae servers with dedicated arrivas to each of the queues. One server can ony process tasks in its own queue, whie a super-server can process tasks from both queues. A threshod poicy is proposed in which the super-server processes tasks from its own queue when the other server s queue ength is beow a threshod, and otherwise the super-server processes the tasks from the other queue. This poicy can be viewed as the scheduing counterpart of the asymmetric routing poicy considered in [23. In a discrete review setting, the author in [4 proved that a sufficient condition for the asymptotic optimaity of this threshod poicy is that the threshod must grow as a constant times the average number of tasks in the system. The same resut was generaized to a continuous review setting with more genera arriva and service distributions in [2. As in the paper by Key and Laws [5, the asymptotic optimaity in [2, 4, 23 hods in a finite time interva since the convergence to the stationary distribution is not vaidated for the Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

5 44:5 diffusion approximations. Considering the simiarity between the scheduing poicies in [2, 4 and the routing poicy in [23, our approach deveoped in this paper might be appied to estabish heavy-traffic deay optimaity in steady state for dynamic scheduing poicies as we. We sha finay point out that the heavy-traffic regime considered in this paper and a the aforementioned papers assumes that the number of servers is a constant, which is different from the Hafin-Whitt heavy-traffic regime (aso known as many-server heavy-traffic regime or quaityand-efficiency-driven regime [2. In the atter regime, the heavy-traffic parameter ϵ approaches zero and the number of servers N goes to infinity at the same time [, 6, 0, 20. For exampe, it has been shown that, on any finite time interva, the imiting process under the JIQ poicy is indistinguishabe from that under the JSQ poicy in the Hafin-Whitt heavy-traffic regime [20. In contrast, in the conventiona heavy-traffic regime considered in this paper, its deay performance is stricty between that of JSQ and random routing as shown by Theorem Notations The dot product in R N is denoted by x, y N n= x n y n. For any x R N, the norm is denoted by x N n= x n and 2 norm is denoted by x x, x. In genera, the r norm is denoted by x r ( N n= x n r /r. Let N denote the set {, 2,..., N }. 2 SYSTM MODL AND PRLIMINARIS This section first describes the system mode and assumptions considered in this paper. Then, severa necessary preiminaries are presented. 2. System mode We consider a discrete-time oad baancing system consisting of a centra dispatcher and N servers. ach server maintains an infinite capacity FIFO queue. At the centra dispatcher, there is aso a oca memory denoted as m(t, through which the dispatcher can have imited information about the system. In each time-sot, the centra dispatcher routes the new incoming tasks to one of the servers, immediatey upon arriva as in [7, 9, 25, 27, 28, 30. Once a task joins a queue, it wi remain in that queue unti its service is competed. ach server is assumed to be work conserving: a server is ide if and ony if its corresponding queue is empty. 2.. Arriva and Service. Let A Σ (t denote the number of exogenous tasks that arrive at the beginning of time-sot t. We assume that A Σ (t is an integer-vaued random variabe, which is i.i.d. across time-sots. The mean and variance of A Σ (t are denoted by λ Σ and σσ 2, respectivey. We further assume that there is a positive probabiity for A Σ (t to be zero. Let S n (t denote the amount of service that server n offers for queue n in time-sot t. Note that this is not necessariy equa to the number of tasks that eaves the queue because the queue may be empty. We assume that S n (t is an integer-vaued random variabe, which is i.i.d. across time-sots. We aso assume that S n (t is independent across different servers as we as the arriva process. The mean and variance of S n (t are denoted as µ n and νn, 2 respectivey. Let µ Σ Σn= N µ n and νσ 2 ΣN n= ν n 2 denote the mean and variance of the hypothetica tota service process S Σ (t N n= S n (t. To iustrate the key ideas behind the resuts, we first assume that both the arriva and service processes have a finite support, i.e., A Σ (t A max < and S n (t S max < for a t and n. However, the main resuts sti hod when the support is infinite, as discussed in Section Queue Dynamics. Let Q n (t be the queue ength of server n at the beginning of time sot t. Let A n (t denote the number of tasks routed to queue n at the beginning of time-sot t according to Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

6 44:6 X. Zhou et a. the dispatching decision. Then the evoution of the ength of queue n is given by Q n (t + = Q n (t + A n (t S n (t + U n (t, n =, 2,..., N, ( where U n (t = max{s n (t Q n (t A n (t, 0} is the unused service due to an empty queue. 2.2 Preiminaries In this paper, we are interested in a genera pu-based poicy formay defined as foows. In words, under this poicy, the arriva is randomy dispatched to one of the servers whose queue engths are beow a threshod r, if there are any; Otherwise, it is dispatched to one of N queues randomy. Definition 2.. Join-Beow-Threshod (JBT poicy is composed of the foowing components: ach server is initiaized with an empty queue, and a corresponding ID in the oca memory of the dispatcher. Upon new arrivas at the beginning of each time-sot, the dispatcher checks the avaiabe IDs in memory. If one or more IDs exist, it removes one uniformy at random, and sends a the new arrivas to the corresponding server. Otherwise, a the new arrivas are dispatched uniformy at random to one of the servers in the system. (c ach server reports its ID to the dispatcher at the end of each time-sot if its queue ength is beow the threshod, and the dispatcher does not contain its ID (see the remark beow on this condition. (d For the case of heterogeneous servers, in (c each server aso sends its µ n to the dispatcher and in instead of choosing the ID uniformy at random, the dispatcher seects the ID in proportion to the service rate. Specificay, if the ID of server i is in m(t, the probabiity for server i to be chosen is µ i / j m(t µ j. Remark. It is easy to see that JIQ is a specia case of JBT with r =. Morevoer, note that in (c the server can easiy know whether or not its own ID exists at the dispatcher. This is because whenever there are new arrivas to a server, the server immediatey knows that its own ID at the dispatcher (if exists has just been removed in order to dispatch the new arrivas. In addition, after each successfu report, the server knows that the dispatcher has just added its ID in the memory. Of course, in the anaysis of JBT, we can simpy assume that the set of servers whose queue engths are beow the threshod are known at the dispatcher without worrying about the impementationa detais. The considered oad baancing system under JBT can be modeed as a discrete-time Markov chain {Z (t = (Q(t, m(t, t 0} with state space Z, using the queue ength vector Q(t together with the memory state m(t. We consider a set of oad baancing systems {Z (ϵ (t, t 0} parameterized by ϵ such that the mean arriva rate of the exogenous arriva process {A (ϵ Σ (t, t 0} is λ(ϵ Σ = µ Σ ϵ. Note that the parameter ϵ characterizes the distance between the arriva rate and the boundary of the capacity region. We are interested in the throughput performance and more importanty the steady-state deay performance in the heavy-traffic regime under the JBT poicy. Reca that a oad baancing system is stabe if the Markov chain {Z (t, t 0} is positive recurrent, and Z = {Q,m} denotes the random vector whose distribution is the same as the steady-state distribution of {Z (t, t 0}. We have the foowing definition. Definition 2.2 (Throughput Optimaity. A oad baancing poicy is said to be throughput optima if for any arriva rate within the capacity region, i.e., for any ϵ > 0, the system is positive recurrence and a the moments of Q (ϵ are finite. Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

7 44:7 Note that this is a stronger definition of throughput optimaity than that in [25, 28, 30, because besides the positive recurrence, it aso requires a the moments to be finite in steady state for any arriva rate within the capacity region. To characterize the steady-state average deay performance in the heavy-traffic regime when ϵ approaches zero, by Litte s aw, it is sufficient to focus on the summation of a the queue engths. First, reca the foowing fundamenta ower bound on the expected sum queue engths in a oad baancing system under any throughput optima poicy [7. Lemma 2.3. Given any throughput optima poicy and assuming that (σ (ϵ Σ 2 converges to a constant σσ 2 as ϵ decreases to zero, then im inf ϵ Q (ϵ n ϵ 0 ζ 2, (2 n= where ζ σσ 2 + ν Σ 2. The right-hand-side of q. (2 is the heavy-traffic imit of a hypothetic singe-server system with arriva process A (ϵ Σ (t and service process N n S n (t for a t 0. This hypothetica singe-server queueing system is often caed the resource-pooed system. Since a task cannot be moved from one queue to another in the oad baancing system, it is easy to see that the expected sum queue engths of the oad baancing system is arger than the expected queue ength in the resource-pooed system. However, under a certain oad baancing poicy, the ower bound in q. (2 can actuay be attained in the heavy-traffic imit and hence based on Litte s aw this poicy achieves the minimum average deay of the system in steady-state. This directy motivates the foowing definition of steady-state heavy-traffic deay optimaity as in [7, 9, 25, 27, 28, 30. Definition 2.4 (Heavy-traffic Deay Optimaity in Steady-state. A oad baancing scheme is said to be heavy-traffic deay optima in steady-state if the steady-state queue ength vector Q (ϵ satisfies where ζ is defined in Lemma 2.3. im sup ϵ ϵ 0 n= Q (ϵ n ζ 2, In the anaysis of the deay performance of JBT, the foowing region R (r in R N pays an instrumenta roe by the virtue of the JBT poicy. where r and R (r = R (r R (r u, (3 R (r { x R N + : x n r for a n N } R (r u { x R N + : x n r for a n N }. By the definition of the JBT poicy, we have that whenever the queue ength vector is within the region R (r, then JBT reduces to (proportionay random routing. On the other hand, when the queue engths vector is outside the region R (r, shorter queues are preferred over onger queues. 3 MAIN RSULTS In this section, we present both necessary and sufficient conditions on the threshod r for the JBT poicy to be heavy-traffic deay optima in steady-state. We first estabish throughput optimaity of the JBT poicy, which serves as a basis for the anaysis of heavy-traffic deay optimaity. Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

8 44:8 X. Zhou et a. 3. Throughput optimaity We first prove the foowing resut, which estabishes that a oad baancing system under the JBT poicy is stabe with bounded moments on the queue engths for any threshod r. Lemma 3.. JBT is throughput optima with the p-th moment of Q (ϵ being O(/ϵ p for any threshod r and integer p. Proof. See Appendix A Besides throughput optimaity, another important aspect of this emma is that it serves as the basis for the discussions on heavy-traffic deay optimaity in the foowing sections. This is because, firsty, a oad baancing poicy that cannot stabiize the system is incapabe of being heavy-traffic deay optima at a. Second, the bounded moments resut aows us to set the mean drift of Lyapunov functions concerning queue engths to be zero in steady state, which pays a pivota part in the framework of Lyapunov drift-based heavy-traffic anaysis. 3.2 Necessary condition In this section, we show that a necessary condition for the JBT poicy to achieve heavy-traffic deay optimaity is that the threshod r shoud grow to infinity as the heavy-traffic parameter ϵ approaches zero. However, as we show it cannot grow too fast. Formay, it is presented in the foowing theorem. Theorem 3.2. Consider a oad baancing system with homogeneous servers under the JBT poicy. ( Suppose the threshod r is any constant in [,, then we have im inf ϵ Q (ϵ n ϵ 0 > ζ (4 2 n= and im sup ϵ Q (ϵ n ϵ 0 < im ϵ ϵ 0 Q (ϵ n,rand n=, (5 n= where Q (ϵ Rand is the steady-state vector under random routing poicy. (2 Suppose the threshod r (ϵ = (/ϵ +α for any constant α > 0, then we have im ϵ ϵ 0 Q (ϵ n = im ϵ ϵ 0 Q (ϵ n,rand n=. (6 n= Proof. See Section 5. Now, we wi present the high-eve intuitions behind the necessary condition with the iustration in Fig.. These intuitions can not ony faciitate understanding of the resuts, but aso motivate the sufficient condition in the next section. To start with, et us consider case (2 when r (ϵ = (/ϵ +α for any α > 0. In this case, a the queue engths are beow the threshod r for high oads since the sum queue ength in the system is ony on the order of /ϵ. As a resut, in case (2, the JBT poicy competey degenerates to random routing, which is not heavy-traffic deay optima [8. An iustration of case (2 for a two-server system is presented in Fig.. Then, we turn to case ( for which the threshod is a constant. In particuar, combing qs. (4 and (5 yieds that the deay performance of JBT under any constant r in heavy-traffic ies stricty between that of a heavy-traffic deay optima poicy (e.g., JSQ and that of random routing. This Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

9 44:9 Q 2 R (r u Q 2 R (r u R (r (r, r (r, r (/ Q R (r (/ Q r =(/ +, > 0 r is a constant in [, Fig.. Geometric iustrations of the necessary condition. reveas an interesting and kind of counter-intuitive insight about the JBT poicy under a constant threshod. For exampe, consider the specia case r =, i.e., the JIQ poicy. At first gance, one might expect that the deay performance of JIQ woud downgrade to that of random routing in the heavy-traffic imit, since in this case there are hardy any ide servers, and hence the dispatcher under JIQ woud just randomy choose one server when aocating arrivas, as in random routing. However, it turns out that this is not true as shown in q. (5. That is, the performance of JIQ is sti stricty better than that of random routing even in the heavy-traffic imit. This demonstrates that JIQ is abe to achieve partia resource pooing due to the fact that it adopts queue ength information to prefer shorter queues whenever possibe. To see this, note that by positive recurrence, there aways exists some time when the queue ength vector is outside the region R (r and hence shorter queues are preferred (i.e., the orange ine in Fig., even though it is much ess than the time within the region R (r (i.e., the green ine in Fig.. This is totay different from the case in Fig. in which the queue-ength state aways competey remains within the R (r for high oads, and hence JBT woud downgrade to random routing in the imit. On the other hand, to expain the iminf resut in q. (4, we wi utiize the foowing resut. That is, the necessary (and sufficient condition for the JBT poicy to be heavy-traffic deay optima is given by [ im Q (ϵ (t + ϵ 0 U (ϵ (t = 0. (7 This is a direct appication of the resuts in [29. Note that since Q n (t + U n (t = 0, the above condition basicay means that the key for JBT to be heavy-traffic deay optima is that it shoud guarantee that no server is iding whie other servers are busy with high oads. In the case when r is a constant, the event that one queue is zero whie others with high oads (denoted by bad happens with a non-negigibe probabiity since the axes are cose to the region R u (r. As a resut, the eft-hand side of q. (7 is stricty positive, and hence JBT is not heavy-traffic deay optima for a constant r. The intuition that we shoud guarantee that the event bad occurs very rarey in heavy-traffic aso motivates our sufficient condition in the next section where we et the threshod r grow in a certain rate to guarantee that the axes are far away from the region R u (r. Remark 2. It is worth noting that in [30, a simiar resut as q. (4 has been estabished for the JIQ poicy (i.e., the specia case r = of JBT in a two-server system under the constraints that the service processes are constant and the variance of arriva process shoud be arger than a particuar vaue. Thus, our contribution is to generaize the resut in [30 to any constant r Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

10 44:0 X. Zhou et a. and any finite number of servers without the constraints on service and arriva process as required in [30. More importanty, we provide new resuts given by qs. (5 and (6, which give us a sharper understanding of genera pu-based poicies. 3.3 Sufficient condition In this section, we now investigate the sufficient condition. In particuar, we show that if the threshod in JBT grows at a ogarithmic rate with respect to the average sum queue engths, i.e., r (ϵ K og(/ϵ for some specified constant K, then the JBT poicy is heavy-traffic deay optima in steady state, which is formay presented in the foowing theorem. Theorem 3.3. Consider a oad baancing system under the JBT poicy. Suppose that the threshod r satisfies r (ϵ K og(/ϵ and r (ϵ = o(/ϵ, where the constant K = 2( + α /θ for any α > 0 and θ is the constant in q. (9, then JBT is heavy-traffic deay optima in steady state. Proof. See Section 5.2 The main contributions of this resut can be summarized as foows. First, it directy resoves and generaizes a conjecture in [5. More precisey, the authors in [5 consider a two-server system with Poisson arrivas and exponentia service under a threshod poicy that has the same impementation as JBT, and conjecture that as ong as the threshod is greater than a specified constant times og(/ϵ, the heavy-traffic asymptotic optimaity of the threshod routing strategy hods. Thus, our resut resoves this conjecture and aso generaizes it to any finite number of servers case with genera arriva and service distributions. More importanty, the asymptotic optimaity defined in [5 hods ony for a finite time interva since the convergence to steady-state distribution is not touched. In contrast, our resut directy gives the steady-state characterization of the deay optimaity in heavy-traffic of the JBT poicy. The key step in estabishing the sufficient condition in Theorem 3.3 is the notion of statespace coapse. In words, it says that in heavy traffic the system state under the JBT poicy woud concentrate around the region R (r as defined q. (3. To that end, we need the foowing property of the distance to the region R (r. The distance of a point x to the region R (r is reated to the distances to the regions R (r and R (r u as foows. ( d R (r (x = min d R (r where the distance of a point x to a set A in R N is defined as { d A (x inf x y }. y A This equaity (8 can be estabished by contradiction. Suppose that ( min (x,d (r R (x = d u R (r (x + α d R (r for some α > 0, then there exists a y R (r such that d R (r (x ( x y < min (x,d (r R (x, (8 u d R (r (x,d (r R (x. u However, since y R (r = R (r R (r u, this eads to a contradiction to the right-hand side of the inequaity above. We say that the system state concentrates around the region R (r if a the moments of the distance d R (r (Q are upper bounded by constants. Formay, we have the foowing definition. Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

11 44: Definition 3.4 (State-space coapse to R (r. Suppose that the system process converges in distribution to a steady-state random vector Q (ϵ. Then, we say that the state-space of a oad baancing system coapses to the region R (r if there exist some positive constants ϵ 0, θ and C such that for a ϵ (0, ϵ 0 [ ( e θ d R (r Q (ϵ C, (9 where both θ and C are independent of ϵ. Note that this notion of state-space coapse is different from previous works, as wi be expained ater. For any constant threshod r, q. (9 triviay hods since the distance to the region R (r is aways bounded by a constant. Thus, in the foowing we ony consider the interesting case when r grows to infinity, which is aso required by the necessary condition in Theorem 3.2. In this case, we have the foowing resut regarding state space coapse of the JBT poicy, which pays a key roe in the proof of Theorem 3.3. Proposition 3.5. Consider a oad baancing system under the JBT poicy. Suppose that the threshod satisfies im ϵ 0 r (ϵ =, then the system state-space coapses to the region R (r. Proof. See Section 5.3 Remark 3. It shoud be noted that besides being a key step in proving the sufficient conditions in Theorem 3.3, Proposition 3.5 has its own contributions. (i First, the region of state-space coapse in this paper, i.e., R (r is not a singe dimensiona ine as in [7, 9, 25, 27, 28, 30, nor a mutidimensiona convex cone as in [7, 8, 24, 29. This not ony brings new chaenges in proving state-space coapse itsef, but aso requires new methods to reate the coapse resut to heavy-traffic deay optimaity. More specificay, on the one hand, in order to prove state-space coapse resut, we need to hande the non-convexity of R (r by choosing the minimum of two distances as the Lyapunov function. The techniques suggested in [29 to hande the non-convex region cannot appy here since the region R (r cannot be covered by the cone defined in [29. On the other hand, in order to utiize the state-space coapse resut to concude heavy-traffic deay optimaity, the conventiona decompositions of parae and perpendicuar components of the queue ength vector Q woud not work. Instead, we need to carefuy divide the system state and then appy Chernoff ( (ϵ bound on the random variabe d R (r Q, which is possibe by the state-space coapse resut in q. (9. (ii Second, the upper bound resut in q. (9 hods even when the system is not at the heavy-traffic imit, and hence it is of independent interest for anayzing the system performance in the pre-imit regime, especiay when combined with optimization techniques. Now, we turn to provide the high-eve intuitions on Proposition 3.5 and Theorem 3.3 with the hep of Fig. 2. This wi faciitate the understanding of the resuts as we as their proofs. To start with, note that by virtue of the JBT poicy, when the queue-ength state Q is outside the region R (r, there aways exists a positive drift towards the region R (r. This is because in this case there exists a positive drift towards the ower region R (r and a positive drift towards the upper region R u (r, respectivey (see Fig. 2 for an iustration. This provides the key intuition as to why the system state woud concentrate around the region R (r since suppose there is no drift (e.g., under random routing the expected distance to the region R (r woud go to infinity as r (ϵ goes to infinity (assuming that the growth rate of r (ϵ is not too fast. In contrast, under the JBT poicy, the distance remains constant (as shown by the gray coor in Fig. 2. This is the reason why we ca it a state-space coapse resut, which is different from much of previous works where Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

12 44:2 X. Zhou et a. d u = r Q 2 µ µ R (r u Q 2 d >r R (r u (r, r (r, r R (r µ µ R (r Q 2r Q Fig. 2. Geometric iustrations of the sufficient condition. the system state coapses to a ower dimensiona space (e.g., a ine or a convex cone whie our state-space coapse region R (r is of the same dimension as the origina queue-ength state vector. Hence, we need to deveop new methods to appy this new type of state-space coapse resut to achieve heavy-traffic deay optimaity of the JBT poicy, as in Theorem 3.3. To this end, we wi utiize the sufficient (and necessary condition in q. (7 again. As discussed before, it basicay requires us to guarantee that no server is iding whie other servers are busy under high oads. To achieve this, a ogarithmic growth rate as in Theorem 3.3 is sufficient. For an iustration of the main ideas behind the proof, et us consider a simpe two-server case. In this case, q. (7 reduces to [ im Q (ϵ (t + U (ϵ 2 + Q (ϵ 2 (t + U (ϵ = 0. (0 ϵ 0 Take the second term above for exampe, it can be rewritten as the summation of the foowing terms (for simpicity we omit the superscript (ϵ Q 2 (t + U I ( Q 2 (t + 2r, Q (t + = 0 ( Q 2 (t + U I ( Q 2 (t + > 2r, Q (t + = 0, (2 where we use the fact that Q n (t + U n (t = 0 again. The expectation of q. ( can be upper bounded by 2r (ϵ ϵ since [ U ϵ. For the expectation of q. (2, we first appy Cauchy-Schwartz inequaity and hence obtain its upper bound as C ϵ 2 P ( Q 2 (t + > 2r, Q (t + = 0, where C is a constant independent of ϵ. Now, we can appy the state-space coapse resut (i.e., q. (9 combined with Chernoff bound to show that the probabiity that one queue is empty and another queue ength is arger than 2r has an exponentia decay rate. In particuar, we have P ( Q 2 (t + > 2r, Q (t + = 0 P ( d R (r ( Q (ϵ r C e θ r, where hods since in this case the distance to the region R (r is r (see Fig. 2 for an iustration; foows directy from state-space coapse resut and Chernoff bound. Therefore, combining the expectations of qs. ( and (2, yieds [ Q (ϵ 2 (t + U (ϵ 2r (ϵ ϵ + C ϵ 2 e, θ r (ϵ which approaches zero whenever r (ϵ = o( ϵ and r (ϵ K og(/ϵ where K = 2( + α /θ for any α > 0. By the same arguments, we can estabish the same resut for the expectation of the first term Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

13 44:3 in q. (0. Therefore, we have reached the sufficient condition for heavy-traffic deay optimaity in Theorem GNRALIZATIONS For the iustration of the key ideas, the main resuts in the ast section are obtained under the assumptions that both arriva and service processes have finite support. However, it is worth pointing out that the same resuts sti hod (with ony a change in constants when the support is infinite. More specificay, we need the foowing weak condition on arriva and service processes, which requires that the tais of both arriva and service processes have an exponentia decay. Condition A (Weaker condition on arriva and service. The i.i.d arriva process A Σ (t and service process S n (t satisfy [ e θ A Σ (t D and [ e θ 2S n (t D 2, for each n where the constants θ > 0, θ 2 > 0, D < and D 2 < are a independent of ϵ. In order to obtain the same main resuts under the weaker condition above, we shoud make some mid changes in our proofs. In the foowing, we wi highight the key steps invoved in this process. (i First, note that in order to estabish condition (C in Lemma 5., we woud use the foowing upper bound in our proofs based on the finite support assumptions. [ A(t 0 S(t 0 2 Z (t 0 L N max(a max, S max 2. However, under the weaker Condition A, we can sti bound the eft-hand side by a constant independent of ϵ. This directy foows from the fact that a the moments of a random variabe are finite if its moment generating function is finite in an open interva containing zero. (ii Second, we shoud now repace condition (C2 in Lemma 5. with the foowing weak stochastic domination condition (C2, (C2 [ V (X X (t 0 = X W for a t 0 and [ e θw = D is finite for some θ > 0. This condition hods under the weaker Condition A since the arriva and service processes both have an exponentiay bounded tai by the finiteness of their moment generating functions. As shown by Theorem 2.3 in [, the combination of (C and (C2 is sufficient to guarantee bounded moments as required in the proof of our main resuts. (iii Third, we now shoud take a carefu treatment of the unused service. For exampe, the foowing resut pays a key roe in estabishing the necessary and sufficient condition in q. (7 [ im U (ϵ 2 = 0. ϵ 0 Under the assumption of finite support for the service process, the eft-hand side can be easiy bounded above by NS max ϵ, which approaches zero as ϵ 0. Now, under the weak condition, we need to adopt the truncation trick to hande the unbounded service. More specificay, et us consider any n N, we have for any t 0 and constant S U 2 n (t U n (ts n (t = U n (ts n (ti (S n (t S + U n (ts n (ti (S n (t > S U n (ts + S 2 n (ti (S n (t > S. Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

14 44:4 X. Zhou et a. In steady state, we have [ U 2 n [ U n S + [ Sn 2 ( I (S n ( > S ϵs + [ Sn 2 (0I (S n (0 > S ϵs + β, [ where foows from the fact that U (ϵ = ϵ and service process is i.i.d.; in, we choose S such that [ Sn 2 (0I (S n (0 > S β, which is possibe by the exponentia decay rate of S n (0 under the weak condition. Thus, we have [ im U 2 n β, for any β > 0. Hence, we have im ϵ 0 ϵ 0 [ U 2 n = 0 for each n. Remark 4. The three highighted key steps coud aso demonstrate their generaization power in previous works where the Lyapunov drift-based framework is adopted under the assumption of finite supports for the arriva and service processes. 5 PROOFS In this paper, we wi adopt the Lyapunov drift-based approach deveoped in [7 to derive bounded moments in steady state. In particuar, the foowing emma, which foows directy from Lemmas 2 and 3 in [8, wi be the main too in our proofs. Lemma 5.. For an irreducibe aperiodic and positive recurrent Markov chain {X (t, t 0} over a countabe state space X, which converges in distribution to X, and suppose V : X R + is a Lyapunov function. We define the drift of V at X as V (X [V (X (t 0 + V (X (t 0 I(X (t 0 = X, where I(. is the indicator function. Suppose the drift of V satisfies the foowing conditions: (C There exists an η > 0 and a κ < such that for any t 0 =, 2,... and for a X X with V (X κ, [ V (X X (t 0 = X η. (C2 There exists a constant D < such that for a X X, P( V (X D =. Then {V (X (t, t 0} converges in distribution to a random variabe V for which there exists a θ > 0 and a C < such that [ e θ V C, which directy impies that a the moments of V exist and are finite. More specificay, we have for any p =, 2,... [ V (X p ( D + η p (2κ p + (4D p p!. (3 η We woud aso utiize the foowing usefu resut in our proofs. Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

15 44:5 Lemma 5.2. For the JBT poicy with threshod r, it is heavy-traffic deay optima if and ony if [ im Q (ϵ (t + ϵ 0 U (ϵ (t = 0. (4 This emma is a direct appication of the resuts in [29, which estabishes that q. (4 is the sufficient and necessary condition for any oad baancing poicy to be heavy-traffic deay optima if the system is stabe with bounded moments. By Lemma 3., we have that the JBT poicy is throughput optima with a the moments being bounded for any r, and hence the above emma hods. 5. Proof of Theorem 3.2 Before we present our proof, we first give the foowing usefu resut, which can be estabished by setting the mean drift a chosen Lyapunov function to zero in steady state. For competeness, the proof is given at Appendix B. Lemma 5.3. Consider a oad baancing system with homogeneous servers under the JBT poicy. For any threshod r, we have [( 2 (Q + i (ϵ U (ϵ j + (Q + j (ϵ U (ϵ i = T (ϵ + T (ϵ 2 T (ϵ 3, where i= j >i T (ϵ 2 T (ϵ 2 T (ϵ 3 i= j >i i= j >i i= j >i Q + Q(t + [( ( Q (ϵ i Q (ϵ j A (ϵ i A (ϵ j [( A (ϵ i A (ϵ j S (ϵ i + S (ϵ 2 j [( U (ϵ i U (ϵ 2 j and A (ϵ i and U (ϵ i are dependent of Q for each i and ϵ > 0. Now, we are ready to present the proof of Theorem 3.2. Proof of Theorem 3.2. To start with, we first note that the sufficient and necessary condition in Lemma 5.2 can be rewritten as foows under the JBT poicy. [ 2 Q (ϵ (t + U (ϵ (t = 2 = 4 (c = 4 i= j >i i= j >i i= j >i [( (Q + i (ϵ U (ϵ j + (Q + j (ϵ U (ϵ i [( (Q + i (ϵ U (ϵ j k= ( ku (ϵ j P (Q + i (ϵ = k, (Q + j (ϵ = 0,U (ϵ j, (5 Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

16 44:6 X. Zhou et a. in which and (c foow from the fact Q i (t + U i (t = 0 for each i and t 0; hods by the symmetry property of JBT poicy for homogeneous servers. Thus, by Lemma 5.2, Lemma 5.3 and the above equation, in order to anayze heavy-traffic deay optimaity of JBT under any constant threshod, a we need to do is to focus on terms T (ϵ, T (ϵ 2 and T (ϵ 3, respectivey. Now, et us first focus on case ( in Theorem 3.2. For T (ϵ, we have T (ϵ 2 = 2 i= j >i i= j >i = 4 [( ( Q (ϵ i Q (ϵ j A (ϵ i A (ϵ j [( ( ( Q i Q j Ai A j I Qi r, Q j r i= j >i i= j >i i= j >i 4λ Σ (c = 4λ Σ [( ( ( Q i Q j Ai A j I Qi < r, Q j < r [( ( ( Q i Q j Ai A j I Qi r, Q j < r [( ( ( Q i Q j Ai A j I Qi r, Q j < r r (k mp ( Q i = k, Q j = m i= j >i m=0 k=r r (k mp ( Q + i = k, Q + j = m, (6 i= j >i m=0 k=r where foows from the definition of the JBT poicy, i.e., when both queues are in memory or both queues are not in memory, they have the same probabiity to be seected in the homogeneous case; is true since when the ID of server j is in m(t whie the ID of server i is not, we have A i (t = 0 and A j (t A Σ (t by the definition of the JBT poicy; (c hods since Q(t + has the same distribution as Q(t in steady state. In order to further simpify the term T (ϵ, we need to define the foowing events in which k r and m r. (k,m { Q + i = k, Q + j = m } + (k,m { Q i (t + 2 = k, Q j (t + 2 = m } (k,0,0 { Q + i = k, Q + j = 0,U j = 0 } + (k,0,0 { Q i (t + 2 = k, Q j (t + 2 = 0,U + j = 0 } (k,0, { Q + i = k, Q + j = 0,U j } + (k,0, { Q i (t + 2 = k, Q j (t + 2 = 0,U + j }. Note that by the assumptions of arriva and service processes, there exists a positive probabiity ˆp (independent of ϵ such that there is no arriva during one time-sot and meanwhie the potentia Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.

17 44:7 service of a the servers are d for some d between and S max. For ease of exposition, we take d = in the foowing proof, and the same techniques appy for the case where d. Now, for each occurrence of event (k,m, there exists a positive probabiity ˆp such that + wi happen. (k,m Therefore, we have P ( (k,m = P ( + (k,m ˆpP ( (k,m, (7 where hods due to the fact that both events are defined in steady state. Simiary, we have P ( ( ( (k,0,0 = P + (k,0,0 ˆpP (k, (8 P ( (k,0, = P ( + (k,0, ˆpP ( (k,0,0. (9 Now, we can further simpify T (ϵ as foows T (ϵ 4λ Σ kp ( (k,0, + kp ( (k,0, i= j >i ˆp k=r k=r r 4λ Σ i= j >i ˆp m= m+ (k mp ( (k m,0, k=r r = 4λ Σ i= j >i ˆp hp ( (h,0, =0 h=r r 4λ Σ i= j >i ˆp P ( (h,0, = h=r r 4λ Σ i= j >i ˆp hu j P ( (h,0, =0 h=0 r 4λ Σ i= j >i ˆp ϵ, (20 = where foows from eqs. (7 to (9; hods since U j (t and [ [ U j U (ϵ = ϵ. The atter fact can be easiy obtained by setting mean drift of ˆV (Z (t Q(t to be zero in steady state, which is true since a the moments of Q are bounded. For T (ϵ 2, we can simpify it as foows. [( T (ϵ 2 A (ϵ i A (ϵ j S (ϵ i + S (ϵ 2 j = i= j >i i= j >i = (N [( A (ϵ i A (ϵ 2 ( j + S (ϵ i S (ϵ 2 j (( σ (ϵ Σ 2 ( + λ (ϵ 2 Σ + ν 2 Σ, (2 where hods since the arriva and service are independent and the servers are homogeneous; is true because A i (ta j (t = 0 for a i j and t 0, and the service is independent and homogeneous. Proc. ACM Meas. Ana. Comput. Syst., Vo. 2, No. 3, Artice 44. Pubication date: January 209.