Heavy Traffic Optimal Resource Allocation Algorithms for Cloud Computing Clusters

Transcription

1 Heavy Traffic Optial Resource Allocation Algoriths for Cloud Coputing Clusters Siva Thea aguluri and R. Srikant Departent of C and CSL University of Illinois at Urbana-Chapaign siva.thea@gail.co; rsrikant@illinois.edu Lei Ying Departent of C Arizona State University lying6@asu.edu Abstract Cloud coputing is eerging as an iportant platfor for business, personal and obile coputing applications. In this paper, we study a stochastic odel of cloud coputing, where obs arrive according to a stochastic process and request resources like CPU, eory and storage space. We consider a odel where the resource allocation proble can be separated into a routing or load balancing proble and a scheduling proble. We study the oin-the-shortest-queue routing and powerof-two-choices routing algoriths with axweight scheduling algorith. It was known that these algoriths are throughput optial. In this paper, we show that these algoriths are queue length optial in the heavy traffic liit. Index Ters Scheduling, load balancing, cloud coputing, resource allocation. I. INTRODUCTION Cloud coputing services are eerging as an iportant resource for personal as well as coercial coputing applications. Several cloud coputing systes are now coercially available, including Aazon C syste 7], Google s Appngine 1], and icrosoft s Azure 3]. A coprehensive survey on cloud coputing can be found in 9], ], 17]. In this paper, we focus on cloud coputing platfors that provide infrastructure as service. Users subit requests for resources in the for of virtual achines Vs. ach request specifies the aount of resources it needs in ters of processor power, eory, storage space, etc.. We call these requests obs. The cloud service provider first queues these requests and then schedules the on physical achines called servers. ach server has a liited aount of resources of each kind. This liits the nuber and types of obs that can be scheduled on a server. The set of obs of each type that can be scheduled siultaneously at a server is called a configuration. The convex hull of the possible configurations at a server is the capacity region of the server. The total capacity region of the cloud is then the inkowski su of the capacity regions of all servers. The siplest architecture for serving the obs is to queue the at a central location. In each tie slot, a central scheduler chooses the configuration at each server and allocates obs to the servers, in a preeptive anner. As pointed out in 15], this proble is then identical to scheduling in an ad hoc wireless network with interference constraints. In practice, however, obs are routed to servers upon arrival. Thus, queues are aintained at each individual server. It was shown in 15] that oin-the-shortest queue-type algoriths for routing, along with the axweight scheduling algorith ] at each server is throughput optial. The focus of this paper is to study the delay, or equivalently, the queue length perforance of the algoriths presented in 15]. Characterizing the exact delay or queue length in general is difficult. So, we study the syste in the heavy-traffic regie, i.e., when the exogenous arrival rate is close to the boundary of the capacity region. In this regie, for soe systes, the ulti-diensional state of the syste reduces to a single diension, called state-space collapse. In 16], 3], a ethod was outlined to use the state-space collapse for studying the diffusion liits of several queuing systes. This procedure has been successfully applied to a variety of ultiqueue odels served by ultiple servers 0], 11], 1], 4]. But these odels assue that the syste is work conserving, i.e., queued obs are processed at axiu rate by each server. Stolyar 1], generalized this notion of state-space collapse and resource pooling to a generalized switch odel, where it is hard to define work-conserving policies. This was used to establish the heavy traffic optiality of the axweight algorith. ost of these results are based on considering a scaled version of queue lengths and tie, which converges to a regulated Brownian otion, and then show saple-path optiality in the scaled tie over a finite tie interval. This then allows a natural conecture about steady state distribution. In 8], the authors present an alternate ethod to prove heavy traffic optiality that is not only sipler, but shows heavy traffic optiality in unscaled tie. In addition, this ethod directly obtains heavy-traffic optiality in steady state. The ethod consists of the following three steps. 1 Lower bound: First a lower bound is obtained on the weighted su of expected queue lengths by coparing with a single-server queue. A lower bound for the singleserver queue, siilar to the Kingan bound 14], then gives a lower bound to the original syste. State-space collapse: The second step is to show that the state of the syste collapses to a single diension. Here, it is not a coplete state-space collapse, as in the Brownian liit approach, but an approxiate one. In particular, this step is to show that the queue length along a certain direction increases as the exogenous arrival rate gets closer to the boundary of the capacity region but the

2 queue length in any perpendicular direction is bounded. 3 Upper bound: The state-space collapse is then used to obtain an upper bound on the weighted queue length. This is obtained by using a natural Lyapunov function suggested by the resource pooling. Heavy-traffic optiality can be obtained if the lower bounds and the upper bounds coincide. In this paper, we apply the above three-step procedure to study the resource allocation algoriths presented in 15]. We briefly review the results in 15] now. Jobs are first routed to the servers, and are then queued at the servers, and a scheduler schedules obs at each server. So, we need an algorith that has two coponents, viz., 1 a routing algorith that routes new obs to servers in each tie slot we assue that the obs are assigned to a server upon arrival and they cannot be oved to a different server and a scheduling algorith that chooses the configuration of each server, i.e., in each tie slot, it decides which obs to serve. Here we assue that obs can be preepted, i.e., a ob can be served in a tie slot, and then be preepted if it is not scheduled in the next tie slot. Its service can be resued in the next tie it is scheduled. Such a odel is applicable in situations where ob sizes are typically large. It was shown in 15] that using the oin-the-shortestqueue JSQ routing and axweight scheduling algorith is throughput optial. In Section III, we show that this policy is queue length optial in the heavy traffic liit when all the servers are identical. We use the three step procedure described above to prove the heavy traffic optiality. The lower bound in this case is identical to the case of the axweight scheduling proble. However, state-space collapse does not directly follow fro the corresponding results for the axweight algorith in 8] due to the additional routing step here. We use this to obtain an upper bound that coincides with the lower bound in the heavy traffic liit. JSQ needs queue length inforation of all servers at the router. In practice, this counication overhead can be quite significant when the nuber of servers is large. An alternative algorith is the power-of-two-choices routing algorith. In each tie slot, two servers are chosen uniforly at rando and new arrivals are routed to the server with the shorter queue. It was shown in 15] that the power-of-two-choices routing algorith with the axweight scheduling is throughput optial if all the servers are identical. Here, we show that the heavy-traffic optiality in this case is a inor odification of the corresponding result for JSQ routing and axweight scheduling. A special case of the resource allocation proble is when all the obs are of sae type. In this case, scheduling is not required at each server. The proble reduces to a routingonly proble which is well studied 18], 5], 6], 13], 19]. For reasons to be explained later, the results, fro Section III cannot be applied in this case since the capacity region is along a single diension of the for < µ. In Section IV, we show heavy traffic optiality of the power-of-twochoices routing algorith. The lower and upper bounds in this case are identical to the case of JSQ routing in 8]. The ain contribution here is to show state-space collapse, which is soewhat different copared to 8]. The results here copleent the heavy-traffic optiality results in 6], 13] which were obtained using Brownian otion liits. Note on Notation The set of real nubers, the set of non-negative real nubers, and the set of positive real nubers are denoted by R, R + and R ++ respectively. We denote vectors in R J or R by x, in noral font. We use bold font x to denote vectors in R J. Dot product in the vector spaces R J or R is denoted by x, y and the dot product in R J is denoted by x, y. II. SYST ODL AND ALGORITH Consider a discrete tie cloud coputing syste as follows. There are servers indexed by. ach server has I different kinds of resources such as processing power, disk space, eory, etc.. Server has R i, units of resource i for i {1,, 3,..., I}. There are J different types of obs indexed by. Jobs of type need r i, units of resource i for their service. A ob is said to be of size D if it takes D units of tie to finish its service. Let D ax be the axiu allowed service tie. Let A t denote the set of type- obs that arrive at the beginning of tie slot t. Indexing the obs in A t fro 1 through A t, we define a t = k A D t k, to be the overall size of the obs in A t or the total tie slots requested by the obs in A t. Thus, a t denotes the total work load of type that arrives in tie slot t. We assue that a t is a stochastic process which is i.i.d. across tie slots, a t] = and Pra t = 0 > ɛ A for soe ɛ A > 0 for all and t. any of these assuptions can be relaxed, but we ake these assuptions for the ease of exposition. Second oents of the arrival processes are assued to be bounded. Let vara t] = σ, = 1,... J and σ = σ 1,...σ J. We denote σ = σ1,...σj. In each tie slot, the central router routes the new arrivals to one of the servers. ach server aintains J queues corresponding to the work loads of the J different types of obs. Let q, t denote the total backlogged ob size of the type obs at server at tie slot t. Consider server. We say that server is in configuration s = s 1, s,..., s J Z + J if the server is serving s 1 obs of type 1, s obs of type etc. This is possible only if the server has enough resources to accoodate all these obs. In other J words, s r i, R i, i {1,,..., I}. Let s ax be the =1 axiu nuber of obs of any type that can be scheduled on any server. Let S be the set of feasible configurations on server. We say that s is a axial configuration if no other ob can be accoodated i.e., for every s + e

3 where e is the unit vector along violates at least one of the resource constraints. Let C be the convex hull of the axial configurations of server. Let C = {s R + J : s s for soe s C}. Here s s eans s s {1,,..., J}. C can be thought of as the capacity region for server. Note that if interiorc, there exists an ɛ > 0 such that 1 + ɛ C. C is a convex polytope in the nonnegative quadrant of R J. Define C = C = {s R + J : s C s.t. s =1 =1 s }. We denote this as C = =1 C. Here s ust denotes an eleent in C and not th power of s. Then, C = C, where denotes the inkowski su =1 of sets. So, C is again a convex polytope in the nonnegative quadrant of R J. So, C can be described by a set of hyperplanes as follows: C = {s 0 : c k, s b k, k = 1,...K} where K is the nuber of hyperplanes that copletely defines C, and c k, b k copletely defines the k th hyperplane H k, c k, s = b k. Since C is in the first quadrant, we have c k = 1, c k 0, b k 0 for k = 1,,...K. It was shown in 15] that C is the capacity region of this syste. Siilar to C, define S = S. =1 Lea 1: Given the k th hyperplane H k of the capacity region C i.e., c k, = b k, for each server, there is a b k such that c k, = b k is the boundary of the capacity region C, and b k = b k. oreover, for every set =1 { } k C such that k = k and k C lies =1 on the k th hyperplane H k, we have that c k, k = b k. Proof: Define b k C = C, we have that b k = =1 = ax s C c k, s. Then, since b k. =1 Again, by the definition of C, for every C, there are k C for each such that k = k. =1 However, these { ay } not be unique. We will prove that for every such k, for each, c k, k = b k. Suppose, for soe server 1, c k, k 1 < b k 1. Then since c k, k = b k, there exists such that =1 =1 c k, k > b k which is a contradiction. Thus, we have the lea. III. JSQ ROUTING AND AXWIGHT SCHDULING In this section, we will study the perforance of JSQ routing with axweight scheduling, as described in Algorith 1. Algorith 1 JSQ Routing and axweight Scheduling 1 Routing Algorith: All the type arrivals in a tie slot are routed to the server with the sallest queue length for type obs, i.e., the server = arg in q,. {1,,...} Ties are broken uniforly at rando. Scheduling Algorith: In each tie slot, server chooses a configuration s C so that s = J arg ax s q,. It then schedules up to a axiu s C =1 of s obs of type in a preeptive anner. Note that even if the queue length is greater than the allocated service, all of it ay not be utilized, e.g., when the backlogged size is fro a single ob, since different chunks of the sae ob cannot be scheduled siultaneously. Denote the actual nuber of obs chosen by s. Note that if q, D ax s ax, then s = s. Let Y, t denote the state of the queue for type- obs at server, where Y, i t is the backlogged size of the i th type- ob at server. It is easy to see that Yt = {Y, t}, is a arkov chain under the JSQ routing and axweight scheduling. Then, q, t = i Y, i t is a function of the state Y, t. The queue lengths of workload evolve according to the following equation: q, t + 1 = q, t + a, t s t = q, t + a, t s t + u,t 1 where u, t is the unused service, given by u, t = s t s t, s t is the axweight schedule and s t is the actual schedule chosen by the scheduling algorith and the arrivals are a, t = { a t if = t. 0 otherwise Here, is the server chosen by the routing algorith for type obs. Note that u, t = 0 when q, t + a, t D ax s ax. 3 Also, denote s = s where s = s. 4 =1 Denote a = a,,, s = s, and u = u,,. Also denote 1 to be the vector with 1 in all coponents. It was shown in 15] that this algorith is throughput optial. Here, we will show that this algorith is heavy traffic optial.

4 Recall that the capacity region is bounded by K hyperplanes, each hyperplane H k described by its noral vector c k and the value b k. Then, for any interiorc, we can define the distance of to H k and the closest point, respectively, as ɛ k = in s H k s 5 k = + ɛ k c k where ɛ k > 0 for each k since interiorc. We let ɛ ɛ k K denote the vector of distances to all hyperplanes. k=1 Note that k ay be outside the capacity region C for soe hyperplanes. So define { } K k {1,,...K} : k C K identifies the set of doinant hyperplanes whose closest point to is on the boundary of the capacity region C hence is a feasible average rate for service. Note that for any interiorc, the set K is non-epty, and hence is welldefined. We further define { } K o k K : k RelintF k where F k denotes the face on which k lies and Relint eans relative interior. Thus, K o is the subset of faces in K for which the proection of is not shared by ore than one hyperplane. For ɛ ɛ k K > 0, let k=1 ɛ be the arrival rate in the interior of the capacity region so that its distance fro the hyperplane H k is ɛ k. Let k be the closest point to ɛ on H k. Thus, we have k = ɛ + ɛ k c k. 6 Let q ɛ t be the queue length process when the arrival rate is ɛ. Define c k R J +, indexed by, as c, = c. We expect that the state space collapse occurs along the direction c k. This is intuitive. For a fixed, JSQ routing tries to equalize the queue lengths across servers. For a fixed server, we expect that the state space collapse occurs along c k when approaching the hyperplane H k, as shown in 8]. Thus, for JSQ routing and axweight, we expect that the state space collapse occurs along c k in R J. For each k K o, define the proection and perpendicular ɛ coponent of q ɛ to the vector c k as follows: q ɛ,k c k, q ɛ c k q ɛ,k q ɛ q ɛ,k In this section, we will prove the following proposition. Proposition 1: Consider the cloud coputing syste described in Section II. Assue all the servers are identical, i.e., R i, = R i for all servers and resources i and that JSQ routing and axweight scheduling as described in Algorith 1 is used. Let the exogenous arrival rate be ɛ InteriorC and the standard deviation of the arrival vector be σ ɛ R J ++ where the paraeter ɛ = ɛ k K k=1 is so that ɛk is the distance of ɛ fro the k th hyperplane H k as defined in 5. Then for each k K o, the steady state queue length ɛ satisfies ] ɛ k c k, qt ζɛ,k + B ɛ,k where ζ ɛ,k = c 1 k, σ ɛ + ɛk, B ɛ,k is o 1 ɛ k In the heavy traffic liit as ɛ k 0, this bound is tight, i.e., li ɛ k c k, q ɛ ] = ζk ɛ k 0 where ζ k = 1 c k, σ. We will prove this proposition by following the three step procedure described in Section I, by first obtaining a lower bound, then showing state space collapse and finally using the state space collapse result to obtain an upper bound. A. Lower Bound Since ɛ is in the interior of C, the process { q ɛ t } has t a steady state distribution. We will obtain a lower bound on c k, q ɛ ] J c k ] = in steady state as =1q =1 follows. Consider the single server queuing syste, φ ɛ t with 1 arrival process c k, a ɛ t and service process given by bk than c k, qt ɛ. Thus, we have at each tie slot. Then φt is stochastically saller c k, q ɛ ] φ ɛ]. Using φ as Lyapunov function for the single server queue and noting that the drift of it should be zero in steady state, one can bound φ ɛ] as follows 8] where c k ɛ k φ ɛ] ζɛ,k = B ɛ,k 1. J c k, B ɛ,k 1 = bk ɛ k and =1 ζ ɛ,k = 1 c k, σ ɛ + ɛk. Thus, in the heavy traffic liit as ɛ k 0, we have that li ɛ k c k, q ɛ ] ζk 7 ɛ k 0 where ζ k = 1 c k, σ. B. State Space Collapse In this subsection, we will show that there is a state space collapse along the direction c k. We know that as the arrival rate approaches the boundary of the capacity region, i.e., ɛ k 0, the steady state ean queue length q]. We will show that as ɛ k 0, queue length proected along

5 any direction perpendicular to c k is bounded. So the constant Lea 3: Drift of W k does not contribute to the first order ter in 1 can be bounded as follows:, in which we ɛ k are interested. Therefore, it is sufficient to study a bound on W k the queue length along c k. This is called state-space collapse. q 1 q k V q V k q q R J + Define the following Lyapunov functions. 8 J Let us first consider the last ter in this inequality. V q q,, W k q q k, W k q q k =1=1 V k V k q c k, q ɛ q k = = 1 q J ɛ q ɛ t = q ɛ] q, c. = V k q ɛ t + 1 V k q ɛ t q ɛ t = q ɛ] =1=1 ] Define the drift of the above Lyapunov functions. = c k, q ɛ t + 1 c k, q ɛ t qt = q ɛ V q V qt + 1 V qt] Iqt = q ] W k q W k k qt + 1 W qt Iqt = q ] W k q W k qt + 1 W k qt Iqt = q ] V k q V k qt + 1 V k qt Iqt = q To show the state space collapse happens along the direction of c k, we will need a result by Haek 10], which gives a bound on q k if the drift of W k q is negative. Here we use the following special case of the result by Haek, as presented in 8]. Lea : For an irreducible and aperiodic arkov Chain {Xt]} t 0 over a countable state space X, suppose Z : X R + is a nonnegative-valued Lyapunov function. We define the drift of Z at X as ZX ZXt + 1] ZXt]] IXt] = X, where I. is the indicator function. Thus, ZX is a rando variable that easures the aount of change in the value of Z in one step, starting fro state X. This drift is assued to satisfy the following conditions: 1 There exists an η > 0, and a κ < such that for all X X with ZX κ, ZX Xt] = X] η. There exists a D < such that for all X X, P ZX D = 1. Then, there exists a θ > 0 and a C < such that ] li sup e θ ZXt] C. t If we further assue that the arkov Chain {Xt]} t is positive recurrent, then ZXt] converges in distribution to a rando variable Z for which e Z] θ C, which directly iplies that all oents of Z exist and are finite. We also need Lea 7 fro 8], which gives the drift of W k k q in ters of drifts of V q and V q. = c k, q ɛ t + a ɛ t s ɛ t + u ɛ t ] c k, q ɛ t qt = q ɛ = c k, q ɛ t + a ɛ t s ɛ t + c k, u ɛ t + c k, q ɛ t + a ɛ t s ɛ t c k, u ɛ t ] c k, q ɛ t qt = q ɛ c k, a ɛ t s ɛ t c k, s ɛ t c k, u ɛ t ] + c k, q ɛ t c k, a ɛ t s ɛ t qt = q ɛ c k, q ɛ c k, a ɛ t qt = q ɛ] s ɛ t qt = q ɛ] c k, s ax 1 = qɛ,k =1 J c a ɛ t qt = qɛ] =1 =1, s ɛ t qt = q ɛ] K = qɛ,k = qɛ,k =1 J c =1 J c =1 ɛ k =1 ɛ k c k s ɛ t qt = q ɛ] K s ɛ t qt = q ɛ] K 10 = qɛ,k J c =1 k =1 =1 s ɛ 9 t qt = q ɛ] K ɛk q ɛ,k 11 = qɛ,k =1=1 J c k s ɛ t qt = q ɛ]

6 K ɛk q ɛ,k K ɛk q ɛ,k 1 where K = Js ax. quation 9 follows fro the fact that the su of arrival rates at each server is sae as the external arrival rate. quation 10 follows fro 6. Fro the definition of C, we have that there exists k C such that k = =1 k. This gives 11. Fro Lea 1, we have that for each, there exists b k and c k, s ɛ b k we have, for each, J c =1 k such that J c k =1 = b k for every s ɛ t C. Therefore, s ɛ t qt = q ɛ] 0 and so 1 is true. Now, let us consider the first ter in 8. By expanding the drift of V q ɛ and using 3, it can be easily seen that V q ɛ q ɛ t = q ɛ] J K + where K = =1=1 q ɛ, a, t s t 13 + σ + Js ax 1 + D ax By definition of a, t, we have J q ɛ, a,t =1=1 J = = ɛ =1 =1 J =1, a t q ɛ q ɛ, ɛ =1 J ɛ =1 =1 Fro 13 and 14, we have, V q ɛ q ɛ t = q ɛ] J K + J =K + J =1 =1 k J q ɛ, =1 ɛ k c k q ɛ, s t =1 =1 q ɛ,. 14 q ɛ, s t =1 q ɛ, 15 =K ɛk q ɛ,k + =K 1 ɛk q ɛ,k + J =1 q ɛ, =1 in =1 J k q ɛ r, C =1 s t k r 16 where K 1 = K + JD ax s ax. quation 16 is true because of axweight scheduling. Note that in algorith 1, the actual service allocated to obs of type at server is sae as that of the axweight schedule as long as the corresponding queue length is greater than D ax s ax. This gives the additional JD ax s ax ter. Assuing all the servers are identical, we have that for each, C = {/ : C}. So, C is a scaled version of C. Thus, = /. Since k K o, we also have that ɛ k K o for the capacity region C ɛ. Thus, there exists δ k > 0 so that B k δ k H k {r R J + : r k / δ k } lies strictly within the face of C that corresponds to F k. Note that this is the only instance in the proof of Proposition 1 that we use the assuption that all the servers are identical. Call this face F k. Thus we have, V q ɛ q ɛ t = q ɛ] K 1 ɛk q ɛ,k in J =1 r B k δ k =1 = in J r B k =1 δ k =1 = in J =1 r B k δ k =1 = δ k J =1 =1 =1=1 q ɛ, q ɛ k r 17 q, k c k q ɛ,k, k r r 18 q ɛ,k, 19 δ k J q, ɛ,k 0 = δ k q k. 1 quation 18 is true because c is a vector perpendicular to the face F k of C whereas both k / and r lie on the face F k 1. So, q k J k c r = 0. quation 19 =1 is true because J q ɛ,k k, r is inner product in R J + =1 which is iniized when r is chosen to be on the boundary of B k k so that δ k r points in the opposite direction

7 to q ɛ,k, =1. Since J q ɛ,k, =1 J =1=1 q ɛ,k,, we get 0. Now substituting 1 and 1 in 8, we get W k qɛ q ɛ t = q ɛ] K 1 + K δ k q ɛ,k δk whenever W k qɛ K 1 + K. δ k oreover, since the departures in each tie slot are bounded and the arrivals are finite there is a D < such that P ZX D alost surely. Now, applying Lea, we have the following proposition. Proposition : Assuing all the servers are identical, for ɛ C, under JSQ routing and axweight scheduling, for every k K o, there exists a set of finite constants ɛ {N r k q ɛ,k } r=1,,... such that r ] N r k for all ɛ > 0 and for each r = 1,,... As in 1], 8], note that k K o is an iportant ɛ assuption here. If k K K o, i.e., if the arrival rate ɛ approaches a corner point of the capacity region as ɛ k 0, then there is no constant δ k so that B k lies in the face δ k F k. In other words, the δ k depends on ɛ k and so the bound obtained by Lea also depends on ɛ k. Reark: As stated in Proposition 1, our results hold only for the case of identical servers, which is the ost practical scenario. However, we have written the proofs ore generally whenever we can so that it is clear where we need the identical server assuption. In particular, in this subsection, up to quation 16, we do not need this assuption, but we have used the assuption after that, in analyzing the drift of V q. The upper bound in the next section is valid ore generally if one can establish state-space collapse for the non-identical server case. However, at this tie, this is an open proble. C. Upper Bound In this section, we will obtain an upper bound on the weighted queue length, c k, q ɛ ] in steady state, and show that in the asyptotic liit as ɛ k 0, this coincides with the lower bound. Noting that the drift of W k is zero in steady state, it can be shown, as in Lea 8 fro 8] that in steady state, for any c R J +, we have c, qt c, st at ] c, st at ] c, ut ] = c, qt + at st c, ut ] 4 We will obtain an upper bound on c k, q ɛ ] by bounding each of the above ters. Before that, we need the following definitions and results. Let π k be the steady-state probability that the axweight schedule chosen is fro the face F k, i.e., π k = P c, st = b k. where s = s =1 γ k = in as defined in 4. Also, define { b k c, r : r S \ F k}. Then noting that in steady state, ] c k, sq c k, ɛ = b k ɛ k, it can be shown as in Clai 1 in 8] that for for any ɛ k 0, γ k, 1 π k ɛk γ k. Then, note that ] b k c, st = 1 π k b k c, st c, st b k ], ɛk b k + c, sax1 5 γ k Define C R J + as C = C 1... C. Then, C is a convex polygon. Clai 1: Let q R J + for each {1,,...}. Denote q = q =1 RJ +. If, for each, s is a solution of ax q, s then s = s is a solution of ax q, s. s C s C Proof: Since s C, q, s ax q, s. s C Note that ax q, s = ax q, s. Therefore, s C =1s C if q, s < ax q, s, we have q, s < s C =1 ax q, s. Then there exists an such that =1s C q, s < ax q, s, which is a contradiction. s C Therefore, choosing a axweight schedule at each server is sae as choosing a axweight schedule fro the convex polygon, C. Since there are a finite nuber of feasible schedules, given c k R J + such that c k = 1, there { exists an angle θ k 0, π ] such that, for all q q R J + : q k q cos θ k}, i.e., for all q R J + such that θ k qq θ k where θ ab represents the angle between vectors a and b, we have c k, st I qt = q = b k /. We can bound the unused service as follows. ] ] c k, ut c k, st at

8 = 1 ] c k, st c k, ɛ = 1 ] c k, st b k ɛ k ɛk 6 where the last inequality follows fro the fact that the axweight schedule lies inside the capacity region and so c k, st ] b k. Now, we will bound each of the ters in 4. Let us first consider the ter in. Given that the arrival rate if ɛ we have, ] c k, qt c k, st at ] = c k b k, qt 1 c k, c k b k ], qt c k, st ] = ɛk c k, qt q k b k t c k, st ]. Now, we will bound the last ter in this equation using the definition of θ k as follows. q k b k ] t c k, st b k ] = qt cos θ k qq c k, st = qt cos θ k qq I θ k qq > θ k b k c k, st ] 7 = q k t cot θ qq k b k c k, st = q k ti θ qq k θ k I ] θ k qq > θ k b > θ k k ] c k, st cot 1 ] q k b t k c k, st cot θ k 8 cot θ k ] b q k t k c k, st ] cot θ k N k ɛ k γ k b k + c, sax 1 9 where 7 follows fro the definition of θ k, 8 follows fro our choice of c k and definition of s, 9 follows fro Cauchy-Schwarz inequality. the last inequality follows fro state-space collapse Proposition and 5. Thus, we have ] c k, qt c k, st at ] ɛk c k, qt cot θ k N k ɛ k b k + c, sax 1 30 γ k Now, consider the first ter in 3. Again, using the fact that the arrival rate is ɛ we have, ] c k, st at ] = c k, at bk b k ] + c k, st ɛk b k + ] c k, st 1 c k, at ɛ + b k ] c k, st c k, ɛ b k = 1 c k, at ɛ ] + ɛk c k, at ɛ ] + 1 ɛ k + 1 ] b k c k, st c k, σ ɛ k ɛ k γ k b k + c, sax1 = 1 ζ ɛ,k + 1 ɛ k γ k b k + c, sax1 where ζ ɛ,k was defined as ζ ɛ,k = 31 3 ɛ k + 1 c k, σ ɛ. quation 31 is obtained by noting that at] = ɛ and so c k, at ɛ ] = var c k, at ɛ = c k, varat ɛ. Consider the second ter in 3. ] ] c k, ut c k, 1s ax c k, ut ɛk c k, 1s ax 33 where the last inequality follows fro 6. Now, we consider the ter in 4. We need soe definitions so that we can { only consider the non-zero} coponents of c. Let L k ++ = {1,,...J} : c k > 0. Define

9 c k = c k L k ++, q = q L k ++ and ũ = u k L. ++ Also define, the proections, q k = c k, q c k and q k = q q k. Siilarly, define ũ k and ũ k. Then, we have ] c k, qt + at st c k, ut ] ] = c k, qt + 1 c k, ut c k, ut ] c k, qt + 1 c k, ut ] = c k, qt + 1 c k, ũt ] = q k t + 1ũ k ] = q k t + 1, ũk t ] = q k t + 1, ũt ] = qt + 1, ũt ] + q k t + 1, ũt ] D ax s ax 1, ũt ] + q k t + 1 ũt ] 34 D ax s ax 1, ũt ] + N k ũt, ũt ] 35 D ax s ax 1, ũt ] + N k s ax 1, ũt ] where 34 follows fro 3 and fro Cauchy-Schwarz inequality. quation 35 follows fro ] fro state-space ] collapse Proposition, since q k q k N k. Note that 1, ũt ] 1 ] c k, ũt where c k in = in L k ++ c k c k in = 1 c k in ɛk ] c k, ut > 0 and the last inequality follows fro 6. Thus, we have ] c k, qt + st at c k, ut ɛ k D ax s ax + N k ɛ s ax k 36 Now, substituting 30, 3, 33 and 36 in 4, we get ] ɛ k c k, qt ζɛ,k + B ɛ,k where b k + c, sax1 + D ax s ax ɛ k B ɛ,k = 1 ɛ k γ k + ɛk c k, 1s ax + N k s ax ɛ k + cot θ k N k ɛ k b k + c, sax 1. γ k Thus, in the heavy traffic liit as ɛ k 0, we have that li ɛ k c k, q ɛ ] ζk ɛ k 0 37 where ζ k was defined as ζ k = 1 c k, σ. Thus, 7 and 37 establish the first oent heavy-traffic optiality of JSQ routing and axweight scheduling policy. The proof of Proposition 1 is now coplete. D. Power-of-Two-Choices Routing and axweight Scheduling JSQ routing needs coplete queue length inforation at the router. In practice, this counication overhead can be considerable when the nuber of servers is large. An alternate algorith is the power-of-two-choices routing algorith. In this algorith, in each tie slot t, for each type of ob, two servers 1 t and t are chosen uniforly at rando. All the type ob arrivals in this tie slot are then routed to the server with the shorter queue length aong these two, i.e., t = arg in q, t. { 1 t, It was shown in 15] that t} power-of-two-choices routing algorith with axweight scheduling is throughput optial if all the servers are identical. Fro the proof of throughput optiality, one obtains V q ɛ q ɛ t = q ɛ] J K q ɛ, + =1 =1 J q ɛ, s t =1=1 Note that this inequality is identical to 15, in the proof of state-space collapse of JSQ routing and axweight scheduling policy. Also note that the reainder of the proof of statespace collapse and upper bound in Sections III-B and III-C is independent of the routing policy. oreover, the proof of lower bound in Section III-A is also valid here. Thus, once we have the above relation, the proof of heavy traffic optiality of this policy is identical to that of JSQ routing and axweight scheduling policy. IV. POWR-OF-TWO-CHOICS ROUTING In this section, we consider the power-of-two-choices routing algorith, without any scheduling. This is a special case of the odel considered in the previous section when all the obs are of the sae type. In this case, there is a single queue at each server and no scheduling is needed. Note on Notation In this section, since J = 1 here, we ust denote all vectors in R in bold font x. The result fro previous section is not applicable here because of the following reason. In Proposition 1, a sequence of systes with arrival rate approaching a face of the capacity region, along its noral vector were considered. The noral

10 vector of the face plays an iportant role in the state space of Q along c 1, i.e., Q = Q, c 1 c 1 where.,. denotes the collapse, and so the upper bound obtained is in ters of this Q canonical dot product. Thus, Q noral. So, this result cannot be applied if the arrival rates = 1. Define Q to be the coponent of Q perpendicular to Q were approaching a corner point where there is no coon, i.e., Q = Q Q. noral vector. In particular, the proof of state space collapse in Section III-B is not applicable here because one cannot define Define the Lyapunov functions V Q = Q = a ball B k as in 17 at a corner point. δ k Let At denote the set of obs that arrive at the beginning of and W Q = Q = 1 Q Q tie slot t. Let D k be the size of k th. ob. We define at = k At D k, to be the overall size of the obs in At or A. Lower Bound the total tie slots requested by the obs. We assue that Consider an arrival process with arrival rate ɛ such that at is a stochastic process which is i.i.d. across tie slots, ɛ = µ ɛ. Let q ɛ t denote the corresponding queue at] = and Prat = 0 > ɛ a for soe ɛ a > 0 for all t. Let σ length vector. Since the syste is stabilizable, there exists = varat]. Let Xt denote the servers chosen at tie slot t. So, Xt can take one of a steady-state distribution of q ɛ t. Again, lower bounding C values of the for, where, Z + and 1 < q ɛ by a single queue length as in Section III-A, we have. Here C denotes the nuber of -cobinations in a set ] of size. Note that Xt is an i.i.d. rando process with σ q ɛ ɛ + ɛ B 1 a unifor distribution over all possible values. Define C ɛ different arrival processes denoted by a, t with 1 < as follows. If xt = ˆ, ˆ where B 1 = sax, then. Thus, in the heavy-traffic liit we have ] { at for = ˆ and a, t = ˆ = li inf. ɛ q ɛ σ ɛ otherwise B. State Space Collapse Thus, {a, t} can be thought of as a set of correlated For siplicity of notation, in this sub-section, we write q arrival processes. They are correlated so that only one of the for q can have a non-zero value at each tie. Let, = a, t]. ɛ. We will bound the drift of the Lyapunov function Then, = W Q, and again use Lea to obtain state space collapse. C. The arrivals in a, t can be routed only to either server or server We again use 8 with c 1 instead of c k to get the drift of. According to the powerof-two-choices algorith, all the obs are then routed to the W k k q in ters of drifts of V q and V q. server with sallest queue aong and Let us first consider the last ter.. Ties are broken at rando. Let a t denote the arrivals to server at tie V q qt = q ] t after routing. = V qt + 1 V qt qt = q ] Let µ be the aount of service available in each tie slot at each server. Not all of this service ay be used either = 1 because the queue is epty or because different chunks of q t + 1 q t qt = q sae ob cannot be served siultaneously. Let s t be the actual aount of service scheduled available in tie slot t at server. Let u t denote the unused service which is = 1 q t + a t µ + u t defined as u t = µ s t. Let q t denote the queue length at server at tie t, and let qt denote the vector q 1 t, q t,...q t Then, we have q t qt = q Note that q t + 1 = q t + a t µ + u t. u t = 0 whenever q t + a t D ax µ. 38 We again follow the procedure used in the previous section to show heavy traffic optiality. Since power-of-two-choices algorith tries to equalize any two randoly chosen queues, we expect that there is a state-space collapse along the direction where all queues are equal, siilar to JSQ algorith. Let c 1 = 1 1, 1,...1 be the unit vector in R along which we expect state-space collapse. Let 1 denote the vector 1,1,...1. For any Q R, define Q to be the coponent = 1 q t + a t µ + u t + q t + a t µ u t q t qt = q Q 1 a t µ + q t a t µ

11 ] µ u t qt = q ] q a t µ qt = q ] µ u t qt = q K 3 + q µ K 3 ɛ q 40 where K 3 = µ is obtained by bounding s t and u t by s ax. Now, we will bound the first ter in 8. xpanding V q qt] and using 38, it is easy to see that V q qt = q] K 4 µ q t ] + X q ta t qt = q, Xt = i,. where K 4 = µ D ax σ +. Let p be a perutation of 1,,... so that q p1 q p... q p. Let p be the inverse perutation. In other words, p is the position of in the perutation p. Let q in = q p1 and q in = q p. Then, we have V q qt = q] K 4 µ q t + q in C 1 + q i tat + q i tat Xt = i, ] C i, p1,p =K 4 µ q t q ax q in + q t. C =K 4 ɛ q t q ax q in C Note that Thus, we have, q = q q q ax q in = q ax q in. V q qt = q] K 4 ɛ q t q C Substituting this and 40 in 8, we have W q] K 3 + K 4 q 1 C. This eans that we have negative drift for sufficiently large W q. Since the drift of W q is finite with probability 1, using Lea, ] there exist finite constants {N r} r=1,,... such that q ɛ r N r for each r = 1,,... C. Upper Bound The upper bound is again obtained by bounding each of the ters in 4. This is identical to the case of JSQ routing Proposition 3 in 8]. So, we will not repeat the proof here, but ust state the upper bound. ] σ q ɛ ɛ + ɛ ɛ where B ɛ = liit, we have N s ax ɛ li inf ɛ ɛ 0 B ɛ + sax. Thus, in heavy traffic ] q ɛ σ. This coincides with the heavy-traffic lower bound in 39. This establishes the first-oent heavy-traffic optiality of powerof-two choices routing algorith. V. CONCLUSIONS We considered a stochastic odel for load balancing and scheduling in cloud coputing clusters. We studied the perforance of JSQ routing and axweight scheduling policy under this odel. It was known that this policy is throughput optial. We have shown that it is heavy traffic optial when all the servers are identical. We also found that using the power-of-two-choices routing instead of JSQ routing is also heavy traffic optial. We then considered a sipler setting where the obs are of the sae type, so only load balancing is needed. It has been established by others using diffusion liit arguents that the power-of-two-choices algorith is heavy traffic optial. We presented a steady-state version of this result here using Lyapunov drift arguents. VI. ACKNOWLDGNTS Research was funded in part by ARO URI W911NF and NSF grant CNS RFRNCS 1] Appngine. ]. Arbrust, A. Fox, R. Griffith, A. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, et al. Above the clouds: A Berkeley view of cloud coputing Tech. Rep. UCB/eeCs-009-8, CS departent, U.C. Berkeley. 3] Azure. 4] S. L. Bell and R. J. Willias. Dynaic scheduling of a parallel server syste in heavy traffic with coplete resource pooling: asyptotic optiality of a threshold policy. lectronic J. of Probability, pages , 005.

12 5]. Brason, Y. Lu, and B. Prabhakar. Randoized load balancing with general service tie distributions. In Proceedings of the AC SIGTRICS international conference on easureent and odeling of coputer systes, SIGTRICS 10, pages 75 86, New York, NY, USA, 010. AC. 6] H. Chen and H. Q. Ye. Asyptotic optiality of balanced routing, lgtyehq/papers/chenye11or.pdf. 7] C. 8] A. ryilaz and R. Srikant. Asyptotically tight steady-state queue length bounds iplied by drift conditions. Queueing Systes, pages 1 49, 01. 9] I. Foster, Y. Zhao, I. Raicu, and S. Lu. Cloud coputing and grid coputing 360-degree copared. In Grid Coputing nvironents Workshop, 008. GC 08, pages 1 10, ] B. Haek. Hitting-tie and occupation-tie bounds iplied by drift analysis with applications. Advances in Applied Probability, pages 50 55, ] J.. Harrison. Heavy traffic analysis of a syste with parallel servers: Asyptotic optiality of discrete review policies. Ann. App. Probab., pages 8 848, ] J.. Harrison and. J. Lopez. Heavy traffic resource pooling in parallel-server systes. Queueing Systes, pages , ] Y. T. He and D. G. Down. Liited choice and locality considerations for load balancing. Perforance valuation, 659: , ] J. F. C. Kingan. Soe inequalities for the queue GI/G/1. Bioetrika, pages , ] S. T. aguluri, R. Srikant, and L. Ying. Stochastic odels of load balancing and scheduling in cloud coputing clusters. In Proc. I Infoco., pages , ].Brason. State space collapse with application to heavy-traffic liits for ulticlass queueing networks. Queueing Systes Theory and Applications, pages , ] D. A. enasce and P. Ngo. Understanding cloud coputing: xperientation and capacity planning. In Proc. 009 Coputer easureent Group Conf., ]. itzenacher. The Power of Two Choices in Randoized Load Balancing. PhD thesis, University of California at Berkeley, ] R. L. D. N. D. Vvedenskaya and F. I. Karpelevich. Queueing syste with selection of the shortest of two queues: An asyptotic approach. Probles of Inforation Transission, 31:15 7, ]. I. Reian. Soe diffusion approxiations with state space collapse. In Proceedings of International Seinar on odelling and Perforance valuation ethodology, Lecture Notes in Control and Inforation Sciences, pages 09 40, Berlin, Springer. 1] A. Stolyar. axweight scheduling in a generalized switch: State space collapse and workload iniization in heavy traffic. Adv. Appl. Prob., 141, 004. ] L. Tassiulas and A. phreides. Stability properties of constrained queueing systes and scheduling policies for axiu throughput in ultihop radio networks. I Trans. Autoat. Contr., 4: , Deceber ] R. J. Willias. Diffusion approxiations for open ulticlass queueing networks: Sufficient conditions involving state space collapse. Queueing Systes Theory and Applications, pages 7 88, 1998.