A Better Model for Job Redundancy: Decoupling Server Slowdown and Job Size

Transcription

1 A Better Moel for Job Reunancy: Decoupling Server Slowown an Job Size Kristen Garner 1 Mor Harchol-Balter 1 Alan Scheller-Wolf Benny Van Hout 3 October 1 CMU-CS School of Computer Science Carnegie Mellon University Pittsburgh, PA School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA Tepper School of Business, Carnegie Mellon University, Pittsburgh, PA, USA 3 Mathematics an Computer Science Department, University of Antwerp, Antwerp, Belgium This work is supporte by the Siebel Scholars Founation, by a Google Anita Borg Memorial Scholarship, by NSF-CMMI-153, NSF-CMMI-13319, an NSF-CSR-111, by the Intel ISTC-CC, by a Google Faculty Research Awar 15/1, an by the FWO grant G51N.

2 Keywors: Reunancy, Replication, Parallel servers, Server slowown, Task Assignment, RIQ, Reunant-to-Ile-Queue, Reunancy-

3 Abstract Recent computer systems research has propose using reunant requests to reuce latency. The iea is to replicate a request so that it joins the queue at multiple servers. The request is consiere complete as soon as any one of its copies completes. Reunancy allows us to overcome serversie variability the fact that a server might be temporarily slow ue to factors such as backgroun loa, network interrupts, an garbage collection to reuce response time. In the past few years, queueing theorists have begun to stuy reunancy, first via approximations, an, more recently, via exact analysis. Unfortunately, for analytical tractability, most existing theoretical analysis has assume an Inepenent Runtimes (IR) moel, wherein the replicas of a job each experience inepenent runtimes (service times) at ifferent servers. The IR moel is unrealistic an has le to theoretical results which can be at os with computer systems implementation results. This paper introuces a much more realistic moel of reunancy. Our moel ecouples the inherent job size (X) from the server-sie slowown (S), where we track both S an X for each job. Analysis within the S&X moel is, of course, much more ifficult. Nevertheless, we esign a ispatching policy, Reunant-to-Ile-Queue (RIQ), which is both analytically tractable within the S&X moel an has provably excellent performance.

4

5 1 Introuction As clou computing an resource sharing become more prevalent, we are face with greater egrees of server variability. Recent computer systems stuies have shown that the same job can take up to 1 or 7 longer to run on one machine than another [3, 5]. This is ue to varying backgroun loa, temporary garbage collection, networking interrupts, an other transient events. This server variability is exacerbate by multiplexing of applications an by our increase reliance on virtual machines (VMs); multiple VMs may share the same host resources, affecting each other in unpreictable ways. In an effort to reuce overall latency, an particularly tail latency, the computer systems community has propose using reunancy [3, 7, 17,,, 1]. Reunancy, also known as job replication, is the iea of ispatching the same job to multiple servers, where the job is consiere one as soon as it completes service on any one server. Reunancy provies two key avantages: First, a job that is ispatche to servers experiences the queue with the least work. This is true even if jobs are ispatche to queues immeiately via a front-en loa balancer that has no knowlege of the number of jobs in each queue, or the jobs sizes. Secon, each job experiences the minimum slowown of the servers on which it runs. Both avantages are important when server-sie variability is high. As reunancy has become more popular in computer systems, a raft of theory papers have attempte to analyze the response time benefits of reunancy [1, 1,, 19,, 13, 9, 1, 11, 15, ]. These results assume an Inepenent Runtimes (IR) moel, where a job s copies have inepenent runtimes (service times) at ifferent servers. The inepenent runtimes typically are assume to be exponentially istribute or more highly variable. Here, the IR moel suggests that (barring cancellation costs), more reunancy is better. While the IR moel makes sense in certain settings, it can be problematic in others. Consier for example a job that is very large, meaning that it comprises a large volume of computation. That job shoul appear to be large on all servers, possibly slowe own more on some servers than others. This oes not happen in the IR moel: the job is assigne a ifferent, inepenent, runtime on ifferent servers. When the job runs on multiple servers it experiences the minimum runtime of all those servers inepenent runtimes. Thus an inherently large job can become arbitrarily small uner the inepenence assumption. There is no concept of an inherently large job which remains large at every server. In this paper we propose a more realistic moel for reunancy, calle the S&X moel (see Figure 1). The S&X moel explicitly ecouples the server slowown (represente by the ranom variable S) from the inherent job size (represente by the ranom variable X). The S&X moel marks a eparture from traitional queueing theory, which uses a single service time variable to jointly represent the server spee an job size. A single ranom variable is insufficient in the context of reunancy: We nee to ecouple the variables so that a job with a large X component (large job size) will have a large X component on every server. The S&X moel shes light, however, on some sa truths: reunancy is not always a win an can in fact be angerous. Consier for example, the Reunancy- policy, which replicates every arriving job to queues [9]. Uner the IR moel assume in [9], mean response time only ecreases as we increase. By contrast, in the S&X moel, mean response time uner 1

6 Figure 1: The S&X moel. The system has k servers an jobs arrive as a Poisson process with rate λk. Each job has an inherent size X. When a job runs on a server it experiences slowown S. A job s running time on a single server is R(1) = X S. When a job runs on multiple servers, its inherent size X is the same on all these servers an it experiences a ifferent, inepenently rawn instance of S on each server. Reunancy- can improve significantly as we a a small mount of reunancy, but the system (typically) eventually becomes unstable because of the increase loa of replication, sening mean response time to infinity (see Figure ). Unfortunately, in general we cannot etermine the values of that will lea to goo performance or to instability, because proviing a performance analysis of policies like Reunancy- in the S&X moel is an open problem that is likely very ifficult (analyzing Reunancy- even within the IR moel requires a very complex state space, since one nees to track all the copies of every job in every queue, see [9]). The ifficulty in tuning an potential instability of Reunancy- in the S&X moel motivate us to look for new reunancy policies which are less sensitive to, are robust in that they provably will not go into overloa, an are analytically tractable within the S&X moel. To this en, we introuce a new reunancy policy, calle Reunant-to-Ile-Queue (RIQ). This policy is similar to Reunancy- in that every arrival queries servers. However replicas are only mae to those servers which are ile. If no server is ile, then the job is sent to a ranom one of the servers (with no aitional replicas). We provie an analytical approximation for the transform of response time uner RIQ as a function of. Our analysis allows for any istribution of S, any istribution of X, an any cancellation time. Our analysis matches simulation very well, provie that is small relative to the total number of servers, which is certainly typical in practice. Most importantly, our analysis shows us that RIQ is extremely robust. While Reunancy- can outperform RIQ at low, we erive an analytical upper boun on the response time of RIQ uner any S an X for any that shows the system oes not go into overloa as gets high. The RIQ policy emonstrates that it is possible to esign provably robust an analytically unerstoo reunancy policies within the S&X moel. But there is still room for improvement. We consier several moifications to the baseline policies, motivate by strategies use in practice. One iea that further reuces response time is to use Join-the-Shortest-Queue ispatching for jobs that o not fin ile servers. Another iea is to replicate only small jobs; unfortunately this hurts RIQ, which is alreay quite conservative. On the other han, replicating only small jobs can prevent instability uner Reunancy-, while still allowing Reunancy- to achieve

7 1 IR moel S&X moel 1 Figure : Uner Reunancy-, each arriving job sens copies to servers chosen uniformly at ranom. Here we show mean response time, E [T ], as a function of uner Reunancy- when the system has k = 1 servers, the total arrival rate is λk where λ =.7, an inherent job sizes, X, follow a two-phase hyperexponential istribution with balance means, E [X] = 1, an.7 CX = 1. In the IR moel (soli green line, from analysis in [9]), each job raws an i.i.. instance of X on each server. As increases, mean response time ecreases. In the S&X moel (ashe pink line, simulate; 95% confience intervals are within the line unless shown), a job raws a single instance of X which is the same on all servers, an an i.i.. instance of S on each server. Here S is an empirically measure istribution escribe in Section 3. While in the S&X moel mean response time initially ecreases as a function of, as becomes high the system eventually becomes unstable. goo performance at low. As a compromise between Reunancy- an RIQ, we propose the THRESHOLD-n policy, uner which each arriving job polls servers an replicates itself to those servers with fewer than n jobs in the queue. Like uner RIQ, the system is stable uner THRESHOLD-n for all. Like uner Reunancy-, mean response time uner THRESHOLD-n can be quite low at the best choice of. The remainer of this paper is organize as follows. In Section we review prior theoretical an empirical work on reunancy. Section 3 introuces the S&X moel. In Section we iscuss Reunancy- an the ifficulties it poses, an in Section 5 we introuce an analyze Reunant-to-Ile-Queue (RIQ). Sections an 7 evaluate the performance of RIQ at low an high respectively. In Section we iscuss improvements upon Reunancy- an RIQ. Section 9 stuies an alternative setting where a job s extra copies are cancelle upon the start, rather than upon completion, of the first copy. Finally, in Section 1 we conclue. Prior Work: The Gap Between Theory an Systems Recently there has been growing interest in the theoretical community in analyzing systems with reunancy, with the goal of unerstaning how the number of copies per job affects response time. All of this theoretical work makes crucial simplifying assumptions for analytical tractability, most commonly assuming the IR moel. In a few cases, other simplifications are aopte instea, 3

8 incluing that the system has no queueing (i.e., it is an M/G/ ) or that all jobs replicate to all servers. As we will see below, these assumptions lea to results that are qualitatively ifferent from those prouce by empirical systems stuies. In the Reunancy- system, every arriving job is replicate to servers chosen at ranom, where is a small constant compare to the number of servers [9]. The full istribution of response time as a function of has been analyze in close form in the IR moel [9]. Exact response time istributions also have been erive in the IR moel in a system in which a job s class etermines where it is replicate [1]. This result has been extene to networks of queues, still in the IR moel []. In [1], jobs are compose of multiple tasks, where tasks may be replicate; again the analysis assumes the IR moel. In all these works, when runtimes are exponential or more variable, mean response time ecreases as the number of copies per job increases: it is optimal to replicate jobs at all servers. This is also in accorance with [1, ]. In a reunancy moel calle the (n, k) system, each job sens copies of itself to all n servers an waits for k n copies to complete service. Bouns an approximations for mean response time in the (n, k) system are erive in [19, 1, 13, 3] in the IR moel. While [1] oes consier a moel in which a job s runtime consists of a eterministic component that is the same on all servers an an exponential component that is inepenent across servers, this moel is only analyze in a system where there is no queueing (i.e., an M/G/ ). In a variation calle the (n, k, r) system, in which each job sens copies to r n of the servers an waits for k r of these copies to complete, increasing the value of r ecreases mean response time whenever runtimes are at least as variable as an exponential istribution [19, 13]. The story tol by empirical systems work is more cautious. While theoretical results suggest that response time ecreases as the number of copies increases, practical stuies have shown that creating too many copies can lea to unacceptably high response times an even instability []. This gap emerges because the strong assumptions require for the above theoretical analysis o not hol in practice. Specifically, a large job remains large when replicate; hence, too much reunancy can lea to overloa. Systems researchers have observe that in realistic settings, more sophisticate ispatching an scheuling policies are neee to leverage the potential benefits of reunancy. One iea is to replicate only small jobs to limit the amount of loa ae to the system; this can lea to a % reuction in mean response time [3]. In MapReuce systems, many algorithms begin running replicate copies of jobs only after waiting for some elay to ientify which jobs are experiencing significant slowown [5, ]. Recent work buils on this iea by combining elaye execution of replicas with scheuling policies that reserve a set of servers on which to run replicas (assuming jobs are non-preemptible) [1, ]. Our goal in this paper is to bring the theoretical moels of reunancy systems closer to the real systems that theoretical work eneavors to analyze. To this en, we propose a new moel calle the S&X moel that removes the problematic assumptions in the IR moel. We revisit policies use in practice in Section, where we iscuss these policies in the context of the S&X moel.

9 Table 1: The Dolly(1,1) empirical slowown istribution [3]. The server slowown ranges from 1 to 1, with mean.7. S Prob The S&X Moel We consier a setting with k homogeneous servers (see Figure 1). Each server has its own queue an works in first-come first-serve orer. Jobs arrive as a Poisson process with rate kλ, where λ is a constant, an are ispatche immeiately upon arrival. We assume that jobs are non-preemptible. We introuce the S&X moel, which captures the effects of both job-epenent an serverepenent factors on a job s runtime. Here S is a ranom variable enoting the slowown that a job experiences at a server an X is the job s inherent size. A job that runs on one server has runtime R(1) = X S (here S is assume to be at least 1; S represents the factor by which the inherent size, X, is stretche ) 1. Unlike in the IR moel, in the S&X moel runtimes are not inepenent across servers because a job s X component is the same on all servers. When a server begins working on a job it raws a new inepenent instance of S. Thus consecutive jobs may see ifferent slowowns on the same server. This reflects the fact that slowowns change over time an are not fixe for a particular server. If a job is ispatche to a single server j, it completes service after time equal to its queueing time, T Q j, plus its running time, X S j, where S j is the instance of S that the job experiences on server j (S j is rawn inepenently across servers for a given job, an across jobs for a given server). If a job is ispatche to multiple servers, its response time is T = min j {T Q j + X S j }. The S&X moel takes a big eparture from traitional queueing theory where there is a single ranom variable for service time, making it impossible to ifferentiate between the inherent job size an the server slowown components. When a job that is running on multiple servers completes service on one of these servers, all of its remaining copies must be cancelle. We moel the time it takes to cancel a copy as taking time Z, where Z is a ranom variable. We use ρ to enote the system loa. Unlike in traitional queueing systems, in systems with reunancy loa an stability epen not only on the arrival rate an mean runtime, but also on the particular ispatching an scheuling policies. Hence we efer a more etaile iscussion of loa an stability to Sections an 5, which introuce specific ispatching policies: Reunancy- an Reunant-to-Ile-Queue. In much of this paper, we focus on the Dolly(1,1) istribution (see Table 1), which was measure empirically by analyzing traces collecte from Facebook s Haoop cluster an Microsoft 1 Alternatively, slowown can be aitive, where R(1) = X + S. The analysis an bouns in Sections 5 an 7 easily exten to this setting. 5

10 Bing s Drya cluster an has mean.7 [3]. The Reunancy- Policy A natural ispatching policy for systems with reunancy is Reunancy- [9]. Definition 1. Uner Reunancy-, each arriving job creates copies of itself an sens these copies to servers chosen uniformly at ranom without replacement. Figure compares mean response time, E [T ], uner Reunancy- in the IR moel (from analysis, see [9]) to that in the S&X moel (simulate; in the S&X moel, we set S Dolly(1, 1)). In the IR moel, as increases E [T ] ecreases provie runtimes are at least as variable as an exponential istribution. This is very ifferent from what happens uner the S&X moel: while at first E [T ] ecreases as a function of, as becomes higher E [T ] starts to increase an ultimately the system becomes unstable. While Figure shows that the system can become unstable uner Reunancy- in the S&X moel if is too high, it is ifficult to prove this analytically. Typically a system is stable as long as the system loa, ρ, is less than 1. We can think of ρ as being the arrival rate, λ, multiplie by the expecte server capacity use per job. But in the S&X moel, eriving the expecte server capacity use per job is not straightforwar because it requires knowing the average number of servers on which a job runs; this is not because some copies may be cancelle while still in the queue. Furthermore, copies can enter service at ifferent times on ifferent servers, so knowing the number of servers on which a job runs is not enough to etermine the uration of time for which the job occupies each server. Figure highlights the importance of making the right moeling assumptions. Unlike in the IR moel, in the S&X moel Reunancy- is not robust to the choice of ; choosing the wrong value of can lea to unacceptably poor performance or even instability. Unfortunately, the analysis of Reunancy- in the S&X moel remains an open problem, meaning that it is very ifficult to know how to choose a goo. The lack of robustness, ifficulty in tuning, an potentially poor performance of Reunancy- motivate the nee for a better ispatching policy for the S&X moel. In esigning such a policy, it is important to avoi the factors that cause poor performance for Reunancy-. In particular, Reunancy- creates copies of jobs even when the system has no extra capacity with which to run these copies. Consequently, Reunancy- can a too much loa to the system, causing queue lengths to buil up. An important consieration when esigning ispatching policies for the S&X moel therefore is to ensure that we o not a excessive loa. The full istribution oes not appear in [3]; we obtaine this from personal communication with the authors.

11 5 Reunant-to-Ile-Queue Here we introuce the Reunant-to-Ile-Queue (RIQ) ispatching policy (Section 5.1). 3 We erive an approximation for the Laplace transform of response time uner RIQ (Section 5.), an evaluate the accuracy of our approximation by comparing our analytical results to simulation (Section 5.3). 5.1 The Reunant-to-Ile-Queue Dispatching Policy Definition. Uner Reunant-to-Ile-Queue (RIQ), each arriving job queries servers chosen uniformly at ranom without replacement. If all querie servers are busy, the job joins the queue at one of the servers chosen at ranom. If < i querie servers are ile, the job enters service at all i ile servers, an its runtime is R(i) = X min{s 1,..., S i }. Uner RIQ, the system loa is ρ = λ E [I R(I)], where I is a ranom variable enoting the number of servers on which a job runs. Here ρ is the average arrival rate multiplie by the average service capacity use per job. Forming an expression for loa uner RIQ is easier than for Reunancy- because uner RIQ any job that runs on multiple servers occupies all of its servers for the same uration. Nonetheless, it is not immeiately obvious how to compute E [I R(I)]; we efer our erivation of ρ to the en of Section 5.. The analysis of RIQ relies on the Asymptotically Inepenent Ileness assumption, efine as follows: Assumption 1. [Asymptotically Inepenent Ileness] The servers are ile -wise inepenently; that is, Pr{server s ile servers s 1,..., s 1 ile} = Pr{server s ile} for all sets of istinct servers s 1,..., s. In Section 5.3 we show that our analysis matches simulation when k, inicating that Assumption 1 is reasonable. 5. Analysis: Laplace Transform of Response Time Our erivation of the transform of response time, T (s), uner RIQ begins by conitioning on whether an arrival to the system fins any ile servers: { } ( ) { } ( ) job fins T (s) = Pr ile servers T s job fins job fins no + Pr ile servers ile servers T s job fins no ile servers = (1 ρ ) T (s job fins ile servers) + ρ T (s job fins no ile servers), (1) 3 RIQ is motivate by the highly practical Join-Ile-Queue (JIQ) policy, uner which each arrival is ispatche to a single ile queue, if one exists [1]. 7

12 where the secon line is ue to Assumption 1. We fin T (s job fins ile servers) by further conitioning on the number of ile servers a job fins: ( T s ) job fins = ile servers = i=1 i=1 { job fins i Pr ile servers (1 ρ) i ρ i( i job fins ile servers ) } R(i)(s) 1 ρ R(i)(s). () To fin T (s job fins no ile servers), we observe that a job that fins no ile servers joins the queue at a single server, which has the following properties: 1. When the server is ile, arrivals form a Poisson process with rate λ. This is because the total arrival rate is λk, each arrival polls the server with probability, an every job that polls the k server while it is ile runs on it.. When the server is busy, arrivals form a Poisson process with rate λ = λρ 1. This is because the total arrival rate is λk, each arrival polls the server with probability, an runs on it if the k job s 1 other servers are also busy (probability ρ 1 ) an the job chooses our particular server (probability 1). 3. All jobs that fin the server ile have runtime R, where: { R(1) w.p. ρ 1 R = R(i) + Z w.p. (1 ρ) i 1 ρ 1( 1 i 1), 1 < i. Here we use the probability that the arrival foun i 1 other ile servers among its 1 other servers. Note that in practice, if a job runs on i servers then only i 1 servers nee to incur a cancellation cost Z. We simplify the analysis by assuming that all i servers incur the cost.. All jobs that fin the server busy have runtime R(1). We call the system escribe above an M*/G/1/efs. Here M* enotes that the arrival rate epens on whether the system is ile, an efs inicates that the system has an exceptional first service (i.e., the first job serve uring a busy perio has a ifferent service time istribution than all other jobs). We now have: ( ) T s job fins no = ile servers R(1)(s) T Q (s queueing) M /G/1/efs, (3) where T Q (s queueing), the Laplace transform of time in queue given that a job waits in the queue, is T Q (s queueing) M /G/1/efs = T (s) M /G/1/efs (1 P M /G/1/efs Q P M /G/1/efs Q R(1)(s) ) R (s), ()

13 Simulation Analysis 1 k Figure 3: E [T ] uner RIQ from analysis (ashe blue line) an simulation (soli re line, 99% confience intervals are within the line) when λ =.7, =, X H with E [X] = 1 an.7 = 1, S Dolly(1, 1), an Z =. When k = 1, the analysis is within 1% of simulation. C X an P M /G/1/efs Q = E [N B] E [N B ] + 1 is the probability of queueing in an M*/G/1/efs an E [N B ] = λ E [R ] 1 λ E [R(1)] (5) () is the expecte number of arrivals uring an M*/G/1/efs busy perio, an T (s) M /G/1/efs is given in Lemma 1. Lemma 1. The Laplace transform of response time in an M*/G/1/efs where the first job experiences service time R, all remaining jobs experience service time R(1), the arrival rate while the server is ile is λ, an the arrival rate while the server is busy is λ is the same as that in an M/G/1/efs where the first job experiences service time R, all remaining jobs experience service time R(1), an the arrival rate is λ. Proof. The M/G/1/efs response time analysis relies on PASTA, so it oes not irectly apply to the M*/G/1/efs, which has a state-epenent arrival rate. However, the arrival rate while the system is ile only affects how close together busy perios occur. It oes not affect the response time of any of the jobs uring the busy perio. Thus to erive response time, we can view the system as an M/G/1/efs. The response time in an M/G/1/efs is well unerstoo (see []). For our system, we have T (s) M /G/1/efs = 1 λ E [R(1)] 1 λ E [R(1)] + λ E [R ] (λ s) R (s) λ R(1)(s) λ s λ R(1)(s). (7) Combining equations (1)-(7) gives a close-form expression for mean response time uner RIQ in terms of ρ. 9

14 1 RIQ simulation RIQ analysis Reunancy- simulation 1 RIQ simulation RIQ analysis Reunancy- simulation (a) CX = 1, λ =.3 (b) C X = 1, λ =.7 RIQ simulation RIQ analysis Reunancy- simulation RIQ simulation RIQ analysis Reunancy- simulation (c) CX = 1, λ =.3 () C X = 1, λ =.7 Figure : E [T ] vs. uner Reunancy- (from simulation) an RIQ (from both simulation an analysis) for X H with E [X] = 1 an.7 C X = 1 (top row) an C X = 1 (bottom row), where E [R(1)] = E [X] E [S] = 1, S Dolly(1,1), an the cancellation time is Z =, uner low (λ =.3, left) an high (λ =.7, right) arrival rate. The simulations have k = 1 servers; 95% confience intervals are within the line where not shown. At low λ, RIQ an Reunancy- perform similarly, but at high λ Reunancy- becomes unstable when is large, whereas RIQ continues to achieve low E [T ]. Finally, we erive ρ, the probability that a server is busy. In Section 3 we efine ρ = λ E [I R(I)], where I is the number of servers on which a job runs. Alternatively, we can compute ρ using renewal-rewar theory, efining a cycle as the time from when a server becomes ile until it is next ile: ρ = E [B] E [B] + 1, () λ where E [B] = E[R ] 1 λ E[R(1)] 1 is the expecte busy perio uration an is the mean interarrival time λ to an ile server. Note that E [R ] is efine in terms of ρ, hence () efines ρ in terms of itself. When =, the equation is of egree an can be solve exactly; for higher we solve for ρ numerically. The system is stable as long as ρ < 1. However, without a close-form expression for ρ, it is ifficult to unerstan this stability conition intuitively. In Section 7 we erive an upper boun 1

15 15 1 Z = Z =.1 Z =.5 Z = Z = Z =.1 Z =.5 Z = (a) RIQ 1 (b) Reunancy- Figure 5: Mean response time as a function of uner (a) RIQ (from analysis) an (b) Reunancy (simulate with k = 1 servers, 95% confience intervals are within the line except where shown), λ =.7, job sizes are X H with E [X] = 1 an.7 C X = 1, an server slowowns are S Dolly(1, 1) for ifferent eterministic cancellation times Z. As Z increases, mean response time increases uner both RIQ an Reunancy-. Uner Reunancy-, this leas to the system becoming unstable at lower values of, whereas RIQ remains stable for all values of Z. on mean response time uner RIQ which gives us an alternative conition: the system is stable if λ E [R(1)] < Quality of Approximation Our approximation matches simulation quite well provie is sufficiently small relative to k. Figure 3 compares mean response time from our analysis to that obtaine via simulation when =, λ =.7, X follows a two-phase hyperexponential istribution with balance means an E [X] = 1 an.7 C X = 1, an S follows the Dolly(1,1) server slowown istribution. Our analysis is more accurate at high k; when k = 1 the analysis is within 1% of simulation. When an λ are lower the analysis converges to simulation more quickly. Results: k In this section we evaluate the performance of RIQ using our analysis from Section 5. Throughout the section, E [R(1)] = 1, k = 1 servers, S follows the Dolly(1,1) istribution (Table 1, E [S] =.7), an X follows a two-phase hyperexponential istribution with balance means: { Exp (µ 1 ) w.p. p X, Exp (µ ) w.p. 1 p where p = 1 p. µ 1 µ 11

16 We consier only the k regime, in which our analysis provies a close approximation for performance uner RIQ. In Section 7 we stuy what happens when is close to k. In Figure we compare RIQ (our analysis in Section 5 matches simulation) to Reunancy- (simulate) at both low an high λ when the inherent job size X has ifferent square coefficients of variation, CX, an the cancellation time is Z =. When is low, Reunancy- outperforms RIQ because Reunancy- allows all jobs to benefit from creating multiple copies, not just jobs fining multiple ile servers, an this outweighs any pain cause by the extra loa ae by these copies. However, as gets higher, Reunancy- becomes unstable, whereas mean response time uner RIQ continues to ecrease. Uner both RIQ an Reunancy-, mean response time increases as the cancellation time Z increases (see Figure 5). Uner Reunancy-, as Z increases the system becomes unstable at lower values of. Uner RIQ, the system remains stable, inicating that RIQ is more robust than Reunancy-. Results are more ramatic at the tail of response time, which in computer systems is often a more important metric than the mean. We numerically invert the transform erive analytically in Section 5 to obtain the c..f. of response time using the proceure escribe in [1]. As increases, the 95th percentile of response time, T 95, rops much more steeply than E [T ] (see Figure ), inicating that RIQ successfully overcomes the negative effects of the variable server slowowns. In fact, all percentiles of response time are much lower at = 5 than at = 1 (see Figure 7). The trens are similar when the server slowown follows istributions other than the Dolly(1, 1) istribution. In Figure we show mean response time as a function of uner Reunancy- an RIQ when S follows a Bimoal istribution, as was measure in a Google cluster [7]: { 1 w.p..99 S 1 w.p..1. With Bimoal slowown, Reunancy- is unstable at lower values of than with the more uniform Dolly(1, 1) istribution, but E [T ] uner both RIQ an Reunancy- follows the same qualitative shape regarless of the istribution of S. 7 Results: high One might think that mean response time shoul be lowest when = k, since when k we saw that E [T ] ecreases as increases. However, this intuitively oes not make sense. As increases, a job achieves ecreasing marginal runtime an queueing time benefit from querying one more server. Eventually, increasing only increases loa without proviing any benefit. As gets high, one might think that the servers are always busy, riving queue lengths to infinity. Inee, we see in Figure 9 that ρ 1 as k. But surprisingly, the system is never unstable, even when = k. This is because RIQ limits the amount of ae loa. Effectively, when a server goes ile we can think of it as being given some extra work, where this extra work is some job s replicate copy. This causes the server to be always busy (sening ρ to 1) but it affects the queue length like a vacation time, which cannot cause instability. We formalize this intuition in Theorem 1, which gives an upper boun on E [T ] uner RIQ, for all an for all k. 1

17 1 T 95 5 T 95 T T (a) λ =.3 (b) λ =.7 Figure : Mean (soli blue line) an 95th percentile (ashe pink line) of response time, T, (via analysis) uner RIQ when X H with E [X] = 1 an.7 C X = 1, S Dolly(1, 1), Z =, an (a) λ =.3 an (b) λ =.7. As increases, both the mean an the 95th percentile of response time ecrease; the improvement is more pronounce in the tail. Pr{T < t} = 1 = t Figure 7: Distribution of response time uner RIQ when = 1 (soli black line) an = 5 (ashe purple line) when S Dolly(1, 1), X H with E [X] = 1 an.7 C X = 1, an λ =.7. Theorem 1. The response time uner RIQ is upper boune by the response time in an M/G/1/vacation queue (see []) with arrival rate λ < 1/E [R(1)], where G = R(1) = S X, an where the vacation time is R(1) + Z: E [T ] RIQ E [T ] M/G/1/vacation where (R(1) + Z) e enotes the excess of R(1) + Z. = E [T ] M/G/1 + E [(R(1) + Z) e ]. Proof. The only part of the analysis of mean response time uner RIQ in Section 5. that is not exact is Assumption 1. We remove our epenence on Assumption 1 by upper bouning each component of the analysis that uses Assumption 1. 13

18 1 RIQ analysis Reunancy- simulation 1 RIQ analysis Reunancy- simulation (a) λ =.3 (b) λ =.7 Figure : Both RIQ an Reunancy- exhibit similar trens to their performance when S Dolly(1, 1) uner alternative S istributions. Here we show results for S Bimoal(1, 1;.99) (E [S] = 1.13) when X H with E [X] = 1 an 1.13 C X = 1, an Z =, for (a) λ =.3 an (b) λ =.7. RIQ results are from analysis; Reunancy- is simulate (95% confience intervals are within the line). 1 ρ =.3 E[I] = 1 ρ =.±.1 E[I] = 5.19±.1 ρ =.95±.5 E[I] = 13.±. ρ =.9±. E[I] = 1.3±.11 ρ =.9±. E[I] = 13.11± ρ =.7 E[I] = 1 ρ =.17±.1 E[I] = 1.9±.1 ρ =.991±. E[I] = 3.9±. ρ =.997±. E[I] = 3.33±.3 ρ =.99±.3 E[I] = 3.59±.1 1 (a) λ =.3 (b) λ =.7 Figure 9: Mean response time uner RIQ (simulate; 95% confience intervals are within the line) as a function of when k = 1, X H with E [X] = 1 an.7 C X = 1, S Dolly(1, 1), Z =, an (a) λ =.3, (b) λ =.7. While mean response time increases as gets close to k, even when = k mean response time is lower than when = 1. The lines are annotate with the values of ρ an E [I] (the expecte number of copies per job) at several ifferent values of. 99% confience intervals are within the line for E [T ] an are explicitly shown for ρ an E [I]. 1

19 We first conition on whether an arrival fins ile servers: { } { } { } job fins E [T ] = P E T job fins job fins no ile servers + P ile servers ile servers { E T } job fins no. ile servers In the analysis in Section 5., we use Assumption 1 to approximate P {job fins ile servers}. We can upper boun E [T ] by instea assuming that no job ever fins an ile server: E [T ] E [T job fins no ile servers] = E [R(1)] + E [T queueing] M /G/1/efs, (9) where the equality uses the same reasoning as in Section 5.. In Section 5., we further use Assumption 1 in two places when analyzing the M*/G/1/efs. First, we sai that when the server is busy arrivals occur with rate λρ 1. We upper boun this by saying that arrivals occur with rate λ. Secon, we sai that the first job serve uring a busy perio has runtime R + Z, where R is efine in (3). We upper boun this by saying that the first job has runtime R(1) + Z. Since (9) forces all jobs to have a non-zero queueing time, the first job in the busy perio can be viewe as a ummy job that oes not contribute to response time. Every time a server goes ile, we can force it to work on a ummy job, so the server is always busy when the next real job arrives to the server. This is an M/G/1/vacation system with vacation time R(1) + Z. Theorem. Uner RIQ, the system is stable for all if λe [R(1)] < 1. Proof. The stability region for the M/G/1/vacation is the same as that for the M/G/1, namely λ E [R(1)] < 1. If the M/G/1/vacation is stable, then RIQ is stable since the M/G/1/vacation gives an upper boun on RIQ. Improving Reunancy Policies In this section we stuy several policy variations esigne to improve performance relative to the baseline RIQ an Reunancy- policies. In some cases, the analysis presente in Section 5 extens easily to the variations on RIQ. In other cases, we evelop new response time analysis. When analysis is not feasible, we stuy performance via simulation..1 JSQ: Better Dispatching for Jobs that Fin No Ile Servers One weakness of RIQ is that when a job fins no ile servers it is ispatche to a single queue chosen at ranom, which is known to yiel poor performance. A superior ispatching policy is Join-the-Shortest-Queue (JSQ), uner which each arriving job joins the queue containing the fewest jobs. A practical variant of JSQ is JSQ-, where each arriving job polls queues chosen at ranom an joins the queue containing the fewest jobs among the polle queues. JSQ- is motivate by the fact that polling all k queues can be expensive. The RIQ+JSQ policy combines RIQ with JSQ- ispatching. 15

20 Definition 3. Uner RIQ+JSQ, each arriving job polls servers chosen uniformly at ranom without replacement. If 1 i of the servers are ile, the job enters service at all i ile servers. If all servers are busy, the job joins the queue with the fewest jobs among the polle queues. Observation 1. RIQ+JSQ has the same stability region as RIQ, since the only ifference between the two policies is that uner RIQ+JSQ we achieve better loa balancing of the jobs that o not fin ile servers. Hence uner RIQ+JSQ, the system is stable as long as λ E [X S] < 1. Response Time Analysis Our analysis of mean response time uner RIQ+JSQ follows the approach use in []. We write a system of equations that escribes the evolution of the RIQ+JSQ system, then use numerical methos to approximate the steay-state behavior. It is easy to moify the analysis presente below to inclue a cancellation time. Let Y l,i (t, t + r) be the fraction of servers with at least l jobs in the queue (incluing the job in service, if there is one) at time t such that the job in service at time t is still in service at time t + r an is running on exactly i servers. Let Y l (t, t + r) = i=1 Y l,i(t, t + r). For l, i > 1, there are two ways in which a server can contribute to Y l,i (t, t + r): 1. The queue length was at least l at time an the same job remains in service on i servers uring time interval (, t + r).. At some time u < t an arrival cause the queue length to go from l 1 to l an the same job is in service on i servers uring the time interval (u, t + r). When i = 1 there is a thir case: a job complete service at time u t an left behin at least l jobs, an the next job stays in service uring time (u, t + r). Hence for l > 1, Y l,i (t, t + r) = Y l,i (, t + r) + λ t u= + I i=1 t (Y l 1 (u, u) Y l (u, u) ) Yl 1,i(u, t + r) Y l,i (u, t + r) u Y l 1 (u, u) Y l (u, u) u= where Ḡ(x) is the probability that a job s runtime excees x an ( r Y l+1 (u, u))ḡ(t + r u)u, (1) r Y l+1 (u, u) = lim r Y l+1 (u, u) Y l+1 (u, u + r) r is the service completion rate of jobs in a queue with length at least l + 1 at time u. When l = 1 the secon term, which correspons to a new arrival, changes because now the arrival joins an ile queue an therefore runs on 1 i servers: t Y 1,i (t, t + r) = Y 1,i (, t + r) + iλ u= t + I i=1 u= ( i ) Y 1 (u, u) i (1 Y 1 (u, u)) i Ḡ (i) (t + r u)u ( r Y (u, u))ḡ(t + r u)u, (11) 1

21 where Ḡ(i) (x) is the probability that a job s runtime excees x given that the job is running on exactly i servers. Summing over i in (1), we fin t Y l (t, t + r) = Y l (, t + r) + λ + u= t u= (Y l 1 (u, u) Y l (u, u) ) Yl 1(u, t + r) Y l (u, t + r) u Y l 1 (u, u) Y l (u, u) ( r Y l+1 (u, u))ḡ(t + r u)u, (1) which correspons to equation () in [] when =. Summing over i in (11) yiels t Y 1 (t, t + r) = Y 1 (, t + r) + λ u= + I i=1 t i=1 u= ( ) Y 1 (u, u) i (1 Y 1 (u, u)) i i iḡ(i) (t + r u)u ( r Y (u, u))ḡ(t + r u)u. (13) We now use equations (1) an (13) to numerically estimate π l, the steay-state probability that a server has at least l jobs (assuming the number of servers k is large). Note that π 1 = ρ is the probability that a server is busy. We introuce a mesh of with δ an approximate Y l (t, r) for t [, δ, δ,..., t ] an r [, δ, δ,..., r ]. Here r is chosen such that Ḡ(r ) is negligible, an t is chosen ynamically so that l,r Y l(t + δ, t + δ + r) Y l (t, t + r) is sufficiently small, e.g., 1 1. We also truncate the queue lengths to at most n jobs. 1 RIQ RIQ+JSQ approximation RIQ+JSQ simulation 1 RIQ RIQ+JSQ approximation RIQ+JSQ simulation 1 1 (a) λ =.3 (b) λ =.7 Figure 1: Comparing baseline RIQ an RIQ+JSQ from our approximation (ashe re line) an simulation (otte black line; 95% confience intervals are within the line). Here S Dolly(1, 1), X H with E [X] = 1 an.7 C X = 1, Z =, an (a) λ =.3 an (b) λ =.7. We start with an empty system at time, so Y l (, r) = for all l an r, an compute Y l (t + 17

22 δ, t + r + δ) base on Y l (t, t + r) as follows: Y 1 (t + δ, t + r + δ) = Y 1 (t, t + r + δ) t+δ + λ + u=t t+δ u=t i=1 ( ) Y 1 (u, u) i i (1 Y 1 (u, u)) iḡ(t + r + δ u)u i ( r Y (u, u))ḡ(t + r + δ u)u Y 1 (t, t + r + δ) (Y (t, t + δ) Y (t, t))ḡ(r + δ ) ( ) + λδ Y 1 (t, t) i (1 Y 1 (t, t)) i i iḡ(i) (r + δ ) i=1 Y l (t + δ, t + r + δ) Y l (t, t + r + δ) (Y l+1 (t, t + δ) Y l+1 (t, t))ḡ(r + δ ) + λδ Y l 1(t, t) Y l (t, t) Y l 1 (t, t) Y l (t, t) (Y l 1(t, t + r) Y l (t, t + r)). The numerical scheme now uses these two approximations in the following manner. First, we efine Ȳ l (t + δ, t + r + δ) = Ŷl(t, t + r + δ) (Ŷl+1(t, t + δ) Ŷl+1(t, t))ḡ(r + δ ) an subsequently we compute Ŷl(t + δ, t + r + δ) as + λδ Ŷl 1(t, t) Ŷl(t, t) Ŷ l 1 (t, t) Ŷl(t, t) (Ŷl 1(t, t + r) Ŷl(t, t + r)), Ŷ l (t + δ, t + r + δ) = Ŷl(t, t + r + δ) (Ȳl+1(t + δ, t + δ) Ȳl+1(t + δ, t + δ))ḡ(r + δ ) + λδ Ȳl 1(t + δ, t + δ) Ȳl(t + δ, t + δ) Ȳ l 1 (t + δ, t + δ) Ȳl(t + δ, t + δ) (Ȳl 1(t + δ, t + r + δ) Ȳl(t + δ, t + r + δ)). A similar approach is use to compute Ŷ1(t + δ, t + δ + r). We now use Ŷl(t, t) to approximate π l. The expecte number of jobs in the system, E [N], is E [N] = π ell λ l=1 i= ( ) π1 i (1 π 1 ) i (i 1)E [ G (i)], i We thank Xingjie Li for helpful suggestions to stabilize the numerical scheme. 1

23 where ( ) i π i 1 (1 π 1 ) i is the probability that a job runs on i servers. Note that the secon term is inclue to avoi overcounting jobs that run on i > 1 servers. Finally, we fin E [T ] by applying Little s Law. Valiation of Approximation We compare our compute π l values with simulation; when k = 1, = 1, CX = 1, λ =.7, an S Dolly(1, 1), an for δ =. an n = 1, the approximation is within 5% of simulation for π 1, π, an π 3 (queue lengths excee 3 less than.5% of the time). The approximation is more accurate when λ,, CX, an δ are lower an when k is higher. Performance We use our new analysis to compare RIQ+JSQ with baseline RIQ, assuming is small (for both policies, this is the regime in which our analysis hols). When λ is low, RIQ+JSQ provies very little benefit over RIQ because most jobs fin ile servers (see Figure 1). When λ is high, RIQ+JSQ beats RIQ because many jobs o not fin ile servers an benefit from being ispatche via JSQ rather than ranomly. Note that it oes not make sense to combine JSQ with Reunancy-; this woul require either polling > servers per job, or limiting the number of replicas to be <.. SMALL: Replicate Only Jobs with Small X A major concern when replicating jobs is that running multiple copies of the same job as loa to the system. This is particularly true of jobs with a large inherent size X. When a large job runs on many servers, it clogs up these servers for a long time even if it experiences a small slowown S on some server. This makes it harer for jobs with small X components to fin any ile servers on which to run. One way to prevent the small jobs from being blocke on many servers by large jobs is to allow only jobs with a small X component to create replicas. That is, we efine a constant threshol x an only replicate jobs with inherent size X x. Definition. Uner RIQ+SMALL, if an arriving job has inherent size X x it polls servers chosen uniformly at ranom without replacement. If 1 i of the polle servers are ile, the job replicates itself on all i ile servers. If none are ile, the job joins the queue at one of the servers chosen uniformly at ranom. If the job has inherent size X > x, it is ispatche to a single server chosen uniformly at ranom. Definition 5. Uner Reunancy-+SMALL, if a job has inherent size X x it replicates itself to servers chosen uniformly at ranom, whereas if a job has inherent size X > x it is ispatche to a single server chosen uniformly at ranom. Theorem 3. Uner RIQ+SMALL, the system is stable if λ E [X S] < 1. Proof. The same M/G/1/vacation system that we use in Theorem 1 to upper boun E [T ] uner RIQ applies for RIQ+SMALL. Hence the stability conition uner RIQ+SMALL is the same as that uner RIQ. 19

24 Response Time Analysis: RIQ+SMALL The analysis of response time uner RIQ+SMALL follows the same approach as that uner the baseline RIQ policy. We begin by conitioning on a job s inherent size: T (s) = T (s X x) P {X x} + T (s X > x) P {X > x}. To fin T (s X > x), observe that any job with inherent size larger than x is ispatche ranomly to a single M*/G/1/efs. As before, the exceptional first service comes from the fact that the first job in the busy perio may run on anywhere between 1 an servers. However, now the job s response time is not simply the response time in an M*/G/1/efs because jobs with size X > x are less likely to be the first job in a busy perio than jobs with size X < x. So we conition on whether the job fins the server ile: ( ) { } T (s X > x) = T s X > x & job job fins P fins ile servers ile servers ( ) { } + T s X > x & job fins job fins no P no ile servers ile servers = R(1)(s X > x) (1 ρ) + T Q (s queueing) M /G/1/efs R(1)(s X > x) ρ. The arrival an service rates in this M*/G/1/efs iffer from those in the original RIQ analysis, an are iscusse below. To fin T (s X x), we follow the original RIQ analysis: ( ) { } T (s X x) = T s X x & job job fins P fins ile servers ile servers ( ) + T s X x & job fins P no ile servers i=1 { job fins no ile servers where P {job fins ile servers} = 1 ρ an ρ is compute numerically using the same approach as in the original RIQ analysis an the M*/G/1/efs parameterization given below. Similarly to the original analysis, we have: ( ) T s X x & job { } job fins i = P job fins fins ile servers ile servers ile servers R(i)(s X x) i=1 (1 ρ) i ρ i( ) i = 1 ρ R(i)(s X x), where R(i)(s X x) = X min{s 1,..., S i }(s X x). If a job with X < x fins no ile servers, it simply joins the queue at a single server, which behaves like an M*/G/1/efs: ( ) T s X x & job fins = no ile servers T (s X x) M /G/1/efs = R(1)(s X x) T Q (s) M /G/1/efs. In this system, our M*/G/1/efs has the following parameters: },

25 1. When the server is ile, arrivals form a Poisson process with rate P {X > x} λk 1 + P {X x} λk k k. When the server is busy, arrivals form a Poisson process with rate = λ(p {X > x} + P {X x} ). P {X > x} λk 1 + P {X x} λk k k ρ 1 1 = λ(p {X > x} + P {X x} ρ 1 ). 3. Jobs that fin the server ile experience runtime { R(1) X > x w.p. P{X>x} R = R(i) + Z X x w.p. P{X>x}+ P{X x} P{X x} P{X>x}+ P{X x} (1 ρ)i 1 ρ i( 1 i 1), 1 i. Jobs that fin the server busy experience runtime { R(1) X > x w.p. P{X>x} R b = R(1) X x w.p. P{X>x}+P{X x} ρ 1 P{X x} ρ 1 P{X>x}+P{X x} ρ 1 Here the probabilities are reweighte because the probability that a job of size > x enters the server is not equal to the probability that a job of size > x enters the system. Performance Figure 11 compares baseline RIQ an Reunancy- to their SMALL variants with inherent size cutoffs at the 1th, 5th, an 9th percentiles of X. The RIQ+SMALL results are from the above analysis; Reunancy-+SMALL is simulate. Surprisingly, replicating only the small jobs oes not reuce mean response time uner RIQ. In fact, as the size cutoff increases (i.e., more jobs are allowe to replicate) E [T ] ecreases, an is lowest uner baseline RIQ. This is because RIQ is esigne to replicate only when the system has spare capacity. Further limiting the number of jobs that replicate leas to a minor increase in the number of ile servers, thereby helping small jobs that can still replicate. But this benefit is outweighe by the harm experience by the large jobs that no longer get to see the minimum of multiple server slowowns. Similarly, when the system is stable uner baseline Reunancy-, lowering the size cutoff (i.e., replicating fewer jobs) leas to an increase in mean response time. But importantly, the system is stable at higher values of uner Reunancy-+SMALL than uner baseline Reunancy-. While Reunancy-+SMALL eventually becomes unstable (e.g., when λ =.3 an C X = 1, instability occurs aroun = ), at practical (low) values of the system is both stable an relatively insensitive to the particular choice of. 1

26 15 1 Cutoff = X 1 Cutoff = X 5 Cutoff = X 9 Cutoff = (RIQ) 15 1 Cutoff = X 1 Cutoff = X 5 Cutoff = X 9 Cutoff = (RIQ) (a) RIQ, λ =.3 (b) RIQ, λ =.7 Cutoff = X 1 Cutoff = X 5 Cutoff = X 9 Cutoff = (Reunancy-) Cutoff = X 1 Cutoff = X 5 Cutoff = X 9 Cutoff = (Reunancy-) (c) Reunancy-, λ =.3 () Reunancy-, λ =.7 Figure 11: Comparing E [T ] uner baseline RIQ (top, from analysis) an Reunancy- (bottom, simulate; 95% confience intervals are within the line where not shown) to that uner the SMALL variant, which replicates only jobs with inherent size below a certain cutoff X i, where X i represents the ith percentile of X. Here S Dolly(1, 1), X H with E [X] = 1 an.7 C X = 1, Z =, an λ =.3 (left) an λ =.7 (right).

27 1 n = (RIQ) n = 1 n = n = 3 n = (Reunancy- 1 n = (RIQ) n = 1 n = n = 3 n = (Reunancy-) (a) λ =.3 (b) λ =.7 Figure 1: Mean response time uner THRESHOLD-n for ifferent values of n (n = is from our RIQ analysis; other values of n are simulate, 95% confience intervals are within the line). Here X H with E [X] = 1 an.7 C X = 1, S Dolly(1, 1), Z =, an (a) λ =.3 an (b) λ =.7..3 THRESHOLD: Replicate to Short Queues Only A major avantage of RIQ is that the system cannot become unstable as gets large. But RIQ is not perfect: with the right choice of, Reunancy- can achieve lower response time than RIQ. RIQ allows very few jobs to replicate, sacrificing potential response time gains to guarantee stability. On the other han, Reunancy- allows all jobs to replicate, offering the potential for large response time improvements, but risking instability. Ieally, our policy woul lie between RIQ an Reunancy-: we can affor to replicate more than uner RIQ, but not as much as uner Reunancy-. To accomplish this, we introuce the THRESHOLD-n policy, which represents a compromise between RIQ an Reunancy-. Definition. Uner THRESHOLD-n, each arriving job polls servers an joins the queue at those servers with queue length n. If all queue lengths are > n, the job joins a single queue at ranom from among the polle servers. Note that RIQ an Reunancy- are the extrema of the THRESHOLD-n policies, where RIQ THRESHOLD- an Reunancy- THRESHOLD-. Unfortunately, THRESHOLD-n is not analytically tractable. The key feature enabling us to analyze RIQ (an the JSQ an SMALL variants) is that uner these policies, all copies of a job start service simultaneously, so we o not nee to track the ages of copies in service. This is not true for THRESHOLD-n, hence we stuy THRESHOLD-n via simulation. THRESHOLD-n achieves the best features of both RIQ an Reunancy- (see Figure 1). Like Reunancy-, THRESHOLD-n allows for enough reunancy to achieve substantial response time improvements. Like for RIQ, we can erive an upper boun on E [T ] that shows that the system oes not become unstable as a function of (see Theorem ). For example, when λ =.7 (Figure 1(b)), THRESHOLD-1 performs nearly ientically to Reunancy- for 7; here both policies outperform RIQ. At λ =, Reunancy- approaches instability, but uner THRESHOLD-1 response time plateaus to a value only slightly higher than that uner RIQ. The 3