Self-Adaptive Admission Control Policies for Resource-Sharing Systems

Transcription

1 Self-Adaptve Admsson Control Polces for Resource-Sharng Systems Varun Gupta Carnege Mellon Unversty Mor Harchol-Balter Carnege Mellon Unversty ABSTRACT We consder the problem of admsson control n resource sharng systems, such as web servers and transacton processng systems, when the job sze dstrbuton has hgh varablty, wth the am of mnmzng the mean response tme. It s well known that n such resource sharng systems, as the number of tasks concurrently sharng the resource s ncreased, the server throughput ntally ncreases, due to more effcent utlzaton of resources, but starts fallng beyond a certan pont, due to resource contenton and thrashng. Most admsson control mechansms solve ths problem by mposng a fxed upper bound on the number of concurrent transactons allowed nto the system, called the Mult- Programmng-Lmt (MPL), and makng the arrvals whch fnd the server full queue up. Almost always, the MPL s chosen to be the pont that maxmzes server effcency. In ths paper we abstract such resource sharng systems as a Processor Sharng (PS) server wth state-dependent servce rate and a Frst-Come-Frst-Served (FCFS) queue, and we analyze the performance of ths model from a queueng theoretc perspectve. We start by showng that, counter to the common wsdom, the peak effcency pont s not always optmal for mnmzng the mean response tme. Instead, sgnfcant performance gans can be obtaned by runnng the system at less than the peak effcency. We provde a smple expresson for the statc MPL that acheves nearoptmal mean response tme for general dstrbutons. Next we present two traffc-oblvous dynamc admsson control polces that adjust the MPL based on the nstantaneous queue length whle also takng nto account the varablty of the job sze dstrbuton. The structure of our admsson control polces s a mxture of flud control when the number of jobs n the system s hgh, wth a stochastc component when the system s near-empty. We show va smulatons that our dynamc polces are much more robust to unknown traffc ntenstes and burstness n the arrval process than mposng a statc MPL. Research supported by NSF SMA/PDOS Grant CCR-066 and a 009 IBM Faculty Award. Permsson to make dgtal or hard copes of all or part of ths work for personal or classroom use s granted wthout fee provded that copes are not made or dstrbuted for proft or commercal advantage and that copes bear ths notce and the full ctaton on the frst page. To copy otherwse, to republsh, to post on servers or to redstrbute to lsts, requres pror specfc permsson and/or a fee. SIGMETRICS/Performance 09, June 9, 009, Seattle, WA, USA. Copyrght 009 ACM /09/06...$.00. Categores and Subject Descrptors D.4.8 [Operatng Systems]: Performance modelng and predcton, queueng theory; G..6 [Numercal Analyss]: Optmzaton stochastc programmng General Terms Performance. INTRODUCTION The noton of tme-sharng has been around snce the earlest days of operatng systems, as descrbed n the frst paper on Unx [3]. Tme-sharng has several benefts. Frst, gven that jobs often need dfferent resources (CPU, I/O) at dfferent tmes, tme-sharng allows for ncreased throughput, typcally allowng two jobs to complete n the same tme as one, snce they aren t lkely to need the same resources at the same tme. Another major beneft of tme-sharng s that t allows small jobs to get out quckly; the small jobs are not stuck queueng behnd bg jobs as they would be n a frst-come-frst-served (FCFS) system, and therefore they don t have to suffer the delays of watng for bg jobs to complete. However, as many researchers have observed, tme-sharng s most effectve when there s a fxed Mult-Programmng- Lmt (MPL) mposed, so that not too many jobs can tmeshare at once. Allowng too many jobs to tme-share can lead to thrashng (due to the context-swtchng overhead), and reduced overall performance. Ths pont has been observed tme and tme agan startng wth operatng systems papers n the 970 s [8] and 980 s [, ], and contnung to more recent Web server desgn papers [9, 3], and database mplementaton papers [4, ]. Specfcally, a system has a servce rate curve whch shows that the speed of the system ncreases when the number of jobs n the system ncreases from to, and ncreases agan as the number ncreases from to 3, but the system speed starts to drop as the number of jobs n the system ncreases beyond some pont. Fgure shows a typcal servce rate curve (see, e.g. [6, Fgure ]). Model To model a tme-sharng system, we start wth a G/G//PS queue where PS denotes processor sharng, meanng that f there are n jobs n the system, they each receve th of n the system s processng capacty. We wll assume that the job szes (or servce requrements) are ndependently and dentcally drawn from a general dstrbuton, and we wll

2 Server speed µ(n) Number of jobs at the server (n) Fgure : A prototypcal servce rate curve. The peak effcency pont for the curve shown s K =. use X to denote such a generc random varable. We wll use C to denote the squared coeffcent of varaton (SCV) of X: C = var(x) E[X] and throughout assume that E[X] = wthout loss of generalty. In order to capture the fact that the speed of the system depends on the number of jobs at the server, we assume that our G/G//P S server has state-dependent servce rates µ(n). That s, when the number of jobs at the server s n, the speed of the server s µ(n), where µ(n) s chosen to match the system s servce rate curve (Fgure ). As an example, a job of sze x seconds whch s sharng the server wth n jobs (ncludng tself) for ts entre duraton x µ(n) would requre n tme to complete. We assume that the µ(n) curve s unmodal, that s, ntally t s non-decreasng and then after some pont the curve swtches to beng nonncreasng. We defne K to be the smallest MPL whch acheves the maxmum speed, and K to be the largest MPL whch acheves the maxmum speed. For the µ(n) curve n Fgure, K = 4 and K =. To complete our model, we now add an MPL parameter whch lmts the number of jobs that are allowed to concurrently share the server to some number MPL=K, and forces all remanng jobs to wat n a Frst-Come-Frst- Served (FCFS) buffer. We assume that the job szes of the jobs n the system are not known, and sze-based prortzaton s not possble. We denote our model by the notaton G/G/PS-MPL. Fgure depcts a G/G/PS-MPL system wth MPL=4. When we addtonally assume the arrval process to be Posson, we wll denote the system by M/G/PS-MPL. Throughout, we assume load-dependent servce rates µ(n). So, for example, f there are n = 0 jobs n the G/G/PS-4 system, the server speed wll be µ(4), snce only 4 jobs tme-share the server, whle f there are n<4 jobs n the system, the speed wll be µ(n). Thus the response tme for a job of sze x wll be ts queueng tme plus ts x servce tme, where the servce tme wll typcally be 4, µ(4) assumng that there are at least 4 jobs n the system durng the job s tme n system. The nterestng queston for the G/G/PS-MPL model s, of course: What s the optmal MPL, so as to mnmze mean response tme? Obvously, the servce rate curve plays a large role n the answer. In fact, computer systems papers would have us FCFS MPL = 4 Fgure : A G/G/PS-MPL queue wth MP L = 4. Only 4 jobs can smultaneously share the server. The rest must wat outsde n FCFS order. beleve that the servce rate curve provdes the entre answer to ths queston: Smply choose that MPL that maxmzes effcency, e.g., [,, 9]. For the curve shown n Fgure, ths would mean choosng the MPL to be K = 4 or K =. In ths paper, we wll show that ths obvous answer can be far from correct, when job sze varablty s hgh. We wll also ask and answer the even harder queston of how to dynamcally vary the MPL when the arrval rate s not known and as load condtons change. When the job sze dstrbuton s Exponental, the answer s straghtforward: We always want to operate at the peak effcency pont, regardless of the arrval process. The queston of choosng an optmal MPL becomes nterestng when the job sze dstrbuton exhbts hgh varablty, snce wth hgh varablty job sze dstrbutons, t s known that PS has a much better performance than FCFS because tme sharng prevents small jobs from gettng blocked behnd bg jobs. However, by lettng too many jobs nto the server, the system effcency drops. Pror Work Unfortunately, the queston of choosng an optmal MPL for hgh varablty job sze dstrbutons s dffcult to answer snce there s no known analyss even under a Posson arrval process wth a fxed arrval rate. Ths s not surprsng because the M/G/PS-MPL model s a generalzaton of the classcal M/G/K multserver system where there are K dentcal servers, each of whch can process at most one job at a tme, and a FCFS buffer where jobs queue up when all the K servers are busy. The M/G/K multserver system can be modeled by an M/G/PS-MPL system wth MPL = K, and µ(n) = µ n, where µ s the speed of the ndvdual servers. The performance analyss of the M/G/K system s stll largely an open problem. Whle the performance analyss of the Processor-Sharng queue has been well understood for years, and research on the M/G//PS queue has been abundant [4,, 7, 6, 7, 9, 6], very lttle s known about the M/G/PS-MPL queue. Most analyses of the M/G/PS- MPL queue do not allow for load-dependent servce rates. For example, Itzhak and Halfn [3] derve a -moment approxmaton for the mean response tme for the M/G/PS- MPL queue where the servce rate s fxed, whle Zhang and Zwart [8] derve a heavy-traffc dffuson approxmaton for M/G/PS-MPL (whch they refer to as the Lmted Processor Sharng queue) wth a fxed servce rate. There s one analyss of the M/G/PS-MPL that does nvolve state-dependent servce rates, see Rege & Sengupta []. However [] requres that job szes are exponentally-dstrbuted whle we are focusng on hgh-varablty job sze dstrbutons whch are more representatve of computer workloads. Whle Fredercks [0] warns that the exponental job sze dstrbuton s not a good ndcator of performance of the M/G/PS-MPL wth hgh varablty, he does not derve an approxmaton PS

3 that allows for hgher varablty. Fnally none of the above theoretcal papers have tred to answer the queston of how to set the MPL so as to mnmze the mean response tme. Whle there s a large body of work on adaptve load control and admsson control n resource-sharng systems, all of the exstng work ether gnores the crucal pont of loaddependent servce rates at the server, or the effect of job sze varablty. Elnkety et al. [9] propose montorng the load of the server and admttng tasks as long as the resultng load does not exceed the peak effcency pont. Blake [] also proposes operatng at the peak effcency pont, but uses the fracton of jobs watng n the vrtual memory queue as an ndcator of thrashng to control the MPL. Kamra et al. [3] model the server as an deal M/G//P S system thereby gnorng the state-dependence of the servce rate. They montor the response tme of the departng jobs, and adjust the droppng probablty of the arrvng requests to acheve target response tme for the admtted tasks. Our solutons dffer from [3] n that we do not drop requests. Hess and Wagner [] propose a feedback mechansm to montor the effect that changng the MPL has on the performance metrc of nterest. However, as the authors observe, ths requres montorng at least hundreds of departures before a control decson can be taken. Another drawback of the soluton proposed n [] s that the authors assume the system reaches statonarty after the control decson has been taken. Ths assumpton s hardly justfed, and can cause ncorrect decsons due to a delay between the tme the control acton s taken, and the tme ts effect s observed. Schroeder et al. [4] consder the problem of settng a statc MPL n the presence of varable job szes, but the emphass of [4] s to fnd a suffcently small MPL so that class-based task prortzaton can be done n the FCFS queue. Schroeder et al. also develop a feedback based controller based on measurng the throughput and response tmes, but gnore the state-dependence of servce rate. Van der Wej, Bhula and van der Me [] also look at admsson control n a PS queue under the assumpton that the job sze dstrbuton s of phase type and the phases of all the jobs n the system are known. The authors assume a constant µ(n), and characterze the optmal admsson control polcy. In contrast, we assume that no nformaton about the job szes s avalable and hence sze-based prortzaton s not possble. Contrbutons of ths paper To the best of our knowledge, we are the frst to consder the queston of controllng the mult-programmng lmt n a resource-sharng system by takng nto account both the servce rate curve, and the hgh varablty of the job sze dstrbuton. Our paper has two prncpal contrbutons:. Optmal traffc-aware statc polces We derve the frst approxmaton for mean response tme for the M/G/PS-MPL queue wth state-dependent servce rates, and extend ths approxmaton for GI/G/PS-MPL systems. The approxmaton enables us to choose the MPL that mnmzes mean response tme. Va extensve smulaton experments presented both n ths paper and n [], we demonstrate that the optmal MPL settng can be much hgher than the peak effcency pont, under job sze varablty characterstc of computer workloads. In fact, we show examples where the optmal MPL operates the system at 8% of the peak effcency, whle droppng the mean response tme by more than 6% []. Our results are verfed across a varety of job sze dstrbutons ncludng Webull, Pareto and Hyperexponental dstrbutons. We refer to the statc polcy whch uses the optmal statc MPL as the Opt- Statc polcy.. Near-optmal traffc-oblvous dynamc polces The above results assume jobs arrve accordng to a Posson process wth a known arrval rate and propose the best statc MPL. However, we are nterested n scenaros where the mean arrval rate may not be known, or the arrval process may not even be Posson, exhbtng burstness or temporal correlatons. Our goal s to desgn lght-weght MPL control polces that adapt to the traffc characterstcs. By lghtweght polces, we mean polces whch take decsons based only on the nstantaneous number of jobs n the buffer, Q(t), and the nstantaneous number of jobs at the server, K(t). We frst consder the settng where the arrval process s known to be Posson, but wth an unknown mean arrval rate. We fnd that, unsurprsngly, statc MPLs are very poor n handlng uncertanty n the mean arrval rate. We then propose two lght-weght MPL control polces, Lght- Approx and Posson-Approx that robustly handle uncertanty n the mean arrval rate. The key dea n our approach s that by consderng a specal class of job sze dstrbutons, the -phase degenerate hyperexponental dstrbuton, we are able to ncorporate the effect of job sze varablty n our optmzaton problem, whle (Q(t), K(t)) remans a Markov process. Thus, the control polces we obtan are a functon only of (Q(t), K(t)). Va smulatons we show that both Lght-Approx and Posson-Approx are robust at adaptng to unknown mean arrval rate, resultng n near-optmal mean response tme (under 9%) for a wde range of arrval rates when compared to the optmal statc MPLs for each arrval rate. Next, we consder the settng where not only s the mean arrval rate not known, but the arrval process s also bursty. We demonstrate that both Lght-Approx and Posson- Approx are smultaneously robust to unknown mean arrval rate and burstness of the arrval process, resultng n less than % hgher mean response tme than the mean response tme for the optmal traffc-aware statc MPL. Surprsngly, we fnd that f the mean arrval rate s known, a statc MPL optmzed for a Posson arrval process wth the gven mean arrval rate s also near-optmal when the arrval process s bursty wth that mean arrval rate (that s, the nterarrval tmes are..d. but not Exponentally dstrbuted). However, burstness can greatly worsen the performance of statc polces when the mean arrval rate s unknown. Outlne In Secton, we solve the problem of choosng the optmal statc MPL for a general job sze dstrbuton under the assumpton that the arrval process s Posson wth a known mean arrval rate. In Secton 3, we begn by demonstratng that the approach of choosng a sngle statc MPL s fundamentally lmted n ts ablty to handle varablty n traffc arrval patterns. In Sectons 3. and 3.3, we construct our dynamc MPL control polces Lght-Approx and Posson- Approx, respectvely. In Secton 3.4, we evaluate these dynamc polces wth respect to () robustness to unknown arrval rate, and () robustness to burstness of the arrval process. Fnally we compare our traffc-oblvous dynamc polces to the optmal traffc-aware statc MPL polcy.

4 . CHOOSING THE BEST STATIC MPL Our frst goal n ths paper s to address the queston of how to optmally set a mult-programmng lmt n a resourcesharng system so as to mnmze the mean response tme (equvalently, mnmze the mean number of jobs n the system). We assume that the arrval process s Posson wth a known mean arrval rate, and that the job sze dstrbuton s known. In Secton., we present some stochastc monotoncty results for the performance of PS-MPL systems under farly general job sze dstrbutons whch motvate the need to approprately choose the MPL based on the job sze dstrbuton. In Secton., we provde a smple approxmaton for the mean number of jobs n an M/G/PS-MPL system wth state-dependent servce rate nvolvng only the frst two moments of the job sze dstrbuton, and demonstrate a job sze dstrbuton for whch the approxmaton s, n fact, exact. In Secton.3, we present the Opt-Statc polcy, whch uses our approxmaton to choose a statc MPL based on the mean arrval rate and the frst two moments of the job sze dstrbuton. Even though our approxmaton nvolves only the frst two moments of the job sze dstrbuton, we show va experments that t leads to optmal or near-optmal MPL selecton for a range of dstrbutons used to model computer workloads.. Stochastc monotoncty results Let F be a dstrbuton functon for a non-negatve random varable X, and f be the correspondng densty functon. Defnton. Dstrbuton F s sad to belong to the class DFR (IFR) f the functon h(x) = f(x) s decreasng (ncreasng). F (x) Defnton. Dstrbuton F s sad to belong to the class DMRL (IMRL) f the functon R(a) = E[X a X a] s decreasng (ncreasng). The classes IMRL (Increasng Mean Resdual Lfe, also referred to as NWUE for New Worse than Used n Expectaton) and DFR (Decreasng Falure Rate) both capture the noton that young jobs (those who have receved less servce) are more lkely to fnsh earler than old jobs. The condton DFR s equvalent to sayng that the resdual lfe of young jobs s stochastcally smaller than the resdual lfe of old jobs, whle IMRL s equvalent to sayng that the mean resdual lfe of young jobs s smaller than the mean resdual lfe of old jobs. The followng s a corollary of [9, Theorem ]. Proposton. In a G/G/PS-MPL system wth a DFR job sze dstrbuton, the number of jobs n the system at any tme s a stochastcally decreasng functon of the MPL K, for K K. For an IFR dstrbuton, the number of jobs n the system s a stochastcally ncreasng functon of the MPL K, for K K. A smlar proposton can be proven for the mean number of jobs (equvalently mean response tme) by relaxng the assumptons on the arrval process and the job sze dstrbuton. Proposton. In an M/G/PS-MPL system wth an IMRL job sze dstrbuton, the mean number of jobs n the system s a decreasng functon of the MPL K, for K K. For a DMRL dstrbuton, the mean number of jobs n the system s an ncreasng functon of the MPL K, for K K. Proof. From [, Theorem 3.4], for IMRL dstrbutons, t suffces to prove that for all x, V x, whch denotes the mean workload n the system due to jobs wth attaned servce less than x, s decreasng n the MPL K for K K. From the proof of [9, Theorem ], ths s easly seen to hold. The proof for DMRL dstrbutons s analogous. Intutvely, when the job sze dstrbuton s DFR or IMRL, we prefer to serve young jobs as they are more lkely to fnsh earler. By choosng an MPL smaller than K, we do not gan servng capacty, snce K acheves the maxmum speed, and smultaneously lmts the ablty of new jobs (whch are lkely to be small) to enter servce. Smlarly, for IFR or DMRL job sze dstrbutons, we prefer to serve old jobs as they are more lkely to fnsh earler. By choosng an MPL larger than K, we do not gan aggregate servng capacty, and we smultaneously reduce the capacty avalable to old jobs, as young jobs are allowed nto servce. Job sze dstrbutons belongng to class DFR and IMRL correspond to dstrbutons whch are more varable than the Exponental dstrbuton, and the above results show that there s no beneft n runnng at an MPL smaller than K n ths case. However, there mght be beneft n operatng at an MPL hgher than K, ncreasng the chance for small jobs to enter servce and fnsh quckly even whle losng aggregate servce capacty n the process, as we show next.. -moment approxmaton for M/G/PS-MPL As mentoned earler, there are no known analytcal expressons or approxmatons for the mean number of jobs n an M/G/PS-MPL system wth state-dependent servce rate. We now propose a smple approxmaton for the mean number of jobs n an M/G/PS-MPL system nvolvng only the frst two moments of the job sze dstrbuton. Proposton 3. Let E[N] denote the mean number of jobs n an M/G/PS-MPL system wth arrval rate λ, statedependent servce rate µ(n) when there are n jobs at the PS server, wth MPL=K, and a general job sze dstrbuton wth mean and SCV C. Then, h E[N] E NExp(K) S + C + E hn QExp (K) () where E hn QExp (K) and E NExp(K) S, respectvely, denote the mean number of jobs n the FCFS Queue and at the PS Server n an M/M/PS-MPL wth the same state-dependent servce rates as the orgnal M/G/PS-MPL system, wth MPL=K and Exponental job sze dstrbuton wth mean. The expressons for E hn QExp (K) and E NExp(K) S are gven by: E hn QExp (K) = h E NExp(K) S = φ K+ P + λ P K P = φ + K = φ + P = φ µ(k)! =K+ φ where φ s are the rato of the statonary probabltes and the dle probablty for an M/M/PS-MPL, and are gven by: φ = 8 < : Π j= φ K λ µ(j) K, K λ µ(k) > K. Proposton 3 can be seen as a generalzaton of the Lee and Longton [7] approxmaton for the mean number of jobs n

5 an M/G/K system, and agrees wth the approxmaton gven by [3] when the servce rate s ndependent of the state. In Proposton 4, we show that approxmaton () s n fact exact for a degenerate hyperexponental dstrbuton, H, wth mean and squared of coeffcent of varaton C. Defnton 3. A degenerate hyperexponental dstrbuton wth mean and SCV C s defned by: ( 0 wth probablty q = C H (C ) Exp C + wth probablty q = C + C + where Exp(ν) denotes an Exponental random varable wth mean /ν. Proposton 4. The mean number of jobs n an M/H (C )/PS-MPL system wth arrval rate λ, state-dependent servce rate µ(n) when there are n jobs at the PS server, and MPL=K s gven by: h E N H (C )(K) = E NExp(K) S + C + E hn QExp (K) where E hn QExp (K) and E N S Exp(K) are as defned n Proposton 3. Proof. We frst observe that the H (C ) dstrbuton conssts of two classes of jobs, those of sze 0 and those belongng to the Exponental branch. The response tme and hence the number of jobs belongng to the Exponental class n the M/H (C )/PS-MPL system s not affected by the presence of zero-szed jobs. Therefore, the contrbuton to the mean number of jobs n the system consstng of jobs n + E h N Q Exp. The the Exponental class s precsely E NExp S zero-szed jobs only contrbute to the mean number n queue. However, snce the schedulng polcy s sze-ndependent, the watng tme dstrbuton of a zero-szed job s the same as the watng tme dstrbuton of a job belongng to the Exponental class, but the arrval rate of zero-szed jobs s C tmes the arrval rate of the Exponental class. Therefore, the contrbuton of the zero-szed jobs to the mean number n system s C E h N Q Exp, provng the proposton. In Secton.4 we extend Proposton 3 to obtan an approxmaton for a GI/G/PS-MPL system nvolvng the frst two moments of the nterarrval tme and job sze dstrbutons..3 The Opt-Statc polcy We now ntroduce the Opt-Statc polcy to choose a near-optmal statc MPL. The Opt-Statc polcy smply sets MPL = κ where κ denotes the MPL that mnmzes the rght hand sde of (): h E κ = arg mn K NExp(K) S + C + E hn QExp (K) We now show that the Opt-Statc polcy s a good heurstc for mnmzng the mean response tme n an M/G/PS- MPL system wth known mean arrval rate. In Fgure 3, we present smulaton results for the followng three job sze dstrbutons all wth mean and C =9 : () Webull dstrbuton wth scale parameter and shape 6 parameter. 3 Bounded Pareto dstrbuton wth shape parameter α =. and support [0.8, 78.79]. A two-phase hyperexponental (H ) dstrbuton whose parameters are chosen so that, r, the fracton of the total load consttuted by the phase wth the smaller mean, s 0.. The results n Fgure 3 assume that the state-dependent servce rates of the PS server are gven by the µ(n) curve shown n Fgure. We wll use the servce rate curve shown n Fgure n all the numercal and smulaton evaluatons n ths paper. In [], we present detaled smulaton results for more scenaros. The man message of Fgure 3 s that the optmal MPL can be much larger than the peak effcency MPL of K =. For example, when λ = 0.8, the optmal statc MPL for the bounded Pareto dstrbuton s wth a resultng mean number of jobs around 3.4, whle K = results n 3% larger mean number of jobs at approxmately 4.6. Second, as can be seen, even though approxmaton () s not extremely accurate at predctng the mean number of jobs n the system for general dstrbutons (n fact, t s possble to show that no approxmaton based on only the frst two moments can be), t s robust n predctng the optmal or near-optmal MPL. Our approxmaton recommends MPL = 4 and the mean number of jobs n the system usng our recommended MPL s around 3.4. Usng approxmaton (), t s easy to see why the mean number of jobs n the system s mnmzed at a larger MPL than the peak effcency MPL of K when job szes have hgh varablty. To see ths, start by consderng the case of low varablty: C =. For ths case, approxmaton () suggests that the optmal MPL s n fact K. As we ncrease the MPL beyond K, f the traffc ntensty s not h very hgh, E N Q Exp falls whle E NExp S ncreases. For a h hgh enough C, the fall n C + E N Q Exp, and hence n the mean watng tme n the FCFS buffer, wll be larger than the rse n E NExp S, whch s the component representng the mean tme to process a job at the PS server. Therefore, settng an MPL hgher than K, and allowng small jobs to overtake the bg jobs, leads to an overall reducton n the mean response tme. We would lke to pont out that the queston of choosng the optmal mult-programmng lmt s closely related to the queston of choosng the optmal number of servers n a multserver system (that s, one fast vs. K slow servers), such as the M/G/K, but wth a fundamentally dfferent trade-off. In the presence of hghly varable job szes, one wants to choose a large number of servers n a multserver system to prevent small jobs from gettng blocked behnd large jobs. Smlarly, n the PS-MPL system, we want to choose a hgh MPL to allow small jobs to overtake large jobs. In both cases, we are lmted n our ablty to ncrease the parallelsm due to capacty wastage. Whle n a multserver system, capacty s wasted when there are less than K jobs n the system, n the PS-MPL system, capacty s wasted when the mult-programmng lmt K s set larger than the peak effcency pont K, and there are more than K jobs n the system. Therefore, n a multserver system, hgh parallelsm (large number of servers) s preferred when the traffc ntensty s hgh, whle n a PS-MPL system a hgh degree of parallelsm (large MPL) s preferred when the traffc ntensty s low..4 Approxmaton for GI/G/PS-MPL

6 E[N] Pareto. Webull H (r=0.) Our approx. E[N] Pareto. Webull H (r=0.) Our approx. E[N] Pareto. Webull H (r=0.) Our approx MPL (a) λ = MPL (b) λ = MPL (c) λ = 0.9 Fgure 3: The mean number of jobs n the system vs. MPL for the followng dstrbutons, all wth mean and SCV 9: () Bounded Pareto dstrbuton wth shape parameter. () Webull dstrbuton () Two-phase hyperexponental dstrbuton wth % of load consttuted by the branch wth the smaller mean. The arrval process consdered s Posson wth the ndcated mean arrval rate, λ. For reference, we have also shown our -moment approxmaton for the mean number of jobs n the system. The optmal MPL for each curve s shown wth a crcle. Proposton. Let E[N] denote the mean number of jobs n a GI/G/PS-MPL system wth state-dependent servce rate µ(n) when there are n jobs at the PS server, MPL=K, a general job sze dstrbuton wth mean and SCV Cs, and a general nterarrval tme dstrbuton wth mean and λ SCV Ca. Then, h + C s + E[N] E N S Exp where E N Exp S and E hn Q Exp h E N Q Exp, denote, respectvely, the mean number of jobs at the PS Server and n the FCFS Queue n a BP P/M/PS-MPL system wth the same state-dependent servce rates as the orgnal GI/G/PS-MPL system, MPL=K, Exponental job sze dstrbuton wth mean, mean arrval rate λ and..d. The expressons for E N S Exp where h E N Exp S = h E and γ = φ = N Q Exp 8 < : λ µ(k). = geometrc batch szes wth mean C s +C a and E hn Q Exp are gven by P K = φ + P P K =K+ φ + φ K+ + P C s +. (3) = φ Cs + C a (4) = φ (Cs + )( γ) Π λ (Cs +)+µ(j ) (C a ) j= K φ K γ (C s +)+C a (C s +C a )µ(j) K C s +C a > K 3. SELF-ADAPTIVE MPL CONTROL POLICIES In the prevous secton, we consdered the queston of choosng the optmal statc MPL under the assumpton that the arrval process s Posson, and that the mean arrval rate, λ, was known accurately. We begn ths secton by showng that the methodology of choosng a statc MPL based on assumng a mean ntensty for the Posson arrval process s very fragle. In Table we consder a Webull job sze dstrbuton wth mean and C = 9, and show the mean number of jobs n the system for varous settngs of MPL and the mean arrval rate λ. We assume the servce rate curve shown n Fgure wth K =. The optmal MPL n Table vares from, when λ = 0.6, to, when λ =.. In fact, choosng the optmal statc MPL assumng a λ 0.8 results n an unstable system when true λ =.. There can be two ways around ths problem: The frst approach s to robustly choose a sngle statc MPL that works well for all λ. Ths necessarly mples operatng the system at peak effcency K, whch we have already seen can be far from the optmal. The second approach s to learn the parameters of the arrval process and then choose the optmal statc MPL for that partcular arrval process. However, ths approach wll fal to adapt to varatons n traffc on small tme scales. In ths secton, we are motvated by the queston: Are there lght-weght, traffc-oblvous MPL control polces whch perform as well as the traffcaware optmal statc MPL polces? By a traffc-oblvous control polcy, we mean a polcy that does not depend on knowng the arrval rate or the hgher order characterstcs of the arrval process. In ths secton, we develop two dynamc MPL control polces - Lght-Approx and Posson-Approx. Secton 3. hghlghts the key deas n our approach. Secton 3. and Secton 3.3, respectvely, present the numercal algorthms nvolved n the constructon of our traffc-oblvous dynamc MPL control polces Lght-Approx and Posson-Approx. In Secton 3.4 we evaluate our dynamc MPL control polces va smulatons and demonstrate that our proposed MPL control polces exhbt robustness to both the traffc ntensty and the burstness of the arrval process. 3. Key Steps n Our Approach Recall that, gven a job sze dstrbuton, our goal s to obtan MPL control polces whch are () lght-weght: adjust the MPL based only on the nstantaneous queue length, Q(t), and the nstantaneous MPL, K(t), and () traffcoblvous: robust to varatons n the arrval process. To acheve our frst goal, we consder a specal class of job sze dstrbutons, the degenerate hyperexponental dstrbuton (H ), whch s a mxture of an Exponental dstrbuton, and a pont mass at 0. Snce the jobs of sze 0 do not spend any tme at the server, and due to the memoryless property of the Exponental dstrbuton, (Q(t), K(t)) s

7 MPL % c.. λ = ± λ = ± 0.00 λ = ± 0.94 λ = ±.483 λ = ± 4.73 λ = ±.606 Table : Numercal results for mean number of jobs n system for dfferent values of MPL and arrval rates. The arrval process was Posson, and the job sze dstrbuton was Webull wth mean, SCV 9. The optmal value for each settng of the mean arrval rate has been boldened. a Markov process. Ths ensures that we can obtan a lghtweght dynamc MPL control polcy, snce any optmal MPL control polcy for the H job sze dstrbuton wll only take decsons based on (Q(t), K(t)). The next step n our approach s solvng a stochastc dynamc programmng problem to construct famles of canddate dynamc MPL control polces. The Lght-Approx and Posson-Approx polces dffer n the famly of canddate polces. Under Lght-Approx, the famly of canddate polces s a set, {π p}, where a partcular polcy π p s constructed by solvng an optmal MPL control problem for an H job sze dstrbuton wth parameter p (Eqn. (7)). Thus, whle there s some unque H (C ) job sze dstrbuton that matches the frst two moments of the true job sze dstrbuton (Defnton 3), the famly s constructed by lookng at a range of H dstrbutons. To solve the optmal control problem, we assume that we start n some ntal state (Q 0, K 0), and fnd the polcy that mnmzes the sum of response tme of jobs n the system gven that there are no further arrvals. In the case of Posson-Approx, the famly of canddate polces s the set, {π λp }, where a partcular polcy π λp s obtaned by solvng an optmal control problem for a Posson arrval process wth ntensty λ p and the H (C ) job sze dstrbuton to mnmze the tme-average mean number of jobs n the system. The fnal step n our approach s choosng one member from the famly of canddate dynamc polces, so that the chosen polcy s robust to the arrval process. To acheve ths goal, we evaluate the canddate polces n the famly for a Posson arrval process wth rate λ [λ, λ] and H (C ) job sze dstrbuton. Let E[N (λ)] denote the mean number of jobs n the system for Posson arrval process wth ntensty λ, and H (C ) job sze dstrbuton, under the Opt-Statc polcy. The quantty E[N (λ)] s gven by Proposton 4. Let E[N π (λ)] denote the mean number of jobs n the system for the H (C ) job sze dstrbuton and Posson arrval process wth ntensty λ under a dynamc MPL control polcy π. We defne the worst-case relatve error for a polcy π as: E[N π (λ)] E[N (λ)] ɛ(π) = max λ [λ,λ] E[N (λ)] Gven a famly of canddate polces {π a} wth parameter a takng values n some set A, we choose the polcy that mnmzes the worst case relatve error: () a = arg mn ɛ(π a) (6) a A Thus, n our case, π p denotes the Lght-Approx polcy, and π λ p denotes the Posson-Approx polcy. 3. The Lght-Approx polcy As a frst step towards dervng the Lght-Approx polcy, we begn n Secton 3.. by formulatng and solvng a lght-traffc optmal MPL control problem. We fnd that the soluton to ths problem exhbts both a flud component, to guarantee stablty, and a stochastc component, to handle varablty n job szes. In Secton 3.., we use the soluton of the lght-traffc optmal control problem to construct a famly, {π p}, of smple, lght-weght MPL control polces, and n Secton 3..3 we sketch the use of Matrx-Geometrc methods to evaluate ths famly of canddate polces to enable selecton of the approprate polcy, Lght-Approx. 3.. A lght-traffc optmal control problem In ths secton we solve an optmal lght-traffc MPL control problem parameterzed by p, by consderng the followng degenerate hyperexponental job sze dstrbuton : ( H 0 wth probablty p (p) (7) Exp () wth probablty p We assume that we start our PS-MPL system n some state (Q 0, K 0) at tme t = 0, where a departure has taken place at tme t = 0. The state varable Q 0 denotes the queue length at t = 0 and K 0 s one more than the number of jobs at the PS server left behnd by the last departure. We assume that multple zero-szed jobs admtted at the same tme leave together. Thus K 0 does not necessarly denote the MPL at tme t = 0. However, by our assumpton of an H (p) job sze dstrbuton, each of the (K 0 ) jobs at the server has remanng servce requrement ndependent and dentcally dstrbuted as Exp(). Note that whle the zero-szed jobs do not spend any tme at the server, they stll experence delays whle watng n the FCFS buffer. We assume that there are no more arrvals (hence the lghttraffc). We can now take one of the followng actons at tme t = 0:. Decrease MPL: We do not admt another job from the queue nto the PS server, decreasng the MPL to K 0.. Keep MPL same: We admt only one job from the queue nto the PS server to replace the departng job, mantanng the MPL at K Increase MPL by k: We admt k + jobs from the queue nto the PS server, ncreasng the MPL to K 0+k. Our am s to take the optmal acton n each state so as to acheve the followng goal: Mnmze the expected sum of response tmes of jobs present n the system at tme t = 0, gven that there are no further arrvals.

8 If our goal was to mnmze the tme untl the system emptes, the optmal control would be to operate at MPL of K. However our performance metrc s the mean response tme. Note that we do not allow preemptng an executng job to decrease the MPL. Ths s mportant because n a transacton processng system, for nstance, kllng an executng task nvolves unrollng the executon trace for the task and s sgnfcantly expensve. In our framework, we can only alter the MPL when a job departs, and hence we assume that there are no costs assocated wth changng the MPL. The soluton of the above optmal-control problem can be obtaned n a straghtforward fashon va stochastc dynamc programmng. To do so, we assocate a cost functon c(q, K) wth each state (Q, K), whch represents the optmal expected sum of response tmes, gven that we start n state (Q, K) at tme t = 0, and an acton functon π(q, K), representng the optmal acton n state (Q, K). The functon π(q, K) takes values n the range {, 0,,,...} wth representng the acton decrease MPL, 0 representng the acton keep MPL same and k > 0 representng the acton ncrease MPL by k. The cost of the states wth zero queue length s smply: c(0, K) = K X = µ() To see why the above s true, note that snce the queue s empty and we do now allow preempton of executng jobs, the cost of state (0, K) s the expected sum of response tmes of the K jobs executng at the server. The mean tme (8) µ(k ) snce untl the departure of the frst job s gven by the server s processng at rate µ(k ). The tme untl the frst departure gets added to the response tme of all the K jobs n the system, and contrbutes to c(0, K), and µ(k ) so on for subsequent departures. We represent by c (Q, K) the cost of state (Q, K) gven that we take acton decrease MPL n state (Q, K). Smlarly, c k (Q, K) (k {0,..., Q }) denotes the cost of state (Q, K) gven that we take acton ncrease MPL by k n state (Q, K). Gven c (Q, K) and c k (Q, K), the optmal acton π(q, K) and the cost functon c(q, K) are: π(q, K) = arg mn c δ (Q, K) δ {,..., Q } (9) δ c(q, K) = c π(q,k) (Q, K) (0) The functon c (Q, K) s gven by: = c (Q, K) = Q + K µ(k ) + c(q, K ) () and c k (Q, K) s gven by: Q + K c k (Q, K) = µ(k + k) + c(q k, K + k) ( p) k+ k+ X k + + c(q k, K + k + ) ( p) k+ p () In dervng the last equaton, we have made use of the assumpton that f multple zero-szed jobs are admtted smultaneously, then they all leave together. Ths mantans the nvarant that the K n state descrptor (Q, K) s one larger than the number of jobs at the server belongng to the Exponental class, and we do not have to keep track or estmate the number of zero-szed jobs. Whle n the problem formulaton above, we have not mposed an upper bound on k, n practce we restrct k max to prevent sudden jumps n MPL. For all the smulaton results n ths paper, we set max =. 3.. A famly of traffc-oblvous MPL control polces In Secton 3.. we formulated an optmal control problem parameterzed by p, the fracton of zero-szed jobs n the H (p) job sze dstrbuton. By varyng the parameter p, we obtan a famly of MPL control polces. Let π p denote the acton functon for the control problem wth parameter p. Fgure 4 shows the structure of π p for p = 0.3 and p = 0. and the servce rate curve shown n Fgure. For example, f the current state s (Q =, K = 0), under the p = 0.3 polcy, the control s to decrease the MPL to 9 by not admttng a new job, whle under p = 0. polcy, the optmal control s to ncrease the MPL to by admttng two jobs. The structure of the optmal soluton has some nterestng features:. For a gven p, there s some mnmum queue length Q(p) such that the optmal acton for Q > Q(p) s to operate at the peak effcency pont. In Fgure 4(a), Q(p) = 0 and the optmal control for Q > Q(p) s to attan the peak effcency MPL of K =. We call ths the flud component of the control polcy. Ths flud component provdes robustness to the dynamc MPL polcy aganst hgh arrval rates. Further, as p ncreases, the threshold Q(p) ncreases.. As the queue length decreases, the stochastc component of the control takes over, gradually ncreasng the MPL to a pont wth lower servce rate than the most effcent pont. Ths stochastc component gves our MPL control polcy the ablty to combat the job-szevarablty when the traffc ntensty s low. The structure of the optmal control s qute ntutve. Whenever a decson to ncrease the MPL has to be taken, there are two scenaros: () wth probablty p the admtted job s of sze zero n whch case the decrease n server speed does not hurt any one, and () wth probablty p, the admtted job belongs to the Exponental class and n ths case adds to the watng tme of everyone n the queue. If we defne the threshold queue length to be the pont when we should ncrease the MPL and move to a less effcent servce rate, then we see that ths threshold queue length s an ncreasng functon of p. Gven any acton functon π, we can translate t nto a dynamc MPL control polcy va the procedure n Fgure. The Lght-Approx control polcy for a dstrbuton wth SCV C s now chosen to be π p such that: p = arg mn ɛ(π p) (3) p where ɛ( ) s gven by (). Expermentally, t suffces to carry out the optmzaton over a small set of parameters p (at a coarse granularty) Evaluaton of dynamc MPL control polces va Matrx-geometrc analyss

9 0 0 MPL 0 MPL Queue Length (a) p = Queue Length (b) p = 0. Fgure 4: The structure of the Lght-Approx control polcy for two values of the parameter p and max =. A + ndcates ncrease MPL, and a o ndcates keep MPL same. At every other pont, the optmal control s to decrease MPL. Algorthm MPL control(π) Case: New arrval Let Q be the queue length and K be the MPL mmedately after the arrval. Let π(q, K + ) = k f k 0: admt k + jobs from the head of the FCFS buffer nto the server and ncrease MPL to K + k + f k < 0: do nothng Case: Departure Let Q be the queue length and K be the MPL mmedately before the departure. Let π(q, K) = k f k 0: admt k + jobs from the head of the FCFS buffer nto the server and set MPL to K + k f k < 0: reduce MPL to K by not admttng any job from the FCFS buffer Fgure : The dynamc MPL control polcy obtaned from the acton functon π. In ths secton, we outlne a method to numercally evaluate the mean number of jobs, E[N π (λ)], for a dynamc MPL control polcy π under the assumpton of the H (C ) job sze dstrbuton (Defnton 3) and a Posson arrval process of ntensty λ. Note that n Proposton 4 wth statc MPL, we were able to smplfy the analyss of the H (C ) job sze dstrbuton by gnorng the zero-szed jobs and focusng on the exponental class. Ths was because the admsson control polcy was ndependent of the queue-length. However, wth a dynamc polcy that looks at the queue-length, we need to keep track of how many zero-szed jobs are n the system. For succnctness, let q =. C + Assumng that under the dynamc polcy π, there s some queue-length Q such that the optmal control for any queue length Q Q s to operate at the hghest effcency pont K, we can express the system as a Markov chan wth a Q = 0 Q Q + K +,? d K + K + K +,? d K + K + K + K,? K,? 0,? 0 a K + q K +,? d K + K + 0,? a K +,? K + K + a K + K + K + q q K +,? K +,? a K + q d K + K + q q( q) ( q) q a K + q a K + d K + d K + q K + K dk K,? q a K + a K + ak d K + d K + K + d K + d K + q K + K dk K,? q a K + a K + ak q d K + d K + q K + K + K dk K,? Fgure 6: The embedded Markov chan for evaluaton of dynamc MPL control polces. We use a n to denote λ and dn = an. For decson states wth multple λ+q µ(n) alternatves (e.g., (, K +,?) and (, K,?)), the dashdotted arcs correspond to the decson to not admt any jobs, dashed arcs correspond to the decson to admt one job, and dotted arcs correspond to the decson to admt two jobs. repeatng structure. The states of the Markov chan are pars (Q, K) wth Q denotng the queue length, and K denotng the number of jobs of the exponental class at the server. However, due to the zero-szed jobs, we can have arbtrarly bg drops n Q. For example, f we are n state (Q = 0, K = ) and a departure takes place, and f all the jobs n the queue have sze 0, whch happens wth nonzero probablty, we jump to state (Q = 0, K = 4). To a K + a K + ak

10 take care of ths problem, we ntroduce decson states represented as (Q, K,?). We transton to the decson state (Q, K,?) mmedately after a departure takes place from the state (Q, K + ), or f an arrval takes place whle n state (Q, K) and Q < Q. The state (Q, K,?) mplements the admsson control polcy π, as well as handlng zero-szed jobs, because now the jumps are bounded. For example, f the control n state (Q, K,?) s to admt job, then wth probablty ( q) the job s of sze 0, and we transton to (Q, K,?); otherwse, wth probablty q we transton to (Q, K + ). However, the rate of transtonng from the decson states s nfnte. Thus we wll fnd t sutable nstead to work n the framework of Sem-Markov processes. We wll consder the embedded dscrete tme Markov chan where the transtons correspond to arrvals, departure and decsons taken n decson states n the orgnal contnuous tme system. The embedded Markov chan s shown n Fgure 6. We then solve for the statonary dstrbuton of ths embedded Markov chan va Matrx-Geometrc method. We would lke to pont out that due to the specal structure of the Markov chan n Fgure 6 (the backward transton matrx s of rank ), the rate matrx nvolved n the Matrxgeometrc soluton has an explct soluton n our case [0]. Fnally, we obtan the statonary dstrbuton of the number of jobs n the system by multplyng the probablty of beng n a state n the embedded chan wth the mean resdence tme n that state, and normalzng. 3.3 The Posson-Approx polcy The Posson-Approx polcy s defned by constructng a famly {π λp }, where the canddate polcy π λp s obtaned as follows: We consder a Posson arrval process of ntensty λ p and the H (C ) job sze dstrbuton, and solve the optmal dynamc MPL control problem to mnmze the mean number of jobs. The polcy π λp s computed va the method of polcy teraton, explaned n Appendx A. Fgure 7 shows the structure of π λp for λ p = 0.9 and λ p =.0. The Posson-Approx MPL control polcy s now chosen to be π λ p where: λ p = arg mn ɛ(π λp ) (4) λ p where ɛ( ) s defned n (). As n the case of Lght-Approx, t suffces to carry out the above optmzaton at a coarse granularty. 3.4 Performance Evaluaton In ths secton we show va smulatons that our dynamc MPL control polces proposed n Sectons 3. and 3.3 guarantee robustness aganst both msestmaton of traffc ntensty, and aganst hgher order characterstcs of the arrval process, such as the burstness Robustness aganst traffc ntensty estmaton We wll now evaluate the Lght-Approx and Posson- Approx polces for a Posson arrval process wth unknown mean arrval rate, λ, and compare them aganst the Opt- Statc polcy that s gven the exact mean arrval rate. To do ths, we show the mean number of jobs, E[N], under dfferent arrval rates, obtaned va smulatons. Recall that Table shows these results for the Webull job sze dstrbuton and varous values of statc MPLs. In Table we show the results for the mean number of jobs for the same Webull job sze dstrbuton under the Lght-Approx polcy, as a functon of λ and the parameter p of the famly {π p} of canddate polces. The optmzaton procedure (3) sets p = 0. from among the values shown n the table (column hghlghted). Observe that the Lght-Approx polcy gves near optmal performance for each arrval rate as compared to Table for λ up to.0 wth approxmately 3% larger mean number of jobs n the system than the optmal traffc-aware statc polcy when λ = 0.8. On the other hand, a sngle robustly chosen statc MPL necessarly has to operate at the peak effcency pont and, as Table shows, exhbts 4% larger mean response tme than the optmal traffc-aware statc polcy when λ = 0.7. Table 3 shows smulaton results for the mean number of jobs wth the Posson-Approx MPL control polcy for varous values of the parameter λ p for the famly {π λp } of canddate polces. The optmzaton procedure (4) sets λ p = 0.9 from among the values shown n the table (column hghlghted). The Posson-Approx polcy also acheves near-optmal performance for each arrval rate as compared to Table wth approxmately 9.% larger mean number of jobs n the system than the optmal traffc-aware statc polcy when λ =.. Note that for these results, we have not completely optmzed the λ p parameter, and the performance of the Posson-Approx polcy s lkely to mprove further. Whle we have seen that both dynamc polces are far superor than any statc polcy when the mean arrval rate s not known, lookng both at Tables and 3, one can observe that nether dynamc polcy sgnfcantly outperforms the Opt-Statc polcy f the mean arrval rate s known Robustness aganst burstness n arrval process wth unknown arrval rate We now evaluate the robustness of our MPL control polces aganst burstness of the arrval process when the mean arrval rate s not known. To do so, we choose a batch Posson arrval process (BPP). The batch szes were..d. geometrc wth mean. Table 4 shows the results for the mean number of jobs n the system wth Webull job sze dstrbuton for varous settngs of statc MPL and mean arrval rate λ of the arrval process. From Table 4, we see that when the arrval rate s not known, a robustly chosen statc polcy has to operate at K =, whch results n 0% hgher mean number of jobs than the optmal traffc-aware statc polcy when the mean arrval rate s λ = 0.6. Therefore a bursty arrval process can exacerbate the nadequacy of statc MPL polces when the mean arrval rate s not known. Table shows the results for mean number of jobs n the system for the same settng as Table 4 for the Lght- Approx MPL control polcy as a functon of the parameter p of the famly {π p} of canddate polces for varous values of the mean arrval rate λ. The column for the parameter chosen by the Lght-Approx polcy has been hghlghted. From Table, we fnd that Lght-Approx polcy s also robust to burstness, whle yeldng at worst % hgher mean response tme than the optmal traffc-aware statc MPL polcy. Therefore, the Lght-Approx polcy s smultaneously robust to both the mean arrval rate and burstness of the arrval process. The Lght-Approx polcy wth parameter p = 0.3 outperforms the polcy wth p = 0. for the chosen settng, but as noted earler, ths s due to the fact that we have not optmzed the parameter completely.