Approximation of Generalized Processor Sharing with Interleaved Stratified Timer Wheels - Extended Version

Transcription

1 TECHNICAL REPORT CS , UNIVERSITY OF WATERLOO - EXTENDED VERSION OF ACM/IEEE TON PAPER 1 Approxmaton of Generalzed Processor Sharng wth Interleaved Stratfed Tmer Wheels - Extended Verson Martn Karsten Davd R. Cherton School of Computer Scence Unversty of Waterloo mkarsten@cs.uwaterloo.ca Techncal Report CS , Unversty of Waterloo Abstract Ths paper presents Interleaved Stratfed Tmer Wheels as a novel prorty queue data structure for traffc shapng and schedulng n packet-swtched networks. The data structure s used to construct an effcent packet approxmaton of General Processor Sharng (GPS). Ths schedules the frst of ts knd by combnng all desrable propertes wthout any resdual catch. In contrast to prevous work, the scheduler presented here has constant and near-optmal delay and farness propertes, and can be mplemented wth O(1) algorthmc complexty, and has a low absolute executon overhead. The paper presents the prorty queue data structure and the basc schedulng algorthm, along wth several versons wth dfferent cost-performance trade-offs. A generalzed analytcal model for rate-controlled rounded tmestamp schedulers s developed and used to assess the schedulng propertes of the dfferent scheduler versons. Some llustratve smulaton results are presented to reaffrm those propertes. Index Terms Communcaton systems, Computer network performance, Packet schedulng, Data structures, Algorthms I. INTRODUCTION Packet schedulng algorthms are a cornerstone for the future development of packet-swtched networks as ubqutous communcaton nfrastructure, ntegratng a wde range of network technologes and offerng a wde varety of applcaton servces. Packet schedulng algorthms ncrease the level of control over packet transmssons and permt the support of dfferent servce polces. There are many applcaton areas for packet schedulng, rangng from detaled qualty of servce guarantees fondvdual applcaton flows [1] to servce assurances for aggregates [2]. In each of these scenaros, a more precse scheduler translates nto more effcent resource usage n relaton to the qualty of the servce guarantees. Because of the sgnfcant complexty and executon cost of packet schedulers, the archtectural sweet spot of network and capacty plannng has been n confguratons wth very smple schedulers, so far. A feasble packet scheduler wth perfect or near-perfect servce propertes has been elusve. Clearly, the avalablty of such a packet scheduler wll go a long way towards establshng accurate traffc control as a basc buldng block for packet-swtched communcaton networks. General Processor Sharng (GPS) has been ntroduced n [3] as a conceptual scheduler wth many desrable propertes. In very basc terms, GPS schedulng works by assgnng fractons of the overall forwardng capacty to flows. These fractons are termed weghts and the GPS scheduler guarantees flud servce n proporton to a flow s weght compared to the sum of all other actve flows weghts. In realty, packets are atomc unts and thus a packet scheduler always devates somewhat from perfect GPS servce. A large varety of GPS emulaton algorthms have been proposed, but no algorthm exsts so far that combnes very close GPS approxmaton wth constant algorthmc complexty and low executon overhead. Ths paper presents a novel data structure called Interleaved Stratfed Tmer Wheels (ISTW) and approprate access operatons. Ths desgn enables the constructon of a set of novel packet schedulers wth effectvely constant complexty, and constant farness and delay characterstcs n all relevant dmensons. The ISTW data structure s used as a compact and effcent prorty queue that enables the vrtual traffc shapng necessary for achevng these characterstcs. Constant complexty n ths context s defned as amortzed executon cost over a certan amount of nput traffc. Amortzed cost can be translated nto extra bufferng and as such, addtonal delay. In contrast to prevous work [4] however, the amortzaton perod for the schedulers presented here s tghtly lmted and practcal. In partcular, the worst-case executon cost for all per-packet operatons s proportonal to the sze of the correspondng packet. In an earler verson of ths work [5], the SI-WF 2 Q scheduler has been proposed and analyzed, whch requres an addtonal small but nontrval amortzaton buffer to acheve constant executon complexty. The schedulers presented here overcome ths lmtaton. In short, the contrbutons of ths paper are: We descrbe the basc ISTW data structure, and summarze and slghtly mprove the prevous SI-WF 2 Q proposal. Based on ths work, we generalze and mprove the analytcal model, and demonstrate ts applcablty. We present a new packet scheduler termed K Packet Scheduler that avods the search problem of SI-WF 2 Q n exchange for only a mnoncrease of the worst-case farness bound. Ths schedules thus the frst of ts knd by combnng a near-optmal packet approxmaton of GPS wth constant and low executon overhead. We also present a smpler varant of the K Packet Scheduler that further reduces the memory footprnt and

2 TECHNICAL REPORT CS , UNIVERSITY OF WATERLOO - EXTENDED VERSION OF ACM/IEEE TON PAPER 2 complexty n exchange for slghtly hgher error terms. The rest of the papes organzed as follows. In Secton II, we revew prevous work and ts relaton to the approach presented here. In Secton III, we ntroduce and analyze a general model for rate-controlled tmestamp schedulers that operate on rounded deadlnes. Ths s followed by the specfcaton of a new prorty queue data structure n Secton IV and resultng packet scheduler desgns and ther assessment n Secton V. We present canoncal smulaton results to verfy and llustrate the new schedulers servce propertes n Secton VI and dscuss furthenterestng detals n Secton VII, before fnshng the paper wth a concluson n Secton VIII. Ths papes an extended verson of [6]. II. BACKGROUND AND RELATED WORK A. Generalzed Processor Sharng Generalzed Processor Sharng (GPS) [3] s a conceptual schedulng dscplne defned for a set of flows F, such that each flow s allocated a weght φ. At each tme, assumng a fxed server capacty C and a set of backlogged flows B, the servce for each backlogged flow B s guaranteed to be φ j B φ j In other words, GPS always shares the avalable lnk capacty n perfect proporton to the flows weghts. Because GPS cannot be mplemented n realty, there s a class of schedulers that attempt to approxmate GPS as good as possble. These schedulers are prmarly assessed through qualty metrcs that descrbe ther devaton from perfect GPS servce. On the other hand, the respectve algorthmc complexty determnes the practcal feasblty of each scheduler. The man qualty metrcs of a GPS approxmaton packet scheduler are the delay bound, especally n a form that can be used to determne an end-to-end delay bound as shown n several analytcal frameworks [2], [7], [8], [9], as well as two farness measures. Relatve farness (ntroduced n [10]) denotes the capablty of a scheduler to dstrbute excess capacty between dfferent sessons n proporton to ther allocated servce rates. Worst-case farness (ntroduced n [11] and refned n [12]) expresses the maxmum devaton from perfect GPS schedulng. Whle the delay bound only characterzes how far the actual servce for a sesson can be behnd the deal GPS scheduler durng a busy perod, worstcase farness essentally provdes an ntegrated bound on how far ahead or behnd the actual servce can be. Farness thus also descrbes the burst characterstcs of the servce allocaton n relaton to the deal smooth servce of GPS. The key metrc for descrbng the qualty of these servce characterstcs as well as the computatonal complexty of a packet schedules the asymptotc relaton between the respectve characterstc and the number of flows n the system, whch s descrbed as constant, logarthmc, or lnear. For example, constant delay descrbes the property that the delay bound s ndependent of the number of flows. Any real-world packet-orented scheduler needs to operate on packets as atomc servce unts and thus C. cannot avod a certan servce devaton from the flud GPS model. Exstng GPS emulaton algorthms can be classfed as tmestamp schedulers, round-robn schedulers, and hybrd versons. Dfferent schedulers provde dfferent combnatons of the aforementoned schedulng qualty characterstcs, whle none of the exstng proposals s optmal n all qualty dmensons and also of low complexty and executon overhead. There s a class of fundamentally dfferent schedulers, termed Servce Curve Schedulers [13], [14], whch can provde delay bounds ndependently of throughput guarantees. These schedulers are nherently more complex than GPS emulaton schedulers, so t seems hopeless to thnk about effcent mplementaton before solvng the GPS emulaton problem. Hence, although the technques presented here may be applcable to such schedulers, detals are out of scope for ths paper. B. Tmestamp Schedulers Tmestamp schedulers approxmate GPS behavour by smulatng the vrtual system tme n the equvalent GPS system. The respectve start and/or fnsh tmes of packets n the reference GPS system are used to decde the orden whch packets receve servce. By the same token, the smulated vrtual tme s used as a startng pont for newly arrvng flows, whch s necessary to acheve at least some bound on unfarness and burstness. Ths challenge s well-known snce the earlest proposals for rate-proportonal packet schedulng [15], [16]. Note that the order of packets wthn a flow s not affected by the scheduler operaton. Therefore, only the frst packet n each flow s queue needs to be consdered for schedulng and the term flow tmestamp s often used when referrng to the packet tmestamp at the head of a flow s queue. C. Worst-case Far Weghted Far Queueng (WF 2 Q) An optmal packet-based approxmaton of GPS s gven by Worst-case Far Weghted Far Queueng (WF 2 Q) [12]. The devatons from GPS schedulng are bound by strctly ratedependent values for the respectve flow (or two flows n the case of relatve farness) wth provably optmal coeffcents. In partcular, all schedulng errors are ndependent of the number of flows n the system. Earler attempts at approxmatng GPS, such as proposed n [3], [10], [15], [17], ncur a potentally lnear devaton from GPS schedulng, n terms of ether farness or delay behavour. In fact, t turns out that when consderng only packet start tmes for schedulng, the startup delay cannot be lmted effectvely and the delay bound depends on the number of flows n the system. When schedulng packets only by ncreasng fnsh tmes, packet bursts and unfarness can occur, also bound only by the number of flows. Conceptually, the WF 2 Q algorthm works by combnng both crtera, start and fnsh tme. In a frst shapng step all elgble packets are selected, that s all packets wth a start tme not later than the current system vrtual tme. From all these packets, the one wth the smallest fnsh tme s sent next. Ths packet selecton polcy, termed Start-elgble Earlest Fnsh-tme Frst (SEFF) ensures tght bounds for all qualty ndces. Stlads and Varma [18] ndependently

3 TECHNICAL REPORT CS , UNIVERSITY OF WATERLOO - EXTENDED VERSION OF ACM/IEEE TON PAPER 3 arrve at the same concluson that the combnaton of traffc shapng and fnsh-tme servce results n optmal schedulng characterstcs. Whle [18] only descrbes a very smplfed mplementaton n the context of fxed-sze packet networks (ATM n ths case), the orgnal WF 2 Q proposal [12] s not at all concerned wth algorthmc complexty or executon overhead. the othen between the processng of two consecutve packets. Snce all flows may end up wth the same or very close start tmes, ths number cannot be lmted and effectvely results n O(N) worst-case complexty. Whle the transfer cost could be amortzed over the number of packets transferred, ths amortzaton would requre O(N) buffer space between scheduler and output lnk and result n a correspondng schedulng error. D. WF 2 Q Approxmaton There are two proposals fomplementng an approxmaton to WF 2 Q wth lower complexty: WF 2 Q+ [19] and Leap Forward Vrtual Clock (LFVC) [4]. The work by Stlads and Varma [18] contans a smlar concept, but does not elaborate on all detals. In general, all SEFF-based algorthms contan three parts that are relevant for ther executon overhead and complexty: Flows are sorted accordng to tmestamps. The SEFF polcy requres consderaton of both the start and the fnsh tmestamp for the schedulng decson. The vrtual tme of the GPS reference system needs to be smulated or approxmated. Sortng and prorty queues are among the best-studed problems n Computer Scence. Wthout further restrctons, mantanng a sorted contaner has O(log N) complexty n the number of elements. In the context of GPS emulaton schedulers, however, t can be a very acceptable trade-off to use rounded tme values (and ncur some addtonal schedulng error) n exchange for a fnte unverse of sortng values, whch enables more effcent solutons to the sortng problem. For example, the van Emde Boas prorty queue [20] has O(log log N) access complexty fonserton, removal and fndng the lowest value. Smlarly, a tmer wheel [21] could operate n O(1) fonserton or removal and, n combnaton wth herarchcal btmaps and a prorty encoder of wdth K, wth O(log K N) complexty for searchng the lowest value (see Secton 5 n [22] for a bref dscusson). In both cases, a fnte tme horzon must be assumed, whch translates nto a maxmum specfable nter-arrval tme of packets. Snce the maxmum packet nter-arrval tme s part of the lower bound on the startup delay, one can assume that there s an upper lmt to ths value and t wll not change n future networks. Then, f the desred delay precson s also fxed n terms of wall-clock tme, one could argue that these algorthms operate wth constant complexty n all relevant dmensons. The orgnal LFVC work [4] presents generalzed proofs for rounded tmestamps, but t requres unform routng of all tmestamps, whch n turn prohbts a constant-tme mplementaton. Unfortunately, the SEFF polcy makes the above consderatons somewhat rrelevant. It bascally requres keepng flows n a two-dmensonal contaner where they are sorted by both ther start and fnsh tmes. Ths approach has been chosen for WF 2 Q+, but nevtably requres a tree-based data structure and consequently O(log N) access complexty. Alternatvely, flows can be kept n two one-dmensonal contaners, as proposed for LFVC, and explot the lower access complexty dscussed above. However, ths two-contaner soluton requres the transfer of all newly elgble flows from one contaner to E. Vrtual Tme Tradtonally, the precse smulaton of GPS vrtual tme has been consdered as beng an operaton wth lnear complexty, snce t needs to keep track of all changes n the set of flows backlogged n the GPS reference system. A recent proposal [23] allows for the exact smulaton of GPS vrtual tme wth O(log N) algorthmc complexty n the number of flows. Independent analyss shows O(log N) to be a lower bound [24]. Both WF 2 Q+ and LFVC (along wth other algorthms) use a smpler approxmaton of GPS vrtual tme. Bascally, the approxmated vrtual tme progresses wth real tme durng actual servce, that s, t s ncremented by the duraton of the current packet at each schedulng step. If however the smallest start tme of all backlogged flows (whch s readly avalable n WF 2 Q+ and LFVC) s larger than the current vrtual tme, the vrtual tme jumps forward to ths mnmum start tme. Ths approxmaton of GPS vrtual tme has some negatve effects on the schedulng qualty of WF 2 Q+ and LFVC [23]. Under certan crcumstances, the approxmaton of vrtual tme could lead to unfarness and burstness beng lnean the number of flows. Ths observaton s not a contradcton of the fndngs for WF 2 Q+ and LFVC, but results from dfferent assumptons. Normally, GPS emulaton schedulers are consdered n a context where some knd of delay guarantees are sought. Such delay guarantees can only be gven f the sum of rates of all flows does not exceed the lnk capacty. In terms of the GPS defnton, ths denotes a stuaton where the sum of weghts s less than or equal to 1. In ths case, all prevous results about WF 2 Q+ and LFVC hold, and burstness s ndependent of the number of flows. If ths restrcton s removed, then the observatons reported n [23] become an ssue. However, t s somewhat questonable whether any applcaton scenaro requres the support for sum of weghts exceedng 1. Certanly, all scenaros that am at provdng some form of delay guarantee do not qualfy. Furthermore, the maxmum spread between the hghest and lowest servce rate s typcally lmted. For such a scenaro, t s possble to fnd a better schedulng soluton wth a small and constant executon overhead, as demonstrated n ths work. There s another seemng contradcton between the results reported by Xu and Lpton [25] and the results presented here. Ths dscrepancy s explaned by the computatonal model by Xu and Lpton, whch does not allow for the floor or celng functon to be used. The lower bounds are crtcally lnked to ths assumpton. In contrast, our algorthm s based on rounded tmestamps usng the floor functon and provdes a superor combnaton of schedulng propertes and algorthmc complexty.

4 TECHNICAL REPORT CS , UNIVERSITY OF WATERLOO - EXTENDED VERSION OF ACM/IEEE TON PAPER 4 F. Low Complexty Implementaton A number of technques for the effcent mplementaton of tmestamp schedulers are presented and dscussed by Stephens et al. [26]. For the partcular case of fxed packet szes n ATM networks, the artcle presents an mplementaton of WF 2 Q+ wth constant executon overhead. In the case of varable packet szes, a dfferent soluton s presented, whch techncally can be regarded as havng O(1) complexty n the number of flows, but there are shortcomngs. The scheduler mplementaton uses stratfcaton n the vrtual servce tme of packets to reduce the complexty of the one-contaner soluton proposed for WF 2 Q+ [19]. Ths results n a number of stratfed groups whch s logarthmc to the rato of the maxmum over the mnmum supported servce rate. For example, f the system were to support servce rates between 16 Kbt/s and 40 Gbt/s at packet szes rangng from 64 to 1500 bytes, ths would result n 26 stratfed groups. The algorthm then specfes that between each schedulng step, t s necessary to nspect the start and fnsh tmes of the front flow n each group to determne the next one to receve servce under the SEFF polcy. Ths sequence of comparsons s a nontrval and costly operaton and s hardly possble wthn the strct tmng bounds of hgh-speed lnks. Instead, the paper refers to hardwarebased tmestamp sortng. In other words, ths sequence of comparsons ntroduces too hgh an absolute constant overhead per schedulng step. Also, the proposal requres addtonal sortng, snce flows change ther group assocaton. Fnally, the theoretcal treatment of rounded tmestamps s not as comprehensve as the model and analyss presented n ths paper. However, the work by Stephens et al. provdes very nterestng nsght nto the mplementaton detals of a packet scheduler at hgh lne rates [26]. The practcal mplementaton of WF 2 Q+ by Rouskas and Dwekat [27] reles on the fact that certan packet szes domnate the overall traffc. The exstence of other packet szes s mentoned, but not addressed thoroughly and no quanttatve results are gven for ths case. Also, t seems that the scheduler can only support a fxed set of flow rates, although the detals of ths are not completely dscussed. As such, the proposal presented there s not as general as ours. Furthermore, the algorthm depends on the ablty to sort tmestamps n constant tme n hardware, whch s a much stronger requrement than a prorty encoder. G. Round Robn and Hybrd Schedulers Round robn schedulers take a fundamentally dfferent am at emulatng GPS schedulng. Instead of tmestamp computatons and sortng, servce slots are assgned n some modfed round robn fashon. Ths dramatcally reduces the algorthmc complexty of such schedulers, but most early proposals suffer from rather large error terms n ther farness and delay propertes. A more recent example, Smoothed Round Robn (SRR) [28], uses a fxed weght matrx to acheve very low computatonal complexty. Its relatve farness only depends on the order of the weght matrx and s thus ndependent of the number of flows. However, the weght matrx cannot be changed easly and the delay and worst-case farness of SRR s lnean the number of flows. The G-3 scheduler [29] s an extenson of the SRR scheduler [28] to overcome the restrctons of a fxed weght matrx. Recent proposals, such Stratfed Round Robn (STRR) [30], Far Round Robn (FRR) [31], and Group Round Robn (GRR) [32], use flow stratfcaton along servce rates and a two-level schedulng herarchy to solve the problem of dynamcally addng and removng flows. The VWQGRR scheduler [33] proposes a new groupng strategy for the GRR scheduler [32] to mprove delay bounds, but does not change the fundamental propertes. The crtcal parameter determnng the trade-off between qualty and complexty of a round robn schedules the sze of the quantum used for each schedulng round. Any round robn scheduler has an error term of a MTU flow rate due to the mnmum quantum. Such a schedulng error poses a problem for lowrate flows wth small packet szes, such as voce. Further, f the quantum s chosen too small, ths can lead to multple processng steps wthout output (slp processng) and thus break O(1) complexty. On the other hand, a larger quantum results n larger error terms. STRR uses a large quantum and has perfect O(1) complexty. However, the algorthm s general delay bound and worstcase farness s lnean the number of flows. FRR does not specfy a partcular quantum, but any quantum that can avod lnear schedulng errors nevtably leads to slp processng n the round robn loop. Ths effectvely breaks O(1) complexty. GRR s a general technque for herarchcal schedulng usng round robn as ntra-group scheduler. The quantum problem s approached dfferently than n other herarchcal round robn schedulers. GRR allocates a large quantum to each group, but nterleaves group servce accordng to the nter-group scheduler. Whle ths results n the best schedulng propertes of all round robn schedulers, t poses the rsk of breakng O(1) complexty through slp processng: A flow that becomes non-backlogged cannot be removed from the round robn lst mmedately. Instead, f t s stll non-backlogged durng the next round, t s consdered departed and effectvely removed from the lst. However, the number of departed flows may be arbtrarly large n any round and therefore, may result n O(N) processng steps between the processng of two consecutve packets. Derved proposals such as VWQGRR and G-3 mprove the respectve basc versons, but stll suffer from the same general quantum-related problems. H. Herarchcal Schedulng Herarchcal schedulng allows foncreased control over lnk sharng and resource allocaton. Some of the shortcomngs of the early far-queueng schedulers become very apparent n the context of herarchcal packet schedulng [19]. Herarchcal rate-based schedulng nvarably ncreases the delay bounds for leaf classes and as such, servce curve schedulers such as SCED [13] and HFSC [14] are more sutable to acheve both lnk sharng and tght delay goals at the same tme, albet wth ncreased complexty. For the purpose of ths work, we do not argue n favour or aganst the usefulness of herarchcal schedulng. However, as noted n [19], herarchcal confguratons can be regarded as an excellent ltmus test for

5 TECHNICAL REPORT CS , UNIVERSITY OF WATERLOO - EXTENDED VERSION OF ACM/IEEE TON PAPER 5 the farness characterstcs of a packet schedulng algorthm. It s strctly for ths reason that the smulaton experments use herarchcal schedulng. III. MODEL AND ANALYSIS We present a model for a General Rate-controlled Packet Servce (GRPS) scheduler that uses some form of tmestamp roundng to reduce algorthmc complexty and analyze the schedulng propertes of the general model. The basc model and Lemmas 1-4 are defned entrely n terms of byte tme, ndependent of the lnk speed or real tme. Therefore, the model apples to fxed and varable btrate lnks. A. System Model The system model s a generalzaton of the analytcal model used for SI-WF 2 Q [5], whch n turn s based on Bennett and Zhang s work on WF 2 Q+ [19] as well as Sur et al. s work on LFVC [4]. A GRPS scheduler system s comprsed of a set of flows F. Each flow s allocated a postve rate fracton wth 1. (1) F Flows are assgned start and fnsh tme labels accordng to the frst packet p n ther respectve queue and the current system vrtual tme V : { S = S p max(v, F p 1 = ) arrval to empty queue F p 1 otherwse F = F p = S p + lp r for next packet wth length l p (3) We assume the exstence of per-flow roundng functons h S (x) for start tmes and hf (x) for fnsh tmes, such that the roundng erros lmted. In partcular, start tmes are rounded down and fnsh tmes are rounded up. It has been observed before that rounded tmestamps work well n ths case [4], [26]. The roundng functons are characterzed by maxmum roundng errors S and F as (2) Ŝ = h S (S ) wth S Ŝ S S, and (4) ˆF = h F (F ) wth F ˆF F + F. (5) We denote wth B the set of backlogged flows. A busy perod s defned as a perod where B s contnuously not empty. In contrast to WFQ or WF 2 Q, but smlar to WF 2 Q+ or LFVC, vrtual tme progresses wth the real lnk speed, but t jumps forward when necessary, so that t never falls behnd the smallest rounded start tme. After servng packet p wth sze l p, vrtual tme ncreases as V = max(v + l p, mn B (Ŝ)). (6) Backlogged flows are separated nto blocked and actve flows, dependng on ther respectve start tmes n relaton to the system vrtual tme. The followng requrements must be satsfed by the schedulng algorthm: Requrement 1: Flow blocked V S. Requrement 2: Flow actve V Ŝ. The flow selecton polcy can be consdered as rounded SEFF,.e., among the actve flows, the one wth the smallest rounded fnsh tme ˆF s chosen for servce. We show that based on these defntons and assumptons, the terms S and F are suffcent to characterze the schedulng propertes of any specfc GRPS nstance n relaton to WF 2 Q. B. Analyss Our analyss closely follows the proof structure from prevous work [5]. However, the analyss s more general and apples to a whole class of schedulers. We have fxed a number of mnonaccuraces and mprove the relatve farness bound gven before [5]. Lemma 1: For a set of flows G wth G : S V, the followng nequalty holds for all V V : l + G(V F ) V V (7) G wth l beng the sze of the frst packet n flow s queue. Proof: By defnton, l = (F S ) for all flows and thus, l (F V ) for flows wth S V. Therefore, l V ) (8) G(F G = (V V ) G G(V F ) (9) (V V ) G(V F ) (10) whch drectly leads to the lemma. The frst step uses F V = V V (V F ) and the second step uses G 1, based on (1). Lemma 2: At any vrtual tme V, the Backlog Inequalty holds for all vrtual tme values V wth L + V V : l + (V F ) L + V V (11) : ˆF V : ˆF V wth l beng the sze of the frst packet n flow s queue and L beng the maxmum transmsson unt (MTU). If a flow j s not backlogged, l j = 0 and F j V. Proof: The proof s by nducton over those events that change varables. The base case s trval. Packet enqueue: Denote the sze of new packet p wth l p and the fnsh tme after enqueue wth F p p. If ˆF > V, nothng changes and the lemma holds. Otherwse, snce ˆF p V after the arrval of the packet, ˆF V before the packet arrval and the flow s already part of the set. In ths case, the frst term on the left sde of (11) s ncremented by l p, but snce the flow s fnsh tme F s ncremented by the vrtual packet tme l p, the second term left s reduced by l p. Vrtual tme jump: After a vrtual tme jump as n (6), all flows n the system have S Ŝ V and Lemma 1 apples for all V V. For V n [V L, V [, the addtonal L term n (11) absorbs any decrement on the rght hand sde. Therefore, the lemma holds. Packet servce: Assume flow j receves servce. Denote the sze of the current packet for servce wth l j, flow rate wth

6 TECHNICAL REPORT CS , UNIVERSITY OF WATERLOO - EXTENDED VERSION OF ACM/IEEE TON PAPER 6 r j, and rounded fnsh tme wth ˆF j. We have to dstngush two cases, dependng on V and ˆF j. Case 1: If V ˆF j, then the frst term on the left sde of (11) s decremented by l j. Snce V s ncremented by l j, the rght sde of (11) s decremented by the same amount. Therefore, the lemma holds. For the next packet n flow j s queue, the case Packet enqueue apples. Case 2: If V < ˆF j, then all flows wth ˆF V have ˆF < ˆF j. All of these flows must have been blocked, otherwse the current packet would not have been chosen. Therefore, Requrement 1 apples and S V holds for all flows wth ˆF V before the packet servce. Assume the vrtual tme before servce s V 1 and after servce V 2. Lemma 1 apples then for all V V 1,.e. l + (V F ) V V 1. (12) : ˆF V : ˆF V Because V 2 = V 1 + l j and we assume L + V V 2 after servce, we only need to consder V wth L + V V 1 + l j before servce. Therefore V V 1 (L l j ) + V (V 2 l j ) = L + V V 2 (13) and the lemma holds after servce. The Servce Tme lemma establshes that a packet s served when the vrtual tme reaches ts rounded fnsh tme wth an error term of at most one maxmum packet sze L n addton to the roundng error. Lemma 3: For any backlogged flow V F + F + L. (14) Proof: We prove V ˆF + L, whch drectly mples the lemma. The proof s by contradcton. The only event that could lead to a volaton of the assumpton s servng a packet durng a busy perod. Assume that at V 1 the lemma holds. A packet p wth rounded fnsh tme ˆF 1 and length l p s served and afterwards at V 2, there s a packet q wth fnsh tme F 2, such that ˆF2 + L < V 2. Denote wth S 1 and S 2 the correspondng start tmes. We need to dstngush three cases. Case 1: Packet q s actve at V 1. Then, ˆF2 ˆF 1 (both packets were elgble at V 1 and p was chosen). Applyng Lemma 2 wth V = V 1 and V = ˆF 2 results n l + ( ˆF 2 F ) L + ˆF 2 V 1 (15) : ˆF ˆF 2 : ˆF ˆF 2 Because F ˆF, the second term on the left sde of the nequalty s non-negatve and therefore l p l L + ˆF 2 V 1 (16) : ˆF ˆF 2 V 2 V 1 L + ˆF 2 V 1 (17) V 2 ˆF 2 + L (18) The step from (16) to (17) uses V 1 + l p = V 2. Case 2: Packet q s not actve at V 1, but becomes actve between V 1 and V 2. Then, Ŝ2 V 1. Vrtual tme advances by at most L and therefore: ˆF 2 Ŝ2 V 1 V 2 L (19) Case 3: Packet q s not actve after servce to p, therefore V 2 s reached by a vrtual tme jump before q can be served. In ths case: ˆF 2 Ŝ2 V 2 V 2 L (20) Ths concludes the proof. Fnally, we need to establsh a bound between vrtual tme V and real tme R, assumng a fxed lnk capacty. Lemma 4: Let I be the last tme when the system was dle and let J be the amount of vrtual tme jumpng that has been done durng the current busy perod. Then, whenever a packet has completed servce: V + I J = R. (21) Proof: At the begnnng of a busy perod, I = R and V = J = 0. Whenever a packet s chosen for servce and transmtted, V and R ncrease by the same amount. When vrtual tme jumps forward, V and J ncrease by the same amount. Usng the above lemmas, we can now prove the man theorems descrbng the servce characterstcs of GRPS schedulers n general. Theorem 1: (End-to-End Delay) GRPS s a Guaranteed Rate (GR) scheduler [7] wth an error term β for flow as β L + F (22) Proof: Let p j and lj denote the jth packet of flow and ts sze. Let A R (p j ) denote the real-tme arrval tme of packet. The guaranteed rate clock values are defned as s [7]: p j GRC(p j ) = max(a R(p j ), GRC(pj 1 )) + lj (23) wth GRC(p 0 ) = 0. Denote wth F j the fnsh tme of the jth packet. Let I(p j ) and J(pj ), respectvely, be the values of I and J from Lemma 4 when servce of p j s completed. Let I (p j ) and J (p j ), respectvely, be the values of I and J when pj arrves at the head of ts queue. We frst prove that durng a busy perod F j + I(pj ) J(pj ) GRC(pj ) (24) We prove (24) by nducton on j. The base case s trval. The vrtual tme of a packet arrval s denoted by A(p j ). Inductve Step: Assume (24) holds for j 1. We need to dstngush two cases. Case 1: A(p j ) > F j 1. Then F j = A(pj ) + lj. (25) Usng Lemma 4 we can characterze the packet arrval tme as A(p j ) + I (p j ) J (p j ) A R(p j ) (26) Addng lj results n A(p j ) + lj + I (p j ) J (p j ) A R(p j ) + lj. (27)

7 TECHNICAL REPORT CS , UNIVERSITY OF WATERLOO - EXTENDED VERSION OF ACM/IEEE TON PAPER 7 Rearrangng the terms usng (25), and replacng A R by a trval maxmum expanson, results n F j +I (p j ) J (p j ) max(a R(p j ), GRC(pj 1 ))+ lj. (28) The rght hand sde shows the defnton of GRC. Snce we are consderng a busy perod, I (p j ) = I(pj ) and J (p j ) J(p j ). Therefore F j + I(pj ) J(pj ) GRC(pj ), (29) whch establshes (24) for Case 1. Case 2: A(p j ) F j 1. In ths case, F j Through the nducton hypothess, we obtan F j 1 + I(p j 1 = F j 1 + lj. ) J(p j 1 ) GRC(p j 1 ) (30) Addng lj and replacng GRC by a trval maxmum expanson results n ), GRC(pj 1 )) + lj. (31) The rght hand sde shows the defnton of GRC. Snce we are F j + I(pj 1 ) J(p j 1 ) max(a R (p j consderng a busy perod, I(p j 1 J(p j ). Therefore ) = I(p j ) and J(pj 1 ) F j + I(pj ) J(pj ) GRC(pj ) (32) whch establshes (24) for Case 2. We can now prove the theorem. A packet p j s served no later than F j +F +L (Lemma 3). At the end of transmsson, the real tme equals F j +F +L+I(p j ) J(pj ) by Lemma 4. By (24), ths s bound by GRC(p j ) + F + L. Theorem 2: (Relatve Farness) The relatve farness [10] of GRPS between any two flows and j s bound by θ,j wth θ,j 2L + ( L x + S x + F x ). (33) r x x=,j wth L x beng the maxmum packet sze for flow x. Proof: We can determne the earlest and latest possble fnsh tmes for a packet from flow that receves servce at a vrtual tme V as follows: F F mn = V F L (34) F F max = V + S + L (35) F mn follows from Lemma 3 and F max follows from Requrement 2. Consequently, for any nterval [V 1, V 2 ] durng whch a flow s backlogged ts maxmum servce s bound by V 2 V 1 + F max F mn, whle the mnmum servce s bound by V 2 V 1 + F mn F max. The maxmum devaton between two backlogged flows and j can be computed by subtractng both terms from each other: (F max F mn ) (Fj mn Fj max ) (36) Insertng (34) and (35) nto (36) gves the lemma. Theorem 3: (Worst-case Farness) Let Q (p) be the backlog of an arbtrary flow mmedately after packet p arrves. The tme δ to clear ths backlog s bound as δ Q + L + L + S + F. (37) Proof: Consder a packet p for flow. Suppose that mmedately after p arrves, the packet at the head of flow s queue s p j. Further assume that there are m 0 packets n the queue n front of p, meanng p = p j+m. Let R 1 denote the real arrval tme and R 2 the real tme when p s servce s complete. Let V 1, V 2 denote the vrtual tmes correspondng to the real tmes R 1 and R 2, and F 1, F 2 the flow s fnsh tmes at R 1 and R 2, respectvely. (35) guarantees the followng nequalty: or F 1 V 1 + L + S (38) V 1 F 1 + L + S. (39) Lemma 3 gves the bound V 2 F 2 + F + L. Subtractng V 1 from V 2,.e. addng (39) to the bound for V 2, results n V 2 V 1 F 2 F 1 + L + L + S + F. (40) Snce flow s backlogged durng the nterval [F 1, F 2 ], we get F 2 = F 1 + m l j+n n=1 r, whch can be nserted nto (40): m l j+n V 2 V 1 + L + r L + S + F. (41) n=1 The queue sze Q (p) satsfes m lr j+n Q (p). (42) n=1 Pluggng ths nto (41) yelds V 2 V 1 Q (p) + L + L + S + F. (43) Usng the bounds between vrtual tme and real tme from Lemma 4, we can obtan a lower bound for R 1 and an upper bound for R 2 : V 1 + I J 1 R 1 and V 2 + I J 2 R 2 (44) wth J 2 J 1. Therefore R 2 R 1 V 2 + I J 2 (V 1 + I J 1 ) (45) = V 2 V 1 (J 2 J 1 ) (46) Q (p) + L + L + S + F (J 2 J 1 ). (47) Snce J 2 J 1 0, ths concludes the proof. Note that the relatve farness bound between two flows s tghter than reported earler [5]. It s the sum of error terms of the absolute farness bounds of both flows. Ths s consstent wth ntuton and prevous fndngs [34]. In comparson to the propertes of WF 2 Q [19], the addtonal error terms can be drectly related to the tmestamp roundng errors S and F.

8 TECHNICAL REPORT CS , UNIVERSITY OF WATERLOO - EXTENDED VERSION OF ACM/IEEE TON PAPER 8 Level 1 Level 2 Level 3 Level 4 Level 5 Level 6 Level 7 Level K Fg. 1. tme n slots Interleaved Stratfed Tmer Wheels (ISTW) bucket/slot numbers f u n c t o n ISTW : : n s e r t ( s l o t, elem ) { k = f f s ( s l o t ) ; l e v e l C o u n t [ k 1] += 1 ; s e t b t ( l e v e l b t s, k ) ; g e t B u c k e t ( s l o t ). push back ( elem ) ; Fg. 2. ISTW Insert Operaton IV. INTERLEAVED STRATIFIED TIMER WHEELS The key challenge resultng from the SEFF polcy (cf. Secton II-A) s that the next flow for servce s chosen from the set of backlogged flows based on two crtera at the same tme: the smallest fnsh tme among those flows wth a start tme less or equal the scheduler s current vrtual tme. For WF 2 Q+ [19], t s proposed to use a tree-based data structure to accommodate both crtera, whle LFVC [4] has ntroduced the dea of two contaners to hold blocked vs. actve flows. Any tree-based data structure can only be mantaned wth logarthmc complexty, whle the two-contaner soluton suffers from a transfer problem, f the system vrtual tme (6) crosses the start tme threshold of many flows n one step. We propose Interleaved Stratfed Tmer Wheels (ISTW) as a prorty queue that supports effcent searchng for the nearest future event and controlled scannng for past events, f necessary. For rate-proportonal schedulng, one ISTW contanes used to sort fnsh tmes and search for the next flow to servce. To facltate the SEFF polcy, a second ISTW nstance keeps flows sorted by start tmes. Besdes searchng for the next start tme, f necessary for (6), ISTW supports scannng for past start tmes wth a meanngful bound for lateness, but wthout the need to transfer many flows at the same tme. An ISTW contanes a collecton of tmer wheels where tme s measured n fxed base tme slots and the bucket wdth s doubled for each level. The top-most wheel n Level 1 has a bucket wdth of two base tme slots. Furthermore, the tmer wheels are nterleaved, such that buckets are never algned wth each other across levels. Ths s shown n Fgure 1. Each bucket can be dentfed by the number of the frst tme slot that t covers, so that the correspondng rounded slot or bucket number for a slot j n level k, k 1, can be computed as j 2 h k (j) = 2 k k k 1. (48) We observe that the roundng erros lmted by 2 k j < h k (j) + 2 k. (49) The fnd-frst-set (ffs) operaton can be used to determne the level of a gven bucket number. The ffs operaton fnds the poston of the least sgnfcant bt n a word. It can be mplemented n software at logarthmc cost of the word length [35]. Further, a prorty encoder can be mplemented n hardware at a very low cycle cost. For example, recent Intel processors mplement the ffs operaton (termed BFS) at 1-3 clock cycles at Ggahertz clock rates, dependng on the archtecture [36], whle the IXP network processor provdes a one-cycle ffs nstructon [37]. Level 1 Level 2 Level 3 Level 4 Level 5 Level 6 Level 7 Level K Fg. 3. start search current tme slot tme n slots occuped occuped Zgzag Search n ISTW An ISTW contaner mantans a btmask levelbts to ndcate whether any bucket n a partcular level s occuped. The nsert operaton s shown n Fgure 2. It assumes a functon getbucket that retreves the bucket correspondng to a gven rounded tme slot number. Searchng for the next occuped slot n an ISTW contaner works by frst locatng the smallest occuped level numben the ISTW contaner by usng the ffs operaton on levelbts and then performng a lnear search of buckets. Because the tmer wheels are nterleaved, teratng through bucket numbers wth a step wdth of half the bucket sze of the smallest occuped level results n a search pattern where the smallest occuped level s searched lnearly, but every second search step consders a bucket from a bgger numbered level. Ths search pattern ensures that any bgger-level bucket along the way s vsted as well. In Fgure 3, the complete search path s llustrated, although n ths example the search would termnate at Bucket 12. Assumng a functon roundslot that mplements h k from (48), the search functon s descrbed by the pseudo-code n Fgure 4. It s easy to see that the worstcase executon complexty of ths lnear search s proportonal to the number of buckets between the current tme and the frst element n the smallest occuped level. We defne the current bucket for each level as the bucket that overlaps wth the current tme slot. The scan operaton for an ISTW contaner retreves an element from the smallest occuped level that has an element n ts current bucket. Ths s done by mantanng a btmask frontbts that ndcates the occupancy of the current buckets for all levels. The pseudocode s shown n Fgure 5. Smlar data structures for sortng and searchng have been used for other schedulers, such as SRR [28], STRR [30], or GRR [32], all of whch are dscussed n Secton II. However, snce all these proposals follow the round robn approach, they do not consder a comprehensve combnaton of searchng and scannng on both start and fnsh tmes. In partcular, nterleavng the tmer wheels s an essental feature n ISTW for mantanng frontbts, as dscussed n SectonV-B. bucket/slot numbers

9 TECHNICAL REPORT CS , UNIVERSITY OF WATERLOO - EXTENDED VERSION OF ACM/IEEE TON PAPER 9 f u n c t o n ISTW : : s e a r c h ( s l o t ) { f (! l e v e l b t s ) return NULL; k = f f s ( l e v e l b t s ) ; s l o t = r o u n d S l o t ( s l o t, k ) ; s t e p = (1 << ( k 1 ) ) ; whle ( g e t B u c k e t ( s l o t ). empty ( ) ) { s l o t += s t e p ; k = f f s ( s l o t ) ; l e v e l C o u n t [ k 1] = 1 ; f ( l e v e l C o u n t [ k 1] == 0 ) c l e a r b t ( l e v e l b t s, k ) ; return g e t B u c k e t ( s l o t ). p o p f r o n t ( ) ; Fg. 4. ISTW Search Operaton f u n c t o n ISTW : : scan ( s l o t ) { k = f f s ( s l o t ) ; f (! g e t B u c k e t ( s l o t ). empty ( ) ) s e t b t ( f r o n t b t s, k ) ; f (! f r o n t b t s ) return s e a r c h ( s l o t ) ; k = f f s ( f r o n t b t s ) ; f ( g e t B u c k e t ( s l o t ). s z e ( ) == 1 ) c l e a r b t ( f r o n t b t s, k ) ; l e v e l C o u n t [ k 1] = 1 ; f ( l e v e l C o u n t [ k 1] == 0 ) c l e a r b t ( l e v e l b t s, k ) ; return g e t B u c k e t ( s l o t ). p o p f r o n t ( ) ; f u n c t o n enqueue ( p k t ) { flow = c l a s s f y ( p k t ) ; flow.q. push back ( p k t ) ; f ( flow.q. s z e ( ) == 1 ) { f ( flow. S < V ) flow. S = V; e l s e f (! A c t v e. l e v e l b t s ) V = flow. S ; n s e r t F l o w ( flow ) ; Fg. 6. Scheduler Enqueue Operaton f u n c t o n dequeue ( ) { Vf = Vs = V/ g ; f ( manq. empty ( ) ) f (! A c t v e. l e v e l b t s ) return ; flow = A c t v e. s e a r c h ( Vs ) ; e l s e { flow = manq. p o p f r o n t ( ) ; p k t = flow.q. p o p f r o n t ( ) ; t r a n s m t ( p k t ) ; / / n t h e background V += p k t. s z e ; flow. S = flow. F ; f (! flow.q. empty ( ) ) n s e r t F l o w ( flow ) ; t r a n s f e r ( ) ; f o r ( ; Vf < V/ g ; Vf += 1 ) { manq. append ( A c t v e. g e t B u c k e t ( Vf ) ) ; Fg. 5. ISTW Scan Operaton Fg. 7. Scheduler Dequeue Operaton A. Basc Operatons V. SCHEDULER DESIGN SI-WF 2 Q, frst descrbed n [5], uses the ISTW data structure to mplement a two-contaner soluton for rate-based schedulng. We frst show the pseudo-code for the enqueue and dequeue operatons n here, before dscussng ts executon complexty and analyzng ts schedulng propertes usng the general model from Secton III. The two contaners are termed Actve and Blocked wth obvous semantcs. We assume a functon nsertflow that computes the fnsh tme F accordng to (3), rounded tmestamps Sx and Fx, and decdes whether to nsert a new or returnng flow nto the Blocked or the Actve contaner. V denotes the system vrtual tme. The enqueue operaton, shown n Fgure 6 reconcles the flow s start tme wth the system vrtual tme, before nsertng a new flow. If a flow s start tme s old, Lne 6 mplements the max functon from (2). Otherwse, f the system s empty (Lne 5), the vrtual tme s mmedately set to the flow s start tme accordng to (6), whch effectvely resets the system. The dequeue operaton, shown n Fgure 7 chooses the next flow for servce and updates state varables as needed. Flows are not necessarly taken from the Actve contaner drectly, but nstead for each ncrease n vrtual tme, the Actve contanes scanned for flows that have an expred fnsh tme and these flows are moved to a central servce queue manq. Ths s a safe operaton, snce any packets that arrve later wll have ther fnsh tmes set to a value greater than the system vrtual tme. Only f scannng comes up empty,.e., the next avalable fnsh tme s greater or equal than the current vrtual tme, the Actve contanes searched. Ths procedure s further dscussed n the next secton. The transfer operaton, shown n Fgure 8 mplements the transfer of flows that become elgble durng the next packet s transmsson, as well as a potental jump n vrtual tme correspondng to (6). Note that vrtual tme only jumps, f the Actve contanes empty, therefore t does not nterfere wth the scannng loop at the end of dequeue. These operatons usng two ISTW contaners collectvely mplement the system model from Secton III. The tme perod represented by one slot s denoted by the constant g. To analyze the schedulng propertes, complance wth Requrements 1 and 2 needs to be shown. However, we frst dscuss the executon complexty of the operatons.

10 TECHNICAL REPORT CS , UNIVERSITY OF WATERLOO - EXTENDED VERSION OF ACM/IEEE TON PAPER 10 f u n c t o n t r a n s f e r ( ) { whle ( Vs < V/ g &&! Blocked. empty ( ) ) { f (! A c t v e. l e v e l b t s &&! Blocked. f r o n t b t s ) { tmp = Blocked. s e a r c h ( Vs ) ; Vs = tmp. Sx ; e l s e { tmp = Blocked. scan ( Vs ) ; Vs += 1 ; f ( tmp ) A c t v e. n s e r t ( tmp. Fx, tmp ) ; f ( Vs > V/ g ) V = Vs g ; Fg. 8. Scheduler Transfer Operaton B. Executon Complexty The scheduler uses stratfed rounded tmestamps n combnaton wth the ISTW data structure to keep the executon complexty low. Stratfcaton groups smlar flows based on ther servce rates and potentally other characterstcs. The executon complexty of rate-based packet schedulers s typcally characterzed n relaton to the number of flows n the system, whch mplctly assumes that per-packet operatons have a small constant executon overhead. However, n certan exstng schedulers, the per-packet operatons do not have truly constant per-packet overhead, but occasonally need to execute loops wth the worst-case number of teratons beng on the order of the number of flows n the system. Snce these occurrences are rare enough, ther cost s amortzed over tme and termed amortzed constant overhead. However, gven the tght tmng requrements wth whch packets have to be released to a hgh-speed output lnk, t s not possble to ensure that the output lnk s always fully utlzed when the runtme of the per-packet operatons cannot be tghtly controlled. Examples of schedulers wth ths property are LFVC and those round robn schedulers that are subject to slp processng, as dscussed n Secton II. In ths work, we use a slghtly dfferent noton of constant complexty, whch works well for packet schedulers. A packet scheduler typcally needs to be desgned for a worst-case traffc mx of only mnmum-szed packets. If the cost of processng a packet s proportonal to the sze of the packet, then the ncreased cost for a larger packet s drectly offset by the fact that a larger packet keeps the lnk busy for a longer tme. Thereby, only the amortzed cost for processng a packet s constant, but n contrast to long-term amortzaton as dscussed above, the amortzaton perod s only one packet. We denote ths as packet-amortzed constant complexty. In partcular, for the schedulers presented below we propose to set the length of a base tme slot n the ISTW contaner to the mnmum supported packet sze. We then only need to consder the loops for assessng the executon complexty. The flow transfer loop n transfer executes n proporton to the ncrease n vrtual tme caused by the packet that has just been chosen for servce and thus, clearly satsfes the requrements of packet-amortzed constant complexty. Ths operaton s the essental motvaton fonterleavng the tmer wheels n ISTW. Only because of nterleavng t s possble to mantan frontbts smoothly,.e., for each slot exactly one bucket n one level needs to be checked. The overhead of the lnear search n an ISTW contaner depends on the dstance n buckets between the startng pont and the tmestamp found (cf. Secton IV). In case of the Actve contaner, Equatons (34) and (35) lmt the range of rounded fnsh tmes at any vrtual tme V to [V L, V +S +F + L ] for any flow. Assumng that S and F are constant and not sgnfcantly larger than L, ths drectly translates nto packet-amortzed constant overhead, because the search cost s mmedately amortzed by transmttng the packet. The only caveat s that the search has to start at V L, and L may be relatvely large compared to the other parameters, especally when comparng to small packets from hgh rate flows. Therefore, the dequeue operaton s enhanced by scannng the Actve contaner for expred fnsh tmes that are smaller than the current vrtual tme. The overhead of the scannng loop at the end of dequeue s proportonal to the sze of the packet beng transmtted. Ths guarantees that a search n Actve, f necessary, can start at the vrtual tme. Smlar reasonng can be used for the Blocked contaner. The mnmum start tme s gven by Requrement 1 and the maxmum start tme follows from (35) by assumng that a flow s renserted nto the blocked contanemmedately after packet servce. Thereby, the range of rounded start tmes n Blocked s lmted to [V S, V + L ]. The part [V S, V ] s covered by scannng and transfer, so that Blocked only needs to be searched, f the next start tme s greater or equal than the current vrtual tme. However, there s a key dfference to the fnsh tme search dscussed above. The lnear overhead of the search (from the L component) corresponds to the prevous packet of flow, rather than the next one. The prevous packet that causes a search mght already be transmtted and thus, search overhead and packet transmsson cannot be drectly related as before. However, the search overhead per flow s lmted to the equvalent of L n buckets and clearly, multple flows n the same level only ncrease the densty of tmestamps and do not add up to a hgher search overhead. Snce the bucket sze s proportonal to 1, an overall amortzaton buffer of nl max, for n levels and an MTU sze of L max, s suffcent to absorb ths worst-case effect of the start search overhead. C. Schedulng Propertes Requrement 2 s satsfed, because the dequeue operaton only transfers flows wth a rounded start tme greater or equal the vrtual tme through scan, or sets the vrtual tme to a flow s start tme when usng search. We denote the tme slot perod wth λ and show that Requrement 1 s satsfed, f the followng two condtons are met: For any flow n level k > 1, and 2k (50) S 2 k λ (51)