FAST TCP: Motivation, Architecture, Algorithms, Performance

Transcription

1 TCP: Motvaton, Archtecture, Algorthms, Performance Davd X. We Cheng Jn Steven H. Low Sanjay Hegde Engneerng & Appled Scence, Caltech Abstract We descrbe TCP, a new TCP congeston control algorthm for hgh-speed long-latency networks, from desgn to mplementaton. We hghlght the approach taken by TCP to address the four dffcultes, at both packet and flow levels, whch the current TCP mplementaton has at large wndows. We descrbe the archtecture and summarze some of the algorthms mplemented n our prototype. We characterze the equlbrum and stablty propertes of TCP. We provde expermental evaluaton of our frst prototype n terms of throughput, farness, stablty, and responsveness. I. INTRODUCTION AND SUMMARY Congeston control s a dstrbuted algorthm to share network resources among competng users. It s mportant n stuatons where the avalablty of resources and the set of competng users vary over tme unpredctably, yet effcent and far sharng s desred. These constrants unpredctable supply and demand and the desre for effcent dstrbuted operaton necessarly lead to feedback control as the preferred approach, where traffc sources dynamcally adapt ther rates to congeston n ther paths. On the Internet, ths s performed by the Transmsson Control Protocol (TCP) n source and destnaton computers nvolved n data transfers. The congeston control algorthm n the current TCP, whch we refer to as Reno n ths paper, was developed n [] and has gone through several enhancements snce, e.g., [], [], [], [], [], []. It has performed remarkably well and s generally beleved to have prevented severe congeston as the Internet scaled up by sx orders of magntude n sze, speed, load, and connectvty. It s also well-known, however, that as bandwdth-delay product contnues to grow, TCP Reno wll eventually become a performance bottleneck tself. The followng four dffcultes contrbute to the poor performance of TCP Reno n networks wth large bandwdth-delay products: ) At the packet level, lnear ncrease by one packet per Round-Trp Tme (RTT) s too slow, and multplcatve decrease per loss event s too drastc. ) At the flow level, mantanng large average congeston wndows requres an extremely small equlbrum loss probablty. ) At the packet level, oscllaton n congeston wndow s unavodable because TCP uses a bnary congeston sgnal (packet loss). ) At the flow level, the dynamcs s unstable, leadng to severe oscllatons that can only be reduced by the accurate estmaton of packet loss probablty and a stable desgn of the flow dynamcs. We explan these dffcultes n detal n Secton II, and motvate a delay-based soluton. Delay-based congeston control To appear n IEEE/ACM Transactons on Networkng,. An abrdged verson appears n [] and an expanded verson n []. has been proposed, e.g., n [], [], [], [], []. See [], [], [], [], [], [], [], [] for other recent proposals. Usng queueng delay as a congeston measure has two advantages. Frst, queueng delay can be more accurately estmated than loss probablty both because packet losses n networks wth large bandwdth-delay product are rare events under TCP Reno (e.g., probablty on the order or smaller), and because loss samples provde coarser nformaton than queueng delay samples. Indeed, measurements of delay are nosy, just as those of loss probablty. Each measurement of packet loss (whether a packet s lost) provdes one bt of nformaton for the flterng of nose, whereas each measurement of queueng delay provdes multbt nformaton. Ths makes t easer for an equaton-based mplementaton to stablze a network nto a steady state wth a target farness and hgh utlzaton. Second, based on the commonly used ordnary dfferental equaton model of TCP/AQM, the dynamcs of queueng delay has the rght scalng wth respect to network capacty. Ths helps mantan stablty as a network scales up n capacty [], [], []. In Secton III, we lay out an archtecture to mplement our desgn, and present an overvew of some of the algorthms mplemented n our current prototype. Even though the dscusson s n the context of TCP, the archtecture can also serve as a general framework to gude the desgn of other congeston control mechansms, not necessarly lmted to TCP, for hgh-speed networks. The man components n the archtecture can be desgned separately and upgraded asynchronously. We evaluate TCP both analytcally and expermentally. In Secton III-B, we present a mathematcal model of the wndow control algorthm. We prove that TCP has the same equlbrum propertes as TCP Vegas [], []. In partcular, t does not penalze flows wth large propagaton delays and t acheves weghted proportonal farness []. For the specal case of sngle bottleneck lnk wth heterogeneous flows, we prove that the wndow control algorthm of s locally asymptotcally stable, n the absence of feedback delay. In Secton IV, we present both expermental and smulaton results to llustrate throughput, farness, stablty, and responsveness of TCP, n the presence of delay and n heterogeneous and dynamc envronments where flows of dfferent delays jon and depart asynchronously. It s mportant to evaluate a congeston control algorthm not only n terms of throughput acheved, but also what t does to network queues and how that affects other applcatons sharng the same queue. We compare the performance of TCP wth Reno, H (HghSpeed TCP []), (Scalable TCP []), and [], usng ther default parameters.

2 In Secton V, we summarze open ssues and provde references for proposed solutons. II. MOTIVATIONS A congeston control algorthm can be desgned at two levels. The flow-level (macroscopc) desgn ams to acheve hgh utlzaton, low queueng delay and loss, farness, and stablty. The packet-level desgn mplements these flow level goals wthn the constrants mposed by end-to-end control. Hstorcally for TCP Reno, packet-level mplementaton was ntroduced frst. The resultng flow-level propertes, such as farness, stablty, and the relatonshp between equlbrum wndow and loss probablty, were then understood as an afterthought. In contrast, the packet-level desgns of H [], [], and TCP are explctly guded by flow-level goals. A. Packet and flow level modelng The congeston avodance algorthm of TCP Reno and ts varants have the form of AIMD []. The pseudo code for wndow adjustment s: Ack: w w + w Loss: w w w Ths s a packet-level model, but t nduces certan flow-level propertes such as throughput, farness, and stablty. These propertes can be understood wth a flow-level model of the AIMD algorthm, e.g., [], [], [], []. The wndow w (t) of source ncreases by packet per RTT, and decreases per unt tme by x (t)q (t) w (t) packets where x (t) := w (t)/t (t) pkts/. T (t) s the round-trp tme, and q (t) s the (delayed) end-to-end loss probablty, n perod t. Here, w (t)/ s the peak wndow sze that gves the average wndow of w (t). Hence, a flow-level model of AIMD s: ẇ (t) = T (t) x (t)q (t)w (t) () Settng ẇ (t) = n () yelds the well-known / q formula for TCP Reno dscovered n [], [], whch relates loss probablty to wndow sze n equlbrum: q = w In summary, () and () descrbe the flow-level dynamcs and equlbrum, respectvely, for TCP Reno. Even though Reno, H,, and look dfferent at the packet level, they have smlar equlbrum and dynamc structures at the flow level; see [] for detaled It should be ( q (t)) packets, where q (t) s the end-to-end loss probablty. Ths s roughly when q (t) s small. Ths model assumes that wndow s halved on each packet loss. It can be modfed to model the case, where wndow s halved at most once n each RTT. Ths does not qualtatvely change the followng dscusson. () dervatons. The congeston wndows n these algorthms all evolve accordng to: ( ẇ (t) = κ (t) q ) (t) () u (t) where κ (t) := κ (w (t), T (t)) and u (t) := u (w (t), T (t)). They dffer only n the choce of the gan functon κ (w, T ), the margnal utlty functon u (w, T ), and the end-to-end congeston measure q. Wthn ths structure, at the flow level, there are thus only three desgn decsons: κ (w, T ): the choce of the gan functon κ determnes the dynamc propertes such as stablty and responsveness, but does not affect the equlbrum propertes. u (w, T ): the choce of the margnal utlty functon u determnes equlbrum propertes such as the equlbrum rate allocaton and ts farness. q : n the absence of explct feedback, the choce of congeston measure q s lmted to loss probablty or queueng delay. The dynamcs of q (t) s determned nsde the network. At the flow level, a goal s to desgn a class of functon pars, u (w, T ) and κ (w, T ), so that the feedback system descrbed by (), together wth lnk dynamcs of q (t) and the nterconnecton, has an equlbrum that s far and effcent, and that the equlbrum s stable, n the presence of feedback delay. The desgn choces n, Reno, H, and are shown n Table I. These choces produce equlbrum κ (w, T ) u (w, T ) q γα /τ α /x queueng delay Reno /T./w loss probablty.b(w )w. H./w. ( b(w ))T loss probablty aw /T ρ/w loss probablty TABLE I COMMON DYNAMIC STRUCTURE: w IS SOURCE S WINDOW SIZE, T IS ITS ROUND-TRIP TIME, q IS CONGESTION MEASURE, x = w /T ; a, b(w ), ρ, γ, α, τ ARE PROTOCOL PARAMETERS; SEE []. characterzatons shown n Table II. x = α q Reno x = α T q. H x = α T q. x = T α q TABLE II COMMON EQUILIBRIUM STRUCTURE: x IS SOURCE S THROUGHPUT IN PACKETS/SEC, T IS EQUILIBRIUM ROUND-TRIP TIME, q IS END-TO-END CONGESTION MEASURE IN EQUILIBRIUM. THE PARAMETERS ARE: α =. FOR RENO, α =. FOR H, AND α =. FOR. FOR, α SHOULD VARY WITH LINK CAPACITY. We next llustrate the equlbrum and dynamcs problems of TCP Reno, at both the packet and flow levels, as bandwdth-delay product ncreases. B. Reno s problems at large wndow The equlbrum problem at the flow level s expressed n (): the end-to-end loss probablty must be exceedngly small to sustan a large wndow sze, makng the equlbrum dffcult to mantan n practce, as bandwdth-delay product

3 ncreases. Indeed, from (), q w =./w,.e., the average number of packet losses (or loss events) per wndow decreases n nverse proporton to the equlbrum wndow sze for Reno. From Table II, ths number for H s q w =./w.. Hence t also decreases wth the equlbrum wndow, but more slowly than for TCP Reno. For, ths number s q w = a( b/)/b, whch s ndependent of, and hence scalable wth, the equlbrum wndow sze. The recommended values n [] for the constants are a =. and b =., yeldng an average loss of. per wndow. Even though equlbrum s a flow-level noton, ths problem wth Reno manfests tself at the packet level, where a source ncreases ts wndow too slowly and decreases t too drastcally. In contrast, H and ncrease more aggressvely and decrease less drastcally. The causes of the oscllatory behavor of TCP Reno le n ts desgn at both the packet and flow levels. At the packet level, the choce of bnary congeston sgnal necessarly leads to oscllaton n congeston wndows and bottleneck queues, and the parameter settng n Reno worsens the stuaton as bandwdth-delay product ncreases. At the flow level, the system dynamcs gven by () s unstable at large bandwdthdelay products [], []. These must be addressed by dfferent means. Congeston wndow can be stablzed only f mult-bt feedback s used. Ths s the approach taken by the equatonbased algorthm n [], where congeston wndow s adjusted based on the estmated loss probablty n an attempt to stablze around a target value gven by (). Ths approach elmnates the oscllaton due to packet-level AIMD, but two dffcultes reman at the flow level. Frst, equaton-based control requres the explct estmaton of end-to-end loss probablty. Ths s dffcult when the loss probablty s small. Second, even f loss probablty can be perfectly estmated, Reno s flow dynamcs, descrbed by equaton () leads to a feedback system that becomes unstable as feedback delay ncreases, and more strkngly, as network capacty ncreases [], []. The nstablty at the flow level can lead to severe oscllatons that can be reduced only by stablzng the flow level dynamcs. We present a delay-based approach to address these problems. C. Delay-based approach The common model () can be nterpreted as follows: the goal at the flow level s to equalze margnal utlty u (t) wth the end-to-end measure of congeston q (t). Ths nterpretaton mmedately suggests an equaton-based packet-level mplementaton where the wndow adjustment ẇ (t) depends on not only the sgn, but also the magntude of the dfference between the rato q (t)/u (t) and the target of. Unlke the approach taken by Reno, H, and, ths approach elmnates packet-level oscllatons due to the bnary nature of congeston sgnal. It however requres the explct estmaton of the end-to-end congeston measure q (t). Wthout explct feedback, q (t) can only be loss probablty, as used n TFRC [], or queueng delay, as used n TCP Vegas [] and TCP. Queueng delay can be more accurately estmated than loss probablty both because loss samples provde coarser nformaton than queueng delay samples, and because packet losses n networks wth large bandwdth-delay products tend to be rare events under See [] for dscusson on congeston sgnal and decson functon. schemes such as Reno. Indeed, each measurement of packet loss (whether a packet s lost) provdes one bt of nformaton for the flterng of nose, whereas each measurement of queueng delay provdes mult-bt nformaton. Ths facltates an equaton-based mplementaton to stablze a network nto a steady state wth a target farness and hgh utlzaton. At the flow level, the dynamcs of the feedback system must be stable n the presence of delay, as the network capacty ncreases. Here, agan, queueng delay has an advantage over loss probablty as a congeston measure: the dynamcs of queueng delay have the rght scalng wth respect to network capacty, accordng to the commonly used ordnary dfferental equaton model. Ths helps mantan stablty as network capacty grows [], [], [], []. It has been found that delay and packet loss can have a weak correlaton, e.g., [], especally when packet losses can be caused by other reasons than buffer overflow. Ths does not mean that t s futle to use delay as a measure of congeston, but rather, that usng delay to predct loss n the hope of helpng a loss-based algorthm adjust ts wndow s a wrong approach to address problems at large wndows. A dfferent approach that fully explots delay as a congeston measure, augmented wth loss nformaton, s needed. Ths motvates the followng mplementaton strategy. Frst, by explctly estmatng how far the current state q (t)/u (t) s from the equlbrum value of, a delay-based scheme can drve the system rapdly, yet n a far and stable manner, toward the equlbrum. The wndow adjustment s small when the current state s close to equlbrum and large otherwse, ndependent of where the equlbrum s. Ths s n stark contrast to the approach taken by Reno, H, and, where wndow adjustment depends on just the current wndow sze and s ndependent of where the current state s wth respect to the target (compare Fgures (a) and (b) n []). Lke the equaton-based scheme n [], ths approach avods the problem of slow ncrease and drastc decrease n Reno, as the network scales up. Second, by choosng a mult-bt congeston measure, ths approach elmnates the packet-level oscllaton due to bnary feedback, avodng Reno s thrd problem. Thrd, usng queueng delay as the congeston measure q (t) allows the network to stablze n the regon below the overflowng pont, when the buffer sze s suffcently large. Stablzaton at ths operatng pont elmnates large queueng delay and unnecessary packet loss. More mportantly, t makes room for bufferng mce traffc. To avod the ond problem n Reno, where the requred equlbrum congeston measure (loss probablty for Reno, and queueng delay here) s too small to practcally estmate, the algorthm must adapt ts parameter α to capacty to mantan small but suffcent queueng delay. Fnally, to avod the fourth problem of Reno, the wndow control algorthm must be stable, n addton to beng far and effcent, at the flow level. The emergng theory of large-scale networks under end-to-end control, e.g., [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [] (see also, e.g., [], [], [], [] for recent surveys), forms the foundaton of the flow-level desgn. The theory plays an mportant role by provdng a framework to understand ssues, clarfy deas, and suggest drectons, leadng to a robust and hgh performance mplementaton. In the next ton, we lay out the archtecture of TCP.

4 III. ARCHITECTURE AND ALGORITHMS A. Archtecture We separate the congeston control mechansm of TCP nto four components n Fgure. These four components are functonally ndependent so that they can be desgned separately and upgraded asynchronously. Fg.. Data Control Wndow Control Estmaton Burstness Control TCP Protocol Processng TCP archtecture. The data control component determnes whch packets to transmt, wndow control determnes how many packets to transmt, and burstness control determnes when to transmt these packets. These decsons are made based on nformaton provded by the estmaton component. More specfcally, the estmaton component computes two peces of feedback nformaton for each data packet sent a multbt queueng delay and an one-bt loss-or-no-loss ndcaton whch are used by the other three components. Data control selects the next packet to send from three pools of canddates: new packets, packets that are deemed lost (negatvely acknowledged), and transmtted packets that are not yet acknowledged. Wndow control regulates packet transmsson at the RTT tmescale, whle burstness control works at a smaller tmescale. Burstness control smoothes out transmsson of packets n a flud-lke manner to track the avalable bandwdth. We employ two mechansms, one to supplement self-clockng n streamng out ndvdual packets and the other to ncrease wndow sze smoothly n smaller bursts. Burstness reducton lmts the number of packets that can be sent when an ack advances congeston wndow by a large amount. Wndow pacng determnes how to ncrease congeston wndow over the dle tme of a connecton to the target determned by the wndow control component. It reduces burstness wth a reasonable amount of schedulng overhead. For detals of these two mechansms, see [], []. An ntal prototype that ncluded some of these features was demonstrated n November at the SuperComputng Conference, and the expermental results were reported n []. In the followng, we explan n detal the desgn of the wndow control component. B. Wndow control algorthm reacts to both queueng delay and packet loss. Under normal network condtons, perodcally updates the congeston wndow based on the average RTT and average queueng delay provded by the estmaton component, accordng to: w mn {w, ( γ)w ( basertt + γ RTT ) } w + α where γ (, ], basertt s the mnmum RTT observed so far, and α s a postve protocol parameter that determnes the total number of packets queued n routers n equlbrum along the flow s path. The wndow update perod s ms n our prototype. We now provde an analytcal evaluaton of TCP. We present a model of the wndow control algorthm for a network of flows. We show that, n equlbrum, the vectors of source wndows and lnk queueng delays are the unque solutons of a par of optmzaton problems () (). Ths completely characterzes the network equlbrum propertes such as throughput, farness, and delay. We also present a prelmnary stablty analyss. We model a network as a set of resources wth fnte capactes c l, e.g., transmsson lnks, processng unts, memory, etc., to whch we refer to as lnks n our model. The network s shared by a set of uncast flows, dentfed by ther sources. Let d denote the round-trp propagaton delay of source. Let R be the routng matrx where R l = f source uses lnk l, and otherwse. Let p l (t) denote the queueng delay at lnk l at tme t. Let q (t) = l R lp l (t) be the roundtrp queueng delay, or n vector notaton, q(t) = R T p(t). Then the round trp tme of source s T (t) := d + q (t). Each source adapts ts wndow w (t) perodcally accordng to: ( ) d w (t) w (t + ) = γ d + q (t) + α + ( γ)w (t) () where γ (, ], at tme t. A key departure of our model from those n the lterature s that we assume that a source s send rate, defned as x (t) := w (t)/t (t), cannot exceed the throughput t receves. Ths s justfed because of self-clockng: wthn one round-trp tme after a congeston wndow s ncreased, packet transmsson wll be clocked at the same rate as the throughput the flow receves. See [] for detaled justfcaton and valdaton experments. A consequence of ths assumpton s that the lnk queueng delay vector, p(t), s determned mplctly by the nstantaneous wndow sze n a statc manner: gven w (t) = w for all, the lnk queueng delays p l (t) = p l for all l are gven by: R l w d + q (t) { = cl f p l (t) > c l f p l (t) = where agan q (t) = l R lp l (t). The next result says that the queueng delay s ndeed well defned. All proofs are relegated to the Appendx and []. Lemma : Suppose the routng matrx R has full row rank. Gven w = (w, ), there exsts a unque queueng delay vector p = (p l, l) that satsfes (). The equlbrum values of wndows w and delays p of the network defned by () () can be characterzed as follows. Consder the utlty maxmzaton problem max x () α log x s.t. Rx c () Note that () can be rewrtten as (when α (w, q ) = α, constant) w (t + ) = w (t) + γ (α x (t)q (t)) From [], TCP Vegas updates ts wndow accordng to w (t + ) = w (t) + T (t) sgn(α x (t)q (t)) where sgn(z) = f z <, f z =, and f z >. Hence can be thought of as a hgh-speed verson of Vegas.

5 and the followng (dual) problem: mn c l p l α log p l l R l p l () Theorem : Suppose R has full row rank. The unque equlbrum pont (w, p ) of the network defned by () () exsts and s such that x = (x := w /(d + q ), ) s the unque maxmzer of () and p s the unque mnmzer of (). Ths mples n partcular that the equlbrum rate x s α -weghted proportonally far. Theorem mples that TCP has the same equlbrum propertes as TCP Vegas [], []. Its equlbrum throughput s gven by x = α q () In partcular, t does not penalze sources wth large propagaton delays d. The relaton () also mples that, n equlbrum, source mantans α packets n the buffers along ts path [], []. Hence, the total amount of bufferng n the network must be at least α packets n order to reach the equlbrum. We now turn to the stablty of the algorthm. Theorem (Sngle-lnk heterogeneous-source): Suppose there s only a sngle lnk wth capacty c. Then the system defned by () () s locally asymptotcally stable. The basc dea of the proof s to show that the mappng from (scaled) w(t) to w(t + ) defned by () () has a Jacoban whose spectral radus s strctly less than, unformly n w; see Theorem n the Appendx. Hence w(t) converges locally to the unque equlbrum. The proof technque seems to be dfferent from those n the current lterature of TCP congeston control. It also reveals some nterestng global propertes of TCP at a sngle lnk. Corollary : Suppose there s only a sngle lnk wth capacty c. ) The equlbrum pont (w, p ) s gven by w = α + α c j α d, p = j c wth x = cα / j α j. ) Startng from any ntal pont (w(), p()), the lnk s fully utlzed,.e., equalty holds n (), after a fnte tme. ) The queue length s lower and upper bounded after a fnte tme. If all sources have the same propagaton delay, d = d for all, then the system converges n fnte tme. The stablty result reported here s lmted to local asymptotc stablty at a sngle lnk wth heterogeneous sources and feedback delay s gnored. In [], the local stablty result s extended to a multlnk network n the absence of feedback delay. Wth feedback delay, local stablty can be mantaned for the case of a sngle lnk, provded the heterogenety of the delays s small. Ths delayed stablty result s extended n [] to a multlnk network; furthermore, global stablty at a sngle lnk s establshed n the absence of delay usng a Lyapunov argument. These results are summarzed n []. In [], a condton s derved under whch a sngle-source sngle-lnk network s globally aysmptotcally stable under. α IV. PERFORMANCE We have conducted experments on our dummynet [] testbed comparng performance of varous new TCP algorthms as well as the Lnux TCP mplementaton. For more complex scenaros that are hard to relably emulate wth our dummynet testbed, we report some smulaton results on NS- []. The experment and smulaton results reported am to zoom n on specfc propertes of. These scenaros may be ncomplete or unrealstc. Experments n producton networks can be found n [], []. Other results not presented n ths paper due to space lmtaton are collected n [] and []. A. Testbed and kernel nstrumentaton Our testbed conssts of a sender and a recever both runnng Lnux, that are connected through an emulated router runnng dummynet under FreeBSD. Each testbed machne has dual Xeon. GHz CPUs, GB of man memory, and dual on-board Intel PRO/ Ggabt Ethernet nterfaces. We have tested these machnes to ensure each s able to acheve a peak throughput of Mbps wth the standard Lnux TCP protocol usng perf. The testbed router supports paths of varous delays and a sngle bottleneck capacty wth a fxed buffer sze. It has montorng capablty at the sender and the router. The recever runs multple perf snks wth dfferent port numbers for connectons wth dfferent RTTs. We confgured dummynet to create paths or ppes of dfferent RTTs,,,, and ms, usng dfferent destnaton port numbers on the recevng machne. We then created another ppe to emulate a bottleneck capacty of Mbps and a buffer sze of, packets, shared by all the delay ppes. To reduce schedulng granularty, we recompled the FreeBSD kernel so the task scheduler runs every ms. We also ncreased the sze of the IP layer nterrupt queue to to accommodate large bursts of packets. For each connecton on the sendng machne, the kernel montor captures the congeston wndow, the observed basertt, and the observed queueng delay. On the dummynet router, the kernel montor captures the throughput at the bottleneck, the number of lost packets, and the average queue sze every two onds. We tested fve TCP mplementatons:, Reno (Lnux), H,, and BIC-TCP usng ther default parameters for all experments. The TCP s based on Lnux.. kernel (α s set to packets), H, Scalable TCP and Reno are based on Lnux.. kernel, was based on.. kernel. We ran tests and dd not observe any apprecable dfference among the three plan Lnux kernels, and the TCP source codes of the three kernels are nearly dentcal. Lnux TCP mplementaton ncludes all of the latest RFCs such as New Reno, SACK, D-SACK, and TCP hgh performance extensons. There are two versons of H [], []. We present the results of the mplementaton n [], but our tests show that the mplementaton n [] has comparable performance. In all of our experments, the bottleneck capacty s Mbps roughly packets/ms, and the maxmum buffer sze s packets. We now present our expermental results. We frst look at two cases n detal, comparng not only the throughput behavor seen at the source, but also the queue behavor nsde the network, by examnng trajectores of throughputs, nstantaneous queue, cumulatve losses, and lnk utlzaton.

6 We then summarze the overall performance n a dverse set of experments n terms of quanttatve metrcs, defned below, on throughput, farness, stablty, and responsveness. x B. Case study: dynamc scenaro I In the frst dynamc test, the number of flows was small so that throughput per flow, and hence the wndow sze, was large. There were three TCP flows, wth propagaton delays of,, and ms, that started and termnated at dfferent tmes, as llustrated n Fgure. x RENO RENO x ms x ms x ms Fg.. tme () Dynamc scenaro I ( flows): actve perods. For each dynamc experment, we generated two sets of fgures. From the sender montor, we obtaned the trajectory of ndvdual connecton throughput (n Kbps) over tme. They are shown n Fgure. As new flows joned or old flows left, TCP converged to the new equlbrum rate allocaton rapdly and stably (left column). Reno s throughput was also relatvely smooth because of the slow (lnear) ncrease between packet losses. It ncurred neffcency towards the end of the experment when t took mnutes for a flow to consume the spare capacty made avalable by the departure of another flow. H,, and BIC-TCP responded more quckly but also exhbted sgnfcant fluctuaton n throughput. From the queue montor, we obtaned average queue sze (packets) shown n Fgure on the rght column. The queue under TCP was qute small throughout the experment because the number of flows was small. H,, and BIC-TCP exhbted strong oscllatons that flled the buffer. Snce BIC-TCP tred to mantan an aggregate wndow to be just below the pont where overflow occurs, t had the hghest average queue length. From the throughput trajectores of each protocol, we calculate Jan s farness ndces (see Secton IV-D for defnton) for the rate allocatons for each tme nterval that contans more than one flow (see Fgure ). The farness ndces are shown n Table III. TCP obtaned the best ntra- Tme (se #Sources Reno H BIC TABLE III DYNAMIC SCENARIO I: INTRA-PROTOCOL FAIRNESS (JAIN S INDEX). protocol farness, very close to, followed by H, Reno, BIC-TCP, and then. It confrms that TCP does not penalze flows wth large propagaton delays. For TCP, each source tres to mantan the same number of packets n the queue n equlbrum, and thus, n theory, each competng source should get an equal share of the bottleneck bandwdth. Even though TCP acheved the best farness ndex, we dd not observe the expected equal sharng of bandwdth (see Fgure ). Our sender montor showed that all the flows measured ther propagaton delays correctly. We found that connectons wth longer RTTs x H x x H Fg.. Dynamc scenaro I: Throughput trajectory (left column) and Dummynet queue trajectory (rght column). consstently observed hgher queueng delays than those wth shorter RTTs. For example, the connecton on the path of ms saw an average queueng delay of ms, whle the connecton on the path of ms saw an average queueng delay of ms. Ths caused the connecton wth longer RTTs to mantan fewer packets n the queue n equlbrum, thus gettng a smaller share of the bandwdth. We conjecture that a larger wndow sze (due to longer RTT) produces a more bursty traffc. Wth bursty traffc arrvng at a queue, each packet would see a delay that ncludes the transmsson tmes of all precedng packets n the burst, leadng to a larger average queueng delay and a smaller throughput.

7 C. Case study: dynamc scenaro II Ths experment was smlar to dynamc scenaro I, except that there were a larger number () of flows, wth dfferent propagaton delays, whch joned and departed accordng to the schedule n Fgure. The qualtatve behavor n throughput, farness, stablty, and responsveness for each of the protocols s smlar n ths case as n scenaro I, and n fact s amplfed as the number of flows ncreases. Fg.. x ms x ms x ms x ms x ms x ms x ms tme () Dynamc scenaro II ( flows): actve perods. Specfcally, as the number of competng sources ncreases, stablty became worse for the loss-based protocols. As shown n Fgure, oscllatons n both throughput and queue sze are more severe for loss-base protocols. Packet loss was more severe. The performance of TCP dd not degrade n any sgnfcant way. Connectons sharng the lnk acheved very smlar rates. There was a reasonably stable queue at all tmes, wth lttle packet loss and hgh lnk utlzaton. Intraprotocol farness s shown n Table IV, wth no sgnfcant varaton n the farness of TCP. Tme () Sources Reno H BIC TABLE IV DYNAMIC SCENARIO II: INTRA-PROTOCOL FAIRNESS (JAIN S INDEX). throughput(kbps) throughput(kbps) throughput(kbps) throughput(kbps) throughput(kbps) x... x RENO x... x H x... x x... x x... x RENO... x H... x... x D. Overall evaluaton We have conducted several other experments, wth dfferent delays, number of flows, and ther arrval and departure patterns. In all these experments, the bottleneck lnk capacty was Mbps and buffer sze packets. We present here a summary of protocol performance n terms of some quanttatve measures on throughput, farness, stablty, and responsveness. We use the output of perf for our quanttatve evaluaton. Each perf sesson n our experments produced fve-ond averages of ts throughput. Ths s the data rate (.e., goodput) applcatons such as perf receves, and s slghtly less than the bottleneck bandwdth due to packet header overheads. Let x (k) be the average throughput of flow n the fve-ond perod k. Most tests nvolved dynamc scenaros where flows joned and departed. For the defntons below, suppose the composton of flows changes n perod k =, remans fxed over perod k =,..., m, and changes agan n perod k = m +, so that [, m] s the maxmum-length nterval over whch the same equlbrum holds. Suppose... x... x Fg.. Dynamc scenaro II: Throughput trajectory (left column) and Dummynet queue trajectory (rght column). there are n actve flows n ths nterval, ndexed by =,..., n. Let x := m m x (k) k= be the average throughput of flow over ths nterval. We now defne our performance metrcs for ths nterval [, m] usng these throughput measurements. ) Throughput: The average aggregate throughput for the nterval [, m] s defned as E := n = x. ) Intra-protocol farness: Jan s farness ndex for the nterval [, m] s defned as [] F := ( n = x ) /(n n = x ). F (, ] and F = s deal (equal sharng).

8 ) Stablty: The stablty ndex of flow s the sample standard devaton normalzed by the average throughput: S := m (x (k) x ) x m k= The smaller the stablty ndex, the less oscllaton a source experences. The stablty ndex for nterval [, m] s the average over the n actve sources S := n = S /n. ) Responsveness: The responsveness ndex measures the speed of convergence when network equlbrum changes at k =,.e., when flows jon or depart. Let x (k) := k t= x (t)/k be the runnng average by perod k m. Then x (m) = x s the average over the entre nterval [, m]. Responsveness ndex R measures how fast the runnng average x (k) of the slowest source converges to x : R := max { max k : x (k) x x } >. Responsveness ndex R measures how fast the aggregate throughput converges to x : { R := max k : (x } (k) x ) x >. For each TCP protocol, we obtan one set of computed values for each evaluaton crteron for all of our experments. We plot the CDF (cumulatve dstrbuton functon) of each set of values. These are shown n Fgures. From Fgures, has the best performance among all protocols for three evaluaton crtera, farness, stablty and responsveness ndex R. It has the ond best overall throughput. More mportantly, the varaton n each of the dstrbutons s smaller under than under the other protocols, suggestng that had farly consstent performance n our test scenaros. We also observed that both H and acheved hgher throughput and mproved responsveness compared wth TCP Reno. had worse ntra-protocol farness compared wth TCP Reno, whle both BIC-TCP and H acheved comparable ntra-protocol farness to Reno. H, BIC-TCP, and showed ncreased oscllatons compared wth Reno (Fgures, ), and the oscllatons became worse as the number of sources ncreased (Fgure ). From Fgure, TCP acheved a much better responsveness ndex R (whch s based on worst case ndvdual throughput) than the other schemes. We cauton however that t can be hard to quantfy responsveness for protocols that do not stablze nto an equlbrum pont or a perodc lmt cycle, and hence the unresponsveness of Reno, H, and, as measured by ndex R, should be nterpreted wth care. Indeed, from Fgure, all protocols except TCP Reno perform well on the responsveness ndex R whch s based The natural defnton of responsveness ndex as the earlest perod after whch the throughput x (k) (as opposed to the runnng average x (k) of the throughput) stays wthn % of ts equlbrum value s unsutable for TCP protocols that do not stablze nto an equlbrum value. Hence we defne t n terms of x (k) whch, by defnton, always converges to x by the end of the nterval k = m. Ths defnton captures the ntutve noton of responsveness f x (k) settles nto a perodc lmt cycle. on aggregate throughput. Ths apparent dscrepancy reflects the fact that lnk utlzaton traces converge more quckly than ndvdual throughput traces. It also serves as a justfcaton for the lnk model (): the aggregate nput rate to a lnk converges more rapdly than ndvdual rates, and hence the queue stablzes quckly to ts new level that tracks changes n wndows. E. NS- smulatons Our dummynet testbed s lmted to experments wth sngle-bottleneck networks and dentcal protocol. We conducted NS- smulaton to study the performance of n more complex envronments. The mplementaton n NS- s from CUBIN Lab []. We set up the parameters so that only the orgnal algorthm as used n the dummynet experments reported above was enabled. To elmnate possble smulaton artfacts, such as phase effect, we ntroduced two-way nose traffc n the smulaton, where a certan number of Pareto on-off flows wth shape parameter. were ntroduced n each drecton. When a nose flow s on, t transmts at a constant rate of Mbps. Each nose flow has an average burst tme of ms and an average dle tme of ms. Hence the average length of a flow s KB, smlar to web traffc. We repeated each scenaro tmes and report both the average rate and the standard devaton (error bars n the fgures). Three sets of smulatons were conducted: wth dfferent nose levels, wth Reno traffc, and on a multlnk network. Due to space lmtaton, we only present a few examples from each set of smulatons. See [] for complete detals. ) wth nose traffc: Ths set of smulatons repeated the scenaro n Secton IV-B, wth dfferent levels of nose traffc. The nose traffc was n the form of multple Pareto on-off flows as descrbed above. We vared the number of nose flows from to, correspondng to an aggregate nose traffc of % to % of the bottleneck capacty. Fgures show the throughput trajectory of three cases: %, % ( nose flows) and % ( nose flows). Each pont n the fgures represents the average rate over a ond nterval. The NS- smulaton wth % Nose (Fgure ) should be compared wth dummynet experment n Secton IV-B. Dfferent from dummynet experments, the NS- smulaton was clean, and new flows mstook queueng delay due to exstng flows as part of ther propagaton delays, leadng to unfar throughputs. However, when the nose was % of the capacty, such unfarness was elmnated. The queue was frequently empted and new flows observed the correct propagaton delays and converged to the correct equlbrum rates, as shown n Fgure. Fgure shows the throughput when the nose was % of the capacty. throughputs oscllated, adaptng to mce that joned and left frequently. In the perod of to ond, the mce traffc generated so much packet loss that the three flows could not keep α packets n the queue and they behaved lke an AIMD algorthm. Such AIMD behavor led to dscrmnaton aganst long RTT flows (flow and flow ). We also conducted smulatons wth exponental on-off traffc. The results are smlar.

9 .. Reno H.. Reno H CDF. CDF. CDF.... Fg..... Throughput(Mbps) Overall evaluaton: throughput..... Fg Farness Overall evaluaton: farness Fg Stablty Overall evaluaton: stablty. Reno H CDF. CDF Reno H Responsveness () Fg.. Overall evaluaton: responsveness ndex R.... Reno H Responsveness () Fg.. Overall evaluaton: responsveness ndex R. x wth % nose flow (ms) flow (ms) flow (ms) x wth % nose flow (ms) flow (ms) flow (ms) x wth % nose flow (ms) flow (ms) flow (ms) tme() Fg.. wth % mce traffc. x vs Reno Fg.. tme() x wth % mce traffc. vs Reno Fg.. tme() x wth % mce traffc. vs Reno throughput Reno throughput Theoretcal Reno throughput Fg.. tme() flows vs Reno flow. Fg.. throughput Reno throughput Theoretcal Reno throughput tme() flows vs Reno flows. Fg.. throughput Reno throughput Theoretcal Reno throughput tme() flow vs Reno flows. ) Interacton between and Reno: In ths set of smulatons, we used the same set of paths as n Secton IV-B, but we reduced the delay on each path to one-ffth of the orgnal value snce Reno took a very long tme to reach congeston avodance wth the delays n the orgnal setup. On each path, we used parallel flows nstead of a sngle flow. We vared the number Reno flows on each path from zero to four (and the number of flows was hence vared from four to zero) to examne s nteracton wth Reno. The equlbrum rates of and Reno sharng a sngle bottleneck lnk are predctable; see [] for detals. Fgures show the aggregate throughputs of flows and Reno flows when the number of Reno flows on each path s,, and. We also present the theoretc predctons on Reno s throughputs on the same fgures for comparson. The aggregate throughputs n smulatons match the model predctons reasonably well. Reno s throughput s slghtly lower than predcton snce the model does not captures Reno s tmeout behavor. The smulaton results also show Fg.. d = ms d = ms d = ms A B d = ms d = ms buffer szes are large enough capacty = Gbps for all lnks on the edge Topology wth multple bottlenecks that may be more aggressve, or less aggressve than Reno, dependng on the network setup. Indeed, the behavor of a general multlnk network share by heterogeneous protocols that use dfferent congeston sgnals, such as (whch uses delay) and Reno (whch uses loss), can be very ntrcate. See [], [] for detals and more comments n Secton V. ) n a network wth multple bottlenecks: We smulated a network wth two LANs connected by a WAN. The topology s shown n Fgure. Three pars of flows ran

10 throughput of one wan flow (Kbps) Fg.. x theory smulaton bottleneck capacty (Kbps) wth multple bottlenecks x smultaneously on three dfferent paths. The frst par of flows (WAN flows) ran on the path A B. The ond par (LAN flows) ran on A. The thrd par (LAN flows) ran on B. All lnks except A B had a capacty of Gbps. The capacty of A B was vared from Mbps to Gbps. The nose traffc ntroduced n each lnk has an average rate of % of the lnk capacty. Lnk A and lnk B were bottlenecks. Lnk A B also became a bottleneck when ts capacty was less than Mbps. In all scenaros, converged stably to ts equlbrum value, fully utlzng A and B. Fgure shows the throughput of the WAN flow A B wth varous capactes on A B, both measured from smulatons and calculated by solvng the utlty maxmzaton problem n (). The measured throughputs match the theoretcal predctons very well, valdatng the theoretcal model and Theorem on the equlbrum behavor of. V. OPEN ISSUES AND PROPOSED SOLUTIONS s a new delay-based congeston control algorthm. We motvate our desgn carefully and support t wth tentatve analyss and experments. We now summarze some open problems and potental solutons. More practcal experence s needed to assess conclusvely the serousness of these problems and the effectveness of proposed solutons n real networks and applcatons. A. Propagaton delay measurement Propagaton delay (basertt ) s used n the wndow control algorthm (). In a clean network, the queueng delay mantaned by exstng flows may be mstaken as part of the propagaton delay by new flows that jon later, as shown n NS- smulatons n []. The effect of ths estmaton error s equvalent to modfyng the underlyng utlty functons to favor new flows over exstng flows; see [], []. Methods to elmnate ths error are suggested n [], [], []. Our experence wth hgh speed networks and dummynet, however, has been that ths error s neglgble n practce because nose or overshoot n the network s often suffcent to occasonally clear the queue, allowng new flows to observe the true propagaton delay. Propagaton delay measurement can also be affected by route change from a shorter path to a longer path durng the lfetme of a connecton. Though route change at the tmescale of TCP connectons may be uncommon, mstakng an ncrease n propagaton delay as congeston wll reduce the connecton s throughput. A soluton s proposed n [] where the propagaton delay s estmated by the mnmum RTT observed n a certan precedng perod, not snce the begnnng of the connecton, so that the estmate tracks route changes. B. Queueng delay measurement Queueng delay measurement may be affected by the burstness of the flows themselves, leadng to slght unfarness among flows wth dfferent RTTs, as shown n IV-B. Such error can be greatly reduced by deployng a burstness control algorthm n the sender, as shown n []. Lke Vegas, s affected by queueng delay n reverse path, as shown n []. There are a number of ways that have been proposed to elmnate the effect of reverse queueng delay for Vegas wthout the need for addtonal supports from recevers, that are applcable to. The method n [] utlzes the TCP tmestamp opton that s wdely mplemented n today s TCP stacks. If the sender and recever have the same clock resolutons on ther tmestamps, the dfference between the sender tmestamp for a data packet, and the recever tmestamp for the correspondng ack s the oneway delay measurement n the forward drecton. Thus, the dfference between each one-way delay measurement and the mnmum one-way delay measurement would be the forward queueng delay. Note that ths calculaton s correct even when the sender and recever s clocks are not synchronzed. As long as the clock drft s not sgnfcant n the lfe of a TCP flow, the clock offset wll be elmnated through subtracton snce both the one-way delay measurement and mnmum one-way delay measurement has the same offset. If the clock drft s sgnfcant, [], [] provde technques to accurately synchronze the clocks wthout GPS (global postonng system). If the sender and recever have dfferent clock resolutons, the sender can estmate the recever clock perod by observng the number of tcks of recever s clock durng a fxed tme nterval. A dfferent method s proposed n [] that does not drectly measure the queueng delay. Instead, they measure that actual throughput n the forward W drecton, and use ths measurement n place of d+q. C. α tunng The parameter α n wndow control equaton () controls the number of packets that each flow mantans n the bottleneck lnks. If the number of elephant flows sharng a bottleneck lnk s large and the buffer capacty s small, these flows may not be able to reach ther equlbra before they observe packet loss. In ths case, the wndow control algorthm oscllates lke a loss-based AIMD algorthm does. D. Heterogeneous protocols Congeston control algorthms that use the same congeston sgnal can be nterpreted as dstrbuted algorthms for network utlty maxmzaton; see e.g., [], [], [], [], [], []. The underlyng utlty maxmzaton problem mples e.g. that there exsts a unque equlbrum (operatng) pont for general networks under mld condtons. It turns out that a network wth heterogeneous protocols that react to dfferent congeston sgnals can behave n a much more ntrcate way. In partcular, we prove theoretcally n [] that there are networks that have multple equlbrum ponts, and demonstrate expermentally n [] ths phenomenon usng TCP Reno and Vegas/. We also prove n [] condtons Alternatvely, a verson of the mplementaton deals wth the problem of nsuffcent bufferng by choosng α among a small set of predetermned values based on acheved throughput. Ths can sometmes lead to unfar throughput allocaton as reported n some of the lterature. Ths verson was used around early, but dscontnued snce.

11 on network parameters that guarantee (global) unqueness of equlbrum ponts for general networks. We show n [] that any desred nter-protocol farness are n prncple achevable by an approprate choce of parameter, and that ntra-protocol farness among flows wthn each protocol s unaffected by the presence of the other protocol except for a reducton n effectve lnk capactes. How to desgn practcal dstrbuted algorthms that use only local nformaton to acheve a desred nter-protocol farness s however an open problem. Acknowledgments: We gratefully acknowledge the contrbutons of the project team and our collaborators, at n partcular, G. Almes, J. Bunn, D. H. Choe, R. L. A. Cottrell, V. Doraswam, J. C. Doyle, W. Feng, O. Martn, H. Newman, F. Pagann, S. Ravot, S. Shalunov, S. Sngh, J. Wang, Z. Wang, S. Yp. We thank J. Wang for pontng out several errors n an earler verson. Ths work s funded by NSF (grants ANI- and ANI-), Caltech Lee Center for Advanced Networkng, ARO (grant DAAD---), AFOSR (grant F---), DARPA, and Csco. VI. APPENDIX: PROOFS A. Proof of Lemma Fx w. Defne U (x ) = w log x d x and consder the followng optmzaton problem: max x U (x ) subject to Rx c () Snce the objectve functon s strctly concave and the feasble set s compact, there exsts a unque optmal soluton x. Moreover, snce R has full row rank, there exsts a unque Lagrange multpler p for the dual problem. See, e.g., [] for detals. We clam that p s the unque soluton of () and, for all, x = w /(d + q ) () Now, () can be rewrtten as, for all, R l p l = q = w x d = U (x ) l whch s the Karush-Kuhn-Tucker condton for (). Hence () holds. Then () becomes R lx c l, wth equalty f p l >. But ths s just the complementary slackness condton for (). B. Proof of Theorem Clearly unque soluton x for () and unque soluton p for ts dual exst, snce the utlty functons α log x are strctly concave and R s full rank (see e.g. []). We need to show that the dual problem of () s ndeed gven by (). Now the dual objectve functon s gven by [] D(p) := ( ) max α log x x R l p l + c l p l x l = l c l p l + α log α (log α ) cf. the proof of a smlar result n []. l R l p l Snce the last term s ndependent of p, mnmzng D(p) over p s the same as mnmzng () over p. Hence there exsts a unque soluton (x, p ) for () (). We now show that (x, p ) s the equlbrum pont of () (). In equlbrum, we have w (t + ) = w (t) =: w. From (), the correspondng queueng delays p l unquely defned by () must be such that the end-to-end queueng delays are strctly postve,.e., q = l R lp l > for all even though some p l can be zero. Then α (w, q ) = α n equlbrum, and, from (), we have q = l R lp l = α /x, where x := w /(d + q ). But ths s the Karush-Kuhn-Tucker condton for (). Moreover, () s the complementary slackness condton. Hence the equlbrum of () () concdes wth the optmal soluton of () (),.e. w = w and p = p. C. Proof of Theorem Let N be the number of sources. Let q(t) = p(t) denote the queueng delay at the sngle lnk (omttng the subscrpts). It s more convenent to work wth normalzed wndow y (t) := w (t) d () Let Y (t) := y (t) be the aggregate normalzed wndow. Then q(t) > f and only f Y (t) > c. The wndow control algorthm () can be expressed n terms of updates on y(t): y (t + ) = ( γq(t) d + q(t) ) y (t) + γ ˆα () where ˆα := α /d. Let ˆα := ˆα. We frst prove that the queue s lower bounded by a postve constant after a fnte tme. Theorem : ) For all t > c/γ ˆα, we have q(t) >. ) Moreover, gven any ɛ > we have ˆα c mn d ɛ < q(t) < ˆα c max d + ɛ for all suffcently large t. Proof (Theorem ). For the frst clam, we wll prove that the queue wll be nonzero at some t > c/γ ˆα, and that once t s nonzero, t stays nonzero. Suppose q(t) =. Summng () over, we have Y (t + ) = Y (t) + γ ˆα,.e., Y (t) grows lnearly n tme by γ ˆα n each perod. Snce Y (), Y (t) > c after at most c/γ ˆα perods. Hence there s some t > c/γ ˆα such that q(t) >. We now show that q(t) > mples q(t + ) >. Snce γ <, we have from () y (t + ) = Summng over gves Y (t + ) ( q(t) d + q(t) d (t) d + q(t) y (t) + γ ˆα ) y (t) + γ ˆα d (t) d + q(t) y (t) + γ ˆα But q(t) > f and only f d (t) d + q(t) y (t) = c ()