An Improved Link Model for Window Flow Control and Its Application to FAST TCP

An Improved Link Model for Window Flow Control and Its Application to FAST TCP Krister Jacobsson, Lachlan L. H. Andrew, A. Kevin Tang, Steven H. Low Royal Institute of Technology KTH, Stockholm, SE- 44, Sweden California Institute of Technology, Pasadena, CA 95, USA Abstract This paper presents a link model which captures the queue dynamics when congestion windows of TCP sources change. By considering both the self-clocking and the link integrator effects, the model is a generalization of existing models and is shown to be more accurate by both open loop and closed loop packet level simulations. It reduces to the known static link model when flows round trip delays are similar, and approximates the standard integrator link model when there is significant cross traffic. We then apply this model to the stability analysis of FAST TCP including its filter dynamics. Under this model, FAST TCP flows over a single link are always linearly stable regardless of delay distribution. This result resolves the notable discrepancy between empirical observations and previous theoretical predictions. The analysis highlights the critical role of self-clocking in TCP stability and the proof technique is new and less conservative than the existing ones. I. INTRODUCTION In the field of network congestion control [], [8], one line of work of fundamental interest is the dynamics of congestion control protocols. Stability is crucial to ensure that the system operating point is indeed the intended equilibrium, with the desired efficiency and fairness. Local stability [8], [4], [6], [9], [], [], [3], [4], [5], [3] and global stability [], [], [3], [34], [36] of congestion control protocols have both attracted much attention. Until recently, stability analyses have been based predominantly on rate-based rather than window based source models, and, almost without exception, on integrator link queue models. In these models, sources control their data rates explicitly and the rate of change of the queueing delay is proportional to the difference between the aggregate incoming traffic and link capacity. Typically, these results show that the system is stable when round trip delays do not exceed some upper bound. This is in line with the intuition that increased feedback delay may have a destabilizing effect on a closed loop system. There are however features of TCP which play important roles on stability but are not captured in the rate-based/ integrator model. Current TCPs are window based algorithms, meaning that each sender controls its window size an upper bound on the number of packets that have been sent but not acknowledged. The actual rate of transmission is controlled or clocked by the stream of received acknowledgments ACKs: a new packet is transmitted only when an ACK is received, thereby keeping the number of outstanding packets constant and equal to the window size. Therefore, sources control the amount of data they inject into the network rather than the rate of doing so. Intuitively, this volume control is safer in terms of stability than rate control. This self-clocking has the consequences that the queue size tracks the changes in the window much faster than the integrator queue model predicts, and that the sending rate cannot exceed the capacity over long time frames. Both enhance stability. For the case where flows round trip delays coincide and no non-window-based cross traffic is present, the self-clocking effect is dominant: the total ACK rate cannot exceed the link capacity at any time and hence a window change translates into a proportional change in the queue. Indeed, as an example, homogeneous FAST TCP [35] flows are reported empirically to be stable with any delay, in stark contrast to the prediction of the integrator link model of [33] which asserts instability when delay is large enough. This observation motivated the proposal of a static link model to capture self-clocking effect under TCP window control [3], [33]. Using the static link model, it has been theoretically shown that FAST TCP flows are always stable for the homogeneous case [], [3], [33]. As the static model fails to hold in heterogeneous cases, we need a better model to study the general case. Both the integrator and the static link models are incomplete; the former lags the true dynamics yielding conservative stability results, while the latter leads the true dynamics yielding optimistic predictions See examples,, 3 and 5. After introducing notation and preliminary knowledge, we will show in Section II that a natural combination of these two models leads to a more accurate one. Open loop experiments validate this joint model and reveal its underlying intuition. The new joint model predicts significantly different delays from existing models on time scales comparable to a round trip time. Recently, many protocols have been proposed which use delay to respond on this time scale; for example, TCP Vegas [7], FAST [35], TCP Africa [7], Compound TCP [3], TCP Adaptive Reno [6] and TCP Fusion [8]. Accurate analysis of all of these protocols with require the use of a model similar to the joint model. To illustrate this, Section III investigates the stability of FAST TCP. Using the joint link model, we prove that FAST running over a single bottleneck link is stable for any heterogeneous delays, and hence resolve the discrepancy between previous experimental results and existing theoretical predictions. Closed loop experiments are also reported where accurate predictions on the stability region are obtained and verified

with packet level simulations. Finally, section IV shows how all the models in this paper are approximations to a more accurate but less tractable model. II. MODEL AND NOTATION To capture the self-clocking effect in window flow control, we avoid working directly with the sources sending rates. Instead, we use the sources window sizes and the bottleneck queue size to represent the state of the closed loop system. A. Preliminaries Consider N window-based TCP sources sending over a bottleneck link with capacity c. Let w n t denote the congestion window of source n at time t, n {,..., N}. Let a packet that is sent by source n at time t appear at the bottleneck queue at time t + τ f n. This forward delay τ f n models the amount of time it takes to travel from source n to the link, and it accounts for the constant forward latency but not queueing delays. The backward delay τ b nt is the time from when a packet arrives at the link to when the corresponding acknowledgment is received at source n, including the time-dependent queueing delay at the bottleneck queue. The round trip delay seen by source n, denoted τ n t = τ f n + τ b nt, is the elapsed time between when a packet is sent and when the corresponding acknowledgment is received. The latency of source n, denoted d n, is defined as the round trip delay when the bottleneck queue is empty. The queueing delay of the bottleneck link is denoted by pt, and c > is the capacity of the link. The queueing delay observed by the nth source at time t is q n t; it relates to the queue delay by qt = p t, where t solves t = t+τ b n t. The bottleneck link may also carry non-window-based traffic such as User Datagram Protocol UDP traffic. Let x c t [, c] be the rate at which non-window-based crosstraffic is sent over the link, leaving c x c t for window based sources. Whenever a time argument of a variable is omitted it represents its equilibrium value, e.g., p is the equilibrum value of pt. When working in discrete time w[k] denotes wt k. B. Link model and validation As described in Section I, previous work differs in how the dynamic map between the window sizes and the buffer size is modeled. Most existing literature on window based congestion control, see [], [5], [4], [], [], [37], assumes that the sending rate is proportional to the window size divided by the round trip delay and may further model the queue as a simple integrator, integrating the excess rate at the link, i.e., Integrator link model: ṗt = N w n t τn f c d n + pt + x ct c. n= To the best of our knowledge, the current status of research on congestion control protocols can provide quantitative results on equilibrium, while for dynamics, most works focus on qualitative study and have not been able to compare predictions with packet level simulations quantitatively. The model does not, however, take into account the selfclocking characteristic of window based schemes, where the sending rate is regulated by the rate of the received ACKs. To model this phenomenon, an alternative model is used in [3], [35] and implicitly in [7], based on the empirical observation that, due to self-clocking, transient effects are negligible. The relation between the window size and the buffer size is then described by the algebraic relation Static link model: N w n t τn f = c x c t. d n + pt n= In, a change in any of the sources congestion windows w n results in a proportional change in the queueing delay p one forward delay τn f later. This is in contrast to the integrator model where a window change gives a smooth queue transition. Obviously the two models are fundamentally different. We now combine their main features to form a more accurate joint model. Joint link model: In this section, the underlying intuition of the joint link model is explained. The same model is derived in [5] from a detailed analysis of the packet level system. To understand the difference between and the accuracy of the integrator model and the static model, the key is to examine cases where flows have heterogeneous round trip delays or where non-window-based cross traffic such as UDP exist. Consider the case of N flows sending over the bottleneck link with constant window sizes, and consider the response to a change in window size by a system initially in equilibrium. The long term effect of a window change is that the queue integrates the excess rate. This is well known, and captured by the integrator model. The short term effect of a window change is more complex, and often wrongly ignored. Since the link is fully utilized in equilibrium, the queue s immediate response to a window increase is a proportional increase in the queue size; this will occur one forward delay τ f n after the window w n is changed. If there is no cross traffic and the sources share the same round trip time RTT, there is no further transient, because sources sending rates are auto-regulated by their individual ACK rates which sum up to the capacity; subsequently the queue input rate will equal its output rate capacity. This is in line with the static queue model which neglects transient behavior. However, if there is stationary UDP cross traffic, sources can affect their ACK rates over time intervals greater than one RTT actually it is a function of the amount of cross traffic, see [5]. This is also true when no cross traffic is present but the heterogeneity among sources RTT is large enough. As individual flows operate on their individual RTT time scales and it takes one RTT before a queue change affects the queue input rate, from the perspective of flows with small round times, flows with larger RTTs can be considered as nonresponsive cross traffic and the system is hence transient in this case as well. The original model was presented in discrete time for multiple bottlenecks, here we use its continuous time version used in e.g. [] and we consider a single bottleneck.

3 Adopting the standard fluid flow approximation, that packets transmitted from a source form a continuous smooth flow, and motivated by the previous discussion we capture both the short and long term behavior in a single model, Joint link model: ṗt = N wn t τ f n c d n + pt +ẇ nt τn f + x c t c 3 n= which can be seen as a superposition of and. The derivative term ẇt τn f models the immediate proportional change in the queue size due to a window change. Note that it is the window size and its corresponding time derivative only that have delayed variable arguments, which furthermore are identical. This is rigorously motivated in the full model derivation in [5]. A similar model was also implicitly used for flow control stability analysis in [3]. Linearizing 3 gives the Laplace domain transfer function for flow n of G pwn s = L[pt] L[w n t] = c s + d n+p e sτ f n s + N c i= w i/d i + p. 4 For the the case with homogeneous delay d n = d and no cross traffic, applying the identity N n= w n/d + p = c shows that the transfer function zero and pole will cancel, whence the map is a pure delay, scaled by /c. This agrees with the description in the previous discussion as well as in the model. Moreover, the dynamics will be more distinguishable with increasing heterogeneity among the sources and cross traffic as in. Finally, note that 4 is open loop stable, as expected due to self-clocking. Open loop validation experiments: The accuracy of the joint model 3 as well as its similarities and differences with the integrator model and the static model is illustrated in the following open loop examples. A closed loop experiment will be reported in Section III-E. The models, and 3 are validated with packet level data generated by using the NS- FAST TCP module [9]. In each experiment we consider window based flows with static windows, sending over a single bottleneck link. Nonbottleneck links provide configurable forward and backward delays. The window sizes are initially set to the same constant size and are not updated dynamically, i.e. there is no dynamic feedback except for the ACK-clocks. The system is started in equilibrium and perturbed at time t = s by a % step change in the first source s congestion window. In all experiments a packet size of 4 bytes is used. Example : Homogeneous sources All window based flows share the same latency d n = ms and the bottleneck link capacity is set to c = Mbit/s. The window size is w n = 5 packets. Source, which is subject to the window change, has a forward delay of τ f = ms. The solid gray line in Figure shows the queue size when the system is simulated in NS-. The black dashed, dotted and solid lines show the integrator model, the static model and the joint model 3 respectively. The fit of the static and the joint models is excellent neglecting packet level Queue size [sec] 8.8 8.6 8.4 8. x 3 9 8 NS Static model Integrator model Joint model..4.6.8..4.6.8 Time [sec] Fig.. Homogeneous sources. Both the joint model and the static model agree with the NS- simulation, but the integrator model lags significantly. Queue size [sec] 9 8.9 8.8 8.7 8.6 8.5 8.4 8.3 8. 8. 8 x 3 NS Static model Integrator model Joint model 3 4 5 6 Time [sec] Fig.. Cross traffic. Both the joint model and the integrator model agree with the NS- simulation, but the static model leads significantly. noise ; they are identical in this scenario. This suggests that the true dynamics in this case is indeed a pure delay. Also observe that the integrator model lags the NS- simulation. Note that it takes ms before the window change affects the queue, as predicted by the models. Example : A cross traffic scenario This scenario is as in Example but with bottleneck link capacity c = 5 Mbit/s which is also shared by 4 Mbit/s of UDP traffic. In this case the dynamics are clearly distinguishable; it takes over four seconds twenty round trip times before the queue settles again. The static model is too rapid in this case, as expected, while the other two are both good. The joint model captures the rapid initial rise in queue size, and initially tracks the upper envelope of the staircase simulation results, while the integrator model tracks the lower envelope. From s after the transient, the joint model tracks the mean while the integrator lags slightly. Example 3: Heterogeneous sources This scenario is as in Example but with d = 5 ms, distributed such that τ f = 5 ms, for the first source, and d n = 5 ms for the remaining 9 sources, and w n = 35 packets. In this case there is a more pronounced initial increase in the queue followed by a transient phase which dies out after about 5 ms. This corresponds to the time until the ACK-clocks of

4.45 q n t q n k ˆq n k ˆq n k estimation window w n k control ZOH.4 Queue size [sec].35.3.5..5 NS Static model Integrator model Joint model...3.4.5.6.7.8.9 Time [sec] Fig. 3. Heterogeneous sources. The joint model captures both the initial jump and the protracted rise; the integrator model misses the former, while the static model misses the latter. the high delay sources adjust to the new conditions. In summary, these three examples, from different aspects, demonstrate that while the integrator link model may lag the true dynamics and the static link model can lead the true dynamics a lot, the joint model 3 succeeds in modeling the two main system characteristic of self-clocking, i.e., the short term proportional change, and the long term integrating effect that are present in the system. For rigorous derivation of the model and more validation experiments, see [5]. III. APPLICATION: STABILITY OF FAST TCP To demonstrate the tractability of the joint link model, we will apply it to the stability analysis of FAST [35]. FAST is a high speed TCP variant that uses delay as its main control signal. So far, all experiments with FAST TCP have operated at a stable equilibrium regardless of what the round trip delays are. This section will show that under the joint link model 3, FAST is indeed locally stable for a single bottleneck with the default step size. Unlike [3], [33], this analysis is for flows with heterogeneous delays. In this case, there is no longer a natural time step, and it becomes simpler to work in continuous time. A. Window and estimator model of FAST TCP The sending rate of FAST TCP is implicitly adjusted via the congestion window mechanism. Each sender n updates its window size in discrete time according to w n [k + ] = γ n w n [k] + γ n d n + qˆ n [k] w n[k] + γ n α n 5 where α Z + and γ are protocol parameters [35]. This update is performed once every RTT, with sampling time h k. The buffers are treated as continuous time dynamical processes, with quantization treated as noise. The aggregate queueing delay q[k] can be approximated by subtracting the latency d n from the measured RTT. However, this gives a noisy measurement of the true queueing delay, which is therefore estimated by the source. We do not consider the problem of estimating d n. The estimate ˆq n [k] is formed from queueing delay samples measured at each acknowledgement arrival, w n [k] times for d n Fig. 4. Split view of the FAST TCP window update mechanism. the kth update of 5. Denote discrete time at this timescale by k, with inter-packet time h k. With the obvious abuse of notation, we refer to values at this sampling time as q[k ] etc.. The estimator is qˆ n [k + ] = σ n [k ] qˆ n [k ] + σ n [k ]q[k ], σ n [k ] = min {κ/w n [k ], ν}. 6a 6b This non-linear filter has the characteristic of a low-pass filter, with a dynamic time constant of τ n /κ n. In the current implementation the filter parameter is κ = 3 and ν = /4 [35]. A split view of the window dynamics is given in Figure 4. The window size, w n [k], is used by the transmission control, which decides the source sending rate based on w n [k] and the rate of the received acknowledgments. Note that the transmission control is included in the ACK-clock link model [5]. No anti-aliasing filtering is performed prior the sampling, or the down-sampling in the FAST TCP implementation [35]. The ZOH block represents a zero-order hold function defined by ft = ft k, t k t < t k+. 7 A sampler with sampling instants t k is mathematically described by f[k] = δs t k fsds, k =,,,..., 8 where δ is a Dirac impulse. The two sampling rates present in the system are related by h k = w n[k] k = h k +k k where k k = k i= w n[k] is the value of k just before the start of the kth RTT. Note that due to the w n [k] in 9, the relation between the two sampling times is time varying. For the linear stability analysis, the dynamics 5 and 6 are modeled as G wnqs = γ nα n d n /q sτn + γ n q, K, sτ n /K + with /K /κ + /, valid for small perturbations around the equilibrium. This is derived rigorously in Appendix B. The first part corresponds to the window update mechanism; while the second filter models the phase-loss due to the estimation procedure and the various holds in the system. Finally, adopting the standard convention that whenever a round trip time, or forward and backward delay, appears in a variable argument, it is replaced by its equilibrium value [3], [33], the backward transport delays are modeled around equilibrium as 9 q n t = pt τ b n.

5 B. Loop Gain Combining the source dynamics and the Laplace transform of the communicated corrupted price queueing delay with the queue dynamics transfer function 4, results in a negative feedback system with open loop transfer function where µ n = α n cq = Ls = L n s = s + /τ n s + /ˆτ N = ˆτ n= α n N µ n L n s a n= c m α m µ n τ n. τ n qγ n e τns τ ns + γ n q b sτ n /K n + c d When x c =, we can interpret ˆτ as a weighted harmonic mean value of τ n. In particular, when all flows have equal α n, giving µ n = /N, ˆτ is the harmonic mean of τ n. C. Stability analysis The full model of the dynamics of FAST with the joint link model contains details concerning the equilibrium queueing delay and the RTT estimator. As a first step toward proving stability, Lemma abstracts away that detail, to give a condition on the gain parameter sufficient for stability. Lemma : Denote the half plane under the line that passes + j with slope /ωˆτ by Hω = { z argz + arctan } π,. 3 ωˆτ If /γ n F min,n min > ReF n > then a system with open loop gain Ljω = N n= µ n γ n F n ωτ n τ n jω + /ˆτ 4 satisfies Ljω Hω for all ω and is closed loop stable for any τ n, µ n >, n =,..., N, with N n= µ n =, and /ˆτ = N n= µ n/τ n. Proof: By definition Ljω Hω is equivalent to argljω + arctan π,. 5 ωˆτ Substituting 4 and noting that arg jω + ˆτ + arctan condition 5 can be further rewritten as N µ n arg γ n F n ωτ n + jω + ˆτ τ n n= = π ωˆτ, 6 π, π 7 which is equivalent to N µ n Re γ n F n ωτ n + jω + ˆτ >. 8 τ n n= but γ n F n ωτ n by hypothesis, and so d establishes Ljω Hω for all ω. Thus, since, ] ω> Hω =, the Nyquist curve for L cannot encircle, and the system is stable. The construction used for Lemma is depicted in Figure 6, for τ =, τ = 5, µ = µ = /, at ωˆτ = 3. Remark: The techniques used here are significantly different from ones in the existing literature on linear stability of TCP, in two respects. First, the usual approach is to find a convex hull that contains all individual L n jω curves and then argue any convex combination of them is still contained by the convex hull. See, e.g., [3], [3], [8]. However, the proof of Lemma deals directly with Ljω instead of L n jω. Second, for each ω, a separate region is found to bound Ljω away from the interval, ]. That is, the half plane Hω defined by 3 depends on ω. In existing works, convex regions are typically used to bound the whole curves and hence are independent of ω. One exception is [5], where the frequency range is divided into two parts and different convex regions are used in the two parts. These two features lead to tighter bounds, which is necessary for the analysis of this problem. In the FAST model c, the case when the queueing delay is q = is intuitively the least stable, as increasing q reduces the gain and introduces phase lead, both of which intuitively improve stability. To formalize that, Lemma will be used to place bounds on the values of F min,n used in Lemma. Lemma : Consider a complex half-plane H = {z : Imz z e jβ } containing. Consider also a function G : R + C with Gν = rνe jν, where r and are continuous decreasing real functions, r is unbounded as ν, and β + = Φ with Φ, π the angle between the edge of the half-plane and the tail of the spiral, G. If {Gν : ν R + } H then {ηνe jφν Gν : ν R + } H for any η : R + [, ], φ : R + [, π Φ. In particular, if = π/ then taking β = π/ gives min Reηνe jφν Gν min ReGν 9 ν ν for any η : R + [, ], φ : R + [, π. Proof: First, consider the tail of the spiral, ν [, ν] where ν = φ. The image of the tail under the rotation and scaling is entirely in the sector {z : argz [, + φ]}, which is entirely in H by the definitions of φ and Φ. The next step is to show that the image of any point with ν > ν is also within H. Since H is convex and H, it suffices to show that for any ν, ηrν rψ where ψ = ν + φ. Note that ψ > since ν > ν. But ψ < ν as is decreasing, whence rν rψ since r is decreasing. The result follows since η. The special case 9 follows when β = π/, Φ =. It is now possible to show that FAST is stable in singlebottleneck networks.

6 Theorem 3: A system with gain Ls given by is stable for arbitrary α n > and arbitrary K n, if for all n and Proof: Let < γ n < so that becomes Further, let.93 + 6/5π G K = e j j + j + j/k τ q G τ,k = G K τ + γqτ/j Ljω = N µ n γ n G τn,k n ωτ n. τ n jω + ˆτ n= F, K = /K + /K cos + /K sin + /K. Noting that ReG K = F, K, Lemma 5 in Appendix C implies G min min,k ReG K + 6/5π. By Lemma, G min G min,n min ReG τn,k n. By, /γ n G min G min,n > and the result follows from Lemma with F n = G τn,k n. Remark: Note that an effective value of γ =.5 is used for the FAST implementation [35]. For this case, Theorem 3 immediately establishes FAST s stability for any pattern of round trip delays. This explains why FAST has been stable for all experimental cases studied. Theorem 3 does not require γ, α or K to be equal for all flows. Individual flows may adjust γ and K, subject to Lemma, depending, for example, on how noisy their estimates of q are. Numerical solution of min F, K as a function of K, plotted in Figure 5 shows several interesting features. The bound in has.6% slackness. The minimum value is approximately F.689, 5.655.644, while the right hand side of is approximately /.75. The bound can be further relaxed for several interesting cases of K, as follows. Let F min = min F,. If no filtering is used, K, the bound on γ can be relaxed marginally to /F min /.639. This agrees with the earlier analysis of this case in [9]. Importantly, for K [, 4.9], min F, K >. Thus, the system is stable for all γ, ]. As the implementation of FAST uses κ = 3 which corresponds to K 6/5, it is stable for γ, ], as stated in [35]. Theorem 3 is proved by finding a uniform bound for all flows τ n. If we have more detailed knowledge about the round trip delay distribution, we may achieve tighter bounds. When q << τ and K >>, the window mechanism is much slower than the rest of the source dynamics, and is hence dominant. Fmin.8.85.9.95.5.459,.854 4.96, 5.655,.644,.63. 3 4 Fig. 5. Numerical solution of min F, K as a function of the filter parameter K..5.5 + j L jω.5.5.5 K slope /ωτ^ = /3 L jω Fig. 6. An example of a line of slope /ωˆτ which bounds Ljω, denoted by the center cross. Note that the individual terms L njω, denoted by the individual crosses, are not all below this line. When q and K, the stability condition reduces to [9] N n= µ n γ n τ n cosωτ n sinωτ n + >, 3 ωτ n ˆτ Let us now consider some special cases. N = : If there is a single flow, µ = in and the joint link model degenerate to the static link model. In this case, FAST is stable for all γ < π/. In this case, and 3 are loose simply because the frequency, ω =.7437/τ, which minimizes F ωτ does not coincide with a frequency at which the Nyquist plot of Ls crosses the real axis. N = : Consider two FAST flows with µ = µ corresponding to the current practice that α is fixed for all flows. Write τ = Γτ, where Γ measures the heterogeneity. Define γ Γ = sup Γ + Γ cos sin + cosγ sinγ Γ 4 From 3, a sufficient condition for stability of is γ < γ Γ. It was stated in [9] that γ Γ increases from γ = /F min to a peak of.94 at Γ = and then decreases towards /F min as Γ. However, it is fact much more complicated, as seen in Figure 7. It consists of a series of distinct peaks starting from plateaux at, with many valleys of varying sizes.

7.3.5..5..5.95.9 3 4 5 6 7 8 9.3.5..5..5.995.4.6.8..4.6.8 Fig. 7. Lower bound, γ Γ, on the threshold for stability, as a function of Γ. Fig. 8. Zooming in on γ Γ. Let us start by understanding the plateaux. Although the minimum value of each term in 3 occurs for small ω, the supremum in 4 can occur for very large. For almost all Γ, the supremum in 4 cannot exceed, because of the following lemma. Lemma 4: Let F Γ = cos sin For irrational Γ, + Γ cosγ sinγ Γ lim inf F Γn + π = Γ + n Γ Proof: For = n + π, F Γ = + /Γ[cosΓ sinγ/γ] Now lim inf sinγ/γ =. Since Γ is irrational, applying Kronecker s theorem [] and using the continuity of cos gives lim inf cosγ = min φ [,π] cosφ =. This shows that all peaks above are for rational values of Γ. For any Γ, a minimum of F Γ less than one can be achieved for a finite, and so the valleys do not contraditc Lemma 4. For the large peak at Γ =, F Γ = + cos cos sin sin. Remark: Conjectures / observations For any irrational Γ, inf F Γ <. Thus, any pair of spikes are separated by a small valley, rather than true plateau. Fractalish. Any peak is an accumulation on one side or both of peaks. 3 Near big peaks, both other peaks and valleys reduce in size, and valley minima occur for increasingly large. 3 N = : In reality, the link is likely to be shared by many flows. It is then interesting to find the statistical mean value of the stability bound for those scenarios. We will now consider the case of many flows with continuously distributed RTTs, letting α n with α m /α n fixed. Let Mτ = τ n τ α n/cq, and let all τ n be in the range Ω = τ, τ, with τ possibly infinite. If there are many flows Fig. 9..6.4...8.6.4..9998.9.9.93.94.95.96.97.98.99 Further zooming in on γ Γ, showing the fractal nature. with RTTs drawn from a continuous distribution, then applying µτ = M τ to 3 gives µτ cosωτ sinωτ dτ + µτ dτ >. Ω τ ωτ γ Ω τ 5 Noting that d sin = cos sin d, and setting = ωτ, 5 becomes γ > = ωω [ µ sin µ d ω µ/ω d ωω ] ω τ sin ω τ + ω ωτ ωτ ω τ ωτ µ/ω µ /ω sin d ω d 6 7 where ωω = ωτ, ω τ and denotes derivative. This must hold for all ω >. As an example, assume RTTs follow a uniform distribution. As units of time are arbitrary, this can be modeled without loss

8.5..5..5.5 Ljω + j.5.9995.5 µ L jω.999.45.46.47.48.49.5 Fig.. Zooming in on γ Γ around the smaller peak at.5. Fig.. Maximum γ for stability 4 3 of generality as 4 6 8 Heterogeneity index b Maximum value for γ for stability with RTTs uniform in, Γ. µτ = { /Γ τ, Γ otherwise, with Γ >. In that case, 6 becomes [ ] / ωγ sin ωγ > max γ ω> ω ω sinωγ sinωγ = max ω> ωγ logγ d 8 9. 3 It is easily verified see Appendix E, that the right hand side approaches F min as Γ, while for Γ > the bound is strictly looser as shown in Figure, and asymptotes to / logγ for large Γ. D. A counter-example Because Theorem 3 proves stability for γ so close to, it is tempting to seek to show stability for all γ. However, the following example breaks that hope. It illustrates the tightness of our result with the current model and how heterogeneity can potentially hurt stability. Example 4: A counter example Consider a network with a single bottleneck link carrying two flows. The flows have RTTs τ =. ms and τ = 75 ms, Fig...5 3 µ L jω 3.5.5.5 a Individual terms. Triangles show the unstable frequency, ω =...5.5..5..5.95.9 b Close-up of the critical point. Nyquist plot of the system of Section III-D. with α = and α = 8 with c = 8 packets per second. This gives q =, µ = 8, µ = µ and ˆτ = 749 ms. 3 With γ =, in contrast to the implemented γ =.5, is unstable in this extreme case. Instability arises because of the high heterogeneity between the RTTs of the flows, and the greater heterogeneity between the weights given to the flows. The Nyquist curve for this network with γ = is shown with the solid line in Figure a. The dashed lines show the individual curves µ L jω and µ L jω, and the triangles show these curves for the frequency ω = at which Ljω first crosses the real axis. The magnified view of this curve near the point + j in Figure b shows that the Nyquist curve does indeed encircle and the resulting system is unstable. In this example, most of the weight is given to flow, and the instability occurs when ωτ.75, minimizing 3 we get F ωτ.6 see, e.g., Figure 5. Although τ is very small, µ is even smaller, making the coefficient µ /τ in 3 negligible, and allowing 3 to be violated by the τ term. However, the extra factor of /τ provided by the numerator of the first factor in c allows the imaginary part of µ L j to balance that of µ L j where the curve crosses the axis. This example shows that two flows are sufficient to cause instability, even though a network with a single flow or multi- 3 The parameters in this subsection are quite extreme. The example here is primarily of theoretical interest as it gives an upper bound on the γ which can yield guaranteed stability. We report unstable cases with realistic parameters in Section III-E.

9 ple homogeneous flows is always stable. It is also possible to construct a network of three flows with slightly less extreme parameters µ =. 5, µ =.98, µ 3 =.79, τ = 3 ms, τ = 5. ms and τ 3 =. ms. The final Nyquist plot looks very similar to that of Figure a. queue size packets 4 8 6 queue size packets 4 8 6 E. Destabilized FAST TCP: closed loop experiment In this subsection, we will use cases with γ > to compare stability predictions of all three models. The objectives here are twofold. First to investigate the critical step size for FAST to maintain stability. This can potentially suggest a larger step size for quicker response. Second, by comparing three models predictions, this closed loop experiment will further strengthen the validation results in Section II-B. where open loop experiments are reported. Example 5: Closed loop validation Two FAST TCP flows share a single link with capacity of pkt/s. The propagation delays of the two flows are 4 ms and 7 ms, respectively. Both flows use α = 5. The open loop transfer functions for all three models and the critical step size γ c for stability predicted by those models are summarized below. The integrator model predicts a critical step size much smaller than the one from the static model, while the joint model yields a prediction in between as expected. Integrator model: γ c =.3 Ls = N n= Static model: γ c =.8 Ls = Joint model: γ c =.69 Ls = N n= /τ n d n γ n e τns µ n s + /ˆτ τns + γ n q N n= µ n d n γ n e τns τ ns + γ n q µ n s + /τ n s + /ˆτ d n γ n e τns τ ns + γ n q 3 3 33 We now report NS- packet level simulations [9]. 4 Figure 3 shows the queue trajectories with γ =.3 and.8, the critical step size predicted by the integrator link model and the static link model. It is clear that the queue is not stable with γ =.8, which means the static model is too optimistic for stability analysis. We further show queue trajectories with γ =.65 and.75 in Figure 4. The case with γ =.65 is still stable which suggests that the integrator model is too conservative, at least in this case, while the queue starts to oscillate with γ =.75, suggesting that the critical step size is indeed approximately γ c =.69. The fluctuations are not due to burstiness; fluctuations of the same magnitude were observed in the window sizes. 4 To validate the link model, the code was modified to update the window once per RTT, and for modeling simplicity the RTT estimate was evaluated over. RTT. All queue trajectories are plotted after initial transients, to emphasize the local stability of the congestion avoidance phase. 4 4 6 8 4 time s a γ =.3 4 4 6 8 4 time s b γ =.8 Fig. 3. Queue trajectories with critical step sizes predicted by the integrator link model and the static link model. queue size packets 4 8 6 4 4 6 8 4 time s Fig. 4. a γ =.65 queue size packets 4 Queue trajectories around critical step sizes. 8 6 4 4 6 8 4 time s b γ =.75 IV. A MORE DETAILED MODEL Although the joint model 3 improves on previous models by considering the window-based ACK-clocking, it still relies on integrating flows rates. Since window flow control doesn t define a flow s rate, the model uses the conventional approximation of the window divided by the round trip time. This section develops a finer timescale definition of the rate of a window-based source. This is used to derive a much more accurate replacement for the transfer function 4, from which the three models in Section II-B are derived as approximations. We will only discuss a single flow sending over a single link and hence drop subscript n for this part. To simplify notation, we assume the source to link delay is τ f =. This is without loss of generality, as it merely translates the congestion window in time. A. The instantaneous queue input rate Instead of, as in [5], specifically studying the evolution of the queue at time instants t k where t k+ = t k + d + p k, we focus on the average rate that a source inputs to the queue over that interval. The amount of data from the source that arrives to the queue in the interval t k, t k+ ] corresponds to the window size at the end of that interval. So, the average source rate x k over such an interval varies interval-wise according to x k+ = w k+ d + p k. 34 The key observation is that this expression holds for all t. For any time t we know the average rate over the interval t, t +d+pt ] as a function of wt +d+pt. Thus, the

Queue size [sec]...8.6.4...8.6.4 NS DAE model.5.5.5 3 3.5 4 4.5 5 Time [sec] Fig. 5. Validation experiment. Solid grey line: NS- simulation. Dashed black line: Continuous model 37. instantaneous rate xt can be defined by the integral equation d + pt t+d+pt t t+d+pt t xsds = wt + d + pt d + pt 35 xsds = wt + d + pt, 36 which relates the trajectory with the constraint on its average value. In terms of rates, the dynamics of the link buffer is simply an integrator, integrating the excess rate at the link. Hence, by using 36, the ACK-clock link dynamics can be described by the following Differential Algebraic Equation DAE, ṗt c xt + x ct c =. + ṗt xt + d + pt ẇt + d + pt xt 37 The plot in Fig. 5 demonstrates the accuracy of the DAE model 37. The scenario is similar to Example, but with only a single window based flow with initial window w = 75 packets that is increased with 75 packets at t = s; and, furthermore, d = 5 ms, c = Mbit/s, x c = 6 Mbit/s. Obviously the model fits extremely well, capturing even the burstiness in the system. B. Relation to existing models Using the standard convention that a delay appearing as an argument is replaced by its equilibrium value, and assuming that ṗt, the linear frequency domain approximation of 37 is ps = G pw sws, G pw s = /c e sτ x c /c. 38 By a Padé approximation of order, of the exponential function, i.e., e sτ, we get G pw s G, pws = c x c. 39 This is exactly the transfer function of the linearized version of the static link model. If we instead use e sτ sτ, this is the, Padé approximant, the transfer function approximation becomes G pw s G, /τx c pws = s + c x c /τx c. 4 This transfer function can also be derived by linearizing and taking the Laplace transform of the nonlinear differential equation x c ṗt = wt d + pt + x c c, 4 which is closely related to the integrator link model. If we put c instead of x c in the left hand side of 4, we get. Obviously the amount of cross traffic scales the dynamics of 4. More cross traffic gives a more sluggish response, and when no cross traffic is present the relation between pt and wt is static. This is line with the observations we made here, in [5], and also in [33], but is not captured by the classical integrator model, with a delayed denominator as in [33] or without as in. In the limit x c c, i.e., when the cross traffic dominates the link, the two approximations become the same. Using a Padé approximation of order,, i.e., e sτ / + sτ, we get G pw s G, s + /τ/c pws = s + c x c /τc. 4 This is equal to the Laplace transform of the linearization of the joint link model 3. Thus we can interpret previous models static, integrator, joint as approximations of 37, and that these approximations have the forms of differential equations. By increasing the order of the Padé approximant and reverse engineering the linear system to the corresponding higher order non-linear differential equation, the accuracy of the resulting simplified model can be increased. We conclude that the joint link model is the best approximation out of the three models compared in this paper, but that it is possible to increase the model accuracy with a higher complexity as a consequence. V. CONCLUSION We have presented a link model which captures the queue dynamics when congestion windows of TCP sources change. The model is shown to be much more accurate than existing ones. It agrees with the known static link model when flows round trip delays are similar, and approximates the standard integrator link model when the heterogeneity of round trip delays is significant. Using this new model, we have shown that FAST TCP is always stable for networks with a single bottleneck link. This extends the existing stability result on homogeneous FAST flows to cases with heterogeneous delays and resolves the notable discrepancy between empirical observations and existing theoretical predictions. The analysis highlights the critical role of self-clocking in TCP stability and the scalability of FAST TCP with respect to delay. Throughout this paper, various open loop and closed loop simulations are used to validate our predictions. In particular, we are able to predict stability region of the closed loop system accurately compared to packet level simulations. There are several possible directions in which to extend this work. For example, we have assumed the number of flows is fixed and it is certainly very interesting to see the effect of this more accurate link model on networks with dynamically

arriving and departing flows [6]. The model itself is of use for any window based TCP algorithm, and it would be useful to investigate its impact on other protocols. So far, we have only looked at its implications for stability. It will be of great interest to see its prediction of performance, such as queue distribution. Finally, the model and analysis remain to be extended to general networks which can potentially have multiple congested links. ACKNOWLEDGMENTS The authors thank David Wei, Fernando Paganini, and Karl Henrik Johansson for valuable discussions. This is part of the Caltech FAST Project supported by NSF, Caltech Lee Center for Advanced Networking, ARO, AFOSR, and Cisco. REFERENCES [] T. Alpcan and T. Basar. Global stability analysis of an end-to-end congestion control scheme for general topology networks with delay. In Proceedings of IEEE Conference on Decision and Control, 3. [] E. Altman, C. Barakat and V. Ramos. Analysis of AIMD protocols over paths with variable delay. In Proceedings of IEEE Infocom, 4. [3] L. L. H. Andrew, S. V. Hanly and R. G. Mukhtar. CLAMP: A system to enhance the performance of wireless access networks. In Proceedings of IEEE Globecom, pp. 44-447, 3. [4] K. J. Åström and B. Wittenmark. Computer-controlled systems 3rd ed. Prentice-Hall, Inc., 997. [5] F. Baccelli and D. Hong. AIMD, fairness and fractal scaling of TCP traffic. In Proceedings of IEEE Infocom,. [6] T. Bonald and L. Massoulie. Impact of fairness on Internet performance. In Proceedings of ACM Sigmetrics, June [7] L. S. Brakmo and L. L. Peterson TCP Vegas: End-to-End Congestion Avoidance on a Global Internet, IEEE Journal on Selected Areas in Communication, Vol. 3, Nov. 8, 995. [8] H. Choe and S. H. Low. Stabilized Vegas. In Proceedings of IEEE Infocom, 3. [9] T. Cui and L. Andrew. FAST TCP simulator module for ns-, version.. Available http://www.cubinlab.ee.mu.oz.au/nsfasttcp. [] J. Choi, K. Koo, J. Lee and S. H. Low. Global stability of FAST TCP in single-link single-source network. In Proceedings of IEEE Conference on Decision and Control, 5. [] S. Deb and R. Srikant. Global stability of congestion controllers for the Internet. IEEE/ACM Transactions on Automatic Control, 486:55-6, June 3. [] G. H. Hardy and E. M. Wright. An Introduction to the Theory of Numbers, 4th ed. Oxford, 96. [3] C. Hollot and Y. Chait. Nonlinear stability analysis for a class of TCP/AQM schemes. In Proceedings of IEEE Conference on Decision and Control,. [4] C. Hollot, V. Misra, D. Towsley and W. Gong. A control theoretic analysis of RED. In Proceedings of IEEE Infocom,. [5] K. Jacobsson, H. Hjalmarsson, and N. Möller. ACK-clock dynamics in network congestion control an inner feedback loop with implications on inelastic flow impact. In Proceedings of the 45th IEEE Conference on Decision and Control, San Diego, USA, December 6. [6] R. Johari and D. Tan. End-to-end congestion control for the Internet: delays and stability. IEEE/ACM Transactions on Networking, 96:88-83, December. [7] R. King, R. Baraniuk and R. Riedi TCP-Africa: An Adaptive and Fair Rapid Increase Rule for Scalable TCP, In Proceedings of IEEE INFOCOM, 5. [8] K. Kaneko, T. Fujikawa, Z. Su and J. Katto, TCP-Fusion: A Hybrid Congestion Control Algorithm for High-speed Networks, In Proceedings PFLDnet, pp. 3 36, 7. [Online] Available http://wil.cs.caltech.edu/ pfldnet7/paper/tcp fusion.pdf. [9] K. Kim, A. Tang and S. H. Low. Design of AQM in supporting TCP based on the well-known AIMD model. In Proceedings of IEEE Globecom, 3. [] K. Kim, A. Tang and S. H. Low. A stabilizing AQM based on virtual queue dynamics in supporting TCP with arbitrary delays. In Proceedings of IEEE CDC, 3. [] S. Liu, T. Basar and R. Srikant. Pitfalls in the fluid modeling of RTT variations in window-based congestion control. In Proceedings of IEEE Infocom, 5. [] S. H. Low, F. Paganini, and J. C. Doyle. Internet congestion control. IEEE Control Systems Magazine, :8 43, Feb.. [3] S. H. Low, F. Paganini, J. Wang, and J. C. Doyle. Linear stability of TCP/RED and a scalable control. Computer Networks Journal, 435:633 647, 3. [4] L. Massoulie. Stability of distributed congestion control with heterogeneous feedback delays. IEEE Transactions on Automatic Control, 476: 895-9, June [5] F. Paganini, Z. Wang, J. C. Doyle and S. H. Low. Congestion control for high performance, stability and fairness in general networks. IEEE/ACM Transactions on Networking, 3:43-56, February 5. [6] H. Shimonishi, T. Hama and T. Murase TCP-Adaptive Reno for Improving Efficiency-Friendliness Tradeoffs of TCP Congestion Control Algorithm In Proc. PFLDnet, 6. [7] R. Shorten, F. Wirth and D. Leith. Modelling TCP in droptail and other environments. Automatica, To appear, 7. [8] R. Srikant. The Mathematics of Internet Congestion Control. Birkhauser, 4. [9] A. Tang, K. Jacobsson, L. L. H. Andrew and S. H. Low An Accurate Link Model and Its Application to Stability Analysis of FAST TCP, in Proceedings of IEEE INFOCOM, 7. [3] K. Tan, J. Song, Q. Zhang, M. Sridharan, A Compound TCP Approach for High-speed and Long Distance Networks, In proceedings of IEEE INFOCOM 6, Barcelona, Spain, Apr. 6. [3] G. Vinnicombe. On the stability of networks operating TCP-like protocols. In Proceedings of IFAC,. [3] J. Wang, A. Tang, and S. H. Low. Local stability of FAST TCP. In Proceedings of IEEE Conference on Decision and Control, Dec. 4. [33] J. Wang, D. X. Wei, and S. H. Low. Modeling and stability of FAST TCP. In IMA Volumes in Mathematics and its Applications, Volume 43: Wireless Communications. P. Agrawal, M. Andrews, P. J. Fleming, G. Yin, and L. Zhang eds., Springer Science, 6. [34] Z. Wang and F. Paganini. Global stability with time-delay in network congestion control. In Proceedings of IEEE Conference on Decision and Control, December. [35] D. Wei, C. Jin, S. H. Low, and S. Hegde. FAST TCP: motivation, architecture, algorithms, performance. IEEE/ACM Transactions on Networking, To appear in 7. [36] L. Ying, G. Dullerud and R. Srikant. Global stability of Internet congestion controllers with heterogeneous delays. In Proceedings of American Control Conference, 4. [37] Y. Liu, F. L. Presti, V. Misra, D. Towsley, and Y. Gu. Fluid models and solutions for large-scale IP networks. In Proceedings of the 3 ACM SIGMETRICS, 3. APPENDIX A RE-ANALYSIS OF THE INTEGRATOR MODEL It was stated in [33] that the integrator link model predicts that for all γ, there is a τ such that a single FAST flow will be unstable. This was the original motivation for proposing the joint model 3. However, the model, also an integrator, actually predicts that FAST should be stable for all γ. This follows from Lemma, as the second term in disappears. Comparing this to the counter-example in Section III-D shows that the more accurate model 3 predicts FAST to be less stable in the worst case than a pure integrator does. The instability predicted in [33] was in fact due to the details of the integrator model studied. There are two important differences between the model of [33] and : a an extra factor of τ in the gain in [33], b an additional delay of τ in the denominator in [33]. We will now show that the extra factor of τ in the gain in [33] is an error. When that is fixed, the model still with the additional delay of τ in fact predicts that for some γ there

Phase margin 3 γ = γ =.75 γ =.5..4.6.8 q/τ ˆq n k ZOH ˆq n t ẇ = G n w n, ˆq n w n t w n k Fig. 7. Continuous time equivalent window control. q n k ZOH q n t ˆq n = H n ˆq n,q n,w n ˆq n t ˆq n k w n t w n k ZOH Fig. 6. Phase-margin plot. Fig. 8. Continuous time split view of the estimator dynamics. is a maximum τ/q for stability, while for others a single flow is always stable. Modeling the link and ignoring estimator dynamics as in [33], the loop gain for the single source case is Ls = s + x c γ x d e sτ c τ τ e sτ τ s + γ q τ. 43 Using τ = d+q, the poles of the closed loop system are given by the characteristic equation s s + γ q τ + x e sτ s + γ = + c τ τ Ls = 44 where Ls = x c e sτ sτ s + γ/τ s + γq/τ = x e sτ c sτ sτ + γ sτ + γq/τ. 45 We can study the poles of the closed loop system i.e., the zeros of the characteristic equation via the Nyquist theorem using either the loop gain Ls or the transformed loop gain Ls. Due to the exponential function in the denominator in Ls it is more convenient to use Ls in the analysis. Note that Ls is similar to the loop gain studied in [33]; what differs is the extra factor of τ used in [33], which arose from the use of a round-trip-time timescale for the window dynamics and an absolute timescale for the link dynamics, without explicit conversion. Let = ωτ. For the case of γ =, the argument of the numerator of 45 when evaluated at s = j is arg e j + argj + = + arctan <. 46 From 45, stability depends on the ratio between the queue delay and the round trip delay, i.e., q/τ. Taking τ/q arbitrarily large gives arg Lj π, and since Lj, the system must encircle + j, and is hence not stable by the Nyquist theorem. Figure 6 shows the phase-margin for different ratios q/τ for different γ s. Observe that for γ =.5 the model system is stable for all q and τ. This shows how sensitive stability is to the delay in the link model. To see why is correct, refer to [5] where it is shown that... APPENDIX B LINEAR CONTINUOUS TIME SOURCE DYNAMICS Since sources work in time-scales of round trip delays, which are usually heterogeneously distributed, and there are different, time-varying sampling times within the source dynamics, we derive continuous time models that are amenable to analysis and equivalent to 5 and 6 at sampling instants. These are used to derive a continuous time approximate mapping qt wt, valid for small perturbations. Consider first the window control block of Fig. 4. We are seeking the nonlinear function Gw, uˆq in Figure 7. Note that 5 has the form Using this insight we assign w[k + ] = Fˆq[k]w[k] + γα. 47 ẇ = Gw, uˆq = guˆq w + g uˆq. 48 Since guˆq is constant over the sample interval t k t < t k+, the state at t k+ is given by wt k+ = expguˆq t k h k wt k +g uˆq hk expguˆq t k s ds where h k = t k+ t k = d + qt k is the observed round trip delay. Identification with 47 yields log Fuˆq guˆq = = g uˆq = h k γαguˆq expguˆq h k log γξ h k, 49 where ξ = uˆq /d + uˆq, so that 48 becomes ẇt = wt log γξt h k t log γξ = α, 5 h k ξ α ξt. 5 Consider now the estimator block in Fig. 4, whose split view is given in Figure 8. We are seeking Hˆq, u q, u w. Repeating the above procedure, but equating to 6 at sampling instants, gives ˆqt = log σt h k t ˆqt u q t, 5 where σt = min{κ/u w t, ν} and u w t is the sampledand-held w[k ] cf. 6b.