IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 17, NO. 6, DECEMBER

IEEE/ACM TRANSACTIONS ON NETWORKING, VOL 17, NO 6, DECEMBER 2009 1697 Understing XCP: Equilibrium Fairness Lachlan L H Andrew, Senior Member, IEEE, Steven H Low, Fellow, IEEE, Bartek P Wydrowski, Member, IEEE Abstract We prove that the XCP equilibrium solves a constrained max-min fairness problem by identifying it with the unique solution of a hierarchy of optimization problems, namely those solved by max-min fair allocation, but solved by XCP under an additional constraint This constraint is due to the bwidth shuffling necessary to obtain fairness We describe an algorithm to compute this equilibrium derive a lower upper bound on link utilization While XCP reduces to max-min allocation at a single link, its behavior in a network can be very different We illustrate that the additional constraint can cause flows to receive an arbitrarily small fraction of their max-min fair allocations We confirm these results using ns2 simulations Index Terms Congestion control, max-min, optimization I INTRODUCTION T CP congestion control [1] has prevented severe congestion while the Internet has undergone explosive growth However, as the network continues to scale in size capacity, the algorithm is struggling with sensitivity to routine packet loss, slow convergence low throughput [2], [3] This has motivated several recent enhancements [4] [10] (See [6] for further references) Of these, XCP [9], [10] has received much attention [11] [15] is especially suitable for private networks where its need for explicit communication between the traffic sources the network is less of a deployment barrier than in the current Internet Unlike proposals that set the flow rates according to the sum of congestion measures at the links of their paths, XCP sets them according to the minimum available capacity in their paths This has the same flavor as MaxNet [16], [17], which sets flow rates according to the maximum of congestion measures in their paths The stability of XCP has been studied when all round-trip times (RTTs) are equal [9], [11] when they differ [13] Specific networks can also be shown to be stable [18] However, that work focuses on very simple topologies This paper takes the complementary approach [19], [20] of reverse engineering XCP to underst its multi-link equilibrium properties Manuscript received February 27, 2008; revised October 24, 2008; approved by IEEE/ACM TRANSACTIONS ON NETWORKING Editor S Shakkottai First published August 18, 2009; current version published December 16, 2009 This work was supported by NSF Grant 0303620 Partial results have appeared in the Proceedings of IEEE INFOCOM, 2005 L L H Andrew is with Swinburne University of Technology, Hawthorn, Vic 3122, Australia (e-mail: lrew@ieeeorg) S H Low is with the California Institute of Technology, Pasadena, CA 91125 USA (e-mail: slow@caltechedu) B P Wydrowski is with Google Inc, New York, NY 10018 USA (e-mail: bwydrowski@googlecom) Color versions of one or more of the figures in this paper are available online at http://ieeexploreieeeorg Digital Object Identifier 101109/TNET20092013242 Although packet networks seldom achieve equilibrium, equilibrium analysis indicates the best-case fairness capacity properties of a protocol Moreover, dynamic analysis typically starts with a linearization around the equilibrium, which requires a clear characterization such as is presented here We do not consider misbehaving nodes, as studied in [14], or errors in capacity estimates, as studied in [11] A deterministic fluid model of a general XCP network with multiple links multiple flows is presented in Section II Section III analyzes the equilibrium rates of XCP shows that all queues are empty in equilibrium We prove the existence uniqueness of XCP equilibrium rates by identifying them with the unique solution to a hierarchy of optimization problems This is the same set of problems solved by the stard max-min fair allocation [21], but XCP solves them under an additional constraint While XCP reduces to max-min allocation at a single link, its behavior in a network can be very different We describe an algorithm to compute this equilibrium derive upper lower bounds on link utilization In Section IV, we use these bounds to investigate the impact of the choice of protocol parameters on link utilization under the additional constraint We show that flows can receive an arbitrarily small fraction of their max-min fair allocations Specifically, with a max-min fair allocation, as long as a link is a bottleneck for some (not necessarily all) flows that pass through it, it will be fully utilized Under XCP, this is no longer true: When the majority of flows using a link are bottlenecked at other links, the remaining flows at that link may not fully use the residual bwidth With the parameters specified in [9], however, link utilization is at least 80% at any link XCP has a shuffling parameter to prevent the network from settling into an unfair state [9] We show that, given any network topology, we can choose sufficiently small so that the resulting allocation is close to max-min fairness For any fixed, however, there are topologies in which some flow rates can be far away from their max-min allocations These properties the accuracy of our algorithm are verified by ns2 simulations in Section V We conclude in Section VI with limitations of this work II MODEL Consider a network with links shared by flows Sources are indexed by, links by packets by Let be the routing matrix: if flow uses link, 0 otherwise Let be the set of links in the path of flow, be the set of flows that use link Note that 1063-6692/$2600 2009 IEEE

1698 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL 17, NO 6, DECEMBER 2009 We will present a continuous-time fluid model of XCP For flows, define the following variables: : window size at time, in packets; : round-trip propagation ( fixed processing) delay; : round-trip time (RTT) at time ; : flow rate at time For links, define the following variables: : capacity, in packets/s : backlog at time, in packets : aggregate input rate at link at time In equilibrium, we sometimes write to emphasize the dependence on equilibrium rates An XCP link divides time into control intervals of duration, which is also used as a time scaling parameter Nominally, is the mean RTT of all the flows at link [9] We assume to be constant in time, which is reasonable as we will show that there is no queueing delay in equilibrium To simplify notation, we assume all packets have size 1 unit We use flow source interchangeably A XCP Description We now summarize the XCP algorithm See [9] for a detailed description The equilibrium is independent of feedback delay, hence such delay is not modeled For each packet, XCP generates a feedback signal prescribing a change in window size Let be the feedback generated by link for packet at time The acknowledgment for packet received by its source contains in its header the smallest feedback generated by links along its path The source adds this quantity to its current window size 1 We now describe how to compute the feedback Let where are constants, is the link capacity, is the aggregate input rate, is the backlog at time Let The feedback on the th packet at link is is a traffic shuffling term with a constant (Note that we are using the definition of from the Appendix of [9], which differs by a factor of from that used in the corresponding equation in [9]) B Dynamic Model We now translate the per-packet feedback into per-flow feedback Let be the feedback generated by link for flow at time In general, a quantity with a tilde pertains to a packet while the corresponding variable without a tilde pertains to a flow Substituting in (1) gives Recall is the total number of packets arriving at link in period For simplicity, we assume that Of these packets, we assume that flow Hence Thus, the feedback (3) yields feedback to flow again, the per-packet feed- per packet Using back (2) becomes (3) packets are from The feedback per packet to flow from link is then where are the increase decrease components, respectively (1) (2) If flow does not use link, then set Let be the minimum feedback along s path Since source receives feedback packets per unit time (assuming every packet carries control information is acknowledged), its window evolves according to where are the round-trip time window size, respectively, of the flow that transmitted packet, is the total number of packets seen by link over the time interval Here Substituting,wehave 1 In practice, the window size has a lower bound of 1 packet, but for notational simplicity, we ignore this Remark: The pseudo code in [9] contains additional residual terms These terms, modeled in [20], have no effect on the equilibrium if the average rate of flows bottlenecked

ANDREW et al: UNDERSTANDING XCP: EQUILIBRIUM AND FAIRNESS 1699 at upstream links is significant Otherwise, the resulting link utilization is slightly increased (by around 4% in Scenario 1 of Section V) Since these terms hardly affect the equilibrium, we ignore them in the analysis in this paper In summary, an XCP network is described by the following set of equations: We distinguish between links that are bottlenecks those that are not Let be the set of links that are bottlenecks for source wrt a given equilibrium rate where if if (4a) (4b) be the set of links in s path that are not bottlenecks for source wrt We also distinguish between sources that are bottleneck locally those that are not Let be the set of sources bottlenecked at link wrt a given equilibrium rate (5a) (5b) (5c) (5d) (5e) (5f) Here, are constants, Stard XCP uses We will study the behavior of the general model, which includes this as a special case As we will see below, the qualitative properties, such as existence uniqueness of equilibrium rates, do not depend on specific values of these parameters (as long as ) be the set of sources bottlenecked elsewhere Let be the number of sources at link, Let be the fraction of flows through link which are not bottlenecked at link, be the fraction of the link s capacity consumed by such flows Note that while, depend only on the routing matrix, depend also on the equilibrium rate through From (4) the definition of, we have the following Lemma 1: The rate backlog vector is in equilibrium if only if 1) for all with equality if, 2) for all Moreover, 3) if then ; 4) if then implies Proof: Parts 1 to 3 are immediate To see part 4, note that implies III EQUILIBRIUM RATES This section characterizes the equilibrium of XCP describes an algorithm to compute it; the next considers the implications of these results on utilization fairness Equations (4) (5) describe the evolution of the window vector, for all the backlog vector for all A pair of rate backlog vectors, with window vector given by, is said to be in equilibrium if both To avoid technicalities, we only consider cases with ; under a minor reformulation, it can be shown that in equilibrium only if, all theorems lemmas still hold We start by defining a bottleneck link other notation for XCP equilibrium In general, quantities without dependence denote equilibrium quantities, eg, Definition 1: A link is said to be a bottleneck for source with respect to (wrt) if is minimum among all the links that uses, ie, In this case, source is said to be bottlenecked at link wrt By definition, every source has a bottleneck Lemma 1 below implies that in equilibrium at bottleneck By part 2, for all Since at most one of is nonzero,, whence A The Need for Bwidth Shuffling Without bwidth shuffling, XCP would have, giving for all, in equilibrium Theorem 1: Suppose Then, with is an equilibrium if only if 1) for all, 2) for all, there exists with Proof: The first condition in the theorem implies that for all Combined with, this implies for all The second condition then implies that for all Hence, the conditions in the theorem are sufficient, by (5b) the first half of Lemma 1 For necessity, there are two cases If then by Lemma 1 part 4, (4b) implies, since, Otherwise, by definition of This implies, whence in equilibrium

1700 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL 17, NO 6, DECEMBER 2009 Remark: Without bwidth shuffling, any (possibly unfair) boundary point of the set would be an equilibrium This is why XCP uses [9] The rest of the paper considers the more complicated case of B Case: Definitions Preliminaries This subsection provides a conceptually simple characterization which is used in the next subsection to prove the existence uniqueness of XCP equilibrium Following that, we provide an iterative algorithm to compute this equilibrium From (4) (5) Lemma 1, is an XCP equilibrium if only if it satisfies conditions 1 2 of Lemma 1 By (5a), condition 2 becomes: for all, for all Hence Since both numerator denominator are positive, This implies whence Type c) links are bottleneck links for some but not all of the flows using them From (6), we have where the inequality follows from Lemma 2 As for type b) links, this implies We next characterize XCP s equilibrium rates Let (6) with equality for some Hence for links with, all flows that are bottlenecked at link must have the common rate This has important implications as we will see below Several of the results will use the following technical lemma, which is proved in Appendix A Lemma 2: For all 1) if ; 2) if ; 3) with equality if only if ; 4) if ; 2 5) with equality if only if Unlike in the case, we characterize the equilibrium backlogs rates separately The following result says that the equilibrium queue under XCP is zero This originates from the definition of in (5b), which is nonnegative in equilibrium The same property is used in REM [22] to drive the queue to zero, or more generally, to a target value Theorem 2: In equilibrium, for all Proof: Links can be of three types: a) ; b) ; c) Each of these will be considered in turn Type a) links are bottlenecks for all flows passing through them, ie, (6) holds with equality for all Since all flows have common rate, whence equality in (6) implies Thus,, (5b) implies, ie, they share the link capacity fully equally, with no queueing delay Type b) links are not bottlenecks for any of the flows they carry Hence, for all, Multiplying both sides by summing over,wehave 2 h > 0 implies y =c >=( + ) Theorem 6 provides a tighter lower bound where Since depends on only through, we will abuse notation also write or Define the feasible set of source rates to be where denotes the set of nonnegative real numbers We will later show that the XCP equilibrium must be in Note that implies To see this, multiply both sides of the inequalities in (7) by sum over to get whence The converse may not be true, ie, may be a strict subset of Intuitively, exceeds the equal-share rate by a factor that increases as the link underutilization increases as the bwidth shuffling decreases It arises since the link tries to shuffle bwidth from flows it controls to ones it does not Thus, restricts the utilization of links which are not bottlenecks for all flows passing through them Our main result is to prove the existence uniqueness of XCP equilibrium in a general network, that this equilibrium solves a constrained max-min fairness problem Definition 2: A rate vector is constrained max-min fair if for any other feasible implies that there is a with Intuitively, a constrained max-min fair vector is such that it is not possible to increase a component without reducing another smaller or equal component This differs from stard max-min fairness only in that the feasible set is a subset of [23] This restriction has important ramifications, as we will see in the next section We will prove constructively that the unique XCP equilibrium is constrained max-min fair by identifying it with the solution of (7)

ANDREW et al: UNDERSTANDING XCP: EQUILIBRIUM AND FAIRNESS 1701 a hierarchy of optimization problems over the feasible set : it maximizes the smallest source rates in, then maximizes the second smallest rates over all rates that solve the first problem, so on These problems are defined inductively, following the idea of [24] Let The sets define the first problem, whose solution is described by the sets These sets in turn define the second problem, so on To simplify notation, let Given sets,if contains all flows, then we stop Otherwise, we define problem its solution, as follows: Let (8) (9) (10) (11) (12) A few important properties are immediate from these definitions First, the rates are monotonic (13) proving the first condition Condition (6) then implies the second condition Conversely, suppose the two conditions are satisfied If, then from (5a) Lemma 1-4 Lemma 1-2 then implies is the minimum among links in source s path, ie, link is a bottleneck On the other h, if, then, as above, Then, is equivalent to, proving that is a bottleneck Motivated by this lemma, we call link a nonbottleneck wrt if either or for all C Case: Main Result We are now ready to state the main result Theorem 3: The problems are well-defined have a unique solution Moreover, the following are equivalent: 1) is an XCP equilibrium 2) is the unique rate vector that solves all the problems 3) is constrained max-min fair 4) every flow has a bottleneck wrt, ie, for all, there is an such that for all The remainder of this subsection is a proof of this theorem It will use lemmas, whose proofs are relegated to Appendix B We start with a simple observation that greatly simplifies the solution of Lemma 4: Suppose is nonempty The maximization in (9) can be taken over that have equal for In view of Lemma 4, we can replace in (12), for, by their subsets Second, are nonempty; moreover, they are disjoint from, respectively Hence, will eventually contain all the flows there are only a finite number of problems Finally, are strictly nested use them instead of in computing : (15) Indeed it will become clear that is exactly the set of solutions to problem, ie, is the set of feasible rates whose smallest rates are maximized, is a subset of whose second smallest rates are also maximized, so on We prove below that if is the last problem, then is a singleton that solves all problems To contrast XCP equilibrium with the stard max-min fair allocation, we derive a bottleneck characterization that is analogous to that for max-min fairness; see the beginning of Section IV Lemma 3: Let be an XCP equilibrium rate vector Link is a bottleneck for source wrt if only if 1), 2) for all Proof: Suppose link is a bottleneck link for source wrt equilibrium Then, Lemma 1-2 implies that, ie, equality holds in (6) Since by Theorem 2 by Lemma 2, (5c) becomes Thus from (6) (14) This greatly reduces the complexity of (9) from maximizing over -vectors to over a scalar Denote an by, with (16) let Note that,is not in according to definition (12), though it is in We will see in Lemma 6 below that plays an important role in the proof of Theorem 3 The vector induces link flows This motivates the following main technical lemma (17)

1702 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL 17, NO 6, DECEMBER 2009 Lemma 5: Given any scalars,, define (18) for some 1) If either or, then there exists a unique such that, where if only if 2) Moreover, over if only if For later reference, we will denote the mapping from to the unique in Lemma 5 by (19) This function is used in the rest of the proof Lemma 5 implies that if link is a bottleneck for some source with respect to an, then the rate of source cannot be increased without violating the feasibility constraint in (7) For instance, let be such that Setting gives is a bottleneck for all wrt Lemma 5-2 then implies that rates greater than are infeasible at link The next lemma implies that all links are bottlenecks wrt all, all links are nonbottlenecks wrt In particular, this implies that are nonempty Lemma 6: For each, 1) if, then for all wrt all 2) if, then either or for all Lemmas 5 6 suggest the following abstract algorithm to compute the solution of problems An analogous concrete algorithm to find the XCP equilibrium will be given in the Subsection III-D At the start of iteration, all links, are bottlenecks for some sources wrt all A source passes through at least one bottleneck, hence its rate cannot be raised further without violating the constraint in (7) All links are nonbottleneck links wrt defined in (16) Sources pass through only these nonbottlenecks, hence their rates can be increased further, starting from At each nonbottleneck link, nonbottlenecked sources can raise their rates to (19), (16), (17) other nonminimizing links remain nonbottleneck wrt the new rates, the cycle repeats, until all sources are assigned their bottleneck rates The solution of each problem fixes the components, to be rate, until all components have been assigned Hence, if is the last problem, then is a singleton The above discussion is summarized in the following lemma, which justifies Theorem 3 Lemma 7: The problems are well defined is exactly the set of solutions to problem There is a unique solution to the hierarchy of problems We now prove Theorem 3 Proof (Theorem 3): Lemma 7 implies that are well defined have a unique solution It is clear that characterizations 2 3 are equivalent, ie, is the unique solution to the hierarchy of problems if only if it is constrained max-min fair We will first prove the equivalence of characterizations 3 4, then that of 1 4 We will use the equivalent definition of bottleneck links in Lemma 3 Equivalence of characterizations 3 4: We will prove that is constrained max-min fair if only if both every flow has a bottleneck link wrt, ie, for all, there is an such that for all The proof follows the same approach as the corresponding result for stard max-min fairness; see [23] The difference is in the use of Lemma 5 because of the more complicated feasible set Suppose every flow has a bottleneck link wrt If is not constrained max-min fair, then there exists another such that for some, if then We will derive a contradiction Let be a bottleneck for wrt Then, for all, hence for all, with Write the link flow due to rates in terms of the link flow due to Let the scalar be the average Then, the rate vector defined by if otherwise induces the same flow rate at link as does (20) to make a bottleneck The smallest of these rates, smallest over, is It is assigned to all previously nonbottlenecked sources going through the new bottleneck links is the optimal objective value for problem These new bottleneck links are collected into, the newly bottlenecked sources into their rates into (or ) The Since, is feasible, we must have, for all (21)

ANDREW et al: UNDERSTANDING XCP: EQUILIBRIUM AND FAIRNESS 1703 is also fea- where the last equality follows from (20) Hence, sible Since link is a bottleneck for source wrt where (25) (22) so applying Lemma 5-1 with gives Since for all, with, Lemma 5-2 (20) imply that contradicting (21) Hence, is constrained max-min fair Conversely, let be constrained max-min fair If there is a source that has no bottleneck link wrt, then for all, either or for all Lemma 5 then implies that there exists a unique scalar, given by such that, given by for, strictly increases component without having to reduce other components, contradicting the fact that is constrained max-min fair Equivalence of characterizations 1 4: We will prove that a vector is an XCP equilibrium if only if every flow has a bottleneck link wrt The discussion at the beginning of Section III-B shows that is an XCP equilibrium if only if, for all, (6) holds for all, with equality for some This, with (14), establishes As observed after Definition 1, every flow has a bottleneck by definition To show characterization 4 implies characterization 1, it suffices to show that the characterization in Lemma 3 implies statements 1 2 of Lemma 1 The discussion after (7), setting, establishes statement 1 This shows for all If then (5c) (5a) give, with equality when Otherwise, giving, by (5a), D Algorithm for Computing Equilibrium The equilibrium rates of XCP can be found using an algorithm analogous to that of [23] for max-min fairness However, because the constraint on the link throughput in (6) depends on the aggregate flow rate through, some extra bookkeeping is required In [20], an approximation for was given in terms of Here, we use the exact expression from [19] Theorem 4: The utilization of a bottleneck satisfies The rates of all sources bottlenecked at satisfy (23) (24) Proof: Substituting into (14) solving the resulting quadratic equation gives (26) where By Lemma 8 in Appendix A, only the larger solution of (26) satisfies part 5 of Lemma 2 is a valid equilibrium Rearranging the term in the square root gives (23) To obtain (24), instead substitute into (14), giving where (27) Since is increasing in, it is again only the larger root that represents the XCP equilibrium Thus where is given in (25) Rearranging the expression in the square root gives (24) Note that the right-h side of (24) depends on the rate vector through Hence, it is not an explicit formula for the throughput of a general flow However, it says that the common bottleneck rate at each link depends on the rate vector only through that are bottlenecked elsewhere These are source rates smaller than the bottleneck rate at link by Lemma 1 This motivates an algorithm similar to the max-min algorithm of [23] that calculates the throughput of each flow in increasing order, without the need for recourse to simulation 1 Set for all 2 repeat 21 For each link, find from (24) using from rates already allocated 22 Set 23 Set 24 foreach 241 Set 242 For each flow, set endfor 25 Set

1704 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL 17, NO 6, DECEMBER 2009 26 Set 27 Set 28 foreach 281 Set 282 Set endfor 29 Set until This solves each of the optimization problems,, in turn The key is that, by keeping track of the used capacity of each link,, it can compute the maximization in (9) in closed form For each, the values vary during the algorithm For the algorithm to be correct, they must have the right values when link is the minimum in step 22 This occurs as long as the link rates are allocated in increasing order, as is guaranteed by the following theorem, proved in Appendix C Theorem 5: The above algorithm calculates the correct equilibrium rates of XCP If, then (24) reduces to, hence the algorithm reduces to the algorithm in [23] to compute the max-min fair allocation This suggests that, given any topology specified by the routing matrix link capacity vector, one can choose to be sufficiently small so that the equilibrium of (4) is close to max-min fair On the other h, with small, the convergence of individual rates to fairness can be very slow We will return to this point in the following section IV UTILIZATION AND FAIRNESS In this section, we discuss some implications of the results in Section III on link utilization fairness of the equilibrium rates Theorem 3 shows that XCP equilibrium is constrained max-min fair It is instructive to compare the XCP equilibrium with the (stard) max-min fair allocation a class of algorithms proposed in [24] It is proved in [24] that a (stard) max-min fair rate vector is the unique solution of the same hierarchy of problems (8) (12) defined in Section III, except that the feasible set in (7) is replaced with the superset (28) The key feature that results from this much simpler feasible set is that the bottleneck links under a max-min fair allocation are all fully utilized Indeed, a rate vector is max-min fair if only if, for every source, there is a link in its path such that [23] 1) 2) for all From Theorem 3, condition 1 is replaced with the fixed point equation for XCP equilibrium This more complex condition has several implications First, it precludes the much simpler proof techniques used in [24] to show the max-min fair vector is the unique solution of the problems Second, the algorithm in Section III-D to compute the constrained max-min fair rate vector is more complex than the (centralized) one for the max-min fair vector [24], [23] Third, most importantly, the XCP equilibrium can underutilize link capacities deviate by an arbitrarily large factor from the max-min fair allocation, as we illustrate below Max-min fairness is generalized in [24] by restricting the feasible set to a (strict) subset of in (28) Like XCP, the restriction is specified as additional constraints on source rates link flows Like XCP, explicit feedback is required: Each link feeds back the spare capacity to sources that go through this link Sources adjust their individual rates based on feedback on its path in a way that is distributed, yet avoids overshoot MaxNet [16] also allows explicit control of link utilization fairness by separate link source parameters We now illustrate the effect of the additional constraint (7) in XCP on link utilization fairness As we explained in the proof of Theorem 2, there are three types of links The first type are bottlenecks for all the flows that go through that link All links of this type, such as all in problem, are fully utilized, The second type are bottlenecks for none of the flows that go through that link They are underutilized,, because the flow rates going through the link are constrained elsewhere The third type are bottlenecks for some, but not all, of the flows that go through the link In contrast to the stard max-min fair allocation, these links are also underutilized, partial bottlenecks Theorem 6: If We can bound the utilization of these for some, then Proof: Noting that ( that ), removing the last term from the square root in (23) gives the lower bound (29) where the second inequality is an equality if To derive the upper bound, first note that from Lemma 2-2 Since, removing the last term from the square root of (24) yields (30) Multiplying both sides by adding lead to the upper bound on utilization Substituting either or into either the exact expressions (23) (24) or the upper lower bounds (29) (30) gives full utilization as in the max-min case: This shows that XCP could be made to approach max-min fairness if the bwidth shuffling were reduced On the other h, link utilization could be arbitrarily low if had been chosen poorly With the values suggested in [9] however the utilization is at least 80% Consider a network of two links Link 1 has carries flows, while link 2 has carries flows, consisting of all the traffic on link 1 plus one other flow As we

ANDREW et al: UNDERSTANDING XCP: EQUILIBRIUM AND FAIRNESS 1705 get This gives in the limit Thus, both terms in the square root of (23) go to zero, whence (29) becomes tight, as However, with [9], for which positive stability results exist, (23) gives Similarly, a given flow may obtain an arbitrarily small proportion of its max-min fair bwidth for any The ratio of the upper bound on XCP bwidth (30) to the max-min fair bwidth,, is minimized with respect to when Substituting this value into (24) dividing by gives Fig 1 Topology for Scenarios 1 2 (31) where Thus Fig 2 Scenario 1: Utilization Applying the identity, for,gives (32) In the limit as, the right-h side tends to 0 for any This demonstrates that, for any nonzero amount of bwidth shuffling, XCP can be arbitrarily unfair for some topology Hence, although the equilibrium of (4) converges to max-min as, this convergence is not uniform with respect to topology In other words, given any topology specified by, we can choose sufficiently small so that the resulting allocation is close to max-min fairness However, for any fixed, such as 01 used by XCP, there are topologies in which some source rates can be far away from their max-min allocations This behavior can be exhibited by a simple two-link network: One link has capacity 1 carries flows, while the other carries of those same flows has capacity This network has Hence, as These asymptotic results will be illustrated confirmed by simulation in the following section V SIMULATION RESULTS This section presents simulation results using the implementation available from [9] for NS-2 [25] These results verify the accuracy of our algorithm in Section III-D confirm our qualitative discussion in Section IV on the utilization fairness properties of XCP All sources always have packets to send Links are romly assigned delays between 5 500 ms, uniformly on a log scale Results for different realisations were indistinguishable The XCP default parameters are used Although the analysis neglects the residual terms, the simulations include them However, as remarked in Section II-B, they have minimal impact on equilibrium properties, yielding a good match between theory simulation Although XCP has been found to be unstable in specific circumstances [26], all of the simulations here exhibited stable dynamics The topology used for Scenarios 1 2 is shown in Fig 1 consists of two links, with sources traversing link L1 sources traversing L2 Using protocols that have nonempty queues in eqiulibrium, flows in each direction are coupled through changes in the RTT a process known as ACK compression Since XCP maintains empty queues at eqiulibrium, it is sufficient to consider unidirectional flows, which we do here for clarity Scenario 1 investigates the utilization of L1 as the number of sources traversing L1 L2 is changed In the experiment, with Mbps Mbps The utilization of L1 for a range of is shown in Fig 2 A max-min fair allocation would result in a full utilization of L1 for all combinations However, as the number of sources bottlenecked at L2 increases, XCP s utilization of L1 decreases Since XCP s residual terms depend on feedback from upstream nodes, the equilibrium rates depend on the order in which links are traversed If the direction of flow in this network were reversed, then the utilization would be 0% 4% higher than for the case considered than the theoretical predictions Scenario 2 demonstrates that XCP can be arbitrarily unfair for some topology Let Mbps,

1706 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL 17, NO 6, DECEMBER 2009 Fig 3 Scenario 2: Unfairness, r =r! 0 as n!1 Fig 6 Scenario 4: Throughputs Fig 4 Scenario 3: Utilization of L1 as function of = Fig 5 Scenario 3 topology The ratio of the rate of the source traversing only L1 to the max-min fair rate is plotted in Fig 3 Indeed the unfairness increases with the number of sources in the network, confirming the theory Scenario 3 studies XCP with nonstard parameters It verifies that as We set Mbps, The parameter is varied from 0512 to 0016 the utilisation of L1 as a function of, as well as the lower bound from (29), are plotted in Fig 4 Scenario 4 tests the rate allocation algorithm for a more complicated topology as shown in 5 The link capacities in Mbps are is varied in this experiment Delay is set to ms The source rates are plotted in Fig 6 There is a good agreement between the predicted measured rates even though the lower bwidth delay product makes the fluid flow approximation more questionable VI CONCLUSION We have presented a dynamic model of XCP used it to completely characterize its equilibrium properties We have shown that XCP clears the queues in equilibrium has unique equilibrium rates that solve a constrained max-min fairness problem The additional constraint under XCP can lead to unfairness for some network topologies XCP gives a utilization of at least 80%, but a poor choice of or could lead to arbitrarily low utilization We have provided an algorithm to compute the equilibrium for general networks, have presented simulation results to illustrate these findings An important question that we have not pursued is the dynamic properties of XCP, such as its stability Even though the residual terms in the XCP code do not seem to affect equilibrium properties drastically, they may be important in determining its dynamic properties, hence, should be taken into account in such an analysis It is important to underst the stability of individual source rates, in addition to the aggregate rate, as studied in [9], in general networks in the presence of nonuniform delay Since equilibrium queues are zero, the usual practice of linearizing around the equilibrium needs caution at the tightest bottlenecks that have zero queue yet full utilization APPENDIX A VALID EQUILIBRIUM RATE Proof (Lemma 2): 1) By Lemma 1, This implies 2) If this were not the case, then the average rate of flows in ( hence the rate of at least one such flow) would be greater than Part 1 would then require the total rate to exceed the capacity

ANDREW et al: UNDERSTANDING XCP: EQUILIBRIUM AND FAIRNESS 1707 3) From part 1 because for original is in For since the with equality if only if 4) Otherwise, since (by Lemma 1, part 4) Then,, whence 5) with equality if only if Lemma 8: The smaller solution to (26) does not give a valid equilibrium rate for link Proof: For a rate,, to be valid requires, by Lemma 2(5) Let Then, dividing the numerator denominator of (26) by gives If the lower root were valid, then where the first inequality follows because is the average of, the last inequality follows because the original is in Hence, if achieves the maximum in (9), the vector with a common value also achieves the maximum Proof (Lemma 5): Define We will show that there exists, with equality if only if, such that under the conditions given in the lemma Consider the two cases separately Case 1: If then Otherwise, it remains to show that there exists a suitable Consider whence However,, which yields a contradiction Thus, the lower root is not valid APPENDIX B PROPERTIES OF OPTIMIZATION PROBLEMS Proof (Lemma 4): Note that depends on only through Now, are all of the form where Write as Hence, we can write also as a function of Case 2: since This implies that there exists sufficiently large such that Since is continuous on, there exists an such that This is illustrated in Fig 7 Then hence Given any, define the average by (33) consider the vector with equal components From (33), this vector produces the same link flow Moreover, defined by this vector also satisfies for all, hence is in This is Moreover, as from above, Since is continuous for, there exists such that, ie, The same argument as in Case 1 shows that there exists an with Since is continuous on, there exists where We now prove that is unique The argument also shows that over if only if First note that may be negative They key observation, illustrated in Fig 7, is that 1) for is negative, concave, approaches as from below;

1708 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL 17, NO 6, DECEMBER 2009 If then Otherwise, We claim for all If not, then Fig 7 ^g () in (18) f () =^g () 0 (s + ) yielding, a contradiction Hence, if, then either or for all Induction hypothesis: Suppose the second assertion holds for We will prove it for Induction: Fix an First note that Consider all links By the induction hypothesis, either or for all For an, we have from (10), for 2) for is positive, convex, attains its minimum at where, from (17) approaches as from above A moment of thought then convinces one that it suffices to show that grows less rapidly than, ie, Letting Since,wehave, if (34) is violated, the above implies that (34) From (18),, hence the induction hypothesis implies that either or Lemma 5 then implies that, for each, there exists a unique such that The minimum over is, these minimizing constitute All the sources that go through a link in are assigned the common rate, they are collected into Since, the corresponding satisfies which implies (35a) (35b) which is a contradiction since Hence, (34) must hold if only if whenever Proof (Lemma 6): The first assertion follows directly from the definitions of in (10), (11) (15), respectively We will prove the second assertion by induction on Base case : Fix any Consider an By (13), Now Hence However, for all Hence (35) Lemma 5-2 (with ) imply that either or The proof is completed by noting that, since APPENDIX C CORRECTNESS OF RATE ALGORITHM To show the correctness of the algorithm in Section III-D to find XCP s equilibrium rates, it suffices to show that the results satisfy the two conditions stated in Section III-B Each source is assigned a rate in the same step as a particular link If is chosen according to (24) with the true equilibrium

ANDREW et al: UNDERSTANDING XCP: EQUILIBRIUM AND FAIRNESS 1709 values of, then, since that is the condition from which (24) was derived Similarly, (24) implies,as established in the proof of Theorem 6 The correctness of the algorithm can thus be established by showing that the true equilibrium values of are used when the final value is calculated in step 241 For each link, the values of depend only on network parameters flows with rates, by Lemma 2-1 That is, if the rates selected by the algorithm are such that the rate of each link is greater than the rates of the flows flowing through but bottlenecked elsewhere, then the rates must form an equilibrium of XCP The theorem then results from the following lemma Lemma 9: For each for which is defined, Moreover, if, then for each for which is defined, Proof: Consider an arbitrary link,, iteration, If does not carry any flows in, then as required Consider now the case that does carry a flow in Let be the rate of flows allocated in iteration, normalized to link Let Then are the fraction of allocated flows allocated capacity on link before step 241 of iteration, while are the values after the update, where Let be the value of calculated from (27) using Note that by Lemma 2-3, since Moreover,, since if then link would have been an element of would not be defined To prove the lemma, it is sufficient to show that that if then Below, the argument will be dropped when no ambiguity can arise Differentiating (27) with respect to, noting that is the larger of the two solutions of (27), gives Now giving If, then, remains greater than This is because, since This establishes the second part of the lemma If, then the right-h side need not be positive However, the second factor is bounded, so approaches exponentially as increases, so can never drop below In particular,, which establishes the first part of the lemma ACKNOWLEDGMENT The authors would like to thank D Katabi of Massachusetts Institute of Technology (MIT), Cambridge, for helpful discussions REFERENCES [1] V Jacobson, Congestion avoidance control, in Proc ACM SIG- COMM, Aug 1988, pp 314 329 [2] C Hollot, V Misra, D Towsley, W Gong, Analysis design of controllers for AQM routers supporting TCP flows, IEEE Trans Autom Control, vol 47, no 6, pp 945 959, Jun 2002 [3] S H Low, F Paganini, J Wang, J C Doyle, Linear stability of TCP/RED a scalable control, Comput Netw J, vol 43, no 5, pp 633 647, 2003 [4] C Casetti, M Gerla, S Mascolo, M Sansadidi, R Wang, TCP Westwood: End-to-end congestion control for wired/wireless networks, Wireless Netw J, vol 8, pp 467 479, 2002 [5] S Floyd, High-speed TCP for large congestion windows, IETF, RFC 3649, Dec 2003 [6] D X Wei, C Jin, S H Low, FAST TCP: Motivation, architecture, algorithms, performance, IEEE/ACM Trans Netw, vol 14, no 6, pp 1246 1259, Dec 2006 [7] I Rhee L Xu, CUBIC: A new TCP-friendly high-speed TCP variant, presented at the PFLDnet, 2005 [8] D Leith R Shorten, H-TCP: TCP for high-speed long-distance networks, presented at the PFLDnet, Argonne, IL, 2004, in Proc [9] D Katabi, M Hley, C Rohrs, Congestion control for highbwidth delay product networks, in Proc ACM SIGCOMM, 2002 [10] A Falk, D Katabi, Y Pryadkin, Specification for the Explicit Control Protocol (XCP), draft-falk-xcp-03txt, 2007 [11] Y Zhang M Ahmed, A control theoretic analysis of XCP, in Proc IEEE GLOBECOM, Mar 2005, pp 2831 2835 [12] F Abrantes M Ricardo, XCP for shared access multirate media, Comput Commun Rev, vol 36, pp 29 38, Jul 2006 [13] Y Sakumoto, H Ohsaki, M Imase, On XCP stability in a heterogeneous network, in Proc Int Symp Comput Commun, 2007, pp 531 537 [14] C Wilson, C Coakley, B Y Zhao, Fairness attacks in the explicit control protocol, in Proc IEEE IWQoS, 2007, pp 21 28 [15] Y Zhang T Henderson, An implementation experimental study of the explicit control protocol (XCP), in Proc IEEE IN- FOCOM, Miami, FL, 2005, pp 1037 1048 [16] B Wydrowski M Zukerman, MaxNet: A congestion control architecture for max-min fairness, IEEE Commun Lett, vol 6, no 11, pp 512 514, Nov 2002 [17] B Wydrowski, L L H Andrew, M Zukerman, MaxNet: A congestion control architecture for scalable networks, IEEE Commun Lett, vol 7, no 10, pp 511 513, Oct 2003 [18] H Balakrishnan, N Dukkipati, N McKeown, C J Tomlin, Stability analysis of explicit congestion control protocols, IEEE Commun Lett, vol 11, no 10, pp 823 825, Oct 2007 [19] S Low, L Andrew, B Wydrowski, Understing XCP: Equilibrium fairness, in Proc IEEE INFOCOM, 2005, pp 1025 1036 [20] P Wang D L Mills, Further analysis of XCP equilibrium performance, in Proc IEEE Globecom, 2006, pp 1 5 [21] J M Jaffe, Bottleneck flow control, IEEE Trans Commun, vol COM-29, no 7, pp 954 962, Jul 1981 [22] S Athuraliya, V H Li, S H Low, Q Yin, REM: Active queue management, IEEE Network, vol 15, pp 48 53, May/June 2001 [23] D Bertsekas R Gallager, Data Networks, 2nd ed Englewood Cliffs, NJ: Prentice-Hall, 1992 [24] E M Gafni D P Bertsekas, Dynamic control of session input rates in communication networks, IEEE Trans Autom Control, vol AC-29, no 1, pp 1009 1016, Jan 1984 [25] NS Network Simulator, [Online] Available: http://wwwisiedu/ nsnam/ns/ [26] L L H Andrew, B P Wydrowski, S H Low, An example of instability in XCP, [Online] Available: http://netlabcaltechedu/lachlan/abstract/xcpinstabilitypdf, unpublished

1710 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL 17, NO 6, DECEMBER 2009 Lachlan L H Andrew (M 97 SM 05) received the BSc degree in computer science, the BE degree in electrical engineering, the PhD degree in engineering from the University of Melbourne, Australia, in 1992, 1993, 1997, respectively Since 2008, he has been an Associate Professor at Swinburne University of Technology, Hawthorn, Australia From 2005 to 2008, he was a Senior Research Engineer in the Department of Computer Science at California Institute of Technology, Pasadena Prior to that, he was a Senior Research Fellow at the University of Melbourne a lecturer at RMIT University, Melbourne, Australia His research interests include performance analysis of congestion control, resource allocation algorithms energy-efficient networking Dr Andrew is a Member of the IET the Association for Computing Machinery He was co-recipient of the Best Paper Award at IEEE MASS 07 Steven H Low (M 92 SM 99 F 08) received the BS degree from Cornell University, Ithaca, NY, the PhD degree from the University of California at Berkeley He is a Professor of the Computer Science Electrical Engineering Departments at California Institute of Technology, Pasadena, an Adjunct Professor of the Swinburne University of Technology, Hawthorn, Australia Dr Low was a co-recipient of the IEEE William R Bennett Prize Paper Award in 1997 the 1996 R&D 100 Award He was on the Editorial Board of the IEEE/ACM TRANSACTIONS ON NETWORKING from 1997 to 2006 Computer Networks Journal from 2003 to 2005 He is on the Editorial Boards of Computing Surveys Foundations Trends in Networking He is a Senior Editor of the IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS a Co-Editor of the Springer book series on Optimization Control of Communication Systems: Theory Applications He was a Member of the Networking Information Technology Technical Advisory Group for the US Presidents Council of Advisors on Science Technology (PCAST) Bartek P Wydrowski (M 03) received the BE degree in electrical engineering the PhD degree in engineering from the University of Melbourne, Australia, in 1998 2003, respectively Since 2007, he has been a Software Engineer at Google, New York, NY, prior to which he worked at FastSoft Inc, Pasadena, CA, as a Research Engineer at California Institute of Technology, Pasadena, between 2004 2005 Dr Wydrowski was awarded a 2003 Victoria Fellowship