Multipath TCP: Analysis, Design and Implementation

Transcription

1 Multipath TCP: Analysis, Design and Implementation Qiuyu Peng, Anwar Walid, Jaehyun Hwang, Steven H. Low arxiv:38.39v3 [cs.ni] 7 Dec 4 Abstract Multi-path TCP () has the potential to greatly improve application performance by using multiple paths transparently. We propose a fluid model for a large class of algorithms and identify design criteria that guarantee the existence, uniqueness, and stability of system equilibrium. We clarify how algorithm parameters impact TCP-friendliness, responsiveness, and window oscillation and demonstrate an inevitable tradeoff among these properties. We discuss the implications of these properties on the behavior of existing algorithms and motivate our algorithm Balia (balanced lined adaptation) which generalizes existing algorithms and stries a good balance among TCP-friendliness, responsiveness, and window oscillation. We have implemented Balia in the Linux ernel. We use our prototype to compare the new algorithm with existing algorithms. I. INTRODUCTION Traditional TCP uses a single path through the networ even though multiple paths are usually available in today s communication infrastructure; e.g., most smart phones are enabled with both cellular and WiFi access, and servers in data centers are connected to multiple routers. Multi-path TCP () has the potential to greatly improve application performance by using multiple paths transparently. It is being standardized by the Woring Group of the Internet Engineering Tas Force (IETF) []. In this paper we present a fluid model of and study how protocol parameters affect structural properties such as the existence, uniqueness and stability of equilibrium, the tradeoffs among TCP friendliness, responsiveness and window oscillation. These properties motivate a new algorithm that generalizes existing algorithms. Various congestion control algorithms have been proposed as an extension of TCP NewReno for. A straightforward extension is to run TCP NewReno on each subpath, e.g. [3], [4]. This algorithm however can be highly unfriendly when it shares a path with a single-path TCP user. This motivates the Coupled algorithm which is fair because it has the same underlying utility function as TCP NewReno, e.g. [5], [6]. It is found in [7] however that the Coupled algorithm responds slowly in a dynamic networ environment. A different algorithm is proposed in [7] (which we refer to as the Max algorithm) which is more responsive than the A preliminary version has appeared in []. Q. Peng and S. H. Low are with California Institute of Technology, Pasadena, CA, 95, USA ( {qpeng,slow}@caltech.edu) A. Walid is with Bell Laboratories, Murray Hill, NJ 7974, USA ( anwar@research.bell-labs.com) J. Hwang is with Bell Labs, Alcatel-Lucent, Seoul, Korea ( jh.hwang@alcatel-lucent.com) Coupled algorithm and still reasonably friendly to single-path TCP users. Recently, opportunistic lined increase algorithm (OLIA) is proposed as a variant of Coupled algorithm that is as friendly as the Coupled algorithm but more responsive [8]. See [9] for more references to early wor on multi-path congestion control. Our goal is to develop structural understanding of algorithms so that we can systematically tradeoff different properties such as TCP friendliness, responsiveness, and window oscillation that can be detrimental to applications that require a steady throughput. For single-path TCP, one can associate a strictly concave utility function with each source so that the congestion control algorithm implicitly solves a networ utility maximization problem [9] []. The convexity of this underlying utility maximization guarantees the existence, uniqueness, and stability of most single-path TCP algorithms. For many proposals considered by IETF, it will be shown that the utility maximization interpretation fails to hold in general, necessitating the need for a different approach to understanding the equilibrium properties of these algorithms. Moreover the relations among different performance metrics, such as fairness, responsiveness and window oscillation, need to be clarified. The main contributions of this paper are three-fold. First we present a fluid model that covers a broad class of MP- TCP algorithms and identify the exact property that allows an algorithm to have an underlying utility function. This implies that some algorithms, e.g., the Max algorithm [7], has no associated utility function. We prove conditions on protocol parameters that guarantee the existence and uniqueness of the equilibrium, and its asymptotical stability. Indeed algorithms that fail to satisfy these conditions, e.g. the Coupled algorithm, can be unstable and can have multiple equilibria as shown in [7]. Second we clarify how protocol parameters impact TCP friendliness, responsiveness, and window oscillation and demonstrate the inevitable tradeoff among these properties. Finally, based on our understanding of the design space, we propose Balia (Balanced lined adaptation) MP- TCP algorithm that generalizes existing algorithms and stries a good balance among these properties. This algorithm has been implemented in the Linux ernel and we evaluate its performance using our Linux prototype. We now summarize our proposed Balia algorithm. Each source s has a set of routes r. Each route r maintains a congestion window w r and measures its round-trip time τ r. The window adaptation is as follows: For each ACK on route r s,

2 w r w r + x r τ r ( x ) ( ) ( ) + αr 4 + αr For each pacet loss on route r s, 5 () w r w r w r min {α r,.5} () where x r : w r /τ r and α r : max{x } x r. The rest of the paper is structured as follows. In Section II we develop a fluid model for and use it to model existing algorithms. In Section III we prove several structural properties, focusing on design criteria that determine the existence, uniqueness, and stability of system equilibrium, TCP-friendliness, responsiveness, window oscillation, and an inevitable tradeoff among these properties. In Section IV we discuss the implications of these properties on existing algorithms. This motivates our new algorithm and we explain our design rationale. In Section V we compare the performance of the proposed algorithm with existing algorithms using Linux implementations of these algorithms. We conclude in Section VI. II. MULTIPATH TCP MODEL In this section we first propose a fluid model of MP- TCP and then use it to model algorithms in the literature. Unless otherwise specified, a boldface letter x R n denotes a vector with components x i. We use x i : (x,..., x i, x i+,..., x n ) to denote the n dimensional vector without x i and x : ( x i )/ to denote the L - norm of x. Given two vectors x, y R n, x y means x i y i for all components i. A capital letter denotes a matrix or a set, depending on the context. A symmetric matrix P is said to be positive (negative) semidefinite if x T P x ( ) for any x, and positive (negative) definite if x T P x > (< ) for any x. For any matrix P, define [P ] + : (P +P T )/ to be its symmetric part. Given two arbitrary matrices A and B (not necessarily symmetric), A B means [A B] + is positive semidefinite. For a vector x, diag{x} is a diagonal matrix with entries given by x. A. Fluid model Consider a networ that consists of a set L {,..., L } of lins with finite capacities c l. The networ is shared by a set S {,..., S } of sources. Available to source s S is a fixed collection of routes (or paths) r. A route r consists of a set of lins l. We abuse notation and use s both to denote a source and the set of routes r available to it, depending on the context. Liewise, r is used both to denote a route and the set of lins l in the route. Let R : {r r s, s S} be the collection of all routes. Let H {, } L R be the routing matrix: H lr if lin l is in route r (denoted by l r ), and otherwise. For each route r R, τ r denotes its round trip time (RTT). For simplicity we assume τ r are constants. Each source s maintains a congestion window w r (t) at time t for every route r s. Let x r (t) : w r (t)/τ r represent the sending rate on route r. Each lin l maintains a congestion price p l (t) at time t. Let q r (t) : l L H lrp l (t) be the aggregate price on route r. In this paper p l (t) represents the pacet loss probability at lin l and q r (t) represents the approximate pacet loss probability on route r. We associate three state variables (x r (t), w r (t), q r (t)) for each route r s. Let x s (t) : (x r (t), r s), w s (t) : (w r (t), r s), q s (t) : (q r (t), r s). Then (x s (t), w s (t), q s (t)) represents the corresponding state variables for each source s S. For each lin l, let y l (t) : r R H lrx r (t) be its aggregate traffic rate. Congestion control is a distributed algorithm that adapts x(t) and p(t) in a closed loop. Motivated by the AIMD algorithm of TCP Newreno, we model by ẋ r r (x s ) (φ r (x s ) q r ) + x r r s s S (3) ṗ l γ l (y l c l ) + p l l L, (4) where (a) + x a for x > and max{, a} for x. We omit the time t in the expression for simplicity. (3) models how sending rates are adapted in the congestion avoidance phase of TCP at each end system and (4) models how the congestion price is (often implicitly) updated at each lin. The algorithm installed at source s is specified by (K s, Φ s ), where K s (x s ) : ( r (x s ), r s) and Φ s (x s ) : (φ r (x s ), r s). Here K s (x s ) is a vector of positive gains that determines the dynamic property of the algorithm. Φ s (x s ) determines the equilibrium properties of the algorithm. The lin algorithm is specified by γ l, where γ l > is a positive gain that determines the dynamic property. This is a simplified model for the RED algorithm that assumes the loss probability is proportional to the baclog, and is used in, e.g., [], []. B. Existing algorithms We first show how to relate the fluid model (3) to the window-based algorithms proposed in the literature. On each route r the source increases its window at the return of each ACK. Let this increment be denoted by I r (w s ) where w s is the vector of window sizes on different routes of source s. The source decreases the window on route r when it sees a pacet loss on route r. Let this decrement be denoted by D r (w s ). Then most loss based algorithms tae the form of the following pseudo code: For each ACK on route r, w r w r + I r (w s ). For each loss on route r, w r w r D r (w s ). We now model the above pseudo codes by the fluid model (3). Let δw r be the net change to window on route r in each round trip time. Then δw r is roughly δw r (I r (w s )( q r ) D r (w s )q r )w r (I r (w s ) D r (w s )q r )w r since the loss probability q r is small. On the other hand Hence δw r ẇ r τ r ẋ r τ r ẋ r x r τ r (I r (w s ) D r (w s )q r )

3 3 From (3) we have { r (x s ) xr τ r D r (w s ) φ r (x s ) Ir(ws) D r(w s) We now apply this to the algorithms in the literature. We first summarize these algorithms in the form of a pseudo-code and then use (5) to derive parameters r (x s ) and φ r (x s ) of the fluid model (3). Single-path TCP (TCP-NewReno): Single-path TCP is a special case of algorithm with s. Hence x s is a scalar and we identify each source with its route r s. TCP-NewReno adjusts the window as follows: For each ACK on route r, w r w r + /w r. For each loss on route r, w r w r /. From (5), this can be modeled by the fluid model (3) with r (x s ) x r, φ r (x s ) τ r x r We now summarize some existing algorithms, all of which degenerate to TCP NewReno if there is only one route per source. EWTCP [3]: EWTCP algorithm applies TCP-NewReno lie algorithm on each route independently of other routes. It adjusts the window on multiple routes as follows: For each ACK on route r, w r w r + a/w r. For each loss on route r, w r w r /. From (5), this can be modeled by the fluid model (3) with r (x s ) x r, φ r (x s ) a τr x r where a > is a constant. Coupled MPTCP [5], [6]: The Coupled MPTCP algorithm adjusts the window on multiple routes in a coordinated fashion as follows: For each ACK on route r, w r w r + w r/τ r ( s w /τ ). For each loss on route r, w r w r /. From (5), this can be modeled by the fluid model (3) with r (x s ) x r, φ r (x s ) τ r ( s x ) Semicoupled MPTCP [7]: The Semi-coupled MPTCP algorithm adjusts the window on multiple routes as follows: For each ACK on route r, w r w r + τ r( s w /τ ). For each loss on route r, w r w r /. From (5), this can be modeled by the fluid model (3) with r (x s ) x r, φ r (x s ) x r τ r ( s x ) Max MPTCP [7]: The Max MPTCP algorithm adjusts the window on multiple routes as follows: For { each ACK on route r, w r w r + max{w /τ min } ( w /τ ), w r }. For each loss on route r, w r w r /. From (5), this can be modeled by the fluid model (3) with r (x s ) x r, φ r (x s ) max{x /τ } x r τ r ( s x ) (5) TABLE I: algorithms C C C, C3 C4 C5 EWTCP Yes Yes Yes Yes Yes Coupled Yes Yes No Yes Yes Semicoupled No Yes Yes Yes Yes Max No Yes Yes Yes Yes Generalized No Yes Yes Yes Yes Theorem 3. 3., 3.3, where we have ignored taing the minimum with the /w r term since the performance is mainly captured by max{w /τ } ( w /τ ). Recently, OLIA algorithm [8] is shown to achieve good performance in many scenarios. OLIA uses complicated feedbac congestion control signals and cannot be modeled by (3)-(4). We do, however, include OLIA in our Linux-based performance evaluation in Section V. III. STRUCTURAL PROPERTIES Throughout this paper we assume, for all x s, r s, s S, r (x s ) > and φ r (x s ) only if x for some s. A point (x, p) is called an equilibrium of (3) (4) if it satisfies, for all r s, s S and l L, or equivalently, r (x s ) (φ r (x s ) q r ) + x r γ l (y l c l ) + p l x r, φ r (x s ) q r and φ r (x s ) q r if x r > (6) p l, y l c l and y l c l if p l > (7) We mae two remars. First an equilibrium (x, p) does not depend on K s, but only on Φ s. The design (K s, s S) however affects dynamic properties such as stability and responsiveness as we show below. Second, since r (x s ) > and φ r (x s ) only if x for some s by assumption, any finite equilibrium (x, p) must have q r > for all r. In the following we always restrict ourselves to finite equilibria. In this section we denote an algorithm by (K, Φ) : (K s, Φ s, s S). We characterize designs (K, Φ) that guarantee the existence, uniqueness, and stability of system equilibrium. We identify design criteria that determine TCP-friendliness, responsiveness and window oscillation and prove an inevitable tradeoff among these properties. We discuss in the next section the implications of these structural properties on existing algorithms. All proofs are relegated to the Appendices. A. Summary We first present some properties of an algorithm (K, Φ) that we have identified. We then interpret them and summarize their implications. C: For each s S and each x s, the Jacobians of Φ s (x s ) is continuous and symmetric, i.e., Φ s (x s ) [ Φs (x s ) ] T

4 4 C: For each s S there exists a nonnegative solution x s : x s (p) to (6) for any finite p such that q r > for all r. Moreover, y s l (p) p l, lim p l ys l (p) where yl s(p) : r s H lrx r (p) is the aggregate traffic at lin l from source s. C: For each s S and each x s, Φ s (x s ) is continuously differentiable; moreover the symmetric part [ Φ s (x s )/ ] + of the Jacobian is negative definite. C3: For each r R, φ r (x s ) if and only if x r. The routing matrix H has full row ran. C4: For each r s, s S, j s [D s] jr (x s ) where [ Φ. D s (x s ) : s (x s )] C5: For each r R and each x r, lim xr φ r (x s ). These design criteria are intuitive and usually (but not always) satisfied; see Table I. Condition C guarantees the existence of utility functions U s (x s ) that an equilibrium (x, p) of a multipath TCP/AQM (3) (4) implicitly maximizes (Theorem 3.). It is always satisfied when there is only a single path ( s for all s) but not when s >. Conditions C C3 guarantee the existence, uniqueness, and global asymptotic stability of the equilibrium (x, p) (Theorems 3. and 3.3). C says that the aggregate traffic rate through a lin l from source s decreases when the congestion price p l on that lin increases, and it decreases to as p l increases without bounds. C implies that at steady state, if x s, q s are perturbed by δx s, δq s respectively, then (δx s ) T δq s <. In the case of single-path TCP ( s for all s), C is equivalent to the curvature of the utility function U s (x s ) being negative, i.e., U s (x s ) is strictly concave. C3 means that the rate on route r is zero if and only if it sees infinite price on that route. Condition C4 is natural and satisfied by all the algorithms considered in this paper. It allows us to formally compare algorithms in terms of their TCP-friendliness (see formal definition below): under C C4, an algorithm (K, Φ) is more friendly if φ r (x s ) is smaller (Theorem 3.4). The existence of D s in C4 is ensured by C. To interpret C4, note that Lemma B. in Appendix B implies that Φ s (x s) q s at equilibrium. The implicit function theorem then implies T xs q r j s D jr at equilibrium for all r s. Hence C4 says that the aggregate throughput T x s at equilibrium over all routes r s of an flow is a nonincreasing function of the price q r. Condition C5 is also satisfied by all the algorithms considered in this paper. It means that the sending rate on a route r grows unbounded when the congestion price q r is zero. Under C C3, an algorithm (K, Φ) is more responsive (see formal definition below) if the Jacobian of Φ s (x s ) is more negative definite (Theorem 3.5). C5 then implies an inevitable tradeoff: an algorithm that is more responsive is necessarily less TCP-friendly (Theorem 3.6). We now elaborate on each of these properties. B. Utility maximization For single-path TCP (), one can associate a utility function U s (x s ) R + R with each flow s (x s is a scalar and s ) and interpret (3) (4) as a distributed algorithm to maximize aggregate users utility, e.g. [9] []. Indeed, for, an (x, p) is an equilibrium if and only if x is optimal for maximize s S U s (x s ) s.t. y l c l l L (8) and p is optimal for the associated dual problem. Here y l c l means the aggregate traffic y l at each lin does not exceed its capacity c l. In fact this holds for a much wider class of algorithms than those specified by (3) (4) []. Furthermore all the main TCP algorithms proposed in the literature have strictly concave utility functions, implying a unique stable equilibrium. The case of is much more delicate: whether an underlying utility function exists depends on the design choice of Φ s and not all algorithms have one. Consider the multipath equivalent of (8): maximize s S U s (x s ) s.t. y l c l l L (9) where x s : (x r, r s) is the rate vector of flow s and U s : R s + R is a concave function. Theorem 3. (utility maximization): There exists a twice continuously differentiable and concave U s (x s ) such that an equilibrium (x, p) of (3) (4) solves (9) and its dual problem if and only if condition C holds. Condition C is satisfied trivially by when s. For ( s > ), the models derived in Section II-B show that only EWTCP and Coupled algorithms satisfy C and have underlying utility functions. It therefore follows from the theory for that EWTCP has a unique stable equilibrium while Coupled algorithm may have multiple equilibria since its corresponding utility function is not strictly concave. The other algorithms all have asymmetric Jacobian Φs and do not satisfy C. C. Existence, uniqueness and stability of equilibrium Even though a multipath TCP algorithm (K, Φ) may not have a utility maximization interpretation, a unique equilibrium exists if conditions C C3 are satisfied. Theorem 3. (existence and uniqueness): ) Suppose C holds. Then (3) (4) has at least one equilibrium. ) Suppose C and C3 hold. Then (3) (4) has at most one equilibrium Thus (3) (4) has a unique equilibrium (x, p ) under C C3. Conditions C-C3 not only guarantee the existence and uniqueness of the equilibrium, they also ensure that the equilibrium is globally asymptotically stable, when the gain r (x s ) is only a function of x r itself, i.e., r (x s ) r (x r ) for all r R. This is satisfied by all the existing algorithms presented in Section II-B.

5 5 Flow Flow c Flow Flow Fig. : Test networ for the definition of TCP friendliness. The lin in the middle is the only bottlenec lin with capacity c. Theorem 3.3 (stability): Suppose C-C3 hold and r (x s ) r (x r ) for all r R. Then the unique equilibrium (x, p ) is globally asymptotically stable. In particular, starting from any initial point x() R R + and p() R L +, the trajectory (x(t), p(t)) generated by the algorithm (3) (4) converges to the equilibrium (x, p ) as t. Our proposed algorithm does not satisfy r (x s ) r (x r ) even though it seems to be stable in our experiments. This condition is only sufficient and needed in our Lyapunov stability proof; see Appendix C. When r (x s ) depends on x s, one can replace r (x r ) in the definition of the Lyapunov function V in () with r (x s) evaluated at the equilibrium and the same argument there proves that (x, p ) is (locally) asymptotically stable. Also see Theorem 3.5 below for an alternative proof of local stability. D. TCP friendliness Informally, an flow is said to be TCP friendly if it does not dominate the available bandwidth when it shares the same networ with a flow []. To define this precisely we use the test networ shared by a flow and a MP- TCP flow under test as shown in Fig.. All paths traverse a single bottlenec lin with capacity c, with all other lins with capacities strictly higher than c. The lins have fixed but possibly different delays. To compare the friendliness of two algorithms ˆM : ( ˆK, ˆΦ) and M : ( K, Φ), suppose that when ˆM shares the test networ with a it achieves a throughput of ˆx in equilibrium aggregated over the available paths (the SP- TCP therefore attains a throughput of c ˆx ). Suppose M achieves a throughput of x in equilibrium when it shares the test networ with the same. Then we say that ˆM is friendlier (or more TCP-friendly) than M if ˆx x, i.e., if ˆM receives no more bandwidth than M does when they separately share the test networ in Fig. with the same flow. From the theory for single-path TCP ( s for all s S), it is nown that a design is more TCP-friendly if it has a smaller marginal utility U s(x s ) Φ s (x s ). The same intuition holds for algorithms even though the utility functions may not exist for algorithm. Theorem 3.4 (friendliness): Consider two algorithms ˆM : ( ˆK, ˆΦ) and M : ( K, Φ). Suppose both satisfy C C4. Then ˆM is friendlier than M if ˆΦ s (x s ) Φ s (x s ) for all s S. E. Responsiveness around equilibrium Suppose conditions C C3 hold and there is a unique equilibrium z : (x, p ). Assume all lins in L are active with p l > ; otherwise remove from L all lins with prices p l. Let δz(t) : z(t) z. The behavior of (3) (4) around the equilibrium is defined by the linearized system: δż J δz(t) () Here J is the Jacobian of (3) (4) at the equilibrium z : [ ] J : J(x Φ Λ ) : x Λ H T Λ γ H where Λ diag{ r (x s), r R}, Λ γ diag{γ l, l L}, and Φ x is evaluated at x. The stability and responsiveness of the linearized system () (how fast does the system converges to the equilibrium locally) is determined by the real parts of the eigenvalues of J. Specifically the linearized system is stable if the real parts of all eigenvalues of J are negative; moreover the more negative the real parts are the faster the linearized system converges to the equilibrium. We now show that the linearized system () is stable (i.e., converges exponentially fast to z locally) and characterize its responsiveness in terms of the design choices (K, Φ). Let Z {z : (x, p) C R + L z }. Theorem 3.5 (responsiveness): Suppose C C3 hold. Then ) The linearized system () is stable, i.e., Re(λ) < for any eigenvalue λ of J. Moreover Re(λ) λ(j ) where { λ(j x [ ] H Φ + } x x ) : max z Z x H Λ x + ph Λ γ p where Λ and Φs are evaluated at the equilibrium point z. ) For two algorithms ( ˆK, ˆΦ) and ( K, Φ), λ(ĵ ) λ( J ) provided ˆK s K s and ˆΦ s Φ s for all s S Theorem 3.5 motivates the following definition of responsiveness. Given two ˆM and M, we say that ˆM is more responsive than M if λ(ĵ ) λ( J ). Theorem 3.5() implies that an [ algorithm with a larger K s (x s) or + Φ more negative definite s (xs)] is more responsive, in the sense that the real parts of the eigenvalues of the Jacobian J have a smaller more negative upper bound. Then the next result suggests an inevitable tradeoff between responsiveness and friendliness. Theorem 3.6 (tradeoff): Consider two algorithms (K, ˆΦ) and (K, Φ) with the same gain K. Suppose both satisfy C-C3 and C5. Then for all s S ˆΦ s (x s ) Φ s (x s ) ˆΦ s (x s ) Φ s (x s ) In light of Theorems 3.4 and 3.5, Theorem 3.6 says that a more responsive design is inevitably less friendly if they have the same K.

6 6 The theorem is easier to understand in the case of, i.e., when s for all s S and Φ s (x s ) U s(x s ). Then it implies that a more concave utility function U s (x s ) has a larger marginal utility, and hence less friendly. F. Window oscillation Window oscillations are inherent in loss-based additive increase multiplicative decrease (AIMD) TCP algorithms. We close this section by discussing informally why a larger design K s (x s ) generally creates more severe window oscillations. This implies a tradeoff between responsiveness (which is enhanced by a large K s (x s )) and oscillation (which is reduced with a small K s (x s )). The effect of K s (x s ) on window fluctuations can be understood by studying how it affects the decrease D r (w s ) per pacet loss in the following pacet level model: For each ACK on route r, w r w r + I r (w s ). For each loss on route r, w r w r D r (w s ). Let Z r {, } be an indicator variable of whether a pacet loss is observed on route r at an arbitrary time in steady state. Then D s (x s ) : ( E x s r s D r (w s ) τ r Z r ) Z s represents the expected relative reduction in aggregate throughput r s D r(w s )/τ r, given that there is at least one pacet loss on some route r s. It is a measure of throughput fluctuation for each pacet loss that an application experiences. For TCP-NewReno (for which s {r} and w s is a scalar), the window size is halved on each pacet loss, D r (w s ) w r /, and hence D s (x s ) /. To understand D s (x s ) for algorithms, we need the following result. Lemma 3.: Let A i : {a i, a i,...} with A i elements. Each element a ij is an independent binary random variable with P(a ij ) P(a ij ) q i. Define D i (A i ) : d i ( j aij ). Then E D (A ) a ij i,j ( ) d q A q + o q A Suppose each route has a fixed loss probability q r. Then within each RTT, Lemma 3. implies ( r s D s (x s ) w ( )) rq r D r (w s )/τ r x s r s q + o q r rw r r s Substituting w r x r τ r and x r D r (w s ) τ r r (x s ) from (5), we get, ignoring the high-order terms, ( r s D s (x s ) τ ) rq r r (x s ) x s r s τ () rq r x r to the first order. Note that r (x s ) does not affect the equilibrium rates x s. Hence, with the assumption that τ r are constants, D s (x s ) is determined by the functions r (x s ) in steady state. Specifically an algorithm with a larger K s (x s ) tends to have a larger D s (x s ) and hence more severe window oscillations. Theorem 3.5 however suggests that a larger K s (x s ) also leads to better responsiveness, suggesting an inevitable tradeoff between responsiveness and window oscillation. IV. IMPLICATIONS AND A NEW ALGORITHM In this section we discuss the implications of these structural properties on the behavior of existing algorithms. They are further illustrated in experiment results in Section V. The discussion motivates a new design that generalizes the existing algorithm. A. Implications on existing algorithms Recall Table I that summarizes the conditions satisfied by the various algorithms. Only EWTCP and Coupled algorithms satisfy C. Their equilibrium properties can be studied in the standard utility maximization model as done for singlepath TCP. Semicoupled and Max algorithms do not satisfy C and therefore analysis through utility maximization is not applicable. However Theorem 4. below implies that, both Semicoupled and Max algorithms satisfy C C3 provided they enable no more than 8 routes. Theorem 3. and 3.3 then imply that they have a unique and globally stable equilibrium. It is also easy to show that EWTCP satisfies C-C3. The Coupled algorithm does not satisfy C and is found to have multiple equilibria in [5]. Next we discuss friendliness of existing algorithms. It can be shown that the φ r (x s ) corresponding to these algorithms satisfy: φ ewtcp r (x s ) φ semicoupled r (x s ) φ max r (x s ) φ coupled r (x s ) for all x s if all routes r s have the same round trip time. Since all of them satisfy C4, Theorem 3.4 implies that their friendliness will be in the same order, i.e., their throughputs in the test networ of Fig. are ordered as follows: EWTCP(a ) Semicoupled Max Coupled This is confirmed by the Linux-based experiment. Third we will discuss responsiveness of existing algorithms. These algorithms have the same gain function r (x s ).5x r and ( Φ s ) ewtcp ( Φ s ) semicoupled ( Φ s ) max ( Φ s ) coupled Theorem 3.5 then implies that their responsiveness should be in the same order, as confirmed by our experiments in section V. Finally we discuss window oscillation of existing algorithms using D s (x s ) as the metric. As mentioned in Section III-F, D s (x s ).5 for TCP NewReno, a benchmar When a <, the source can obtain even smaller throughput than the competing single-path TCP source.

7 7 single-path TCP algorithm. According to (), if r (x s ).5x r x s, we have, to the first order D s (x s ) r s τ rq r x r x s x s r s τ rq r x r All existing algorithms have the same r (x s ).5x r.5x r x s, with strict inequality if s > and x r > for at least two r s. Thus enabling always tends to reduce window oscillation for existing algorithms compared to TCP NewReno. Moreover, the window oscillation is always reduced compared to TCP NewReno when r (x s ).5x r x s. B. A generalized algorithm Consider the class of algorithms parametrized by (β, n, η) as follows: { r (x s ) x r(x r + η( x s x r )), η φ r (x s ) (( β)xr+β xs n) (), n N τr +, β xr xs This class includes the Max (β, η, n ), Coupled (β, η ), and Semicoupled (β, η, n ) algorithms as special cases when all RTTs on different paths of the same source are the same, i.e., τ r τ s, r s. The next result characterizes a subclass that have a unique and locally stable equilibrium point. Theorem 4.: Fix any η and n N +. For any s S, the φ r (x s ) in () satisfies ) C if β. ) C C3 if < β, s 8 and τ r are the same for all r s (assuming H has full row ran). The requirement that s 8 is not restrictive since in practice a device may typically enable no more than 3 paths. The requirement that τ r are the same for all r s is used in proving the negative definiteness of the (symmetric part of the) Jacobian of Φ s (x s ). Since a negative definite matrix remains negative definite after small enough perturbations of its entries, Theorem 4. holds if the RTTs of the subpaths do not differ much. This (sufficient) condition seems reasonable as two paths between the same source-destination pair often have similar RTTs if both are wireline paths. Note that our experiments in Section V show that the algorithm also converges even if the RTTs on different paths differ dramatically, e.g. the RTT of WiFi is usually much smaller than that of 3G. For the class of algorithms specified by (), Theorem 4. motivates a design space defined by β (, ], η, n N +, where β and n control the tradeoff between friendliness and responsiveness and η controls the tradeoff between responsiveness and window oscillation. In Table II, we summarize how the parameters (β, η, n) affect the performance. We now describe our design philosophy. As discussed above the design of algorithms involves inevitable tradeoffs among responsiveness, friendliness, and the severity of window oscillation. Specifically a design is more responsive if it has a higher gain K s or a more negative definite Jacobian [ Φ s / ] + (Theorem 3.5). However a larger K s usually creates a bigger window oscillation; a more negative definite [ Φ s / ] + implies a larger Φ s, usually hurting friendliness TABLE II: How design choices affect performance. Performance Parameter Parameters in () TCP friendliness φ r(x s) β, n Responsiveness r(x s), Φ s/ β, n, η Window oscillation r(x s) η Fig. : Networ for our Linux-based experiments on TCP friendliness and responsiveness, with N flows and N single-path TCP flows sharing lins of capacity c, c and propagation delay (single trip) T, T. flows maintain two routes with rate x, x. Single-path TCP flows maintain one route with rate x 3. (Theorems 3.6 and 3.4). This is summarized in Table II. Since enabling multiple paths already reduces window oscillation compared to single-path TCP (section IV-A), can afford to use a relatively large gain K s for responsiveness. This does not compromise too much on window oscillation, but allows us to use a less negative definite Jacobian [ Φ s / ] + with a smaller Φ s to maintain sufficient TCP friendliness. Moreover, responsiveness is mainly affected by subpaths with small throughput while window oscillation is mainly affected by subpaths with large throughput. The parameter η in the generalized algorithm () scales r (x s ) in the right way: a path r that has a large x r has r (x s ).5x r and hence a similar degree of window oscillation as existing algorithms, while a path r with a small x r has larger r (x s ) than that under a design with zero η and therefore is more responsive. Our experiments show that Max algorithm ((β, η, n) (,, )) overtaes too much of the competing single-path TCP flows. Hence, we can only use a smaller β since n is already infinite in order to improve friendliness. To compensate the responsiveness performance, we will use a larger η, which will sacrifice window oscillation performance. The Balia MP- TCP algorithm given at the end of Section I corresponds to the choice (β, η, n) (.,.5, ). Instead of allowing the window size to drop to for a pacet loss, we add a cap for the decrement of window size, which improves the performance according to our experiments. Note that there is no best parameter settings since there are tradeoffs among all the performance metrics and we choose (β, η, n) (.,.5, ) based on our experiments in Section V, which show that this parameter setting stries a good balance among responsiveness, friendliness, and window oscillation. V. EXPERIMENT In this section we summarize our experimental results that illustrate the above analysis. In addition to the MP- TCP algorithms illustrated in section II-B, we also include

8 8 the recently developed OLIA algorithm [8]. We evaluate the algorithms using a reference Linux implementation of, Multipath TCP v.88 [3]. Since it currently includes only Max and OLIA algorithms, we implement EWTCP, Semicoupled, Coupled, and the proposed Balia algorithm in the reference implementation. For the Coupled and our algorithm, the minimum ssthresh is set to instead of when more than path is available. The networ topology is shown in Fig.. In the testbed, all nodes are Linux machines with a quad-core Intel i5 3.33GHz processor, 4GB RAM and multiple Gbps Ethernet interfaces, running Ubuntu 3. (Linux ernel 3..8). The networ parameters such as c, c, T, and T are controlled by Dummynet [4]. Our experiments are divided into three parts. First we compare TCP friendliness of Balia algorithm and prior algorithms. The result confirms that the Couple algorithm is the friendliest, Balia algorithm is close to the Coupled algorithm and friendlier than the other algorithms. Second we compare the responsiveness of each algorithm in a dynamic environment where flows come and go. The result shows that the Coupled and OLIA algorithms are unresponsive (illustrating the tradeoff between responsiveness and friendliness). EWTCP is the most responsive; Balia is similar in responsiveness but friendlier to single-path TCP flows. Finally we show that all algorithms have smaller average window oscillations than single-path TCP. These experiments confirm our analytical results and suggest our design choice stries a good balance among friendliness, responsiveness, and window oscillation. A. TCP friendliness We study TCP friendliness of each algorithm, first with paths of similar RTTs and then with paths of different RTTs, which emulates the wireless scenario. We assume all the flows are long lived and focus on the steady state throughput. In the first set of experiments, we let T T 5ms, c c 6Mbps and N N 3. We repeat the experiments times, the average aggregate throughput of and single-path TCP users and the 95% margin of error for confidence interval (CI) are shown in Table III. The Coupled algorithm is the friendliest and Balia algorithm is closer to Coupled algorithm than the others. TABLE III: TCP friendliness (same RTTs): Average throughput (Mbps) and 95% confidence interval of and single-path TCP users. (T T 5ms, c c 6Mbps and N N 3) ewtcp semi. max balia coupled olia mp-tcp (throuput) mp-tcp (CI) sp-tcp (throuput) sp-tcp (CI) In the second set of experiments, we assume a highly heterogeneous RTTs by emulating the scenario of a mobile device with both 3G and WiFi access. WiFi access usually has higher capacity and lower delay compared to 3G. Specificially, we set T ms, c 8Mbps for the first lin to emulate WiFi access and T ms, c Mbps for the second lin to emulate 3G access. When there exists single-path TCP flows, i.e. N >, the behaviors of all the algorithms are similar to the equal RTT case in the first set of simulation. The Coupled algorithm is the friendliest and Balia algorithm is closer than other algorithms. However, when there is no singlepath TCP flow, i.e. N and N, the performance of OLIA is not stable to effectively tae all the available capacity while the other algorithms do not have such problem. We repeat the experiments times and we find sometimes OLIA does not use the 3G access lin. The average throughput of user and the 95% margin of error for confidence interval is shown in Table IV. TABLE IV: Basic behavior (WiFi/3G): throughput (Mbps) of a user and 95% confidence interval. (T ms, T ms, c 8Mbps, c Mbps and N, N ) ewtcp semi. max balia coupled olia throughput confidence interval B. Responsiveness We use the networ in Fig. with c c Mbps, T T ms and N, N 5. To demonstrate the dynamic performance of each algorithm, we assume the MP- TCP flow is long lived while the single-path TCP flows start at 4s and end at 8s. We record the aggregate throughput of the single-path TCP flows from 4-8s, which measures the friendliness of. We also measure the time for the congestion window on the second path to recover of users. It measures the responsiveness of. These measurements are shown in Table V and the congestion window and throughput trajectories of all algorithms are shown in Fig. 4. To clearly show the responsiveness performance, we record the longest convergence time found in our experiment in Table V and the corresponding trajectories are shown in Fig. 4. TABLE V: Responsiveness: convergence time (s) of and total throughput (Mbps) of all single-path TCP users. (T T ms, c c Mbps and N, N 5) ewtcp semi. max balia coupled olia Convergence EWTCP is the most responsive among all the algorithms. Ours is as responsive as the Max algorithm, yet significantly friendlier than EWTCP. Both Coupled and OLIA algorithms tae an excessively long time to recover. For Coupled algorithm, the excessively slow recovery of the congestion window on the second path (see Fig. 4) is due to the design that increases the window roughly by w r /( s w ) on each ACK assuming the RTTs are similar. After the singlepath TCP flow has left, w is small while w is large, so Defined as the first time the congestion window on the second path reaches the average congestion window (e.g., 6) after the single-path users have left.

9 9 that w /(w + w ) is very small. It therefore taes a long time for w to increase to its steady state value. In general, under the Coupled algorithm, a route with a large throughput can greatly suppress the throughput on another route even though the other route is underutilized. The reason of the poor responsiveness performance of OLIA can be explained using similar argument as Coupled algorithm since they have the same increment/decrement for each ACK/loss in this scenario. C. Window oscillation We use a single-lin networ model to compare window oscillation under and single-path TCP. First a MP- TCP flow initiates two subpaths through that lin, and we measure the window size of each subpath and their aggregate window size. Then a TCP-Reno flow traverses the same lin and we measure its window size. The results are shown in Fig. 3 for our algorithm in comparison with single-path TCP (other algorithms have a similar behavior). They confirm that enabling multiple paths reduces the average window oscillation compared with only using single path Balia Subpath Subpath Total Linux TCP (Reno) TCP-Reno Fig. 3: Window oscillation: the red trajectories represent throughput fluctuations experienced by the application in the case of and the case of single-path TCP EWTCP MP-Subpath MP-Subpath Semicoupled MP-Subpath MP-Subpath Max MP-Subpath MP-Subpath Balia MP-Subpath MP-Subpath Throughput (Mbps) Throughput (Mbps) Throughput (Mbps) Throughput (Mbps) 4 3 EWTCP Semicoupled Max Balia VI. CONCLUSION We have presented a model for and identified designs that guarantee the existence, uniqueness and stability of the networ equilibrium. We have characterized the design space and study the tradeoff among TCP friendliness, responsiveness, and window oscillation. We have proposed Balia algorithm that generalizes existing algorithms and stries a good balance among these properties. We have implemented Balia in the Linux ernel and used it to evaluate the performance of our algorithm. REFERENCES [] Q. Peng, A. Walid, and S. H. Low, Multipath tcp algorithms: theory and design, in Proceedings of the ACM SIGMETRICS/international conference on Measurement and modeling of computer systems. ACM, 3, pp [] A. Ford, C. Raiciu, M. Handley, and O. Bonaventure, Tcp extensions for multipath operation with multiple addresses, IETF MPTCP proposal, 9. [3] M. Honda, Y. Nishida, L. Eggert, P. Sarolahti, and H. Touda, Multipath congestion control for shared bottlenec, in Proc. PFLDNeT worshop, 9. [4] J. R. Iyengar, P. D. Amer, and R. Stewart, Concurrent multipath transfer using sctp multihoming over independent end-to-end paths, Networing, IEEE/ACM Transactions on, vol. 4, no. 5, pp , Coupled MP-Subpath MP-Subpath OLIA MP-Subpath MP-Subpath Throughput (Mbps) Throughput (Mbps) 4 3 Coupled OLIA Fig. 4: Responsiveness Performance: congestion window trajectory of for each path (left column). starts at time 4s and ends at 8s. The throughput of and total throughput of are shown in the right column. Parameters: T T ms, c c Mbps and N, N 5.

10 [5] F. Kelly and T. Voice, Stability of end-to-end algorithms for joint routing and rate control, ACM SIGCOMM Computer Communication Review, vol. 35, no., pp. 5, 5. [6] H. Han, S. Shaottai, C. Hollot, R. Sriant, and D. Towsley, Overlay tcp for multi-path routing and congestion control, in IMA Worshop on Measurements and Modeling of the Internet, 4. [7] D. Wischi, C. Raiciu, A. Greenhalgh, and M. Handley, Design, implementation and evaluation of congestion control for multipath tcp, in Proceedings of the 8th USENIX conference on Networed systems design and implementation. USENIX Association,, pp [8] R. Khalili, N. Gast, M. Popovic, and J.-Y. Le Boudec, Mptcp is not pareto-optimal: Performance issues and a possible solution, IEEE/ACM Transactions on Networing (ToN), vol., pp , 3. [9] S. Shaottai and R. Sriant, Networ optimization and control, Foundations and Trends R in Networing, vol., no. 3, pp , 7. [] F. P. Kelly, A. K. Maulloo, and D. K. Tan, Rate control for communication networs: shadow prices, proportional fairness and stability, Journal of the Operational Research society, vol. 49, no. 3, pp. 37 5, 998. [] S. H. Low and D. E. Lapsley, Optimization flow control-i: basic algorithm and convergence, IEEE/ACM Transactions on Networing (TON), vol. 7, no. 6, pp , 999. [] S. H. Low, A duality model of tcp and queue management algorithms, Networing, IEEE/ACM Transactions on, vol., no. 4, pp , 3. [3] Multipath TCP Linux implementation. [Online]. Available: http: //multipath-tcp.org [4] M. Carbone and L. Rizzo, Dummynet revisited, ACM SIGCOMM Computer Communication Review, vol. 4, no., pp.,. [5] H. K. Khalil, Nonlinear Systems, nd ed. Prentice-Hall, Inc., 996. ACKNOWLEDGMENTS This wor was supported by ARO MURI through grant W9NF-8--33, NSF NetSE through grant CNS 94, Bell Labs, Alcatel-Lucent and Seoul R&BD Program funded by the Seoul Metropolitan Government through grant WR895. APPENDIX A PROOF OF THEOREM 3. (UTILITY MAXIMIZATION) The Lagrangian of (9) is: L(x, p) s S U s (x s ) l L p l (y l c l ) U s (x s ) s S l L ( s S U s (x s ) r s p l ( r R H lr x r c l ) x r q r ) + l L p l c l where p are the dual variables and q r : r R H lrp l. Then the dual problem is D(p) max {B s(x s, p)} + p l c l p x s s S l L where B s (x s, p) U s (x s ) r s x rq r. The KKT condition implies that, at optimality, we have U s (x s ) x r < q r x r and x r > U s(x s ) x r q r (3) y l < c l p l and p l > y l c l (4) Comparing with (6) (7) we conclude that, if a algorithm defined by (3) (4) has an underlying utility function U s, then we must have U s (x s ) x r φ r (x s ) r s, x r > (5) Given φ r (x s ), (5) has a continuously differentiable solutions U s (x s ) if and only if the Jacobian of Φ s (x s ) is symmetric, i.e., if and only if Φ(x s ) [ Φ(xs ) APPENDIX B PROOF OF THEOREM 3. (EXISTENCE AND UNIQUENESS) A. Proof of part For any lin l L, let ] T p l {p,..., p l, p l+,..., p L }, whose component composes of all the elements in p except p l. For l L, let g l (p) : c l x r c l yl s (p l, p l ) r:l r s:r s,l r and h l (p) : gl (p). According to C, we have the following two facts, which will be used in the proof. g l (p) is a nondecreasing function of p l on R + since yl s(p) is a nonincreasing function of p l. lim pl g l (p l, p l ) c l since lim pl yl s (p). Next, we will show that h l (p) is a quasi-concave function of p l. In other words, for any fixed p l, the set S a : {p l h l (p) a} is a convex set. If g l (, p l ), then g l (p l, p l ) g l (, p l ) p l, which means h l (p l, p l ) is a nonincreasing function of p l, hence is a quasi-concave function of p l and arg max p l h l (p l, p l ). (6) On the other hand, if g l (, p l ) <, then there exists a p l > such that g l (p l, p l) since g l ( ) is continuous and lim pl g l (p l, p l ) c l >. Note that g l (p) is a nondecreasing function of p l, then h l (p l, p l ) is nondecreasing for p l [, p l ] and nonincreasing for p l [p l, ). Hence, h l (p l, p l ) is also a quasi-concave function of p l in this case and max p l h l (p l, p l ). (7) By Nash theorem, if h l (p l, p l ) is a quasi-concave function of p l for all l L and p is in a bounded set, then there exists a p R L + such that p l arg max p l R + h l (p l, p l). According to (6) and (7), for any l L, either p l > or g l (p ) > but not both holds at any time. Therefore p satisfies Eqn. (7). Since q R T p, there exists an x to (6). Hence there exists at least one solution (x, p) that satisfies (6) and (7).

11 B. Proof of part Lemma B.: Assume a function F : R n R n is continuously differentiable and [ F x (x)] + is negative definite for all x. Then for any x x R n, (x x ) T (F (x ) F (x )) <. Proof: Fix any x x R n. Define A(t) : F (tx + ( t)x ). Since F/ x is continuous, there exists a λ < such that the eigenvalues of [ F/ x] + λ over the compact set {tx + ( t)x t }. Then (x x ) T (F (x ) F (x )) (x x ) T da (τ) dτ dt (x x ) T F x (τx + ( τ)x ) (x x ) dτ λ x x < Lemma B.: Suppose C3 holds. Then x r > at equilibrium for all r R. Proof: Suppose x r. Then qr φ r (x r) by C3 and hence there is a lin l r with p l. But then, for all paths r l, qr and hence x r by C3. This implies yl < c l, and hence p l by (7), contradicting p l. Recall the vector notations that x : (x s, s S) : (x r, r s, s S) and Φ(x) : (Φ s (x s ), s S) : (Φ r (x s ), r s, s S). To prove uniqueness of the equilibrium, suppose for the sae of contradiction that there are two distinct equilibrium points (x, p) and (ˆx, ˆp). By Lemma B. we have x > and ˆx >. Hence (6) implies Φ(x) q H T p and Φ(ˆx) ˆq H T ˆp. By Lemma B. and assumption C we then have Hence > (x ˆx) T (Φ(x) Φ(ˆx)) (x ˆx) T H T (p ˆp) (p ˆp) T (y ŷ) p T y + ˆp T ŷ < p T ŷ + ˆp T y (8) Equilibrium condition (7) implies p T (c y) and ˆp T (c ŷ) (9) Substituting (9) into (8) yields y c and ŷ c () p T c + ˆp T c < p T ŷ + ˆp T y p T (c ŷ) + ˆp T (c y) < But () implies that the left-hand side of the last inequality is nonnegative (since p, ˆp ), a contradiction. Hence the equilibrium is unique. APPENDIX C PROOF OF THEOREM 3.3 (STABILITY) We will construct a Lyapunov function and use LaSalle s invariance principle [5] to prove global asymptotic stability of the unique equilibrium point (x, p ). Define δx : x x, δp : p p. Consider the candidate Lyapunov function: V (x, p) r R xr x r z x r r (z) dz + l L δp l γ l () By definition, V (x, p) > for all (x, p) (x, p ) and V (x, p) if (x, p) (x, p ). Furthermore V is radially unbounded, i.e., V (x, p) as (x, p). Finally V (x, p) r (x r ) δx rẋ r + δp l ṗ l γ l r R l L If δx r then we have (since r (x s ) r (x r )) r (x r ) δx rẋ r δx r (φ r (x s ) q r ) + x r δx r (φ r (x s ) q r ) δx r (φ r (x s ) φ r (x s) δq r ) The first inequality holds since (φ r (x s ) q r ) + x r φ r (x s ) q r if x r > and φ r (x s ) q r, δx r x r if x r. The last equality holds since φ r (x s) qr by Lemma B. and (6). Hence r R r (x r ) δx rẋ r δx T (Φ(x) Φ(x )) δx T δq < δx T H T δp where the last inequality holds since δx T (φ(x) φ(x )) < by Lemma B. and assumption C. Similarly γ l δp l ṗ l δp l (y l c l ) + p l δp l (y l c l ) δp l δy l where the last inequality holds since δp l c l δp l yl by the equilibrium condition (7). Hence δp l ṗ l γ l δp T Hδx l L Therefore if δx then V (x, p) < δx T H T δp + δp T Hδx and if δx then V (x, p). This means V (x, p) and V is indeed a Lyapunov function. Consider the set Z : { (x(t), p(t)) V (x(t), p(t)) for all t } of trajectories on which V. If the only trajectory in Z is the trivial trajectory (x, p) (x, p ) then LaSalle s invariance principle implies that (x, p ) is globally asymptotically stable. We now show that this is indeed the case. As shown above V implies δx, i.e., any trajectory (x(t), p(t)) in Z must have x(t) x for all t. This means ẋ and hence, for all t, q(t) Φ(x(t)) since x(t) x > by Lemma B.. That is, for all t, H T p(t) Φ(x ) and hence p(t) p since H has full row ran by C3. Therefore (x, p) (x, p ) is indeed the only trajectory in Z. This completes the proof of Theorem 3.3.

12 APPENDIX D PROOF OF THEOREM 3.4 (FRIENDLINESS) Let the source be defined by φ r (x s ; µ) µ φ r (x s ) + ( µ) ˆφ r (x s ), µ [, ] Algorithm ˆM and M corresponds to µ and µ respectively. Let x g and τ g be the throughput and RTT of the TCP NewReno source in Fig.. The equilibrium is defined by F (x, µ) where x : (x s, x g ) and F is given by: Φ s (x s ; µ) τg x g T x s + x g c where the first equation follows from p τ g x g φ r (x s ; µ), r s and p is the congestion price at the bottlenec lin. Applying the implicit function theorem, we get ( ) dx F dµ F x µ ] [ Φs (x s ) ˆΦ ] s (x s ) T [ Φs x 3 g where the inverse exists by condition C. C also guarantees Φ the inverse of s (x s ; µ), denoted by D(µ); C4 ensures i s D ij(µ). Let A : Φ s x 3 T and d : g x 3 D ij (µ) g Then Thus [ Φs x 3 p ] T T µ [ A d T A ( ) F [ T F ] x µ i,j ] Dd d T A ( Φ s (x s ) ˆΦ s (x s )) () By matrix inverse formula, A ( Φs ) x 3 T g D(µ) + D(µ) T D(µ) x 3 g T D(µ) Substitute it into (), we have T A (ˆΦ s (x s ) Φ s (x s )) ( ) + T D(µ) T D(µ)( Φ x 3 s (x s ) ˆΦ s (x s )) g T D(µ) ( ) x 3 g x 3 g T D ir (µ) ( D(µ) φ r (x s ) ˆφ r (x s )) r s i s where the inequality follows because D(µ) is negative definite, i s D ir(µ) < and φ r (x s ) ˆφ r (x s ). Thus we have T xs µ for µ [, ], i.e., the aggregate throughput of the over its available paths is increasing in µ. This means M (corresponding to µ ) will attain a higher throughput than ˆM (corresponding to µ ) when separately sharing the test networ in Fig. with the same. APPENDIX E PROOF OF THEOREM 3.5 (RESPONSIVENESS) A. Proof of part Fix any eigenvalue λ of J. Let z : (x, p) Z be the corresponding eigenvector with z. Then we have [ [ ] [ x Λ Φ ] [ λ x H p] T x Λ γ H p] Hence [ ] [ Λ λ x Λ γ p] ] [ H T x H p] [ Φ x Premultiplying z H on both sides, we have λ xh Φ x x + (ph Hx x H H T p) x H Λ x + ph Λ γ p The denominator is real and positive, and (p H Hx x H H T p) in the numerator is imaginary. Hence Re(λ) Re ( x H Φ x x) x H Λ x + ph Λ γ p x [ ] H Φ + x x x H Λ x + ph Λ γ p < where the last inequality holds because the numerator is negative by condition C and the denominator is positive. Since this holds for all eigenvalues λ of J, the linearized system () is stable. Moreover Re(λ) λ(j ) as desired. B. Proof of part Consider two algorithms ( ˆK, ˆΦ) and ( K, Φ) such that ˆK s K s and ˆΦ s Φ s For any (nonzero) z (x, p) Z we have for all s S x H ˆΛ x xh Λ [ x (3) x H ˆΦ ] + [ x x H Φ ] + x < (4) x x Hence λ(ĵ ) λ( J ).

13 3 APPENDIX F PROOF OF THEOREM 3.6 (TRADEOFF) Fix an s. Let f r (x s ) : ˆφ r (x s ) φ r (x s ) and F (x s ) : (f r (x s ), r s) ˆΦ s (x s ) Φ s (x s ). Suppose for the sae of contradiction that ˆΦ s (x s )/ Φ s (x s )/ but ˆΦ s (x s ) Φ s (x s ) does not hold, i.e., there exists a finite x s and a r s such that f r (x s) ˆφ r (x s) φ r (x s) < (5) Since [ F/ ] + by assumption, a trivial modification of Lemma B. shows that, for all x s x s, (x s x s) T (F (x s ) F (x s)), i.e., r s (x r x r ) (f r (x s) f r (x s)) (6) Choose an x s as follows: for all r r, choose x r x r and then use condition C5 to choose an x r < large enough so that x r > x r and f r (x s ) > f r (x s)/. With this x s, (6) becomes (x r x r) (f r (x s ) f r (x ( s)) > (x r x r) f r(x ) s) > where the last inequality follows from (5). This is a contradiction and hence ˆΦ s (x s ) Φ s (x s ). APPENDIX G PROOF OF THEOREM 4. We will show the results hold for any n N +. Since lim n x s n x s, the results also hold for n. When [ ] β, it is easy to show that φ r satisfies C and + Φ s is negative semidefinite under the conditions of the theorem. We hence prove the theorem for β >. A. Proof of part Fix any n N + and β >. Fix any finite p such that q r > for all r. Fix any s S. We now show that there exists an x s > that satisfies (6), in particular φ r (x s ) q r, in two steps. First, there exists an x s that satisfies φ r (x s ) q r if and only if ( ( )) xs n φ r (x s ) τr x s + β q r, (7) x r which is equivalent to x r x s n β β + q r τ r x s (8) Since this holds for all r s, we have ( ) n xr (9) x r s s n ( ) n β β + q r s r τr x s : ψ ( x s ) Clearly ψ (C) as C. Let C : min r s q r τ r (3) Then C < since q r > for all r by assumption. Moreover q r τ r C for all r s and hence ψ (C) + ( ) n β β + q r τ > r r r C where r is a minimizing r s in (3). Since ψ(c) is continuous, there exists an C [C, ) with ψ( C). Moreover such a C is unique since ψ(c) is strictly decreasing. Finally consider the set of x s with x s C. All such x s satisfy (8) with x r β β + q r τ r C x s n : a r x s n (3) But C xs ( r s a r x s n ), implying x s n C r s a r In summary, given any finite p such that q r > for all r, a solution x s > to (8) is uniquely given by a r x r C, r s (3) s a where a r : β β + q r τ r C and C x s is the unique value at which ψ( C). We now prove the other conditions in C: y s l (p) p l, lim p l ys l (p) According to (9), we can show that C is a decreasing function of q r and q r τr C is an increasing function of q r for r s. Thus, C is a decreasing function of pl and q r τr C is an increasing of p l if l r because q r l L H lrp l. For each l L, let s l : {r l r, r s}, then by definition and (3), we have y s l (p) r s l a r r s a r C r s l a r r s l a r + r s l a r C. Since a r is a decreasing function of q r τr C, it is also a decreasing function of p l if l r. Recall that C is also a decreasing function of p l, yl s (p) is thus a decreasing function of p l, in other words, ys l (p) p l. On the other hand, as p l, q r for all paths r traversing l. Then x r by (7) for l r, which shows lim pl yl s (p).

14 4 B. Proof of part To prove φ r (x s ) satisfies C and C3 for β >, we will show that the Jacobian Φ s (x s )/ is negative definite if < β, s 8 and τ r are the same for r s. Other properties of C and C3 are easy to prove and we omit the proof. Fix an s and let τ r τ, the common round-trip time for all r s. Let Λ s : diag{x s } and a s : ( xr x s Then the Jacobian of Φ s at x s is xn r x s n n ), r s Φ s 4( β) τ x s 3 T β x s n τ x s Λ ( s I s + a T ) s Λ s and it is negative definite for β > if [ ] I s + a T + s is positive definite. We now show that this is indeed the case when s 8, i.e., for any z s R s, z T s (I s + a T s )z s z s + z r a r z r > (33) r s r s By Lemma G. below, T a s and a s. Then (33) follows from Lemma G. below provided s 8. Hence the Jacobian is negative definite. 3 The proof of Theorem 4. is complete after Lemmas G. and G. are proved. To show that it satisfies C3, it follows directly from (7) that if x r then φ r (x s ). It is also clear from (7) that the converse holds. This proves C3. Lemma G.: Fix any integer p. Given any x R m +, define a vector a in R m as follows: a i x i x p m j x i m, i m j j xp j Then m i a i and m i a i. m Proof: It is obvious that i a i. To show, we have m i a i m i a i xp i ( ) + 4 i x i 4 ( ( i i j xp j j j) xp+ i ) ( x j xp j j j) x + 4 i x i 4 ( ( i j j) xp+ i ) ( x j xp j j j) x ( i<j m x ix j (x i x j ) i 4 ( ) ( ) j x j j xp j x p ) x p j Lemma G.: Let a R m that satisfies m i a i and m i a i. Then for any nonzero z Rm we have m m f(z) : zi + i i z i m i a i z i > provided m 8. Proof: Given any M let Z M : {z m i z i M}. It then suffices to show that, for every M R, f(z) > for z Z M. Given any M, consider m m min f(z) min zi + M a i z i (35) z Z M z Z M i i i Its Lagrangian is ( m m m ) L(z, µ) zi + M a i z i + µ z i M i where µ is the Lagrange multiplier. Setting L/ z i for all i m and substitute it into m i z i M, we obtain the unique minimizer given by µ 3M/m and z i M ( 3 m a i ). Then ( ) min f(z) M 9 m z Z M 4 m a i M ( ) 9 4 m i i Hence, when M, min z ZM f(z) > if n < 9. When z is nonzero but M, then f(z) > from (35). APPENDIX H PROOF OF LEMMA 3. By the definition of D (A ), we have E D (A ) a ij d P i,j j a j i,j P( j d a j ) P( i,j a ij )) ( ) q A d i q i A i + o q i, i a ij where the last equality follows from the independence of a ij and P( j a j ) ( q ) A A q + o(q ), P( ij a ij ) i ( q i) Ai i A i q i + i o (q i). Thus, E D (A ) ( ) a ij d q A i i,j q + o q. A 3 If β the Jacobian degenerates to Φ s 4 τ x s 3 T, (34) which is merely negative semidefinite.

15 5 Qiuyu Peng received his B.S. degree in Electrical Engineering from Shanghai Jiaotong University, China in. He is currently woring towards the Ph.D. degree in Electrical Engineering at California Institute of Technology, Pasadena, USA. His research interests are in the distributed optimization and control for power system and communication networs. Anwar Walid Anwar Walid is a Distinguished Member of Technical Staff with the Mathematics of Networ Systems Research, Bell Labs, Murray Hill, New Jersey. He received a B.S. degree in electrical and computer engineering from Polytechnic of New Yor University, and a Ph.D. in electrical engineering from Columbia University, New Yor. He holds ten patents on computer and communication networs and systems. He received the Best Paper Award from ACM Sigmetrics, IFIP Performance. He has contributed to the Internet Engineering Tas Force (IETF) with RFCs. He served as associate editor of the IEEE/ACM Transactions on Networing (ToN). He is associate editor of the IEEE/ACM Transactions on Cloud Computing and the IEEE Networ Magazine. He was co-chair of the Technical Program Committee of IEEE INFOCOM. Dr. Walid is an IEEE Fellow and an elected member of Tau Beta Pi National Engineering Honor Society and the IFIP Woring Group 7.3. Jaehyun Hwang Jaehyun Hwang (M ) received the B.S. degree in computer science from The Catholic University of Korea, Korea in 3, and the M.S. and Ph.D. in computer science from Korea University, Seoul, Korea in 5 and, respectively. Since September, he has been with Bell Labs, Alcatel-Lucent, Seoul, Korea as a Member of Technical Staff. His current research interests include data center protocols, software-defined networing, multipath TCP, and HTTP adaptive streaming. Steven H. Low (F 8) is a Professor of the Department of Computing & Mathematical Sciences and the Department of Electrical Engineering at Caltech. Before that, he was with AT&T Bell Laboratories, Murray Hill, NJ, and the University of Melbourne, Australia. He was a co-recipient of IEEE best paper awards, the R&D Award, and an Oawa Foundation Research Grant. He is on the Technical Advisory Board of Southern California Edison and was a member of the Networing and Information Technology Technical Advisory Group for the US President s Council of Advisors on Science and Technology (PCAST) in 6. He is a Senior Editor of the IEEE Transactions on Control of Networ Systems and the IEEE Transactions on Networ Science & Engineering, is on the editorial boards of NOW Foundations and Trends in Networing, and in Electric Energy Systems, as well as Journal on Sustainable Energy, Grids and Networs. He received his B.S. from Cornell and PhD from Bereley, both in EE.