DIMENSIONING BANDWIDTH FOR ELASTIC TRAFFIC IN HIGH-SPEED DATA NETWORKS

Transcription

1 Submitted to IEEE/ACM Transactions on etworking DIMESIOIG BADWIDTH FOR ELASTIC TRAFFIC I HIGH-SPEED DATA ETWORKS Arthur W. Berger and Yaakov Kogan AT&T Labs 0 Crawfords Corner Rd. Holmdel J, U.S.A. {awberger, yaakovkogan}@att.com Abstract Simple and robust engineering rules for dimensioning bandwidth for elastic data traffic are derived for a single bottleneck link via normal approximations for a closed-queueing-network (CQ) model in heavy traffic. Elastic data applications adapt to available bandwidth via a feedback control such as the Transmission Control Protocol (TCP) or the Available Bit Rate transfer capability in ATM. The dimensioning rules satisfy a performance measure based on the mean or tail-probability of the per-flow (per-connection) bandwidth. We provide a new derivation of the normal approximation in CQs using more accurate asymptotic expansions and give an explicit estimate of the error in the normal approximation. A CQ model was chosen to obtain the desirable property that the results depend on the distribution of the file sizes only via the mean, and not the heavy-tail characteristics. We model the load in terms of the file sizes and the think-times between transfers, and consider the resulting flow of packets as dependent on the presence of other flows and the closed-loop controls. We compare the model with simulations, examine the accuracy of the asymptotic approximations, quantify the increase in bandwidth needed to satisfy the tail-probability performance objective as compared with the mean objective, and show regimes where statistical gain can and can not be realized.. Introduction This paper considers the problem of dimensioning bandwidth for elastic data applications in packetswitched communication networks, such as Internet Protocol (IP) or Asynchronous Transfer Mode (ATM) networks. Elastic data applications can adapt to time-varying available bandwidth via a feedback control such as the Transmission Control Protocol (TCP) or the Available Bit Rate transfer capability in ATM. Typical elastic data applications are file transfers supporting or the world wide web. A contribution of this paper is to derive simple, closed-form dimensioning rules that satisfy a performance objective on per-flow or per-connection bandwidth. We have in mind both IP and ATM networks that are offering a best-effort service. In the context of ATM, we are considering Unspecified Bit Rate (UBR) or Available Bit Rate (ABR) transfer capabilities where the minimum frame rate or cell rate is at a default value of zero, []. The network may be supporting other services that network nodes serve at higher priorities; in which case the engineering rules herein can be used by a network operator that wishes the elastic-data applications to have some bandwidth even during periods when the higher-priority connections are fully utilizing their own portion. We assume that the closed-loop controls for the elastic data applications are performing well. For the present work, the key attribute of a well performing control is that it maintains some bytes in queue at the bottleneck link, with minimal packet or cell loss. In contrast, poorly performing controls oscillate between under-controlling (leading to excessive packet or cell loss) and over-controlling (where, for example, TCP needlessly enters a slow-start or time-out state). Thus, poorly performing controls limit 05//98 :39 AM

2 the user s throughput below what would have been possible, given the available bandwidth. Our assumption of well-performing controls is consistent with ongoing research efforts to improve current controls and leads to significant simplification in the network model. We also assume that the output ports of the network nodes use some type of fair queueing across the class of elastic-data flows/connections; for a recent review of implementation of various fair queueing algorithms see Varma & Stiliadis []. This assumption reflects the trend in the industry to implement such service disciplines in order to provide some fairness. In the model, we assume the network node uses processor sharing. As described in the next section, with the above assumptions we obtain a simple, product-form, closed queueing network (CQ) model in heavy traffic. CQ models have been used extensively for the analysis and design of computer systems and networks, see for example [3] or [4], and our model is one of the simplest; though the entity modeled by a job is non-standard, see Section. Our CQ model consists of one inifinite server (IS) and one processor sharing (PS) server and was motivated by Heyman, Lakshman, and eidhardt, [5], who also use a CQ model where focus is on analyzing the performance of the closed-loop control TCP in the context of Web traffic. An important practical feature of these CQ models is their insensitivity: the distribution of the underlying random variables influences the performance only via the mean of the distribution. Thus, given the assumptions of the model, our engineering rules pertain in the topical case when the distribution of file sizes is heavy tailed [6], though with finite mean, and thus the superposition of file transfers is long-range dependent [7]. Since a key ingredient to the design problem, the forecasted demand, has significant uncertainty, we believe that simple, ball-park dimensioning rules are appropriate. In order to attain such closed-form rules we use the normal approximation, which pertains in the regime of interest: heavy traffic with many flows/connections and with high-speed links. A second contribution of the paper is new derivations of the normal approximation that in particular provide an estimate on the error of this approximation. The dimensioning problem focused on here is actually just a piece of the overall network design problem. In the larger context, there is a candidate topology, forecasted point-to-point demand, where the demand would be terms of the number of simultaneous flows or connections to be supported in a given load-set period, and a routing of the demand across the network. This then gives the number of concurrent flows/connections to be supported on each link of the network. In the present paper we focus on a single link and assume all of the flows/connections on the link are bottlenecked at this link. This is equivalent to the conservative procedure of sizing each link for the possibility that it can be the bottleneck for all of the connections on it. In subsequent work, we plan to extend the results to multiple links and to the overall network design problem. One aspect of this problem is the identification of bottleneck nodes in general networks, see [8] for initial results.. Related Topics One can view the above dimensioning problem as a variation of the well known capacity assignment problem in the literature on design of computer networks, see e.g. [9]. In the present paper, the exogenous inputs are not the offered flows (bits/second) but rather the file sizes, as here we consider the offered flows as dependent on the state of the network via the closed-loop control. One can also make a comparison with traditional traffic engineering of telephone networks. Traditional dimensioning for telephone circuits uses the Erlang blocking formula, and, for multi-rate circuits, the generalized Erlang blocking model and associated recursive solution of Kaufman [0] and Roberts [] and asymptotic approximations of Mitra and Morrison []. These formulas assume constant-rate connections, which is natural for traditional circuits. More recently, the concept of effective bandwidth extends the applicability of these formulas by assigning a constant, effective rate to variable-rate Berger & Kogan 05//98

3 connections, see. Chang & Thomas, [3], de Veciana, Kesidis & Walrand, [4],or Kelly [5] for recent summaries. Assigning an effective bandwidth is reasonable (in some parameter regions) for classes of traffic such as packet voice, packet video, frame relay, and Statistical Bit Rate service in ATM. However, for elastic data applications that do not have an inherent transmission rate, but rather adapt to available bandwidth, the concept of effective bandwidth seems dubious. (However, see remarks below in Section 4 after equation (7).) Also, the traditional performance criterion is the blocking probability of a new connection request. This criterion is rather irrelevant for best-effort services if, roughly speaking, all connection requests are granted, or if a request is not even made. In contrast, we have chosen a more pertinent performance criterion based on the bandwidth obtained per-connection or flow. Although we make the conservative assumption that the closed loop control is performing well, one should note that a topic of on-going research is the enhancement of closed-loop controls to improve the throughput that is indeed realized in various contexts. The implemented changes in TCP of congestion avoidance, fast retransmit, and fast recovery have this aim, see for example [6]. Further changes are also being proposed, such as explicit congestion notification, [7]. The design of the feedback control for the Available Bit Rate transfer capability in ATM received tremendous attention at the ATM Forum, an industry forum; for a summary see Fendick [8]. Also, of much interest is the performance of TCP/IP over an ATM connection that is either UBR or ABR: Significant degradation in throughput has been observed and various enhancements have been considered, see for example Romanov and Floyd [9], and Fang et al. [0]. The previously cited Heyman-Lakshman-eidhardt s model predicts behavior both when the performance is poor and good, [5]. Thus, we view as complementary: () network design that assumes well-performing closed-loop controls, and () control implementations that make good use of the deployed bandwidth.. Outline of Remaining Sections Section describes the CQ model and performance objectives. Section 3 summarizes pertinent asymptotic approximations and presents new results that provide more accurate approximations and an estimate of the error of the normal approximation. Section 4 presents the proposed dimensioning rules. Section 5 contains numerical results on the accuracy of the model and on the implications of the dimensioning rules. Section 6 contains the conclusions.. Closed Queueing etwork Model For simplicity we use the term connection to apply to either an ATM connection, or more generally a virtual circuit in a connection orientated packet network, or, with some blurring of meaning, to an IP flow of packets, where a flow is defined at the discretion of the network designer and would likely include the source and destination IP address, or sub-net thereof, and possibly additional descriptors such as the protocol field in the IP header or port number in the TCP header. More informally, one can think of a connection as representing a user. Also we use the term link for the object to be dimensioned, which may indeed be an entire transmission path devoted to the elastic-data traffic, or may be a portion thereof, perhaps a Virtual Path Connection in the context of ATM. The link is to be sized to support connections. Thus, for the dimensioning step, we take the viewpoint that there is a static number of connections present. Each connection alternates between two phases: active and idle. During the active phase packets are transmitted from the source to the destination. The active phase completes, and the idle phase begins, when no packets are in transit. In turn, the idle phase ends when the source begins to transmit another packet. The important case we have in mind is where the packets in an activity period constitute the transmission of a file between two higher-layer applications, across a wide-area high-speed network where multiple packets are typically in transit at a given time. Though, one could also consider the simpler case where at most one packet would be in transit. Berger & Kogan 3 05//98

4 The fixed number of connections and their alternation between active and idle phases makes plausible the use of a closed queueing network (CQ) model with two types of servers. The first type, referred to as a source, is an infinite server (IS) (equivalently the number of servers equals to the number of connections) that models connections while they are in the idle phase. The second type is a processorsharing (PS) server that models the queueing and emission of packets on the link. A diagram of the CQ model is given in Figure. Infinite-Server (IS) node, mean service time = λ, representing the sources (flows, connections) that share the bottleneck link. Processor-Sharing (PS) node, mean service time =µ, representing a network node s output port and buffer at the bottleneck link. Figure. Closed Queueing etwork (CQ) model, wherein a job represents the flow of packets that transfers a higher layer digital object such as a file or message. The one non-standard aspect of the CQ model is the entity represented by a job. For network dimensioning, we are interested in scenarios where the data network is heavily loaded. During such times, network resources (as opposed to end-system resources) will tend to be the limiting factor on the throughput obtained for the elastic-data connections. Moreover, the feedback controls of TCP and ABR will tend to seek out and fill up the available bandwidth. At heavily loaded links, a connection's feedback control, when properly designed and functioning, will attempt to keep at least one packet queued for transmission on the link (otherwise the control is needlessly limiting the throughput). We assume that this is the case. Thus, at an arbitrary point in time, the number of connections that are in the active phase equals the number of connections that have a packet in queue at the bottleneck node, which equals the number of connections that have a packet in queue under the hypothetical scenario that the stream of packets of an active phase arrived as a batch to the network node. A job in the CQ model represents this hypothetical batch arrival. Thus, a job represents all of the packets of an active phase of a connection. ote that a job in the CQ does not capture the location of all of the packets of a file transfer, since at a given moment some of these packets may have reached the destination, while other packets are in transit, and others are still at the source. Clearly, with this notion of job, the CQ can not model packet queue lengths or packet losses. However, it does model the number of connections that are in the process of transferring a file, given the assumption of well-performing controls. And this latter entity is just what we need to model the per-connection performance objective. Thus, the beginning of an active phase (or equivalently the hypothetical batch arrival to the network node) corresponds to a job arriving to the processor sharing (PS) node in the CQ model. A job departing from the PS node and returning to the IS server corresponds to the event when a connection completes its active period and becomes idle. From the number of active connections at the network node (the number of jobs at the PS node in the CQ) one can derive the performance measure of the bandwidth received per connection. Berger & Kogan 4 05//98

5 Our CQ model begins to break down when a connection's feedback control is over-constraining, and the otherwise bottleneck node empties of packets of a connection that is still in the process of transferring a file. Also, note that the CQ does not model a lightly loaded network. A connection (a source) is characterized by two parameters: () the mean time in seconds between completion of transmission of a file and the start of the transmission of the next file, denoted λ, and () the mean size of a file in bits, denoted f. λ is the mean service time in the IS node (the mean think time). In future work we will consider the case of different classes of sources, distinguished by their value of f and/or λ. ote that the characterization of a connection only depends on the mean value of the file sizes and the idle times. Moreover, for the distribution of jobs at the PS node and for the key dimensioning equation, only the product λf is pertinent. This product is the throughput of the source given that the network is imposing no restriction on this flow. Heyman et al. in their modified Engset models also find that the impact on performance by the distribution of file sizes and think times is only via the product λf, [5]. Given the mean file size and the capacity of the link, denoted B, the mean service time of a job in the PS node, assuming no other jobs present, is f / B and is denoted µ. Let Q o be the number of jobs at the IS node, and let Q be the number jobs at the PS node at an arbitrary time. The steady state distributions for Q o and Q are well known, e.g. [9], and the following forms are useful for the sequel. and n ( λ / µ ) µ / λ Pr( Q = n) = = G( ) ( n)! H( ) ( n)! ( λ / µ ) µ / λ Pr( Q0 = n) = Pr ( Q = n) = =, G( ) n! H( ) n! n 0 5, () n n k ( / ) / where G( ) = λ µ, and H( ) = µ λ. ( k)! k!. Performance Criteria k = 0 k = 0 6 () k 6 (3) We are interested in satisfying a performance criterion on the bandwidth that an arbitrary connection obtains in steady state, denoted B c. This bandwidth per connection is defined as B Bc =, (4) Q where B is the link bandwidth and Q is the conditional number of jobs in the PS node given that the PS node is not empty. By its definition, Pr Pr( Q n ( Q = n) ) Pr( Q ), n,..., = = =. (5) > 0 We consider performance criteria on the mean and on the tail probability of B c : or E[ Bc ] b (6) Pr( Bc < b) < α (7) Berger & Kogan 5 05//98

6 for given b and α, where we have in mind typical values for b in the range of 0 4 to 0 6 bits/second and α in the range of 0.0 to 0.. Consider the performance criterion on the mean (6). Given (4) and applying Jensen s inequality, then (6) is satisfied if E[ Q ] B / b. (8) We use (8) in the heavy traffic region where is large and Pr( Q > 0) is exponentially close to, i.e. the distribution of Q is close to Q, and Q / is approximately an atom, in which case (8) holds if and only if (6) does. As for the tail performance criterion, since Pr( B ) Pr( c < b = Q > B / b), (7) is satisfied if and only if Pr ( Q > B / b) < α. Using (5), the performance criteria (6) and (7) are satisfied, respectively, if the following conditions in terms of Q pertain: E[ Q ] Pr( Q > 0 3. Asymptotic Approximations 0 B / b, (9) ) Pr( Q > B / b) < α. (0) Pr( Q > ) Although the exact probability mass function (pmf) for Q, (), can be used in a numerical iteration to compute exactly the bandwidth B such that the performance criterion (6) or (7) is satisfied, it does not yield the simple, closed-form dimensioning rules that we are seeking. To obtain these closed-form expressions, we use the asymptotic approximations summarized in the present section. In our application, is large, λ does not depend on, while the service rate µ is large and of the same order as (a regime of many connections and of high-speed links). We assume that ρ λ / µ is constant. Under these assumptions, the asymptotics of the normalization constant! G( ) and performance measures that can be derived from it (such as queue length, moments and utilization) is given by Ferdinand []. In particular, see [], ρ E[ Q ] for ρ <, and () ρ E[ Q ] ( ρ ), for ρ >, () for >>. Moreover, the PS-node utilization approaches ρ when ρ <, while its utilization is exponentially close to when ρ >. Thus, when ρ <, the mean queue length and PS-node utilization is approximately the same as in an open queueing system with Poisson arrivals, First-In-First-Out service discipline, and exponential service times (M/M/) with offered load ρ. In fact, the queue length distribution at the PS node converges to that of the M/M// queue. For ρ >, there is no corresponding M/M/ system. However, ρ still can be considered as a characterization of the load at the PS node because the mean queue length increases with ρ, as shown in (). ote that () makes sense only for finite, while the M/M/ approximation, for ρ <, is obtained as. Here we are interested only in the heavy traffic regime defined by the condition Berger & Kogan 6 05//98

7 ρ >. (3) We use () to determine the dimensioning rule for the case of the mean performance criterion, see the following section. For the tail performance criterion, we need asymptotics for the right-hand tail of the distribution for Q 7. An asymptotic for the general product-form CQ has been derived given by Pr Q ( ρ ) > k by Pittel [, Lemma ]. For our simple CQ this asymptotic can be written in the following form: where and Here ρ (4) Pr ( Q = n) = c( ) exp F( x) F( x*), >, x n /, x ( 0, ), and c( ) has the property that F( x) = ( x) ln( x) + x( ln ρ ), (5) x* = ρ = arg min F( x). (6) x ( 0, ) F( x*) = ρ ln ρ, (7) ln c( ) 0 as. Thus, the asymptotic (4) is logarithmic in the sense that = as. ln Pr( Q n) F( x) F( x*) Equation (4) implies that Q / converges to x * as, as shown in Pittel [, Theorem ]. The function F( x) is known as a quasi-potential in the theory of large deviations, and it can be formally obtained as a solution of a non-linear differential equation, [3]. By approximating F( x) by its second order Taylor expansion about x *, one can obtain a normal approximation for Q. More precisely, Q x * Z / ρ as, (8) where Z has a normal distribution with mean zero and variance, and denotes convergence in distribution. This result is proved by Pittel [] for general CQs. The following two lemmas provide more accurate approximations for the distributions of Q 0 and Q, and from which the normal approximation easily follows. Moreover, the second lemma provides an explicit estimate for the error in the normal approximation. The proofs are provided in the appendix. Lemma. For ρ > and >>, H( ) defined in (3) has the following asymptotic expansion: 7, (9) / F ( x*) H( ) = e ρ + o( e ) where F( x) is the quasi-potential, and F( x*), given in (7), is negative, and Berger & Kogan 7 05//98

8 6 7 (0) n ρ ρ / / F ( x*) Pr ( Q0 = n) = e + o( e ). n! Thus, Q 0 is asymptotically Poisson with parameter / ρ. In the sequel, we call this approximation the Q 0 -Poisson approximation. A Poisson distribution with parameter / ρ can be considered as the distribution of the sum of independent random variables each with Poisson distribution with parameter / ρ. The application of the central limit theorem results in: Corollary. Under the conditions of Lemma : Q / ρ Z / ρ 0 as. () Lemma. Let k = x where x is constant, ρ > and >>, and 0 < ρ( x c ) 6 <, () where cis a constant independent of. Then where I β ( x) y β x Pr( Q > k) = e dy e + O π 3 π( x)! ( ) 3/ β ( x) = F( x) F( x*) = ρ ρ x + ρ x ln ρ x 7 ", (3) $# < A, (4) where F( x), which is the quasi-potential, and F( x*), are given in (5) and (7) respectively. Lemma provides a numerically convenient and accurate calculation of the tail probabilities of Q in the region of interest. This can be used in an iterative calculation for the bandwidth B that satisfies the performance criterion. To derive from (3) the normal approximation, we consider only the first term, which provides: / Pr( Q > k) = Pr ( Z > ( x) ) + O( ), β 7 (5) where Z has a normal distribution with mean zero and variance, and β( x ) is given by (4). If we in turn approximate β( x ) by expanding the log in (4) to second order (i.e. ln( y) y y ), we 0. 5 obtain β( x) ρ ρ( x) , which results in Corollary. Under the conditions of Lemma, k ( ) Pr( Q > k) Pr Z > ρ. / ρ Remark. In contrast with (4), asymptotic approximations (0) and (3) are exact in the sense that the ratio of the left hand side to the first term in the right hand side tends to as. Remark. In Lemma in condition (0), note that 0 < ρ( x ) if and only if k > ( ρ ), i.e. the quantile k is greater than the asymptotic mean. Also, ρ( x) < c / if and only if (6) Berger & Kogan 8 05//98

9 c k ( ρ ) < ρ O 7., and thus the difference between the quantile k and the asymptotic mean is Remark 3. Asymptotic expansion (3) is obtained using the uniform asymptotic approximation for the partition function, which was initially derived in [4], see also [5]. A new element in (3) is dependence of the lower integration limit β on the parameter x = k/, which can be viewed as a variable. Remark 4. Asymptotic expansion (3) reveals two sources of inaccuracy of the normal approximation. First, we take only the first term in the expansion (3). Then, we approximate the quasi-potential F( x) by the second order Taylor expansion. umerical results in Section 5. show that in our application the inaccuracy of the normal approximation is mainly caused by the second step, however, when is relatively small, both steps affect the accuracy. Remark 5. Asymptotic approximation (3) can be similarly derived for the multiclass Erlang and Engset models where the uniform asymptotic expansion for the partition function is also available, [5, 6]. Moreover, the form of (3), where the lower integration limit is explicitly expressed in (4) through the quasi-potential, suggests its generalization for product-form multiclass CQs and loss networks, by taking advantage of explicit expressions for the quasi-potential that are available in many important particular cases (see e.g. [8], [7]). 4. Dimensioning Rules In this section we use the asymptotic approximations to derive engineering rules for dimensioning the bandwidth. The dimensioning problem is simply stated as: Minimize B such that the chosen performance criterion (6) or (7) is satisfied. From the asymptotic limit (4) - (6), where ρ >, we obtain the approximate solution to (P) for the mean performance criterion. In this asymptotic limit, the distribution of Q / is the atom x *. Thus, Q (and E[ Q ]) is approximately ( ρ ) / ρ, which equals B µ / λ =, and (9) and (6) are λ f satisfied if B = h, where h = + b λ f. Equation (7) is a simple, approximate solution to (P). ote that equation (7) has the classic effectivebandwidth form where each connection has an effective rate of h. As mentioned in the introduction, the concept of effective bandwidths seems unlikely to be suitable for elastic data traffic as the inherent objects are the file sizes, and not the rates that packets traverse network. However, the parameter h is the harmonic mean of two rates that are indeed naturally associated with elastic data and will occur under, respective, extreme network conditions. Suppose the sources have a lot to send and the link is significantly restraining on the potential throughput of the sources, such that all sources have a job at the PS node almost surely, and Q equals and (6) holds as an equality, then Bc = b and thus each source would be transmitting at the rate b. At the other extreme, suppose the data network is very lightly loaded and imposes no constraint on the traffic from the sources - from the user s perspective the time to (P) (7) Berger & Kogan 9 05//98

10 transfer a file is negligible compared to the user s think time. Then each source would be transmitting at an average rate of λf. ow consider the performance criterion based on quantiles: Pr( Bc < b) < α, (7). ote that one can numerically compute the exact solution to (P) by various methods. One could iterate over B for the smallest value that satisfies (7), where for a given B, the tail probability of Q is computed and the inequality (0) is tested. For this iteration, the asymptotic approximation (6) provides a numerically * * stable method to compute the pmf of Q when is large. Start at n = x, set Pr(Q = n*) = and iteratively compute the log of the numerators in (), incrementing up from and down from n*, and then normalize the terms to sum to. If the computation is to be done many times as part of a larger design problem, one could increase the speed by using the standard error function and Lemma above. For numerically accurate and stable methods for more general CQ models see Choudhury, Leung & Whitt [8]. One can also express the exact solution, implicitly, in terms of the partition function H( ), (3). Since Pr( Q > B / b) / Pr( Q > 0) = Pr( Q B / b / Pr( Q = 0), from (3) we have that the tail performance criterion (7) is satisfied if H( B / b ) / H( ) H( 0) / H( ) < α. (8) Thus, the solution to (P) is the smallest B that satisfies (8). However, this is not just a simple iteration over the argument of H( ) as the terms in the sum H( ) depend on µ, which in turn depends on the unknown B. A more explicit solution uses the asymptotic approximation (5), and setting Prob( Q > 0) equal to one, (7) is satisfied if: β q, (9) where q α is the ( α) -quantile of a ormal(mean=0, variance=) random variable, i.e. Pr( Z > q ) = α for Z distributed ormal(0,). Substituting in the expression for β, (4), we have that (7) is satisfied if: = B α ρ x + x ln ρ x qα. (30) Thus, one can numerically solve for the ρ (, ( x ) ) where (30) holds as an equality, and hence determine B, since ρ = λf / B. If we make the further approximation of expanding the log to second order, and use the normal approximation (6), i.e. Q is ormal with mean µ / λ and variance µ / λ, we obtain that (7) is satisfied if The minimum B that satisfies (3) is B / b ( B / λf ) qα B / λf. (3) α B = h + γ + γ + γ, (3) where γ = q h / λf = q b / ( b + λf ). α α Berger & Kogan 0 05//98

11 Summarizing, we have: Proposition : An approximate solution to (P) is: where h = + b λ f that: B = h given the mean performance criterion (6) and (33) B = h + γ + γ + γ given the tail performance criterion, (7), (34) and γ = q h / λf = q b / ( b + λf ), and where the input parameters are such α α λf / b > q. (35) α ote that when B is chosen by the mean criterion, then B < λ f and we have the self consistency that ρ >, and thus the asymptotic approximation and the CQ model apply. However, when B is chosen by the tail-criterion guideline (34) without restrictions on the possible input parameters, then one can have the inconsistent outcome where ρ < and the asymptotic approximation does not apply. However, if the possible input parameters are restricted by (35), then the desired self consistency pertains. ote also that (35) holds in cases of interest, as q α is O() and >> ; for a discussion of magnitude of the ratio λf / b see Section 5.5. Proposition is our proposed dimensioning rule. It provides explicit, simple, closed-form expressions for the dimensioned bandwidth. It is, of course, less accurate than an exact calculation, or the asymptotic approximation (30), but given the uncertainty in the forecasted input parameters, greater accuracy does not seem needed. (We use the more accurate calculations in the next section to obtain a perspective on the inaccuracy of the normal approximation.) An illuminating form for (33) and (34) is to normalize B by what would have been a full allocation of capacity, namely b times the number of connections. B b! B b λf = b + λf λf = + γ / + γ / + γ / b + λf given the mean performance criterion (6), and (36) 0 5 " given the tail performance criterion (7). (37) $# 4. Connection Admission Control After the network has been dimensioned and put into service, the network operator may exercise no connection/flow admission control (CAC), as is the case in the present best-effort-service IP-based networks. In which case the performance objectives (6) and (7) should be viewed as design objectives. The network operator could advertise that the network has been designed based on such objectives. However, since the realized traffic will differ from the forecast, it would be imprudent for the network operator to offer a per-connection service commitment to individual users. If such a service commitment is desired as part of a business strategy, the network operator would likely wish to exercise a CAC policy on the realized traffic. To exercise a CAC policy in IP-based networks the network nodes would need to be able to identify a flow. If the network node is indeed using some form of fair-share scheduling, then the identification of flows is operationally already occurring based on the network node establishing and removing queues (link lists) for the arriving IP packets. Berger & Kogan 05//98

12 Equations (33) and (34) can be used for a connection admission control (CAC) policy. For the CAC policy, we need to determine the maximum number of connections allowed on the link, *, given bandwidth B. Equation (33) gives this value directly for the mean performance criterion, (6). For the tail performance criterion, (7), solving (34), or (3), for yields: * = B / h qα B / λ f, (38) where x is the integer part of x. The stricter performance criterion reduces the number of connections that could be admitted, but this reduction is relatively small for the pertinent case of a large link bandwidth where the standard deviation of Q, B / λ f, is relatively small compared with its mean. An improved, though more complicated, CAC would make use of real-time measurements. By monitoring the current number of connections established (the current number of queues in the fair-share scheduler), and averaging over the recent past, the network node can obtain a current estimate for the E[ Q ] or the Prob( Q k > ). These estimates could then be used, respectively, in (9) or (0) to test whether the performance objectives (6) or (7) are currently being met. If they are close to being violated, the network node could then deny the establishment of new connections. 5. umerical Examples 5. Comparison of CQ model to simulation of TCP/IP sessions In this section we compare the simple CQ model, (), with simulations of file transfers control by TCP. Heyman et al. [5], also compare their more detailed TCP-modified Engset model with simulations of TCP transfers, and they have kindly shared their simulation results with us. Heyman et al. simulate sources that transfer files across access links and then across a shared T link of.5 Mbps. A source alternates between an idle state and an active state, where during the latter the source transfers a file using TCP/IP. The TCP code used in the simulations closely parallels the prevalent Tahoe with Fast Recovery and Reno implementations, and thus includes the dynamics of slow start and time outs. One of the outputs from the simulation is the number of sources, A, that are in the active state at an arbitrary time. In cases where the TCP control is working well, then the number of active sources equals the number of sources that have packets queued at the T link, and thus would equal the number of jobs at the processor sharing node in the CQ model. Thus in this case, A should equal Q. However, in cases where the TCP control is not working as well, and packet losses are causing the sources to enter the slow-start phase, or to time-out, then there would be intervals where the number of the active sources (the number of sources presently attempting to transfer a file) would be greater than the number of sources that have packets queued at the bottleneck link; in which case, the random variable A would be greater than Q, in some sense such as the mean. Of course, by design the CQ does not distinguish between these two cases, and assumes that the former pertains. Thus, it is of interest to compare the CQ model with simulated scenarios where the TCP is working well and where it is not. For all of the simulations in [5], the mean idle time, λ, is assumed to be 5 seconds, and the mean file size, f (including overhead) is 00 Kbytes. Thus, in the CQ model, the mean service time at the PS node, µ, is.0667 seconds ( f divided by the link speed of.5mbps). The number of sources,, is either 5 or 50. This then specifies the distribution for Q in the CQ model, (). Further pertinent aspects of the simulation include the assumption that the buffer feeding the T links is served First-In- First-Out (FIFO) and the choice of buffer management strategy - When the buffer is full and an additional packet arrives, discard: () the new packet, or () the oldest packet, i.e. the packet at the front of the queue. Heyman et al. also consider different distributions for the idle times and the file sizes, including the pareto distribution. The simulations show that these distributions influence the distribution Berger & Kogan 05//98

13 of A only via their means, as predicted by their Engset models, as well as the CQ herein. Figure shows the prob. mass function for two of the scenarios considered in [5], as well as the prediction for Q in the CQ model. Scenario # has 50 sources, the buffer management strategy of discarding from the front, Pareto distributed file sizes and deterministic idle times, and a TCP degradation factor of 0.99, where the latter is a parameter in the Heyman et al. s model that captures the performance impairment due to TCP not accurately tracking the bottleneck link rate, and whose ideal value is one. Thus, in scenario, TCP is working excellently, and the match to the CQ model is surprisingly good. Scenario # has 50 sources, and discard-from-front buffer management, with deterministic idle times and file sizes, but where the TCP/IP is running over ATM and the TCP degradation factor is As expected, the number of active sources increases (the distribution shifts to the right) and the CQ model is no longer a good match. ote however that from the viewpoint of dimensioning, since there are more active sources than there are sources with bytes queued at the bottleneck link, those sources that do have bytes queued will still be receiving (better than) the bandwidth objective for which the link was dimensioned. Of course, in scenario some users are nevertheless obtaining a degraded goodput, but the cause is primarily the interaction of TCP with ATM and buffer size and buffer management policy. This degradation in throughput ought to be corrected by improvements in TCP and/or in ATM layer controls. p.m.f TCP simulation, scenario # TCP simulation, scenario # CQ model # of active sources, A Figure. Comparison of CQ model with simulation 5. Accuracy of Asymptotic Approximations Table illustrates the accuracy of the Q 0 -Poisson approximation, the asymptotic approximation (3), the first term of (3), and the normal approximation. For various numbers of connections,, and for ρ chosen so that the PS node is relatively heavily loaded, the quantile n was then chosen so that the exact tail probability is around 0.0. ote that although ρ is greater than, the CQ is stable since is finite, see Section 3 for further discussion. Table gives the percent error in the approximate calculation of Pr(Q > n), as compared with the exact calculation. The Q 0 -Poisson approximation uses exp 0µ / λ5 for the partition function H(), (3), and is clearly the most accurate approximation of the four listed. The asymptotic approximation (3) also uses exp 0µ / λ5 for H() and in addition approximates the summation in (). Table shows that the asymptotic approximation (3) is qualitatively closer to the exact calculation than is the first-term alone of the asymptotic approximation (3). The normal Berger & Kogan 3 05//98

14 approximation gives a rather poor estimate of the tail probability for these parameter values, although, as illustrated in Figure 3, the overall match to the pmf appears reasonable, even for the case of =5. In any case, as shown and discussed in the subsequent sections, the bandwidth dimensioned via the normal approximation is rather close to the correct value. Figures 4a and 4b illustrate the relative accuracy of the approximations for two of the cases in Table. For a relatively few connections, =5, Figure 4a visually shows that the correction term (the second term) in (3) significantly adds to the accuracy of the approximation. For a larger number of connections, =00, Figure 4b shows that the asymptotic approximation is accurate to tail probabilities as small as 0-6. # of connections, Input parameters ρ λ / µ quantile n Q 0 - Poisson approx. % Error in approximation for Pr(Q > n ), as compared with exact computation, (): Asymptotic approx. (3) First-term of asymptotic approx. (3) ormal approx % -4.0% 48% 80% % -6.0% 3% 96% % -.6% 0% 7% % -.5% 5% 57%, % -0.7% 0% 39% Table. Comparison of exact and approximate calculations of (). Prob. mass function for # at PS node, & associated normal approximation X prob. mass func normal approx # at processor-sharing node =5, mu/lambda=0 Figure 3 [] Berger & Kogan 4 05//98

15 Prob(# in PS node > k) Prob(# in PS node > k) log(prob(# in PS node > k) ormal approx Asymptotic approx, first term only Exact computation Asymptotic approx with correction term log(prob(# in PS node > k) 0^-6 0^-5 0^-4 0^-3 0^- 0^- 0^0 ormal approx Asymptotic approx, first term only Exact computation Asymptotic approx with correction term quantile k quantile k Figure 4a. Comparison of asymptotic approximations with exact calculation, for = 5, rho =.5 Figure 4b. Comparison of asymptotic approximations with exact calculation, for = 00, rho = Impact of Choice of Performance Criterion The tightness of the performance objective (7) impacts the dimensioning rule (34) via the parameter γ, which in turn depends on q α, the (-α )-quantile of a ormal(0,) random variable. ote that in the case of the normal approximation, the mean performance criterion is given by the 50% quantile, i.e. when q α is zero. Table illustrates the increase in the required bandwidth as α decreases. The key observation is that although tighter performance criteria (smaller values of α ) increase the required bandwidth, B, the percentage increase is relatively minor for the relevant cases of a few hundred connections. This is evident from (34) where >> γ and the square-root term is dominated by the linear term of, for large. Heuristically, one would expect this, since the pmf for Q is fairly concentrated around the mean, see Figure. Moreover, in the asymptotic limit where and ρ λ / µ > (many connections and fast service at the PS node), the distribution of Q / becomes an atom. Thus, the model suggests that a network operator can design for a rather strong sounding objective - with 99% probability the perconnection throughput is greater than a threshold - without requiring the deployment of much more bandwidth than for an objective on the mean. If the network operator also implements a connection admission control, see section 4.,then the design objective could be strengthened to be a service objective. Also note that the percentage increase based on the normal approximation is close to that from the exact calculation. This is due to the fact that the normal approximation describes deviations from the mean of the order of. If a network operator would like to use the tail performance criterion, but would prefer a simpler dimensioning rule than (34), then the dimensioning rule for the mean criterion could again be used with a larger value of the effective bandwidth parameter h, where this increase is chosen to bound the percentage increases for parameter values of interest, such as in Table. Berger & Kogan 5 05//98

16 umber of connections, Required bandwidth [Mbps], given mean performance criterion, (6) Percent increase in required bandwidth, relative to that for the mean criterion, for the tail criterion (7) with α equal to: % (3.7%) 6.3% (7.9%) 4.0% (6.%) % (9.5%).0% (.3%) 7.% (7.9%) % (6.6%) 8.3% (8.6%).0% (.3%) % (4.%) 5.% (5.3%) 7.6% (7.6%), % (.9%) 3.7% (3.7%) 5.% (5.3%) 5, % (.3%).6% (.7%).3% (.4%) Table. Required bandwidth, for given number of connections,, where λf = b = 00Kbps. The percents not-in-parentheses are from the exact calculation of bandwidth, while the percents in parentheses are from the normal approximation, Proposition, (33) and (34). 5.4 Engineered Bandwidth versus umber of Connections From equation (34), Figure 5 plots the engineered bandwidth, B, versus the number of connections,, for λf = 0 5 and α = 0.0, and for various values of the per-connection bandwidth objective, b, ranging from 0 3 to 0 7 bps. The exact calculation of the engineered bandwidth matches that shown in the figure to the thickness of the plotted lines. Figure 5 illustrates the basic points that (34) is essentially linear in for parameters of interest and that the engineered bandwidth depends strongly on the per-connection objective. Engineered bandwidth 0 *0^8 4*0^8 6*0^8 8*0^8 bandwidth objective b =.e6 bandwidth objective b =.e5 bandwidth objective b =.e4 bandwidth objective b =.e # of connections, Figure 5. Engineered bandwidth versus number of connections, for lambda*fbar =.e5, and alpha = Comparison to Full Allocation of Bandwidth It is instructive to compare the engineered bandwidth B with what would have been required if a full allocation of the per-connection objective, b, were provided for each of the connections. Setting x = b / λ f, equation (37) becomes: Berger & Kogan 6 05//98

17 ! B / b = + γ / + γ / + γ /, + x 0 5 " (39) $# where γ = qα x / ( + x). As γ / is typically small, B / b is roughly a simple hyperbola in x. For an analogous viewpoint, one can define the statistical gain to be the ratio of the full allocation of bandwidth to the engineered bandwidth, b / B. For the mean performance criterion, b / B is approximately + b / λ f. For a numerical example, Table 3 shows the statistical gain, given the tail performance criterion with α = 0.0 and =,000, for various values of λf and b. For the given parameter values, ρ is greater than one, and the normal approximation pertains. For comparison, the engineered bandwidth is computed both iteratively using () and via the normal approximation (34). As shown in Table 3, the normal approximation is quite close to the exact numerical computation. Also, the statistical gain is minor when the ratio b / λ f is small, but is significant when b / λf. From the viewpoint that a user will want to send much more than the provided bandwidth objective, then b / λf is small, and B is roughly as big as the full-allocation of capacity. Thus, in such cases, the model predicts the intuitive, negative result that a network operator can not attain significant statistical gain given a performance objective on per-connection bandwidth. However, although the per-connection bandwidth objective, b, applies at the epochs the user has initiated a file transfer, λf is the user s desired throughput averaged over time that includes the idle times (think times). Thus, it is plausible that a service provider could offer a bandwidth objective that is the same magnitude as λf, or even larger. When the ratio b / λ f is greater than one, the network operator can realize significant savings in the engineered bandwidth B as compared with the full allocation. An obvious issue is what are realistic values for λf, and we have begun measurement studies to estimate λf in various contexts. 6. Summary Input parameter: b / λf Statistical Gain, b / B exact computation simple normal approx Table 3. Ratio of full allocation to engineered bandwidth, b / B, for =,000, and α =0.0. We have derived simple and robust engineering rules for dimensioning bandwidth for elastic, data traffic for a single bottleneck link. The derivation is based on normal approximations for a closed-queueingnetwork (CQ) model in heavy traffic. We believe that simple, ball-park dimensioning rules are appropriate because of the uncertainty in the forecasts of the traffic demands. The robustness of the dimensioning rules follows from the insensitivity property of the CQ model whereby the distribution of Berger & Kogan 7 05//98

18 the underlying random variables is pertinent only via the mean, and of particular interest, the mean of the file-sizes and not their heavy-tail characteristics. We model the load in terms of the mean file sizes, f, and the mean think-times between transfers, λ, and consider the resulting flow of packets as dependent on the presence of other flows and the closedloop controls. Moreover, the dimensioning rules depend on the parameters f and λ only via their product λf. Thus the dimensioning rules have the nice feature that they depend on the user's traffic characteristics via a single parameter. Recall that λf is the average bit rate from a source during periods when the network is very lightly loaded. During these times, the user's traffic flow could be constrained for other reasons, such as the access line speed or the TCP window size set by the destination endsystem. Estimates of the parameter λf can be obtained from measurements studies of networks during periods of light loads. We compared our CQ with simulations of file transfers regulated by TCP. Despite the simplicity of our CQ model, it accurately predicted the distribution for number of active connections at the bottleneck link, given the condition that the feedback control was performing well. The dimensioning rules satisfy a performance measure based on the mean or the tail-probability of the per-flow (per-connection) bandwidth. In the case of the mean performance measure, (6), the dimensioning rule, (33), has the linear, effective-bandwidth form, where the effective bandwidth, h, is the harmonic mean of two rates, which respectively would occur when the data network is very lightly loaded or very heavily loaded. For the tail performance measure, (7), the dimensioning rule is still in closed-form, (34), though no longer linear in. If the network designer wishes to use a linear rule for the sake of simplicity, then the dimensioning rule for the mean criterion could again be used with an increased value of the effective bandwidth parameter h, where the increase is estimated for parameter values of most interest, see Section 5.3. The dimensioning rules are easily inverted to obtain a connection admission control policy for the number of connections/flows that can be admitted on a link of given bandwidth. We illustrated the increase in bandwidth needed to satisfy the tail-probability performance objective as compared with the mean objective. The key observation is that although tighter performance criteria increases the required bandwidth, the percentage increase is relatively minor, particularly for the relevant cases of a few hundred connections. This occurs since the pmf of Q is rather concentrated around the mean. Thus, the model suggests that a network operator can design for a rather strong sounding objective - with 99% probability the per-connection throughput is greater than a threshold - without requiring the deployment of much more bandwidth than for an objective on the mean. We also showed regimes where statistical gain can and can not be realized. A key parameter is the ratio of the per-connection bandwidth objective, b, to the user s ideal transfer rate given a lightly load network, λf. When b / λ f is less than one, the statistical gain is minor, but when b / λ f is as large as one, the statistical gain is around two, and if b / λ f as big as five, then the gain is greater than five. Values of b / λ f greater than one are plausible since the per-connection bandwidth objective, b, applies at the epochs the user has initiated a file transfer, while λf, in contrast, is the user s desired throughput averaged over the idle times (think times), and also may be limited by other constraints such as access speed. We provide a new derivation of the normal approximation in CQs using more accurate asymptotic approximations and give an explicit estimate of the error in the normal approximation. For the region of applicability, the asymptotic expansion (3) is accurate to within a few percentage points, and the Q 0 - Poisson approximation, (0), is accurate to within a tenth of a percentage point. The derivation of the Berger & Kogan 8 05//98