Statistical Estimation of Internal Available Bandwidth

Transcription

1 Statistical Estimation of Internal Available Bandwidth Ian H. Dinwoodie ISDS Box 9025, Duke University, Durham, NC June revised April Abstract This paper presents a method for using end-to-end available bandwidth measurements in order to estimate available bandwidth on individual internal links. The basic approach is to use a power transform on the observed end-to-end measurements, model the result as a mixture of spatially correlated exponential random variables, do estimation by moment methods, then transform back to the original variables to get estimates and confidence intervals for the expected available bandwidth on each link. Because spatial dependence leads to certain parameter confounding, only upper bounds can be found reliably. Simulations with ns2 show that the method can work well and that the assumptions are approximately valid in the examples. Keywords: Available bandwidth, network tomography, Weibull distribution, mixture models, spatial correlation, moment method MSC: 62M0, 62P30, 94A2 0

2 Introduction This paper is about how to use end-to-end available bandwidth measurements in order to estimate available bandwidth on individual internal links. Network bandwidth relates to economics and quality of service issues [3] and to traffic bottlenecks [2, 7]. There exist several tools for estimating end-to-end available bandwidth, including for example pathload [9, 0] and IGI [8], and some comparisons are made in [9]. We present a method for estimating internal available bandwidth on edges of a rooted tree within the network using simultaneous end-to-end observations on this tree. Simulations with ns2 [2] show that the method is useful but not perfect when different links have somewhat different traffic flows. There are flow patterns where the method does not work. The basic idea is the following. First, simultaneously measure end-toend available bandwidth from the root to the leaves (receiver nodes) of a tree embedded in the network. Then use a power transform on the observed end-to-end measurements and model the result as a mixture of spatially correlated exponential random variables. Finally, do estimation by moment methods and transform back to the original variables to get estimates and confidence intervals for the expected available bandwidth on each link. We use bandwidth in the sense of traffic rate measured in Mega bits or Mega bytes per second (Mb/s, MB/s) ([4], p. 40). Available bandwidth on a particular link is the difference between link capacity and prevailing traffic rate (or traffic load). End-to-end available bandwidth over a network path is the minimum of the link-specific available bandwidth quantities. Estimating internal available bandwidth from end-to-end measurements is a problem of network tomography. This class of problems involves using either synthetic or natural end-to-end network traffic features in order to study internal network properties. The data is hidden in that the quantities of direct interest on specific edges are not observed, but rather one observes a many-to-one function of them (in this case a minimum over edges in paths). With the appropriate statistical model and a multivariate approach that includes correlations among coordinates of the observed vector, one may still be able to make complete inferences. The method of statistical inference used in this paper is based on first and second moments, which is convenient with lots of hidden data when likelihood functions are impossible to calculate. Absolute deviations for fitting moments are used instead of least squares for robustness. Also, we compare ratios of moments instead of differences in order to get a more natural scale for comparison. The resulting objective function for parameter

3 fitting does not lend itself easily to confidence interval calculations, so these are done using quantiles from a bootstrap resampling step. The statistical model specifies the form of the moments, and the main idea is that a power transform of the available bandwidth is assumed to have moments like a mixture of correlated exponential random variables. This roughly translates into an assumption of Weibull distributions on the available bandwidth on each link with spatial dependence. The main results of the paper are in Section 3, where we derive the first and second moments of the end-to-end transformed available bandwidth for a certain precise mixture model in Proposition 3.. Then in Theorem 3. we show that consistent estimation of parameters is possible for a moment model given in (3.2) in which the variances are slightly adjusted from those derived in Proposition 3.. Section 4 gives the results of simulation with ns2.. Related Work There is an increasing amount of research on computer network traffic at the interface of statistics and computer science. Inferring internal bandwidth from end-to-end quantities falls in the class of network tomography problems that started with [2]. This class of problems is more fully described in [4]. A seminal tomography project was MINC (www-net.cs.umass.edu/minc/), a collaboration between researchers at ATT Research and UMASS Amherst. This project focused on using multicast probes for purposes including estimating internal loss rates and topology discovery mostly by maximum likelihood methods. The present paper does not use multicast probes nor maximum likelihood methods, and our goal of estimating bandwidth is different as well, but the fundamental idea of using correlations of multiple simultaneous traffic flows through a network to make internal inferences comes out of their work. A shift away from maximum likelihood methods appears in [], a paper which has other similarities with the present work including the use of queueing models for proposing moment models. Other network applications of moment methods are in [5, 20]. Recently there has been an increase in research on bandwidth estimation focused on algorithms for end-to-end estimation [2, 8, 9] and bottleneck identification [2, 7]. Research institutes on network traffic include CAIDA ( ICIR (The ICSI Center for Internet Research, and Merit ( where a wide range of network problems are studied including queueing, reliability, routing, and security issues, some of which relate to bandwidth. 2

4 2 Notation and Data Transformations A communication network G will be represented as a graph with vertices and edges G = (V, E), and each edge e E has a capacity c e in Mb/s. Time is theoretically continuous. Sampling is done at instants t, t 2,..., t k,..., t n. On edge e at a particular time index k there is also a traffic load L k e (also in Mb/s), which we will consider to be random. Then the available bandwidth on the link e is x k e := c e L k e. Our point of view is that the available bandwidth on each edge is a random variable with a distribution that is unchanging over the time interval of sampling. We are seeking to estimate its mean. A deeper discussion of variability, sampling and stationarity is in section 6 of [9] and in [8], and the details can be quite complicated both in theory and in practice. A tree T G will have a root 0, leaf nodes R called receivers, and d := R is the number of receivers and the dimension of the data. The statistical method will use simultaneous traffic flows on the tree (called statistical multiplexing in [9]) in a procedure that relates means and covariances of end-to-end available bandwidth at receivers to internal available bandwidth. The root node will have one child node. In Figure 2 for example, which shows the embedded tree within the network of Figure, the receivers are R = {9, 0, 7}, and the things to be estimated are the five expected available bandwidths (µ 2, µ 6, µ 7, µ 9, µ 0 ) for undirected links 0-2, 2-6, 2-7, 6-9, and 6-0. Then µ 2 = E(x k 2 ), the expected available bandwidth on edge 2-6, for example. In the path P r through the tree T that connects the root node 0 to a receiver node r R, the available bandwidth is yr k = min e Pr {x k e}. The first observation is a vector y R d +, a simultaneous measurement of the available bandwidths on all the tree branches. In the Figure 2 example, the observed vectors y k are in dimension d = 3 = R. The observation y is the image under a many-to-one nonlinear map of a hidden outcome x indicating available bandwidth on each edge on the entire route from root to receivers. This experiment is repeated identically n times, and observations y = (y, y 2,..., y r ), y2,..., y n are idealized as a sample of identically distributed vectors at the receiver nodes with nonnegative real components. A routing matrix A will be useful. The matrix A will have d = R rows, one for each leaf node in the tree T, and c columns indexed by edges in T. The row for leaf node i will have in column j if j is on the path from 0 3

5 Figure : Communication network simulation in ns Figure 2: Embedded tree for available bandwidth measurement Figure 3: Links for which only upper bounds can be estimated. 4

6 to leaf node i. For the tree in Figure 2, the matrix is given by A = (2.) We will use the notation P r for the collection of edges on the tree T from the root to node r, which are the edges indicated by in row r of the routing matrix A. The exponential distribution has the property that the minimum of independent exponential random variables is again exponential, which makes moment computations quite easy. If the link-specific available bandwidths were independent exponential random variables, the end-to-end available bandwidth y r = min e Pr {x e } would also be exponentially distributed. This exponential property turns out to not hold in practice, but a power transformation of the data leads to a distribution that is approximately exponential. Consider a monotone transformation of the type T (y r ) := (y r /κ) p (2.2) which gives T (y r ) = ( (min e Pr {x e } ) /κ) p = min e Pr {T (x e )} (κ > 0, p > 0). This will work well at transforming to exponential random variables if the link-specific available bandwidth random variables of primary interest x e have approximately the Weibull distribution with common shape parameter p, but possibly different scale parameters β e on each edge e, in the density f(t) t p e (t/βe)p. In the simulations of Section 4, we chose the power p by comparing means and standard deviations of the transformed bandwidth data T (yr k ), adjusting p to a common value over all receivers that makes the ratios of standard deviation to mean all close to, as required by the exponential law. A further extension allows for dependence across links, which is quite important for reasonable model fit. The details follow in Section 3. 3 Moments for Available Bandwidth Measurements In this section, the embedded tree will have root node 0 joined by edge 0 to its only child node. This notation contradicts the labelling of Figure 2, but will simplify the presentation and is no loss of generality. An edge will carry the label of its child node. 5

7 3. Moment Formulas Let t,..., t c, t E, t C be independent exponential random variables with rates γ,..., γ c, γ E, θ, and suppose the transformed available bandwidth a e := T (x e ) on edge e with T (x) := (x/κ) p as in (2.2) is a mixture of the quantities t e t E := min(t e, t E ) with probability α and t C with probability α. One can think of an underlying variable C flagging Common bandwidth (C = ) of exponential rate θ with probability α, and Separate traffic (C = 0) with probability α, and the conditional distribution of the link e bandwidth a e is then exp(γ e + γ E ). The Separate traffic components are correlated because of the common regulating or bounding variable t E in each edge variable t e t E. The model presented above is informally described as follows. For a fraction of time α, the (transformed) available bandwith is exactly the same on all links of the tree, and this common quantity t C of mean /θ may be thought of as a gap between background cross traffic and capacity. This is the state when C =. In the remaining fraction of time α when C = 0, there are separate correlated random variables t e t E for the available bandwidth on each link e. The random variable t E is the same for all edges. It can be thought of as a common regulating bound. The estimation procedure can get α and θ, but the common variable t E has a parameter γ E which cannot be identified separately (Theorem 3.). This will mean that only upper bounds on the expected available bandwidth for edges e can be estimated under separate traffic. The variable t E is the cause of identifiability problems and also the source of spatial correlation in the model. If α is near 0, then x e has approximately a Weibull distribution, a scaled power transform of an exponential random variable. The Weibull distribution has been justified for tail asymptotics of queue lengths in some cases [7], but available bandwidth is different so at present we do not know of a theoretical basis for its use in this application. Examination of histograms in our simulations and those in [2] make it reasonable, and the qualitative descriptions in [9] do not contradict its use here. Define λ := γ + γ E, and for edge e, := γ e. The vector λ = ( ) e T has c components, where c is the number of edges in the embedded 6

8 tree T. Then it follows that a e C=0 exp(γ E + γ e ) a e C= exp(θ) E(a ) = ( α) + α γ + γ E θ = ( α) + α λ θ E(a e ) = ( α) + α γ e + γ E θ ( α) + α θ (e ). We will be able to estimate the variables, but not γ E by itself because it is confounded with γ, appearing in every formula as γ + γ E. The consequence is that we will only get upper bounds on means E(x e ) for e. By elementary integration, we have ( µ e := E(x e ) = E(T (a e )) = κ Γ( + /p) ( ) α µ = κ Γ( + /p) + α µ e κ Γ( + /p) λ /p θ /p ( α λ /p + α e θ /p ) α (γ E + γ e ) /p + α θ /p, e. (3.) If one attempts to use a spatial independence model without introducing the variable t E (which gives the spatial correlation in Separate traffic and is responsible for the confounding issue), the results are misleading and the expected available bandwidth on edges is overestimated. Now let b r := T (y r ) = min{a e : e P r } be the transformed observations of end-to-end available bandwidth, which is also a mixture: { min{t e t E, e P r } if C = 0 b r = t C if C = Let A r be the r th row of the routing matrix A, so the real inner product A r λ is the r th row of the matrix product Aλ. Proposition 3.. Let b r be the (transformed) available bandwidth on path P r in the network. Then under the mixture model of dependent exponential random variables described above, ) 7

9 E(b r ) = ( α) A r λ + α θ E(b r b r2 ) = α ( e P r P r2 ) e P r2 P r f P r λ f + e P r P r2 f P r2 λ f + α 2 θ 2. Proof. For the mean one needs simply that the mean of the minimum of independent exponential random variables is the reciprocal of the sum of the rate parameters : E λ,θ,α (b r ) = ( α) γ E + e P r γ e + α θ (see [6]) and γ E + e P r γ e = e P r = A r λ. The second formula follows from the following result which can be verified with calculus. If Z i exp(η i ), i =, 2, 3 are independent, then 2 E(min(Z, Z 2 ) min(z, Z 3 )) = (η + η 2 + η 3 ) 2 η 3 + (η + η 2 + η 3 ) 2 (η + η 2 ) η 2 + (η + η 2 + η 3 ) 2 (η + η 3 ). To prove this, represent min{z, Z 3 } = min{z, Z 2, Z 3 } + (min{z, Z 3 } Z 2 )I {Z2 <min{z,z 3 }}. Let Z := min e Pr P r2 {t e t E }, Z 2 := min e Pr P r2 {t e }, Z 3 := min e Pr2 P r {t e }, and use b r = min{z, Z 2 }, b r2 = min{z, Z 3 } to get the formula for E(b r b r2 C = 0), the second moment for the separate traffic: E(b r b r2 C = 0) = ( e P r P r2 ) e P r2 P r f P r λ f + e P r P r2 f P r2 λ f. Finally use the mixture representation to get the weighted sum of the second moments 2/θ 2 when C = with probability α and the above expression when C = 0 with probability α. 8

10 Parameter identifiability for consistent estimation is hard to prove with the variance expression derivable from Proposition 3., so an adjustment will be used in the moment model below. The moment model on the transformed data b = T (y) is: E λ,θ,α (b r ) = α + α e P r θ Cov λ,θ,α (b r, b r2 ) = α ( ) 2 e P r P r2 2 + e P r2 P r f P r λ f + E λ,θ,α (b r )E λ,θ,α (b r2 ) (r r 2 ) e P r P r2 f P r2 λ f + α 2 θ 2 Var λ,θ,α (b r ) = E λ,θ,α(b r ) e P r (3.2) with α (0, ) and parameters > 0, e T, θ > 0. Observe that the variance terms from Proposition 3. would be 2( α) Cov λ,θ,α (b r, b r ) = ( e P r ) 2 + 2α ( α θ 2 + α ) 2, e P r θ whereas the actual formula in (3.2) is α ( e P r ) 2 + α θ. e P r These expressions are identical when α = 0 (no mixture case), and their difference is O(α) with an expansion in α that depends on λ, θ. When θ = e P r for each r, then the coefficient on α in the expansion vanishes, so the approximation is particularly good in this setting. There are values of λ, θ where the two expressions may differ substantially. Finally, if one goes ahead and uses the variance from the covariance formula Cov λ,θ,α (b r, b r ) rather than the adjusted version in (3.2), then in practice it seems to work well even though we have not been able to prove parameter identifiability. 3.2 Estimation Let us describe in more detail the moment-fitting procedure by which we attempt to choose the c + 2 parameters λ, θ, α (see [3] for a description of the method of moments). Define the objective function f bn,s 2 for parameter n estimation by f bn,s 2 (λ, θ, α) := bn n E λ,θ,α (b) + s 2 n Cov λ,θ,α (b) (3.3) 9

11 where denotes the L norm (the sum of the absolute values), bn is the d-dimensional sample mean of transformed end-to-end available bandwidth, and s 2 n is the d d sample covariance matrix of the transformed data (b k r) r R = (T (y k r )) r R, k =, 2,..., n. Then the following is a nonlinear minimization problem for getting estimates ˆλ, ˆθ, ˆα: min λ>0,ɛ<θ<θ,0<α< f b n,s 2 (λ, θ, α). (3.4) n The bounds ɛ and Θ (which can be any positive values) on θ are for theoretical reasons in order to avoid asymptotic problems of consistency when n. Theorem 3.. (Consistent Estimation) Suppose the process x, x 2,... of available bandwidths on edges of T is i.i.d. and the moments of transformed minimums b, b 2,... satisfy (3.2) for true parameter values λ > 0, ɛ < θ < Θ, 0 < α <. Suppose each node except for the root node 0 in T has at least two children. If the minimum of (3.4) is attained at interior values ˆλ n > 0, ɛ < ˆθ n < Ω, 0 < ˆα n <, then ˆλ n λ, ˆθ n θ and ˆα n α as n a.s. Proof. Let (ˆλ n, ˆθ n, ˆα n ) be a positive solution to the minimization of (3.4). By the law of large numbers, b n E λ,θ,α (b) a.s. and s2 n Cov λ,θ,α (b) a.s. Hence f bn,s 2 n (λ, θ, α ) 0. By the definition of the estimators, 0 f bn,s 2 n (ˆλ n, ˆθ n, ˆα n ) f bn,s 2 n (λ, θ, α ) 0. If any component ˆλ n,e were unbounded, then the quantity /A rˆλn for a receiver node r with e P r would cluster around 0 as n, and so Varˆλn,ˆθ n,ˆα n (b r ) would cluster around 0 (the numerator in its definition at (3.2) being bounded above 0 since ˆθ n > ɛ). But since f bn,s 2 n (ˆλ n, ˆθ n, ˆα n ) 0 and Var λ,θ,α (b r) > 0, the contradiction implies that ˆλ n is bounded. By definition, we have that ˆθ n, ˆα n are bounded as well. Since the estimators are bounded sequences, it is enough to show that any convergent subsequence must converge to the true values (λ, θ, α ). If any subsequence converged to another limit, say (λ, θ, α ), then this limit would satisfy E λ,θ,α (b) = E λ,θ,α (b) Cov λ,θ,α (b) = Cov λ,θ,α (b). 0

12 Now it is an exercise in algebra to check that these equalities imply that λ = λ, θ = θ, α = α. First, equality of the expectations and the form of the variance terms in (3.2) imply that e P r λ e = A r λ = A r λ for each receiver node r. Now the differences of expectations imply that α = α (look at E λ,θ,α (b r ) E λ,θ,α (b r 2 ) = E λ,θ,α (b r ) E λ,θ,α (b r 2 )). Finally comparing expectations once again shows that θ = θ. It remains now to show that λ = λ, knowing that Aλ = Aλ, α = α, θ = θ. For this we must use the off-diagonal covariance terms. Now take the difference of the covariance expressions for sibling nodes r, r 2 to get (λ r + s 2 ) 2 (2 + λ r 2 s + λ r s 2 ) = (λ r + s 2 ) 2 (2 + λ r 2 s + λ r s 2 ) where s = e P r λ e = e P r λ e, s 2 = e P r2 λ e = e P r2 λ e. Note that s 2 s = λ r 2 λ r = λ r 2 λ r, so we have the equation (λ r + s 2 ) 2 (2+ λ r + s 2 s s + λ r s 2 ) = Now, the function g : R + R + given by g(x) := (λ r + s 2 ) 2 (2+ λ r + s 2 s s + λ r s 2 ). (x + s 2 ) 2 ( 2 + x + s 2 s s + x s 2 ) is strictly monotone decreasing for any positive values s, s 2, and is therefore one-to-one. This implies that λ r = λ r. The covariances for receivers that share incrementally shorter paths from show that all components of λ equal those of λ. The expectations in the model are nonlinear in the parameters and the fitting function (3.3) uses the L norm for robustness, and so the setup does not lend itself to explicit confidence interval formulas. To get confidence intervals for the parameters λ, θ, α from the estimates that minimize (3.3), we use the percentile bootstrap method [6]. If the sample size is n, we resample n data points n times and recompute the estimates on each of the n resampled data sets. By (3.), the quantity κ Γ( + /p) ( ˆα ˆλ /p e + ˆα ˆθ /p )

13 with estimate recomputed on the bootstrap sample is an estimate on the upper bound for µ e. Then the 0.99 quantiles of all n of the recomputed estimates are used for an upper 99% confidence interval on µ e. This works well in practice, even though a certain percentage of the bootstrap refitting computations will not terminate properly when an unconstrained optimizer is used because the parameter estimates may occasionally go negative. We have attempted to use least squares estimators instead of L estimators because they have certain speed advantages in the computation. For example, they are used in large-scale estimation problems of atmospheric modelling []. However, least squares estimators never produced results as accurate or consistent as the L methods in our experience. In the particular simulations of Section 4, the sequence b r, b 2 r,..., b n r shows some small autocorrelation at the time intervals used for sampling (0.0 s), slightly above ρ = 0.0 in some cases. We treated them as temporally independent. 4 Simulations The method of Section 3 has been implemented with scripts in the language R [5] on data generated by the ns2 simulator. An outline of the steps in the simulation is. code the network traffic in the file sim.tcl for ns2 2. add flow monitors on each link to sim.tcl with file output at intervals of 0.0 s and run sim.tcl in ns2 3. code and execute the process.r R script to process the flow monitor file output to get available bandwidths on paths over time intervals [.20,.30] (k = ), [.40,.50] (k = 2),... [.20 n,.20 n+.0] (k = n),with time in seconds, and store them in R as a data matrix with n rows and a column for each path. 4. transform the data matrix with T in R, after choosing κ and p 5. compute the sample mean vector b and the sample covariance matrix s 2 on the transformed data matrix 6. fit the model with the objective function (3.3) using an unconstrained nonlinear minimizer, and invert the transformation for output 7. resample n rows repeatedly and refit for the bootstrap calculations. 2

14 The use of 0.0 s for time interval length of course may be adjusted depending on the situation. We present the results of one typical simulation on the network of Figure. In practice the flow monitors started after a period of stabilization of 5 seconds. There were four Pareto traffic sources and one TCP source, with configuration details in Table. Queue monitors were placed on each link and the true available bandwidths were measured knowing the physical link capacity and traffic rate. The histograms of available bandwidth are in Figure 4. Close inspection shows that some quantities are slightly negative. The reason for this may be the effect of the long queues, which may make traffic appear sometimes to flow at a faster rate than the physical capacity of the link as a queue empties, but we do not know the reason for sure. In the transformation T of (2.2), we took the scale κ = 0 5 and the power p = 3. The result on one path is in Figure 5. Quantile-quantile plots are given for all three paths in Figure 6, which show varying degrees of success in attaining an exponential distribution with the same transformation T, with path 0-0 the worst. The true mean available bandwidths over the 25 second simulation are given in bytes/s on links 0-2, 2-6, 2-7, 6-9, 6-0 in the table below. These are computed as a long time average by dividing total traffic over 25 seconds by 25, and then subtracting from the known physical capacity. The estimation procedure of Section 3 on n = 500 observations taken over intervals of 0.0 seconds yielded estimates ˆα = and for the means at (3.) we have Expected Available Bandwidth µ and Estimates ˆµ (bytes/s) link: µ 53, , , , , ˆµ 2, , , , , lower 8, upper 56, , , , , capacity 25, , , , , These results show a small mixture probability α.02 indicating that the network spends little time in the state of common available bandwidth (C = ). The upper bounds are reasonable for all links, except the second upper bound estimate 29, is slightly below the true value 29, on this particular run. For link 0-2, the two-sided confidence interval contains the true value. The upper bound on link 6-0 is greater than the capacity, which may happen because the fitting method does not use unknown capacities, the model does not bound the available bandwidth, and the inequality at (3.) may become significant. 3

15 Table : Traffic Configuration on Figure Source Destination Type Configuration 5 9 UDP Pareto burst time 25 ms idle time 25 ms rate 200k bits/s shape parameter UDP Pareto burst time 0 ms idle time 0 ms rate 200k bits/s shape parameter UDP Pareto burst time 5 ms idle time 5 ms rate 300k bits/s shape parameter ftp TCP window size 5 packet size 200 bytes 9 7 UDP Pareto burst time 0 ms idle time 0 ms rate 200k bits/s shape parameter 2.0 In the fitting procedure, the initial point for the nonlinear minimization of (3.4) seems to make a difference. Some choices lead to very wrong estimates. This is also an issue in the computation of bootstrap confidence intervals, where one must refit n times (n = 500 in this example). Our procedure used the estimate from the original fitting as the starting point for all of the bootstrap fittings. Prior knowledge about reasonable parameter values is useful for initializing the fitting procedure. The above simulation is one where the method worked relatively well compared to some examples. A situation where the method may not work well is when the intervals of available bandwidth on different edges do not overlap or do not show statistical variability, eliminating any kind of nontrivial correlation structure. If the intervals do not overlap, then the minimum 4

16 Frequency 0 Histograms of Available Bandwidth on Specific Links 0e+00 2e+04 4e+04 6e+04 8e+04 e+05 Link 0 2 Frequency 0 0e+00 2e+04 4e+04 6e+04 8e+04 e+05 Link 2 6 Frequency 0 0e+00 2e+04 4e+04 6e+04 8e+04 e+05 Link 2 7 Frequency 0 0e+00 2e+04 4e+04 6e+04 8e+04 e+05 Link 6 9 Frequency 0 0e+00 2e+04 4e+04 6e+04 8e+04 e+05 Link 6 0 Figure 4: Histograms of link available bandwidth 5

17 Histograms of Original and Transformed Available Bandwidth Frequency Path 0 9 Available Bandwidth Frequency Path 0 9 Transformed Available Bandwidth (p=3, k=0000) Figure 5: Effect of the transformation T on one path 6

18 Weibull Probability Plot of Link 0 9 Available Bandwidth Probability Bytes/s Weibull Probability Plot of Link 0 0 Available Bandwidth Probability Bytes/s Weibull Probability Plot of Link 0 7 Available Bandwidth Probability Bytes/s Figure 6: Weibull quantile-quantile plots of original data 7

19 at a receiver over the path links will always be picking up the available bandwidth on a single link, and the correlation structure is lost. Also, if the traffic is highly deterministic, as in constant bit rate UDP traffic without much background randomness, then the variance structure is not there to help with the parameter fitting. A possible way to deal with these situations is by adding a known amount of traffic with desirable statistical properties on lightly used links, thereby reducing available bandwidth on these links and creating overlap and dispersion. We have not tried this yet. A final bad situation for the model is when the transformation T (y) = (y/κ) p cannot be chosen so that a single exponent p produces data that is approximately exponentially distributed for all receiver nodes it may be that the exponents are too different across receivers and cannot yield a single transformation T which makes the model work. 5 Conclusions We have developed a model for available bandwidth that handles many essential complexities of network traffic and we have presented a method for internal estimation using end-to-end observations. The network complexities include spatial correlation, mixture components, and hidden data. The model combines several elementary parametric distributions. Parameters are found using an L method on moments, and confidence intervals are computed with the bootstrap percentile method. While far from perfect, the model has a theoretical foundation and it seems to work reasonably well for upper bounds on nontrivial simulated traffic. We have found that still simpler models that ignore spatial dependence do not fit the data well and the resulting parameter estimates are not accurate. On the other hand more complex models seem to be difficult or impossible to use because of problems with parameter identifiability. There are interesting algebraic geometry problems connected with complex models, since the identifiability is related to counting solutions of rational or polynomial systems. Also, there may be untapped connections with tropical geometry, a theory of polynomials with operations of min and + instead of + and. 6 Acknowledgments We thank Nick Hengartner and Stephan Eidenbenz for discussions on the topic of this paper. We have used the software ns2 and R for simulation and 8

20 analysis. A referee has pointed out that end-to-end available bandwidths are lower bounds on internal link-specific available bandwidths, which would complement the upper bounds in this paper. A method of getting the best lower bound by searching over paths that use a specific link may be a topic for future research. References [] S. Alouf, P. Nain, and D. Towsley, Inferring network characteristics via moment-based estimators, Proceedings of IEEE INFOCOM 0, Anchorage AL, Apr 200, (200). [2] R. Carter, and M. Crovella, Measuring bottleneck link speed in packetswitched networks, Performance Evaluation, 27 (996) [3] G. Casella, and R. L. Berger, Statistical Inference, 2nd Edition. Duxbury, Belmont CA (200). [4] M. Coates, A. O. Hero, R. Nowak, and B. Yu, Internet tomography. IEEE Signal Processing Magazine, 9 (2002) [5] I. H. Dinwoodie, and E. Vance, Network delay tomography with spatial dependence. Performance Evaluation, 64 (2007) [6] B. Efron, The Jackknife, the Bootstrap, and Other Resampling Plans, CBMS 38, SIAM, Philadelphia (982). [7] A. Erramilli, O. Narayan, and W. Willinger, Experimental queueing analysis with long-range dependent packet traffic. IEEE/ACM Transactions on Networking, 4 (996) [8] N. Hu, and P. Steenkiste, Evaluation and characterization of available bandwidth probing techniques. IEEE Journal on Selected Areas in Communications, 2 (2003) [9] M. Jain, and C. Dovrolis, End-to-end available bandwidth: measurement methodology, dynamics, and relation with TCP. Proceedings of the 2002 ACM/SIGCOMM conference on applications, technologies, architectures, and protocols for computer communications, Pittsburgh PA, (2002) [0] M. Jain, C. Dovrolis, and R. Prasad, pathload.2 (2004). 9

21 [] E. Kalnay, Atmospheric Modeling, Data Assimilation, and Predictability. Cambridge University Press, New York (2003). [2] ns2 The Network Simulator. Version 2.28, nsnam/ns/ (2005). [3] A. Odlyzko, Data networks are lightly utilized, and will stay that way, Review of Network Economics, 2 (2003) [4] L. L. Peterson, and B. S. Davie, Computer Networks. Morgan Kaufmann, New York (2003). [5] R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN , URL (2004). [6] S. Ross, Introduction to Probability Models 7th Edition. Academic Press, New York (2000). [7] A. Shriram, and J. Kaur, Identifying bottleneck links using distributed end-to-end available bandwidth measurements, First ISMA Bandwidth Estimation Workshop (BEst 03), San Diego CA, December 2003 (2003). [8] A. Shriram and J. Kaur, Empirical study of the impact of sampling timescales and strategies on measurement of available bandwidth, Proceedings of the Seventh Passive and Active Measurements Conference, Adelaide, Australia, Mar 2006, (2006). [9] A. Shriram and J. Kaur, Empirical evaluation of techniques for measuring available bandwidth, Proceedings of IEEE INFOCOM, Anchorage AK, May 2007 (2007). [20] A. Soule, A. Nucci, R. Cruz, E. Leonardi, N. Taft, How to identify and estimate the largest traffic matrix elements in a dynamic environment, Proceedings of the joint international conference on measurement and modeling of computer systems, New York NY, June 2004 ACM Press, New York (2004) [2] Y. Vardi, Network tomography: estimating source-destination traffic intensities from link data, JASA, 9 (996)