Robust Estimation of the Self-similarity Parameter in Network Traffic

Transcription

1 Robust Estimation of the Self-similarity Parameter in Network Traffic Haipeng Shen Department of Statistics and Operations Research University of North Carolina at Chapel Hill * joint work with Thomas Lee (UC Davis), Zhengyuan Zhu (Iowa State) June 24, 2010 Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 1 / 48

2 Outline 1 Motivation 2 Parameter Estimation Wavelet Estimators Level Shifts Removal Robust Regression 3 Simulation Studies 4 Analysis of Real Traces Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 2 / 48

3 Outline 1 Motivation 2 Parameter Estimation Wavelet Estimators Level Shifts Removal Robust Regression 3 Simulation Studies 4 Analysis of Real Traces Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 3 / 48

4 Example: Abilene Trace (a) Abilene: original trace packet count X(t) sampling time t (100ms interval) 2-hour trace, 100ms sampling unit eastbound traffic on the Abilene Backbone Network between KC and Indianapolis Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 3 / 48

5 Long-range Dependence X(t): a second-order stationary stochastic process X(t) is long-range dependent (LRD) with parameter α, if its autocorrelation function satisfies γ X (k) c γ k (1 α) as k, for α (0, 1) k γ X (k) = ; γ X (k) goes to zero very slowly Alternative characterization: its spectrum function f X (ν) c f ν α as ν 0, for α (0, 1) (1) Example: Var(X n ) = 2cγnα (1+α)α Doukhan, Oppenheim and Taqqu (2003) 1 n Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 4 / 48

6 Self-similarity Y (t) is self-similar (SS) if and only if c H Y (ct) d = Y (t) for all c > 0 d: equality in finite-dimensional distributions H: self-similarity parameter, or Hurst parameter Close connection between LRD and SS If Y (t) has finite variance, and 1/2 < H < 1, then its increments are LRD H = (α + 1)/2, or α = 2H 1 Example: fractional Brownian motion (fbm): self-similar fractional Gaussian noise (fgn): LRD Estimation of H or α (and c γ ) Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 5 / 48

7 Example: Abilene Trace (Ĥ = 119) (a) Abilene: original trace packet count X(t) sampling time t (100ms interval) Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 6 / 48

8 Example: UNC Trace I (Ĥ = 128) (a) UNC02 APR 09: original trace packet count X(t) sampling time t (100ms interval) Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 7 / 48

9 Example: UNC Trace II (Ĥ = 151) packet count X(t) (a) UNC02 APR 13: original trace sampling time t (100ms interval) Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 8 / 48

10 Example: UNC Trace III (Ĥ = 092) (a) UNC02 APR 11: missing values packet count X(t) sampling time t (100ms interval) Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 9 / 48

11 Example: UNC Trace III, zoomed-in view (a) UNC02 APR 11: missing values packet count X(t) sampling time t (100ms interval) Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 10 / 48

12 Practical Challenges for Parameter Estimation Non-stationarity gradual diurnal trends, (maybe polynomial) abrupt mean level shifts (of various magnitudes) sudden drop of traffic level, missing values confounding between non-stationarity and LRD Extreme values, both large and small Our goals: robust estimation some characterization of the non-stationarity Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 11 / 48

13 Haipeng Shen (UNC-CH) Figure 1: One Problematic Statistics Real of Networks Trace with Level Shifts Issac Newton Institute 12 / 48 Example: Abilene Trace, combined (a) Abilene: original trace (b) Abilene: Logscale Diagram packet count X(t) sampling time t (100ms interval) y_j LD RLD Octave j (c) Abilene: level shifts (d) Abilene: level shifts removed level shift alpha(t) sampling time t (100ms interval) detrended trace beta(t) sampling time t (100ms interval)

14 Outline 1 Motivation 2 Parameter Estimation Wavelet Estimators Level Shifts Removal Robust Regression 3 Simulation Studies 4 Analysis of Real Traces Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 13 / 48

15 Outline 1 Motivation 2 Parameter Estimation Wavelet Estimators Level Shifts Removal Robust Regression 3 Simulation Studies 4 Analysis of Real Traces Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 13 / 48

16 Existing Estimators for H or α Earlier estimators: the aggregated variance estimator the periodogram based estimator the Whittle estimator see Taqqu, Teverovsky and Willinger (1995) Our focus: wavelet-based estimators Abry and Veitch (1998), Veitch and Abry (1999) Soltani, Simard and Boichu (2004) Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 13 / 48

17 The Abry-Veitch (AV) Wavelet Estimator Consider a family of wavelet basis functions: {ψ j,k (t) = 2 j/2 ψ 0 (2 j t k), j = 1,, J, k Z} X(t): second-order stationary, with LRD parameter α Discrete wavelet transform coefficients: d X (j, k) = X(t), ψ j,k Veitch and Abry (1999): d X (j, ) s are iid Gaussian for a fixed j dx (j, ) and d X (j, ) are independent when j j for large j, Ed 2 X (j, ) = 2 jα c f C, (2) where C = C(α, ψ 0 ) Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 14 / 48

18 The AV Estimator Equation (2) suggests that log 2 ( Ed 2 X (j, ) ) = jα + log 2 (c f C) To estimate EdX 2 (j, ), consider µ j = 1 n j dx 2 (j, k), n j k=1 where n j : number of coefficients at scale j Hence, log 2 µ j d = jα + log2 (c f C) log 2 (n j ) + ln X nj / ln 2, where X nj is Chi-squared with n j degrees of freedom E ( log 2 µ j ) = jα + log2 (c f C) + g j, var ( log 2 µ j ) = ζ(2, nj /2)/ ln 2 2 Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 15 / 48

19 The AV Estimator To correct for bias, denote y j log 2 µ j g j Consider linear regression y j = jα + log 2 (c f C) + ɛ j (3) ɛ j has mean 0 and variance ζ(2, n j /2)/ ln 2 2 The AV estimator of α: the weighted-least-squares (WLS) estimate from Model (3) Then, Ĥ = (ˆα + 1) /2 Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 16 / 48

20 50 0 The AV Estimator: Logscale Diagram sampling time t Logscale Diagram (LD): a plot of y j against j (c) H= sampling time t (d) H=09 X(t) Synthetic fractional Gaussian trace: sampling time t y_j True LD LD RLD Octave j Figure 3: Sample fgn with level shifts For LRD processes, the upper part of the LD forms a straight line of slope α; a diagnostic tool for the existence of LRD For the trace shown in Figure 3(c), Figure 3(d) plots the Logscale Diagrams of the original fgn (True LD) and the level-shifts-added fgn (LD) along with the robust LD Selection of the onset of scaling: Veitch, Abry and Taqqu (2003), Park and Park (2009) (RLD) Approximate Gaussian confidence intervals are also provided along the True LD using the variance of ɛ j (Section 32) As one can see, the upper part of the RLD looks Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 17 / 48

21 The SSB Estimator Soltani et al (2004) consider ( ) D j,k = dx 2 (j, k) + d X 2 (j, k + n j/2) /2 Then, D j,k log 2 D j,k d = jα + log 2 (c f C) 1 + ln X 2 / ln 2, (4) where X 2 : Chi-squared with two degrees of freedom It follows that D j,k has a negative Gumbel distribution mean: jα + log2 (c f C) δ/ ln 2 variance: π 2 /(6 ln 2 2) Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 18 / 48

22 The SSB Estimator Define j 1 n j /2 n j D j,k k=1 Then, j d = jα + log2 (c f C) δ/ ln 2 + ɛ j (5) Cental Limit Theorem suggests that ( ( )) ɛ j N 0, π 2 / 3n j ln 2 2 Finally, to estimate α, perform WLS regression of j on j Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 19 / 48

23 Comparison of the Two Estimators AV mean log SSB log mean SSB performs slightly better than AV more immune to heavy-tailed fluctuations Stoev, Pipiras and Taqqu (2002) Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 20 / 48

24 Advantages of Wavelet Estimators naturally incorporate the scaling unbiased, asymptotically efficient fast computation due to DWT (O(n)) robust to polynomial trends, depending on the number of vanishing moments of the wavelet basis robust to (moderate) level shifts in mean and variance (Roughan and Veitch, 1999) Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 21 / 48

25 LRD under Non-stationarity Roughan and Veitch (1999) considered a class of non-stationary LRD models X(t; m, σ, H, c γ ) = m(t) + σ(t)w (t; H, c γ ) where W (t; H, c γ ): mean zero, unit variance LRD They specifically looked at models of mean level shifts X(t) = T (t; J, S, n/2) + W (t; H, c γ ) where T (t; J, S, L) = 1 + J 2 + J ( ) t L π arctan S L: location of the shift J: size of the shift S: smoothness of the shift Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 22 / 48

26 specifically, we begin with a stationary LRD model, and define a class of non-stationary variations by transforming it to induce a Achange Class in the meanof and/or Mean variance, whilst Level the parameters Shifts mea- Models suring the LRD, including H, remain well defined and constant 1 Smoothness=300 In this way some time-varying properties are allowed, and are well defined, but important features of the original stationary 05 Smoothness=0 model remain, and remain well defined also A class of non-stationary LRD models for the traffic rate X(t) is again given by transformation of the mean zero, unit variance LRD3 W (t H c r ), resulting in time from Roughan and Veitch (1999) X(t m H c r )=m(t) +(t) W (t H c r ) (6) Fig 1 The transition functions, with jump size J =1:0 25 where m(t) and (t) are positive functions of time Comparing with (5), we see that the location and scale parameters have f Non stationary FGN, model 1: H = 080, c = 028, sd = become 2 time varying, but the shape function ; W, and its associated parameters (H c jump size = 40 Smoothness=40 r ), do not change In fact m X (t) =m(t), 9 8 smoothness = 1200 and 15 X 2 (t) =2 (t) and 7 Smoothness= R X (t s) = (t)(s); W (t ; s H c r ) ; X (t s) = ; W (t ; s H c r ) Smoothness=300 ; W (k H c r ): (7) Thus, although the autocovariance function is no longer a function 05of the lag Smoothness=0 only, the autocorrelation function retains this t property despite the non-stationarities in location and scale Since we have used a definition of LRD based on such an autocorrelation Fig 2 Non-stationary FGN (parameters shown on each subplot) The white 0 function, it remains well defined, and gives a precise lines show the mean, while the dashed lines show one standard deviation meaning to the notion of non-stationary time LRD models, where the about the mean The left (right) figure shows NS FGN s constructed according to Model I (resp II) LRD parameters (H c r ) retain their physical meanings, and remain constant Fig 1 TheThus, transition in this functions, framework withthe jump estimation size J =1:0 of (H c r ) members of the family are illustrated with smoothness values has the meaning of measuring the stationary part of the non- S = f g, each with J =1and L = 8192 The Non stationary traffic FGN, model model In this 1: H paper = 080, wecconcentrate f = 028, sd = on10 the same smoothness values are used in simulations although, due robust estimation of H Although the estimation of c 12 r in the to space limitations, typically only results for S = f0 300g will normal 11 stationary context is well understood ([21], [20]), the estimation 9 10 jump size = 40 be shown The case S =0corresponds to the limit of the above 8 smoothness of c r in the non-stationary = 1200 context is more difficult and function as S! 0 from above, namely a step function The will7be studied elsewhere 6 smoothness parameter has the dimensions of time and gives a For 5 the remainder of the paper we will consider a particular 4 measure of the duration of the transition region A dimensionless Networksmeasure of the rapidity Issac of change Newtonacross Institutethe region, 23 / form3 of m(t) and (t), namely that of a level shift a monotone 2 Haipeng Shen (UNC-CH) Statistics of 48 process mean X t process me X t 15 Smoothness=40 Smoothness=1200

27 Robustness against Mean (and Variance) Level Shifts Stationary FGN Estimates NS FGN: Mean Shift Variance Shift TABLE I from Roughan and Veitch (1999) the exact size/location of shifts: helpful for deciding LRD region multiple shifts Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 24 / 48

28 Outline 1 Motivation 2 Parameter Estimation Wavelet Estimators Level Shifts Removal Robust Regression 3 Simulation Studies 4 Analysis of Real Traces Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 25 / 48

29 Our Model Suppose observing X(t) at a set of n discrete time points The model: X(t) = α(t) + β(t), α(t) = i µ i1 [ti,t i+1 )(t), t 0 = 1 < t 1 < < t m 1 < t m = n α(t): mean level shifts T = {t1,, t m 1 }: the collection of the shift locations β(t): stationary LRD with a Hurst parameter H Connection with the non-stationary LRD model (Roughan and Veitch, 1999), and the alpha-beta model (Sarvotham, Riedi and Baraniuk, 2001) The issue of model selection: select model for α(t): m and T estimation of H Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 25 / 48

30 Defining A Best Fitting Model Need to define the best combination of m and T Bayesian Information Criterion (BIC) (Schwartz, 1978): BIC(m, T H 0 ) = 2{l(m, T H 0 )} + (2m 1) ln n l(m, T H0 ): the conditional log likelihood for the fitted model 2m 1: number of parameters in the fitted model trade-off between maximum likelihood and model size Suppose β(t) is fgn with a Hurst parameter H 0 Then, l(m, T H 0 ) = C(n) n 2 ln t,s {X(t) ˆα(t)}{X(s) ˆα(s)}W ts, where W ts is the (t, s)-th element of the inverse correlation matrix The best model has (m, T ) that minimizes BIC Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 26 / 48

31 Defining A Best Fitting Model Once (m, T ) is decided, ˆµ i = 1 t i+1 t i t i+1 1 t=t i X(t), ˆα(t) = i ˆµ i 1 [ˆt i,ˆt i+1 ) (t) Residual Sum of Squares: RSS m = t {X(t) ˆα(t)}2 Note that the BIC depends on an initial value H 0 To solve this (cyclic) problem consider a set of initial candidate values for H0 derive the best BIC under each candidate select H0 as the one that gives the smallest BIC value determine the final level shifts under the corresponding (m, T ) Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 27 / 48

32 Locating the Best Model Non-trivial to minimize BIC n is typically huge, m varies amongst the models Consider a greedy merging algorithm begin with an over-fitting model for example, mean shifts at every other time point at each step, merge two adjacent segments to form a single bigger segment, which increases RSS m the least continue the merging until there is only one segment left, corresponding to no mean shift Choose the smallest BIC from the above nested sequence of models Efficient updating of RSS m exists Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 28 / 48

33 Outline 1 Motivation 2 Parameter Estimation Wavelet Estimators Level Shifts Removal Robust Regression 3 Simulation Studies 4 Analysis of Real Traces Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 29 / 48

34 Robust Regression Consider linear model y j = x t j β + ɛ j Least-squares (LS) regression estimates β by ˆβ LS = argmin j (y j x t j β)2 Both AV and SSB estimators use LS However, when the errors have a heavier tail than Gaussian, LS procedures can be very inefficient and unstable Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 29 / 48

35 Robust Regression: the SSB estimator The SSB estimator is based on D j,k Quantiles of Standard Normal Empirical Quantiles Figure 2: Gaussian Quantile Plot of the D j,k s at Octave the sum of the first h ordered squared residuals respectively LTS can 50%, but it is numerically more difficult to obtain As described in Section 2, the AV estimator explores the log linear d 2 X (j, k) (ie µ j) and j, and fits a WLS of log 2 (µ j ) (ie y j ) against j, wh j, the average of D j,k, against j Our simulation results (Section 4) sho The response j 1 n j /2 n j k=1 D j,k only j at large octaves are useful for estimating H however, not many wavelet coefficients at those scales the effect of CLT may not be significant We propose to use robust regression techniques Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 30 / 48

36 Robust Regression L 1 regression (Edgeworth, 1887): M-estimation (Huber, 1981): ˆβ L1 = argmin y j x t j β ˆβ M = argmin ρ(y j x t j β) ρ( ) down-weights extreme observations Iteratively Re-weighted Least Squares (IRLS) algorithm (Heiberger and Becker, 1992) Least trimmed squares estimation (Rousseeuw, 1984): h ˆβ LTS = argmin r(j) 2, where r (j) is the jth order statistic of the residual r j = y j x t j β Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 31 / 48 j=1

37 Robust Wavelet Estimation of H 1 For each candidate of H 0, say H k, 1 Use the level shift removing method to determine the number and locations of mean level shifts (m k, T k ) in X(t) 2 Choose the combination (m k, T k, H k ) that gives the smallest BIC 2 Remove the estimated level shifts ˆα(t) from X(t), and obtain the estimated β(t) as ˆβ(t) = X(t) ˆα(t) 3 Apply robust regression to the wavelet coefficients of ˆβ(t) to obtain a final robust estimate Ĥ Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 32 / 48

38 Robust Logscale Diagram A robust version of the Logscale Diagram Instead of y j or j, find a robust center" of D j,k, the logged wavelet coefficients ling time t H=09 X(t) depend on the robust regression technique sampling time t Fit the robust regression to the upper part of the diagram (d) H=09 y_j True LD LD RLD ling time t Octave j Figure 3: Sample fgn with level shifts Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 33 / 48

39 Outline 1 Motivation 2 Parameter Estimation Wavelet Estimators Level Shifts Removal Robust Regression 3 Simulation Studies 4 Analysis of Real Traces Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 34 / 48

40 Simulation Setup Model: X t = µ t + ɛ t ɛ t : fractional Gaussian noise (fgn) with variance σ 2 and autocorrelation function γ(k) = ( k + 1 2H + k 1 2H 2 k 2H )/2, k 0 (6) H = 1/2: ɛt white noise 1/2 < H < 1: LRD Simulation I: fgn Simulation II: fgn with missing values Simulation III: fgn with mean level shifts Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 34 / 48

41 Simulation I: fgn Simulate 50 fgn using the circulant embedding method of Dietrich and Newsam (1997) n = 2 14, µ = σ = 20 H = {05, 075, 09} Compare bias, standard error (SE), root mean squared error (RMSE) Consider SSB, L 1, IRLS, LTS Level-shifts removal (correctly) found no or very few level shifts Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 35 / 48

42 Fig 3 Sample fgn Simulation I: fgn H=05 X(t) X(t) t H= t X(t) se RMSE H = 09 bias se RMSE H=09 in Table I From Table I we can see that all yield very accurate estimates of H when are exact fgn IRLS gives marginally be than LTS and L 1 regression It is also b regression when H is large In sequel we IRLS in the simulation studies for robust IRLS is computational quicker than LTS an all three robust procedures are expected to performances 0 We also 5000 applied the 10000level shifts removi t the simulated traces For H = 05 and 075 correctly found no level shifts for all 50 sim The bias, Forstandard H = 09, error the(se) method and root foundmean no leve squ errors (RMSE) traces, of one the 50 level estimated shift for H s16 under traces, eachtwo co bination offor Hfive andtraces, the estimation and threemethod level shifts are presen for o all cases, the estimated H for the level sh traces are almost identical to those of the o Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 36 / 48 X(t) t

43 e 1: Comparison of the SSB estimator and three robust regression estima Comparison of SSB, IRLS, LTS, L 1 estimators for fgn SSB IRLS LTS L 1 H = 05 bias se RMSE H = 075 bias se RMSE H = 09 bias se RMSE true Haipeng valueshen of(unc-ch) H, Ĥ i is the estimated Statistics of Networks H for the ith trace, Issac Newton and Institute H is37 the / 48

44 0 Simulation II: fgn with missing values 200 time points set to 0, (1% missing) ure 4 plots two typical fgn traces with H = 05 at the beginning missing values added Vertical dashed lines are erimposed to in highlight the middle the location of the missing es X(t) Fig 4 Sample fgn with missing values H= t X(t) X(t) TABLE II COMPARISON OF THE LS, IRLS AND RW EST WITH MISSING VALUES Missing at the beginning LS IRLS t H = 05 bias H=05 se RMSE H = 075 bias se RMSE H = 09 bias se RMSE Missing in the middle LS IRLS H=05 Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 38 / t

45 Comparison of SSB, IRLS and RW estimators for fgn with missing values : Comparison of the SSB, IRLS and RW estimators for fgn with missing Missing at the beginning SSB IRLS RW H = 05 bias SE RMSE H = 075 bias SE RMSE H = 09 bias SE RMSE Missing in the middle SSB IRLS RW H = 05 bias SE RMSE H = 075 bias SE RMSE H = 09 bias SE RMSE mator employs the mean level shifts removing algorithm first before appl Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 39 / 48

46 Simulation III: Sample fgn with level shifts Take the fgn traces in Simulation I Add level shifts: for H = 05, 075, 1 unit of the grand mean for H = 09, 4 unit of the grand mean Selection of number of shifts: correct %: 98% (H = 05), 84% (H = 075), 68% (H = 09) For larger H, harder to distinguish level shifts from natural variation caused by strong auto-correlation Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 40 / 48

47 according to two-sample t-tests Simulation III: Sample fgn with level shifts (a) H=05 (b) H=075 X(t) X(t) sampling time t (c) H= sampling time t (d) H=09 X(t) y_j True LD LD RLD sampling time t Octave j Figure 3: Sample fgn with level shifts Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 41 / 48

48 1%Þ A timators ller bias, gnificant e occurs dramatic Comparison of SSB, IRLS and RW estimators for fgn with leveltable shifts 2 used by undaries wavelet omment y of the der the or level ss and a ao lower t all the e CRLB, iple, one 16 level shifts of magnitude one unit of the grand Comparison of the SSB, IRLS, and RW estimators for fgn with level shifts SSB IRLS RW H ¼ 0:5 bias SE RMSE H ¼ 0:75 bias SE RMSE H ¼ 0:9 bias SE RMSE Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 42 / 48

49 Outline 1 Motivation 2 Parameter Estimation Wavelet Estimators Level Shifts Removal Robust Regression 3 Simulation Studies 4 Analysis of Real Traces Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 43 / 48

50 Comparison of the AV and RW estimators for the four real traces AV RW Trace Octave Ĥ m Ĥ Abilene UNC I UNC II UNC III Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 43 / 48

51 Example: UNC Trace I (Ĥ = 128) t (a) UNC02 APR 09: original trace packet count X(t) sampling time t (100ms interval) y_j Octave j Table 4 reports the estimated Hur Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 44 / 48

52 reports the estimated Hurst parameters of the four t thods Haipeng (A/V Shen (UNC-CH) and RW) Statistics Theof Networks automatic octave Issac Newton Institute selection 45 / 48 Example: UNC Trace II (Ĥ = 151) t (a) UNC02 APR 13: original trace packet count X(t) t y_j y_j sampling time t (100ms interval) Octave j Octave j

53 Example: UNC Trace III (Ĥ = 092) t (a) UNC02 APR 11: missing values packet count X(t) t y_j y_j sampling 5 6time 7t (100ms 8 interval) Octave j Octave j ose to use a robust procedure to estimate the Hurst all Haipeng robust Shen (UNC-CH) wavelet (RW) Statistics ofestimator Networks We Issac Newton consider Institute 46 / 48t

54 Subtrace consistency Trace Estimator S1 S2 S3 S4 RMSE Abilene AV RW UNC I AV RW UNC II AV RW UNC III AV RW Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 47 / 48

55 Summary Take home messages proposed a robust wavelet estimator for H or α level-shift removal + robust regression illustrated its performance via simulated and real traces Future work model magnitude/duration of level-shifts, connection with bursty periods different model selection criterion incorporate changing variance Haipeng Shen (UNC-CH) Statistics of Networks Issac Newton Institute 48 / 48