Accurate Recovery of Internet Traffic Data Under Dynamic Measurements

Transcription

1 Accurate ecovery of nternet Traffic Data Under Dynamic Measurements un Xie,, Can Pen, Xin Wan, Gaoan Xie 3, ian Wen 3 Collee of Computer Science and Electronics Enineerin, Hunan University, Chansha, China Department of Electrical and Computer Enineerin, State University of New Yor at Stony Broo, USA 3 nstitute of Computin Technoloy, Chinese Academy of Sciences, Beijin, China xieun@hnu.edu.cn, pencancaroline@mail.com, x.wan@stonybroo.edu, xie@ict.ac.cn, wenjian@ict.ac.cn Abstract The inference of the networ traffic matrix from partial measurement data becomes increasinly critical for various networ enineerin tass, such as capacity plannin, load balancin, path setup, networ provisionin, anomaly detection, and failure recovery. The recent study shows it is promisin to more accurately interpolate the missin data with a threedimensional tensor as compared to interpolation methods based on two-dimensional matrix. Despite the potential, it is difficult to form a tensor with measurements taen at varyin rate in a practical networ. To address the issues, we propose eshape- Alin scheme to form the reular tensor with data from dynamic measurements, and introduce user-domain and temporal-domain factor matrices which taes full advantae of features from both domains to translate the matrix completion problem to the tensor completion problem based on CP decomposition for more accurate missin data recovery. Our performance results demonstrate that our eshape-alin scheme can achieve sinificantly better performance in terms of two metrics: error ratio and mean absolute error MAE. ndex Terms nternet traffic data recovery, Matrix completion, Tensor completion. NTODUCTON A traffic matrix TM is often applied to trac the volume of traffic between oriin-destination OD pairs in a networ. Estimatin the end-to-end TM in a networ is an essential part of many networ desin and traffic enineerin tass, includin capacity plannin, load balancin, path setup, networ provisionin, anomaly detection, and failure recovery. Due to the lac of measurement infrastructure, direct and precise end-to-end flow traffic measurement is extremely d- ifficult in the traditional P networ []. Thus previous wor on TM estimation focus on inferrin the TM indirectly from lin loads [], [3], and the methods taen are often sensitive to the statistical assumptions made for models and the TMs estimated are subject to lare errors [4]. As an alternative, TM is directly built throuh the collection of the end-to-end flow-level traffic information usin flow monitorin tools such as Cisco NetFlow, and the recent OpenFlow [5]. Unlie commodity switches in traditional P networs, flow-level operations are streamlined into OpenFlow The wor is supported by the National Natural Science Foundation of China under Grant Nos.65784, 6473, and 6473, and U.S. NSF CNS switches, which provides the possibility of queryin and obtainin the end-to-end flow traffic statistics. Despite the proress in flow-level measurements, the collection of the traffic information networ wide to form TM at fine time scale still faces many challenes: Due to the hih networ monitorin and communication cost, it is impractical to collect full traffic volume information from a very lare number of points. Sample-based traffic monitorin is often applied where measurements are only taen between some random node pairs or at some of the periods for a iven node pair. Measurement data may et lost due to severe communication and system conditions, includin networ conestion, node misbehavior, monitor failure, transmission of measurement information throuh an unreliable transport protocol. As many traffic enineerin tass such as anomaly detection, traffic prediction require the complete traffic volume information i.e., the complete traffic matrix or are hihly sensitive to the missin data, the accurate reconstruction of missin values from partial traffic measurements becomes a ey problem, and we refer this problem as the traffic data recovery problem. Various studies have been made to handle and recover the missin traffic data. Desined based on purely spatial [6] [8] or purely temporal [9], [] information, the data recovery performance of most nown approaches is low. ecently matrix-completion-based alorithms are proposed to recover the missin traffic data by exploitin both spatial and temporal information [] [7]. Althouh the performance is ood when the data missin ratio is low, the performance suffers when the missin ratio is lare. Based on the analyses of real traffic trace, recent wor in [8] reveals that the traffic data have the features of temporal stability, spatial correlation, and periodicity. Specially, the periodicity features indicate that users usually have similar nternet visitin behaviors at the same time of a day, so the measurements for an OD pair taen at the same time slots of two consecutive days are similar. The authors [8] thus model the traffic data as a 3-way tensor to concurrently consider the traffic of different days for more accurate missin data /7/$3. 7 EEE

2 interpolation. Tensors are the hiher-order eneralization of vectors and matrices. Tensor-based multilinear data analysis has shown that tensor models can tae full advantae of the multilinear structures to provide better data understandin and information precision. Tensor-based analytical tools have seen applications for web raphs [9], nowlede bases [], chemometrics [], sinal processin [], and computer vision [3], etc. Compared with matrix-based data recovery, the tensor-based approach can better handle the missin traffic data and will be used in this paper. Althouh promisin, the traffic tensor model in [8], [4] is built with a stron assumption that the networ monitorin system adopts a static measurement stratey with a fixed samplin rate. However, in a practical networ monitorin system, the rate of measurements is often adapted accordin to the traffic conditions i.e., varyin in different periods of a day and some traffic enineerin requirements i.e., to more timely detect anomaly. The dynamic measurements mae it hard to form a reular traffic tensor for further processin. Some challenes due to the variation of the measurement rate are: Difficult to alin the matrices of different days. The traffic matrices of different days would have different number of columns, which maes it hard to interate the traffic matrices of different days to form a standard tensor and recover the missin data. Difference in the lenth of the time slot. The sample data in a column of the traffic matrix may correspond to a time slot with a different lenth, which further brins the difficulty of recoverin the missin items throuh the temporal and spatial correlation amon traffic data. Despite the challenes, the traffic matrix has some special features: The traffic matrices of different days record the data of the same OD pairs in the networ, and The user traffic data follow a daily schedule. Therefore, there should exist some common user-domain and time-domain features that can be exploited for more accurate interpolation. n this paper, we propose a novel traffic data recovery scheme in the presence of variation of traffic measurement rate. Our scheme will first construct a reular tensor with the reshapin and alinment of traffic matrices with inconsistent number of columns and different lenth of time slots, and then enable more accurate traffic data recovery tain advantae of the data correlation in a three dimensional tensor. The contributions of this paper can be summarized as follows: We propose a matrix division alorithm for time alinment, which exploits our novel time rule to efficiently divide the traffic matrices into sub-matrices with each correspondin to one time sement with the same samplin rate. We reshape and alin traffic matrices from dynamic measurements to form a reular tensor, tain advantae of multi-dimensional data correlation for more accurate traffic data recovery. To address the challene of interatin matrices of different dimensions into a tensor, we introduce user-domain and temporal-domain factor matrices to translate the problem of matrix completion for different days to the problem of tensor completion based on CP decomposition. We compare the proposed eshape-alin scheme with the state of the art matrix-based alorithms, and our results demonstrate that our scheme can achieve sinificantly better performance in terms of two metrics: error ratio and mean absolute error MAE. To the best of our nowlede, our eshape-alin scheme is the first one that considers the traffic recovery problem under dynamic measurement in a practical networ system, and provides a novel reshapin and alinment technique that allows the interation of inconsistent traffic matrices to form a standard tensor for more accurate missin data recovery. The rest of the paper is oranized as follows. We introduce the related wor in Section. The preliminaries of tensor are presented in Section. We present the problem and our overview solution in Section V. The proposed alorithms on matrix division for time alinment, and matrix reshapin and alinment for tensor completion are presented in Section V and Section V, respectively. Finally, we evaluate the performance of the proposed alorithm throuh extensive simulations in Section V, and conclude the wor in Section X.. ELATED WO A set of studies have been made to handle the missin traffic data. Desined based on purely spatial [6] [8] or purely temporal [9], [] information, most of the nown approaches have a low data recovery performance. To capture more spatial-temporal features in the traffic data, SMF [] proposes the first spatio-temporal model of traffic matrices TMs. To recover the missin data, SMF is desined based on low-ran approximation combined with the spatio-temporal operation and local interpolation. Followin SMF, several other traffic matrix recovery alorithms [] [5], [7] are proposed to recover the missin data from partial traffic measurements. Compared with the vector-based recovery approaches [6] [], as a matrix could capture more information and correlation amon traffic data, matrix-based approaches achieve much better recovery performance. However, a two-dimension matrix is still limited in capturin a lare variety of correlation features hidden in the traffic data. For example, althouh the traffic matrix defined in [] can catch the spatial correlation amon flows and the smallscale temporal feature, it can not incorporate other temporal features such as the feature of the traffic periodicity cross day. Therefore, a matrix is still not enouh to capture the comprehensive correlations amon the traffic data, and the data recovery performance under the matrix-based approaches is still low. To further utilize the traffic periodicity feature for accurate traffic data recovery, the recent studies [8], [4] combine the traffic matrices of different days to form a tensor to recover the missin data. Several tensor completion alorithms [5] [8] are proposed for recoverin the missin data by capturin the lobal structure of the data via a hih-order decomposition such as CANDECOMP/PAAFAC CP decomposition [9], [3] and Tucer decomposition [3]. Tensor has proven to be

3 ood data structure for dealin with the multi-dimensional data in a variety of fields [9] [3]. Althouh promisin, the traffic tensor model in [8], [4] is built with a stron assumption that the networ monitorin system adopts a static measurement stratey with a fixed samplin rate. The proposed methods may fail to wor in a practical networ monitorin scenario where the rate of measurements varies over time. Moreover, several novel techniques are proposed in the scheme such as matrix division alorithm for time alinment, mechanism to reshape and alin matrices, and the technique to solve the matrix completion problem throuh tensor CP decomposition. To address this practical challene, we propose a novel eshape-alin scheme with several novel techniques, includin matrix division for time alinment, mechanism to reshape and alin matrices, and the technique to solve the matrix completion problem throuh tensor CP decomposition. The simulation results demonstrate that eshape-alin scheme can achieve sinificantly better performance in terms of two metrics: error ratio and mean absolute error MAE.. PELMNAES The notation used in this paper is described as follows. Scalars are denoted by lowercase letters a, b,, vectors are written in boldface lowercase a, b,, and matrices are represented with boldface capitals A, B,. Hiherorder tensors are written as calliraphic letters X, Y,. The elements of a tensor are denoted by the symbolic name of the tensor with indexes in subscript. For example, the ith entry of a vector a is denoted by a i, element i, j of a matrix A is denoted by a ij, and element i, j, of a third-order tensor X is denoted by x ij. Definition. A tensor is a multidimensional array, and is a hiher-order eneralization of a vector first-order tensor and a matrix second-order tensor. An N-way or Nth-order tensor denoted as A N is an element of the tensor product of N vector spaces, where N is the order of A, also called way or mode. The element of A is denoted by a i,i,,i N, i n {,,, n } with n N. Definition. Slices are two-dimensional sub-arrays, defined by fixin all indexes but two. n Fi., a 3-way tensor X has horizontal, lateral and frontal slices, which are denoted by X i::, X :j: and X ::, respectively. n this paper, we denote the frontal slice X :: as X. Fi.. Tensor slices Definition 3. The outer product of two vectors a b is the matrix defined by: a b ij = a i b j. Definition 4. The outer product A B of a tensor A N and a tensor B N is the tensor of the order N + N defined by A B i,i,,i N,i,i,,i N = a i,i,,i N b i,i,,i N for all values of the indexes. Since vectors are first-order tensors, the outer product of three vectors a b c is a tensor iven by: a b c ij = a i b j c for all values of the indexes. Definition 5. A 3-way tensor X is a ran one tensor if it can be written as the outer product of three vectors, i.e. X = a b c. Definition 6. The ran of a tensor is the minimal number of ran one tensors, that enerate the tensor as their sum, i.e. the smallest, such that X = r= a r b r c r. Definition 7. The idea of CANDECOMP/PAAFAC CP decomposition is to express a tensor as the sum of a finite number of ran one tensors. A 3-way tensor X can be expressed as X = with an entry calculated as x ij = r= a r b r c r, 3 r= a irb jr c r 4 where >, a ir, b jr, c r are the i-th, j-th, and -th entry of vectors a r, b r, and c r, respectively. By collectin the vectors in the ran one components, we have tensor X s factor matrices A = [a,, a ], B = [b,, b ], and C = [c,, c ]. Usin the factor matrices, we can rewrite the CP decomposition as follows. x ij x ij X = ab c r ir jr r r= a r b r c r = [[A, B, C]], 5 a i c b j a i c ran one tensors b j C A i-th row -th row j-th row Fi.. CP decomposition of three-way tensor as sum of outer products ran one tensors. CP decomposition can be written as a triplet of factor matrices A, B, C, i.e, the r-th column of which contains a r, b r, and c r, respectively. The entry x ij can be calculated as the sum of the product of the entries of the i-th row of the matrix A, the j-th row of the matrix B, and the -th row of the matrix C. Fi. illustrates the CP decomposition. n this paper, we desin traffic data recovery alorithm based on the CP decomposition. V. POBLEM DESCPTON AND SOLUTON OVEVEW n this section, we first formulate the traffic data recovery problem as a matrix factorization problem, and then present the benefit and methodoloy of transformin this problem further to the tensor factorization problem alon with the difficulty of this transformation in a practical networ monitorin system. 3

4 A. Traffic recovery problem based on matrix factorization For a networ consistin of N nodes, there are n = N N OD pairs. We define a monitorin data matrix, X n m, to hold the traffic data measured in the th day for =,,,. m is the total number of time slots captured in the th day. n the matrix, a row corresponds to an OD pair, a column corresponds to a time slot, and the ij-th entry x :ij represents the monitorin data of the OD pair i at the time slot j. To reduce the networ monitorin overhead, only a subset of measurements are taen. We apply the matrix factorization to infer the missin entries of the matrices correspondin to days. Specifically, to recover the missin data, a monitorin matrix X is factored into a production of an n r factor matrix U for the user domain, an r r diaonal matrix Σ, and a m r factor matrix V for time domain under the condition of minimizin the loss function as follows: min f U, Σ, V U,Σ,V s.t. f U, Σ, V = X U Σ V T F Ω = 6 where r denotes the matrix ran, f U, Σ, V = X U Σ V T F Ω is the loss function defined = based on the Frobenius norm F, Ω is the index set of the observed samples of the matrix X. After obtainin the factor matrices U, the diaonal matrix Σ, and the factor matrix V, the monitorin matrix can be recovered as follows: ˆX = U Σ V T 7 where ˆX denotes the recovered traffic matrix. B. From matrix factorization to tensor factorization As traffic data are observed to possess the features of temporal stability, spatial correlation, and periodicity features [8], [4], rather than only recoverin the data throuh the t- wo dimensional matrix, it is promisin to more accurately Fi. 3. Tensor based traffic model interpolate the missin data with a three-dimensional tensor tain advantae of the periodicity feature of traffic across days. Despite the potential, in a practical networ monitorin system, the measurement stratey may often vary accordin to the traffic conditions. There exist some challenes to combine multiple matrices to form a tensor: nconsistent number of columns across the matrices. As a column represents a sample in a time slot, the variation of measurement rate in different days would mae their traffic matrices to have different number of columns Fi.4a. This introduces the challene to formin the standard tensor with these matrices. nconsistent lenth of time slot within the matrix. Different samplin rate maes columns in a matrix to correspond to different time-slot lenths as shown in Fi.4b, which further brins the difficulty of recoverin the missin items throuh the temporal and spatial correlation amon traffic data Fi. 4. Traffic matrices with inconsistent number of columns C. Characteristics in multiple data matrices Althouh the variation of the samplin rate brins the challene of interatin the measurement matrices of different days to form a reular tensor, these matrices have some characteristics that can be exploited for more accurate data recovery. Traffic matrices of different days record the measurement data of the same OD pairs in the networ, and the row indexes of these matrices are the same. Thus these matrices should have some common OD-domain i.e., user-domain features, so in 8, we use the same factor matrix U for different traffic matrices. Althouh the number of columns and the time-slot lenths may be different for matrices of different days, the user traffic in these matrices vary followin a daily schedule in the temporal domain, as users usually have similar nternet access behaviors. The aims of this paper are to investiate and tae advantae of the common features in the traffic matrices to reshape and alin traffic matrices with inconsistent number of columns and time-slot lenths to form reular tensors, and desin efficient tensor-based traffic data recovery alorithms for more accurate data recovery. D. Solution overview To fully exploit the common features hidden in monitorin matrices for more accurate missin data interpolation, we propose a matrix reshapin and alinment scheme in the presence of varyin networ measurement frequency. Fi. 5a shows example traffic matrices to recover. The time slots in a matrix may have different lenths. To well exploit the common time-domain features hidden in the traffic data within a day, the matrices should be divided and alined in the physical time domain as explained in Section V. Accordinly, we propose a matrix division alorithm with the example shown in Fi.5 b, where the matrices are divided in temporal 4

5 domain to satisfy the time alinment requirement. The submatrices formed after the division in Fi.5 c will be further utilized to form tensors. To exploit correlations across days for more accurate data recovery, we further translate the factor matrices of each submatrix to common ones tain advantae of the user domain and temporal domain features hidden in the sub-matrices, and then interate the reshaped and alined sub-matrices to form the tensor in Fi. 5 d. We apply the tensor completion alorithm to interpolate the missin data in Fi. 5 e, and then tae the reverse procedure of reshapin to obtain the recovered sub-matrices in Fi. 5 f, which will be combined to form the final recovered lare matrices in Fi. 5. X X X X T T t T T T T T T T T T T X X X X Fi. 6. t t t Time alinment problem rate is adopted in X for the whole day. For matrices X, X 3, X 4, two different samplin rates are assumed for the first half day and the next half day, respectively. To alin the data in the time domain, we divide the whole day time into two time sements, each correspondin to one half day and adoptin the same measurement stratey. Accordinly, the oriinal traffic matrices are divided into two parts with X = [ X, X], X = [ X, X], X3 = [ X 3, X3], X4 = [ X 4, X4]. Similarly, in Fi.6b, the time in the day is divided into three time sements and thus oriinal traffic matrices are divided into three parts with X = [ X, X, X] 3, X = [ X, X, X] 3, X 3 = [ X 3, X 3, X3] 3, X4 = [ X 4, X 4, X4] 3. As shown in Fi.6, obviously, the divided sub-matrices X, X, X 3, and X 4 record the traffic data of the same time duration in different days, and can be combined for more accurate traffic data recovery. Our reshapin and alinment scheme exploits these temporal domain features hidden in the matrices to more accurately recover the missin data. X X X X Fi. 5. T T T T T T T Overview solution of eshape-alin scheme T V. MATX DVSON FO TME ALGNMENT T T T T We first present our matrix division alorithm, then reformulate the recovery problem for the sub-matrices by tain consideration of the common features of matrices in both the user OD domain and the time domain. A. Matrix division Althouh the difference in the traffic measurement rate may result in different time-slot lenths, we can still observe that the user traffic patterns often chane daily followin the user daily nternet access behaviors. To well exploit the time-domain features hidden in the traffic data, we divide daily measurements into multiple time sements each havin a different samplin rate. Fi. 6 utilizes two examples to illustrate the time alinment problem, where X, X, X 3, X 4 denote the measurement traffic matrices of four days. n Fi. 6a, a fixed measurement B. Problem reformulation with common user and temporal domain features After the time alinment, the oriinal matrices are divided into multiple sub-matrices. We denote sub-matrices that record the traffic data of the same time sement of different days as one sub-matrix roup. f the duration of a day is partitioned into G time sements, we have X = [X, X, XG ] for =,,,. Accordin to the time division and alinment requirement, all matrices are divided at the same timespot to cover the same time sement. Therefore, after the matrix division, there are G roups of sub-matrices with each roup havin sub-matrices, that is {X, X,, X } for =,,, G. Accordin to the partition, the problem in 6 can be transformed to the problem of minimizin the loss function on each sub-matrices. min f U, Σ U,Σ,V, V s.t. f U, Σ, V = G X U Σ V T = = Ω F 8 where X n m :, U n r, V m : r, Σ is r r diaonal matrix, Ω is the index set of the observed samples of matrix X. m : is the number of columns of X. r is the matrix ran of X. The problem above can be solved by recoverin each matrix independently. However, with the data correlation across days, 5

6 a better recovery can be made if the set of matrices can be interated into a tensor to recover toether. This is not possible with each matrix havin different number of columns. To address the issue, we first exploit the common data features hidden in the user domain and temporal domain to translate the problem. As different monitorin matrices record the traffic data of the same set of n OD pairs of different days, they should share some common features in the user domain. Tain advantae of these features for more accurate traffic recovery, we use the same factor matrix U for different sub-matrices of different days in Eq8. Beside the common feature in user domain, as we have discussed in Section V-C, traffic data also have common feature in the time domain, which is not captured in the problem 8. To reflect the feature, enlihtened by Harshman [3], we impose an invariance constraint on the factor matrices V in the time domain: the cross product V T V is constant over different days, that is, Φ = V T V for =,,,. Before we update the problem formulation in 8 to incorporate this invariance constraint, the followin theorem reformulates the constraint. V. MATX ESHAPNG AND ALGNMENT FO TENSO COMPLETON To solve the problem 9, we propose an alternatin least squares alorithm that alternately solves the followin two sub-problems: Sub-problem : minimize 9 over P for a iven set of U, Σ, V Sub-problem : minimize 9 over U, Σ, V for fixed P A. Sub problem The sub-problem can be written as follows. min f P P s.t. f P = G X U Σ P V T = = Ω F Let B = U Σ P V T, we have B=U Σ V T P T and B T = P V Σ U T. The loss function on each submatrix i.e. X can be written as X U Σ P V T = tr X B F X BT = tr X X B T B T = tr X X T tr X BT + tr BB T As tr X X T and trbb T = tru Σ V T V Σ U T do not depend on P, minimizin 9 is equal to solvin the followin problem: max tr X P BT s.t. P T P = B = U Σ P V T As tr X BT = tr tr V Σ U T X P further transformed to X P V Σ U T =, the problem in can be V Σ U T X P Theorem. For the invariance constraint V T V max tr to hold, P it is necessary and sufficient to have V = P V where V 3 s.t. P r r does not chane in different days and P T P = m : r is a column-wise orthonormal matrix with P T P =. Let V Σ U T X = M N T be sinular value decomposition SVD. Accordin to [33], we have that P Due to the limited space, the proof is omitted. = N To exploit the common feature in time domain, based on M T is the column wise orthonormal solution for the Theorem, replace V = problem 3. P V, the problem in 8 can be further transformed as follows: min f U, Σ U,Σ,V,P, V, P B. Sub-problem s.t. f U, Σ, V, P = G X U Σ The sub-problem of minimizin 9 over U, Σ P V T, V for fixed P = = Ω reduces to the followin problem. F 9 min f U, Σ That is, the difference of the matrix X U for different days,σ,v, V =,,, are captured by the matrix Σ and P. n s.t. f U, Σ, V G = X P U Σ V T = = Ω Section V-B, we will show that the problem formulation in 9 F 4 provides the possibility of translatin the matrix completion Let Y = X P, problem in 4 can be further written as problem to the tensor completion throuh CP decomposition. follows. min f U, Σ U,Σ,V, V s.t. f U, Σ, V G = Y U Σ V T = = Ω F 5 As X n m : and P m : r, obviously, Y n r. t is easy to see Eq.5 corresponds to G 3-way tensors as illustrated in Fi.7b, where each slice Y has the identical size of n r. Fi.7 shows that multiple sub-matrices can be reshaped and alined to the tensor-style. However, the problem in 5 is still a matrix completion problem. We would lie to solve the problem throuh the tensor completion tain advantae of correlation across days for more accurate data recovery. Before introducin our solution, we first investiate the relationship between tensor CP decomposition and frontal slice decomposition. As shown in Fi.8, iven a 3-way tensor X, matrices A, B, C 6

7 Y Fi. 7. Y Y X P Transform G roups of sub-matrices to G tensors. Y be transformed to the followin tensor completion problem: min f U,V,C U, V, C G s.t. f U, V, C = = Y [[U, V, C ]] Ω F 6 where Y is the tensor with its frontal slices bein submatrices Y for =,,,, U,V,C are the factor matrices of Y, Ω is the index set of the observed samples of tensor Y. As this paper dose not focus on CP decomposition, we utilize the solution in [34] to solve the tensor completion to obtain the optimal factor matrices U,V,C by minimizin the loss function in 6. After obtainin U,V,C, the reshaped slice can be recovered throuh followin calculation Ŷ = U Σ V T where Σ = dia C : and C : is the -th row of the factor matrix C. Then throuh the reverse procedure of reshapin, we can obtain the recovered sub traffic matrix ˆX = Y P T = U Σ V T P T. are the factor matrices in CP decomposition in Fi.8a. The frontal slice can be decomposed as X = AΣ B T where Σ = dia C : and C : is the -th row of the factor matrix C, as shown in Fi.8b. This relationship provides the way to recover the roup of matrices throuh tensor completion. ij r ab ir jrcr x x ij X A C X A B i-th row T A -th row j-th row C diac B T -th row: C: Fi. 8. The relationship between tensor CP decomposition and frontal slice decomposition. The problem in 5 aims to find the matrix decomposition for matrix completion. The problem can be solved with hiher accuracy if all the matrices can be interated into a tensor. Fortunately, to reflect the common user domain and time domain features, we have introduced the same user factor matrix U and time factor matrix V across different days. This translation maes the formulation in 5 satisfy the relationship between tensor CP decomposition and frontal slice decomposition. Therefore, utilizin the above relationship, the problem can V. COMPLETE SOLUTON The complete data recovery based on reshapin and alinment is shown in Alorithm. The sub-problems and are iteratively solved and 3-9 Steps are repeated until it converes. Specially, iven traffic matrices of days, if there are G timespots besides the timespots at : and 4: in the time rule involved in these days, in Step, the lare matrix of each day is divided into G sub-matrices accordin to the time alinment requirement. As there are days, after such a division, there are G roups of sub-matrices with each roup havin sub-matrices. The Step initializes the factor matrices needed in the alorithm. Step 4 solves the sub problem of minimizin over P for fix U, Σ, V. Step 5 builds the tensor with the shaped sub-matrices. Step 6 solves the sub problem and updates U, V, C by solvin the tensor completion problem min f U,V,C U, V, C = Y [[U, V, C ]] Ω F. Step 7 builds the diaonal matrix needed in the matrix decomposition Σ di C : where C : is the -th row of factor matrix C obtained in Step 6. After obtainin U, V, Σ, and P, Step 8 calculates the recovered sub matrices in the iterative step. n Step 8, M is an indicator matrix whose entry i, j is one if the entry i, j in X is sampled i.e., measured and zero otherwise. is an all ones matrix that has the same size as M. in Step 8 represents a scalar product or dot product of two matrices. For example, iven that A, B have the same size and C = A B, we have c ij = a ij b ij. X = M X + M U Σ V T P T uarantees that the sample entry already measured remains unchaned and only the missin data are updated durin the iterative procedure. V. PEFOMANCE EVALUATONS We use the public traffic trace data Abilene [35] to evaluate the performance of our proposed eshape-alin scheme. Two different metrics are considered: Error atio and Mean Absolute Error MAE, which are defined as Table. 7

8 Alorithm Complete reshape and alin traffic recovery alorithm : Accordin to the time alinment requirement, lare matrices are divided into G roups of sub-matrices with each roup havin sub matrices X T by SVD, V : nitialize U principal eienvectors = X, Σ 3: while not convere do 4: Sub problem : Compute the SVD V Σ U T X = M N T and update P = N M T 5: Generate tensor Y whose slices are Y = X P 6: Sub problem : Update U, V, C by solvin min f U,V,C U, V, C = Y [[U, V, C ]] Ω F for all the =,,, G tensors throuh CP decomposition 7: Σ di C : 8: Update X = M X + M U Σ V T P T 9: end while : Combine the recovered sub-matrices and obtain the recovered lare matrices. Error atio TABLE PEFOMANCE METC = i,j Ω x :ij ˆx :ij = i,j Ω x :ij MAE = n m = i,j x :ij ˆx :ij n the table, x :ij and ˆx :ij denote the raw data and the recovered data at i, j-th element of the matrix X where i n, j m, and. Only entries not observed i, j Ω are counted in the Error atio. Different from Error atio, the total data entries i.e., T are counted in the MAE. MAE is an averae of the absolute errors after the interpolation. Althouh some limited very recent studies consider the traffic data recovery throuh tensor completion, they cannot be applied in the practical networ with dynamic measurement rate. Therefore, we implement four matrix completion alorithms for the performance comparison:n M F [36], SM F [], SV T [37], LMaF it [38]. To alin measurement data under different measurement rate for data recovery, in all the above matrix completion alorithms, our temporal division scheme is taen to form the sub-matrices of each day. Then we combine the recovery results of different days to evaluate the performance. Accordin the time alinment requirement in Section V, different measurement rates will result in different partitions. We tae 3 measurement scenarios as examples to show the performance: The measurement rates are different in different days while the measurement rate of the same day is the same. The measurement rates are different in different days while the measurement rate chanes at the noon every day. 3 The measurement rates are different in different days while the measurement rate chanes at 8:, and 6: every day. Obviously, for time alinment, matrices in Scenario form one roup. n Scenarios and 3, the traffic matrices are partitioned into two roups and three roups, respectively. Fi.9 shows the performance in terms of error ratio and MAE with different samplin ratios. Note that, samplin ratio is the fraction of the total samplin entries to the total Error atio eshape-alin SMF a Error atio MAE 3 x -3 Fi. 9. ecovery performance under Scenario. Error atio a Error atio eshape-alin SMF MAE x -3 Fi.. ecovery performance under Scenario. Error atio a Error atio eshape-alin SMF MAE 4 b MAE eshape-alin SMF Fi.. ecovery performance under Scenario 3. b MAE eshape-alin SMF eshape-alin SMF b MAE number of matrix entries iven a measurementsamplin rate. As expected, with the increase of the samplin ratio thus sample data, the error ratio and MAE decrease and thus better recovery performance is obtained. Our eshape-alin scheme can transform the data recovery throuh the traffic matrix to the tensor completion to well exploit the multi-dimensional correlations hidden in the traffic data. Therefore, compared with other four matrix completion alorithm, our eshape- Alin scheme achieves the best recovery performance with the lowest error ratio and MAE in all the fiures. Amon all the matrix completion alorithms, SMF achieves the best performance. Besides usin low ran matrix to approximate the traffic matrix, SMF also utilizes spatial and temporal constraint matrices in the problem formulation to express the nowlede about the spatio-temporal structure of the traffic matrix. Amon all the 3 scenarios, the Scenario achieves the best recovery performance while Scenario 3 achieves the worst performance. This is because their matrix sizes are different. The matrices in the Scenario cover the time sement of the whole day, while the sub-matrices for Scenario correspond to half a day, and the sub-matrices operated in Scenario 3 cover one third of a day. A loner time period maes more data available to abstract the temporal feature for missin data recovery, and thus the best performance is achieved in Scenario. n Scenario 3, the performance ap between our eshape- 8

9 Alin scheme and other matrix completion alorithms becomes lare which demonstrates that eshape-alin scheme can well exploit multi-dimensional correlations hidden in the traffic data to accurately recover the missin data even with a short time period. X. CONCLUSON Accurate inference of the traffic matrix in the presence of chanin measurement frequency is of practical importance. n this paper, we propose a eshape-alin scheme which can reshape the inconsistent traffic matrices observed in different days into consistent ones, alin and interate these matrices to form tensor, and tae advantae of the user-domain and temporal domain features hidden in the traffic data to translate the matrix completion problem to the tensor completion problem with CP decomposition for more accurate missin data recovery. The performance studies demonstrate that our scheme achieves sinificantly better performance compared with the state of art matrix-completion alorithms to handle the missin data. EFEENCES [] Q. Zhao, Z. Ge,. Wan, and. Xu, obust traffic matrix estimation with imperfect information: main use of multiple data sources, in ACM SGMETCS Performance Evaluation eview, vol. 34, no.. ACM, 6, pp [] A. Gunnar, M. ohansson, and T. Telamp, Traffic matrix estimation on a lare ip bacbone: a comparison on real data, in Proceedins of the 4th ACM SGCOMM conference on nternet measurement. ACM, 4, pp [3] Y. Zhan, M. ouhan, N. Duffield, and A. Greenber, Fast accurate computation of lare-scale ip traffic matrices from lin loads, Acm Simetrics Performance Evaluation eview, vol. 3, no., pp. 6 7, 3. [4] A. Medina, N. Taft,. Salamatian, S. Bhattacharyya, and C. Diot, Traffic matrix estimation: Existin techniques and new directions, ACM SGCOMM Computer Communication eview, vol. 3, no. 4, pp. 6 74,. [5] A. Tootoonchian, M. Ghobadi, and Y. Ganjali, Opentm: traffic matrix estimator for openflow networs, in nternational Conference on Passive and Active Networ Measurement. Spriner,, pp.. [6] A. Lahina,. Papaiannai, M. Crovella, C. Diot, E. D. olaczy, and N. Taft, Structural analysis of networ traffic flows, in ACM SGMETCS, 3. [7] Y. Zhan, M. ouhan, C. Lund, and D. Donoho, Estimatin point-topoint and point-to-multipoint traffic matrices: An information-theoretic approach, in EEE/ACM Trans. Netw, 5, pp [8] A. Lahina, M. Crovella, and C. Diot, Dianosin networ-wide traffic anomalies, Acm Sicomm Computer Communication eview, vol. 34, no. 4, pp. 9 3, 4. [9] Y.Vardi, Networ tomoraphy,. Amer. Statist. Assoc., vol. vol. 9, no. 433,, p. pp , 996. [] P. Barford,. line, D. Plona, and A. on, A sinal analysis of networ traffic anomalies, ACM MW,. [] M. ouhan, Y. Zhan, W. Williner, and L. Qiu, Spatio-temporal compressive sensin and internet traffic matrices extended version, Networin EEE/ACM Transactions on, vol., no. 3, pp ,. [] M. Mardani and G. Giannais, obust networ traffic estimation via sparsity and low ran, in EEE CASSP, 3. [3]. Du, C. Chen, B. Yan, and X. Guan, Vanet based traffic estimation: A matrix completion approach, in EEE GLOBECOM, 3. [4] G. Gürsun and M. Crovella, On traffic matrix completion in the internet, in ACM MC. [5] Y.-C. Chen, L. Qiu, Y. Zhan, G. Xue, and Z. Hu, obust networ compressive sensin, in ACM MobiCom, 4. [6]. Xie, X. Nin, X. Wan, D. Xie,. Cao, G. Xie, and. Wen, ecover corrupted data in sensor networs: a matrix completion solution, EEE Transactions on Mobile Computin, DO:.9/TMC , 6. [7]. Xie, L. Wan, X. Wan, G. Xie, G. Zhan, D. Xie, and. Wen, Sequential and adaptive samplin for matrix completion in networ monitorin systems, in EEE NFOCOM, 5. [8]. Xie, L. Wan, X. Wan, G. Xie,. Wen, and G. Zhan, Accurate recovery of internet traffic data: A tensor completion approach, in EEE NFOCOM, 6. [9] T. olda and B. Bader, The tophits model for hiher-order web lin analysis, in Worshop on lin analysis, counterterrorism and security, vol. 7, 6, pp [] A. Carlson,. Betteride, B. isiel, B. Settles, E.. Hruscha r, and T. M. Mitchell, Toward an architecture for never-endin lanuae learnin. in AAA, vol. 5,, p. 3. [] C.. Appellof and E. Davidson, Strateies for analyzin data from video fluorometric monitorin of liquid chromatoraphic effluents, Analytical Chemistry, vol. 53, no. 3, pp , 98. [] A. Cichoci, D. Mandic, L. De Lathauwer, G. Zhou, Q. Zhao, C. Caiafa, and H. A. Phan, Tensor decompositions for sinal processin applications: From two-way to multiway component analysis, Sinal Processin Maazine, EEE, vol. 3, no., pp , 5. [3] S. Aja-Fernández,. de Luis Garcia, D. Tao, and X. Li, Tensors in imae processin and computer vision. Spriner Science & Business Media, 9. [4] H. Zhou, D. Zhan,. Xie, and Y. Chen, Spatio-temporal tensor completion for imputin missin internet traffic data, in EEE PCCC, 5. [5]. Liu, P. Musialsi, P. Wona, and. Ye, Tensor completion for estimatin missin values in visual data, EEE Transactions on Pattern Analysis and Machine ntellience, vol. 35, no., pp. 8, 3. [6] E. Acar, D. M. Dunlavy, T. G. olda, and M. Mørup, Scalable tensor factorizations for incomplete data, Chemometrics and ntellient Laboratory Systems, vol. 6, no., pp. 4 56,. [7] E. Acar and T. G. Dunlavy, Daniel M.and olda, A scalable optimization approach for fittin canonical tensor decompositions, ournal of Chemometrics, vol. 5, no., p. 6786,. [8] S. Gandy, B. echt, and. Yamada, Tensor completion and low-n-ran tensor recovery via convex optimization, nverse Problems, vol. 7, no., p. 5,. [9]. Carroll and.-. Chan, Analysis of individual differences in multidimensional scalin via an n-way eneralization of ecart-youn decomposition, Psychometria, vol. 35, no. 3, pp , 97. [3]. A. Harshman, Foundations of the parafac procedure: Models and conditions for an explanatory multi-modal factor analysis, UCLA Worin Papers in Phonetics, vol. 6, no., p. 84, 97. [3] L. Tucer, Some mathematical notes on three-mode factor analysis, Psychometria, vol. 3, no. 3, pp. 79 3, 966. [3]. A. Harshman, Parafac: Mathematical and technical notes, UCLA worin papers in phonetics, vol., no. 344, p. 5, 97. [33] B. F. Green, The orthoonal approximation of an oblique structure in factor analysis, Psychometria, vol. 7, no. 4, pp , 95. [34] T. G. olda and B. W. Bader, Tensor decompositions and applications. Siam eview, vol. 5, no. 3, pp , 9. [35] The abilene observatory data collections. internet.edu /observatory/data-collections.html. [36] D. D. Lee and H. S. Seun, Alorithms for non-neative matrix factorization, in Advances in neural information processin systems,, pp [37].-F. Cai, E.. Candès, and Z. Shen, A sinular value thresholdin alorithm for matrix completion, SAM ournal on Optimization, vol., no. 4, pp ,. [38] Z. Wen, W. Yin, and Y. Zhan, Solvin a low-ran factorization model for matrix completion by a nonlinear successive over-relaxation alorithm, Mathematical Prorammin Computation, vol. 4, no. 4, pp ,. 9