A New Virtual Indexing Method for Measuring Host Connection Degrees

Transcription

1 A New Vrtual Indexng Method for Meaurng ot Connecton Degree Pnghu Wang, Xaohong Guan,, Webo Gong 3, and Don Towley 4 SKLMS Lab and MOE KLINNS Lab, X an Jaotong Unverty, X an, Chna Department of Automaton and NLIST Lab, Tnghua Unverty, Beng, Chna 3 Department of Electrcal and Computer Engneerng, Unverty of Maachuett, Amhert, MA 4 Department of Computer Scence, Unverty of Maachuett, Amhert, MA Emal: {phwang, xhguan}@e.xtu.edu.cn, gong@ec.uma.edu, towley@c.uma.edu Abtract We preent a new vrtual ndexng method for etmatng hot connecton degree for hgh peed lnk. It baed on the vrtual connecton degree ketch where a compact ketch of network traffc bult by generatng the aocated vrtual btmap for each hot. Each vrtual btmap cont of a fxed number of bt elected randomly from a hared bt array by a new method for recordng the traffc flow of the correpondng hot. The hared bt array effcently utlzed by all hot nce t every bt hared by the vrtual btmap of multple hot. To reduce the noe contamnated n a hot vrtual btmap due to harng, we propoe a new method to generate the fltered btmap ued to etmate hot connecton degree. Furthermore, t can be ealy mplemented n parallel and dtrbuted proceng envronment. The expermental and tetng reult baed on the actual network traffc how that the new method accurate and effcent. Keyword-Btmap; Data treamng; ot connecton degree; Vrtual Indexng. I. INTRODUCTION The n-degree/, defned a the number of dtnct ource/detnaton that a network hot connect to durng a gven tme wndow, an mportant tattc metrc of network traffc that provde wth nght nto network meaurement and montorng applcaton uch a hot proflng [], and ad fat detecton of ecurty attack [], [3], etc. To obtan the total number of flow generated by a hot, one need to buld a hah table that keep track of extng flow to avod duplcatng flow record for packet from the ame flow. In th paper, a flow defned a the et of all packet wth the ame ource and detnaton addree n a tme wndow. owever, t not practcal to obtan hot connecton degree by buldng per-hot hah table that are reource ntenve to mantan for hgh peed lnk carryng a huge number of multaneou actve hot and flow. It may not be poble to accurately meaure and montor mave network traffc mply by upgradng the performance of meaurng devce. ence, t derable to develop new method to meet the challenge of montorng hgh peed network traffc onlne. Etan and Varghee [4] propoed a famly of btmap algorthm for etmatng the total number of dtnct flow on hgh peed lnk. To etmate each hot connecton degree, t need to buld a btmap for each hot, whch may not be calable to hgh peed lnk carryng flow aocated a huge The reearch preented n th paper upported n part by Natonal Natural Scence Foundaton (657487), 863 gh Tech Development Plan (7AAZ475, 7AAZ48, 7AAZ464, 8AAZ45) number of hot. Zhao et al. [5] propoed a data tream method to meaure hot connecton degree, whch a varant of Bloom flter [6] and cont of n m -dmentonal bt array. Each hot aocated wth column n the bt array randomly elected by hah functon, and one bt n each of t aocated column et a one for each of t packet comng. The correpondng column of each hot can be ued to etmate t connecton degree, nce each column can be vewed a a drect btmap a propoed n [7]. The drect btmap method ndcate that the number of row repreented n the bt array hould be et n the order of thouand to perform the tak of etmatng connecton degree of ource wth thouand of flow. owever, mot hot have only everal flow and only a very mall number of hot have thouand of flow, whch mple that mot column n the bt array are agned to hot wth a mall connecton degree, and are motly zero. To reduce th pace neffcency, Yoon et al. [8] buld a vrtual btmap for each hot by takng bt randomly from a hared bt array ung a group of hah functon. Each hot vrtual btmap ued to etmate t connecton degree mlar to what propoed n [7] ung a drect btmap. The number of dtnct bt n a hot vrtual btmap vare due to hah collon. There no guarantee on the qualty of the etmate of the of a hot whoe vrtual btmap generated wth many hah collon. Furthermore, each bt n the hared bt array may be hared by everal hot and the noe contamnated n a hot vrtual btmap ext whle etmatng t connecton degree. In partcular, a hot wth a mall number of connecton entve to the noe, nce t ha more bt n t vrtual btmap that are probably contamnated by other ource. In th paper, we preent a new vrtual ndexng method to accurately etmate hot connecton degree over hgh peed lnk. A vrtual connecton degree ketch data tructure degned to buld a very compact ketch of network traffc, whch can be ued to etmate the connecton degree of each hot baed on the aocated vrtual btmap. Each vrtual btmap cont of a fxed number of bt elected randomly from a hared bt array by a new vrtual btmap generaton method. The hared bt array mall, and effcently utlzed by all hot nce t every bt hared by the vrtual btmap of multple hot. The new method computatonally effcent nce t only need to et everal bt for each ncomng packet. To reduce the noe contamnated n each

2 hot vrtual btmap becaue of harng, we preent a new method for generatng the fltered btmap ued to etmate the hot connecton degree. Furthermore, t can be ealy mplemented n parallel and dtrbuted proceng envronment. Experment baed on the actual network traffc how that the new method truly accurate and effcent. It hould be noted that the algorthm for meaurng hot can alo be appled for hot n-degree. Th paper organzed a follow. In Secton II the new vrtual ndexng method decrbed n detal. The performance evaluaton preented n Secton III. Concludng remark then follow. II. A. Data Structure VIRTUAL CONNECTION DEGREE SKETC A vrtual connecton degree ketch (VCDS) cont of a bt array A[k] ( k m ) aocated wth ndependent group of hah functon {f,, f,,, f,l- }, {f,, f,,, f,l- },, and {f,, f,,, f,l- }. Each f, (, L ) a hah functon: {,,, N-} {,,, m-}, where N the ze of ource pace S. Each ource ha correpondng vrtual btmap B () ( ) where B () defned a a bt array contng of L bt elected randomly from A by the group of hah functon {f,, f,,, f,l- }, that B () = (A[f, ()], A[f, ()],, A[f,L- ()]),. B () can be vewed a a drect btmap [7] occuped only by. The length of the bt array n a drect btmap contant and a vtal parameter to etmate the of. owever the number of dtnct bt n B () maller than L, when A[f, ()], A[f, ()],, A[f,L- ()] are elected from A wth hah collon. There no guarantee on the qualty of the etmate of the of a hot whoe vrtual btmap generated wth many collon. For example, when m= 6 and L= 4, the number of dtnct bt elected from A by the group of hah functon {f,, f,,, f,l- } maller than L wth a probablty of - - and t expectaton 995. Th problem alo ext but not notced n [8]. To addre th ue, we propoe a vrtual btmap generatng method by degnng f, (, L ) baed on the double hahng cheme [9] f = ψ + ψ mod m,,, where ψ, a hah functon that map the ource pace unformly to the range {,,, m-}, ψ, a hah functon that map the ource pace unformly to the range {,,, m-}, and m a prme. It ealy valdated that each f, alo map the ource pace unformly to the range {,,, m-}. The followng theorem how that each vrtual btmap hahed nto L dfferent bt n A. Theorem. For a ource, L dfferent bt are elected from A by each group of hah functon {f,, f,,, f,l- } ( ), that, f, f,, < L. Proof. (Proof by contradcton) Aume to the contrary that there f, = f,, for then ( ) ψ mod m., Snce L m and m prme, we have mod m and ψ mod, m. Note ψ, m, o we have ψ, =. Th contradct wth the defnton of, m ψ {,,..., }. Theorem. For each B () ( and S ), denote et SB ()={f, () L }. Then the probablty that any bt A[k] ( k m ) ncluded n B () L P{ k SB } =. () m Proof. If k SB and k the the -th ( L ) bt n B ()=(A[f, ()], A[f, ()],, A[f,L- ()]), we have k= ψ, + ψ, mod m. For each ψ, n {,,, m-} and each n {,,, L-}, there one and only one vrtual btmap contaned A[k], nce ψ, determned a follow: ψ = k ψ mod m.,, Therefore the total number of dtnct vrtual btmap contaned A[k] L(m-). Snce each vrtual btmap B () elected unformly from the total m(m-) dtnct vrtual btmap, we have (). B. Update Procedure Each bt n A ntally et to zero. When a packet p = (, d) arrve, we et the g( d)-th bt n each vrtual btmap B () to one. ere g a unform hah functon wth the range {,, L-}, and the flow label d the concatenaton of ource and detnaton d. A the g( d)-th poton n B (), correpond to the f,g( d) ()-th poton n A, we only need to et bt for each ncomng packet a follow: A[f,g( d) ()] =,. C. Connecton Degree Etmator The bt n B () ( ) that the flow of ource hah nto ung hah functon {f,, f,,, f,l- } are denoted a the bt ued by n the followng part. They are et to one to tore the flow nformaton of, o each B () can be ued to etmate the of mlar to the drect btmap propoed n [7]. Snce each bt n B () elected randomly from A and alo hared by other ource, the other bt n B () not ued by mght alo be et to one by flow belongng to other ource. Th ntroduce noe nto the etmaton of the of. Therefore the more bt n B () are not ued by, the more noe generated. The ze of the vrtual btmap, L, uually et to thouand to guarantee the hgh accuracy of etmatng the of a ource aocated wth a huge number of flow, whch wll generate a large number of bt contanng noe epecally for the ource aocated wth a mall number of flow. In what follow, we ntroduce a new fltered btmap generaton method to reduce the noe generated by other ource. The

3 fltered btmap B defned a a bt vector computed from B (), B (),, and B () a follow: B = B B B where the bt-and operaton. For any flow (, d) of, the g( d)-th bt n each B () ( ) et to one, o the g( d)-th bt n B tll one. Defne ϕ a follow: m L L ϕ = + () m m L where OD the of ource. Then for any gven bt n each B () not ued by, the probablty that t et to zero by any other ource ϕ baed on Theorem. Therefore t a noe bt when all correpondng bt n B () ( ) are one wth probablty Φ q = ϕ where Φ = ϕ. Suppoe m and L are gven, we want to optmze to mnmze the noe contamnated n the fltered btmap B. There are two competng force: for each bt n B not ued by, ung larger gve u a greater chance of fndng a zero bt n t correpondng bt n B (); but ung more btmap reult n more bt n A beng et to one, whch ncreae the probablty that a hot vrtual btmap contamnated wth noe. When m>>l, we have q τ ( ) = Φ. The mpact of on the noe decrbed n the followng lemma. Lemma. τ ( ) = ( Φ ) decreae wth, ln Φ, ncreae wth, +, and obtan the mnmum ln Φ at = ln Φ. Proof. Defne y = ln τ ( ), then t frt dervatve and econd dervatve are Φ ln Φ = ln ( Φ ) d Φ d y d ( ) ( Φ ) OD Φ Φ ln Φ ln Φ =. It can be ealy hown that d < when, ln Φ and d > when, +, therefore functon ln Φ τ ( ) decreae wth, and ncreae wth ln Φ, + ln Φ. Meanwhle we have = and d = Φ ln d y >, o the optmal value of τ ( ) obtaned at d = Φ ln = ln Φ. In what follow a new method propoed to etmate the of ource. For any gven bt n B not ued by, the probablty that no other ource ue t ether p = q. Denote the total number of bt n B not ued by a U, the expectaton of zero bt n B ( B ) ( ) ( ) U B defned a the total number of E U = E U p = p E U. (3) The probablty that a gven bt n B not ued by L OD, o the expectaton of U computed a follow: L ) = L Le. (4) L p computed wth nput of each hot whch are unknown. We fnd that p can be etmated from A. Denote AB ( ) a the bt n A[k] ( k m ) except A[f, ()], A[f, ()],, A[f,L- ()]. For each bt n AB ( ), the Φ probablty of beng et to zero α =. Let U B be the ϕ total number of zero bt n AB ( ). Then OD E U = ( m L) α. (5) B Snce ( p q ) ( α ) equaton from (5): = =, we have the followng ( B ) E U p =. (6) m L From (3), (4) and (6), we have ) B ) B OD L ln + L ln. (7) L m L Replacng B E U and ( B ) OD E U n (7) by the ntantaneou value, U B and U B obtaned from B and AB ( ) repectvely, we have the followng etmate of ( ) OD ( ): U B U B OD = L ln + L ln. (8) L m L The frt term of the rght hand de of (8) correpond to the etmator ued n the drect btmap [7]. The econd term of the rght hand de of (8) ued to compenate for the error caued by the noe generated by the other ource.

4 Fnally the followng equaton gve a more accurate etmator of the of : et OD = medan OD. D. Combnaton Operaton A VCDS A can be dtrbuted acro G router a follow. Each router ( G ) can mantan a VCDS A, baed on the traffc that t oberve. Baed on all network traffc at all G router, VCDS A can be calculated by the followng equaton A = A A AG where the bt-or operaton. Each dtrbuted node ( G ) only need to end A to the control center, whch greatly reduce the amount of communcaton between the dtrbuted node and the control center. E. Parameter Confguraton Snce the total complexty for updatng each packet O(), hould be mall. If the of ource much larger than LlnL + O(L), we wll obtan all n t correpondng vrtual btmap wth hgh probablty due to the reult of the coupon collector problem []. In th cae, the outdegree of cannot be etmated accurately, and we only know that t not maller than LlnL. To addre th ue, we can drectly ue a larger L or apply the pre-amplng method propoed n [8]. III. A. Data Collecton TESTING AND PERFORMANCE EVALUATION. Expermental data ued n th paper baed on the network traffc of the backbone of CERNET (Chna Educaton and Reearch Network) Northwet Regonal Center and the campu network of X an Jaotong Unverty wth more than 3 hot. The actual traffc data wa collected at an egre router wth a bandwdth of.5gbp from a B- cla network by ung TCPDUMP. The total number of collected packet about. 8. There are.4 5 dtnct ource and dtnct flow. We further compare VCDS wth the tate-of-the-art method compact pread etmator (CSE) [8]. B. ot Out-Degree Etmaton The followng experment are ued to evaluate the performance of etmatng hot provded by VCDS. The relatve error of etmated of hot et defned a OD OD / OD, where OD t etmated and OD t actual. Fg. and how the relatve error of hot etmate for dfferent, where m=4 M and L=48. The accuracy of VCDS frt ncreae wth and then decreae wth, whch content wth Lemma. The mpact of L hown n Fg. 3, where =5 and m=4 M. For a hot wth a mall, the error of t etmated ncreae wth L, nce a larger value of L ntroduce more unued bt n the hot vrtual btmap, whch mght be contamnated. owever, a maller value of L et generate a larger etmaton error when the large mlar to the drect btmap algorthm [7]. etmated etmated n-degree actual 5 5 etmated n-degree actual (a). = (b). = actual etmated actual (c). = 5 (d). = 5 Fgure. Actual v etmated for dfferent average relatve error average relatve error 5 5 CSE VCDS = VCDS =3 VCDS =5 5 5 (a). Average relatve error for mall CSE VCDS = VCDS =3 VCDS =5 5 5 (b). Average relatve error for large Fgure. Average relatve error for dfferent

5 average relatve error 3 L=56 L=5 L=4 average relatve error L=56 L=5 L= (a). Average relatve error for hot wth mall (b). Average relatve error for hot wth large Fgure 3. Average relatve error for dfferent L average relatve error 4 3 m= M m= M m=4 M m=8 M average relatve error m= M m= M m=4 M m=8 M 5 5,,,,3 (a). Average relatve error for hot wth mall (b). Average relatve error for hot wth large Fgure 4. Average relatve error for dfferent m Fg. 4 how the average relatve error of hot etmate for dfferent m, where =5 and L=48. We oberve that the accuracy of etmated mproved by ncreang m, whch reduce the noe, epecally for hot wth mall. For a hot wth a mall, the number of unued bt n t vrtual btmap larger than that of a hot wth a large. Therefore the etmated of a hot wth a mall outdegree more entve to the noe. IV. CONCLUSIONS In th paper we preent a data treamng method VCDS for etmatng hot connecton degree over hgh peed lnk. VCDS can generate a very compact ketch of network traffc, and the connecton degree of each hot etmated by the aocated vrtual btmap contng of a fxed number of bt randomly elected from a hared bt array. The expermental and tet reult baed on the actual network traffc how that the new method accurate and computatonally effcent. REFERENCES [] K. Xu, Z. L. Zhang, and S. Bhattacharyya, Proflng Internet backbone traffc: behavor model and applcaton, n Proceedng of ACM SIGCOMM 5, Phladelpha, PA, 5, pp [] M. Roech, Snort lghtweght ntruon detecton for network, n Proceedng of the USENIX LISA Conference on Sytem Admntraton 999, Seattle, WA, 999, pp [3] D. Plonka, Flowcan: A network traffc flow reportng and vualzaton tool, n Proceedng of USENIX LISA, New Orlean, LA,, pp [4] C. Etan and G. Varghee, Btmap algorthm for countng actve flow on hgh peed lnk, n Proceedng of ACM SIGCOMM IMC 3, Mam, FL, 3, pp [5] Q. Zhao, A. Kumar, and J. Xu, Jont data treamng and amplng technque for detecton of uper ource and detnaton, n Proceedng of ACM SIGCOMM IMC 5, Berkeley, CA, 5, pp [6] B.. Bloom, Space/tme trade-off n hah codng wth allowable error, Communcaton of the ACM, vol. 3, no. 7, pp. 4 46, July 97. [7] K. Y. Whang, B. T. Vander-zanden, and. M. Taylor, A lnear-tme probabltc countng algorthm for databae applcaton, IEEE Tranacton of Databae Sytem, vol. 5, no., pp. 8 9, June 99. [8] M. Yoon, T. L, S. G. Chen, J. Per, Ft a pread etmator n mall memory, n Proceedng of IEEE INFOCOM 9, Ro de Janero, Brazl, 9, pp Journal veron to be appeared n IEEE Tranacton on Newtworkng. [9] T.. Cormen, C. E. Leeron, R. L. Rvet, and C. Sten, Introducton to algorthm (nd ed.), MIT Pre, Cambrdge, MA,. [] R. Motwan and P. Raghavan, Randomzed Algorthm, Cambrdge Unverty Pre, 995.