Persistent Spread Measurement for Big Network Data Based on Register Intersection

Transcription

1 Peritent Spread Meaureent for Big Network Data Baed on Regiter Interection YOU ZHOU, Univerity of Florida YIAN ZHOU, Google Inc. and Univerity of Florida MIN CHEN, Google Inc. and Univerity of Florida SHIGANG CHEN, Univerity of Florida Peritent pread eaureent i to count the nuber of ditinct eleent that perit in each network flow for predefined tie period. It ha any practical application, including detecting long-ter tealthy network activitie in the background of noral-uer activitie, uch a tealthy DDoS attack, tealthy network can, or faked network trend, which cannot be detected by traditional flow cardinality eaureent. With big network data, one challenge i to eaure the peritent pread of a aive nuber of flow without incurring too uch eory overhead a uch eaureent ay be perfored at the line peed by network proceor with fat but all on-chip eory. We propoe a highly copact Virtual Interection HyperLogLog VI-HLL architecture for thi purpoe. It achieve far better eory efficiency than the bet prior work of V-Bitap, and in the eantie dratically extend the eaureent range. Theoretical analyi and extenive experient deontrate that VI-HLL provide good eaureent accuracy even in very tight eory pace of le than bit per flow. 4 CCS Concept: Network Network eaureent; Network onitoring; Denial-of-ervice attack; Network anageent; Additional Key Word and Phrae: Peritent Spread Meaureent; Big Network Data; Network Traffic Meaureent; Network Security ACM Reference forat: You Zhou, Yian Zhou, Min Chen, and Shigang Chen. 27. Peritent Spread Meaureent for Big Network Data Baed on Regiter Interection. Proc. ACM Mea. Anal. Coput. Syt.,, Article 4 June 27, 29 page. DOI: INTRODUCTION Maive and ditributed data are increaingly prevalent in odern network a high-peed router forward packet at hundred of gigabit or even terabit per econd. Big data alo happen at the network edge. For a few exaple, Google handle over 4, earch querie every econd [], and illion tweet are produced per day [2]. Traffic eaureent and claification at uch high peed and with uch aive volue poe ignificant challenge [3 4]. Exact eaureent of big network data i often infeaible due Thi work i upported in part by the National Science Foundation under grant STC-6248 and a grant fro Florida Center for Cyberecurity. Correponding author: Yian Zhouyianzhou@google.co. Author addree: Y. Zhou and S. Chen, Departent of Coputer and Inforation Science and Engineering, Univerity of Florida; Y. Zhou and M. Chen, Google, Mountain View. Periion to ake digital or hard copie of all or part of thi work for peronal or claroo ue i granted without fee provided that copie are not ade or ditributed for profit or coercial advantage and that copie bear thi notice and the full citation on the firt page. Copyright for coponent of thi work owned by other than ACM ut be honored. Abtracting with credit i peritted. To copy otherwie, or republih, to pot on erver or to reditribute to lit, require prior pecific periion and/or a fee. Requet periion fro periion@ac.org. 27 ACM /27/6-ART4 $. DOI:

2 4:2 Y. Zhou et al. Multi-period Analyi Traffic Sketche S S S S Raw Traffic Data Tie One Period One Period One Period Fig.. Multi-period analyi of data ketche. to exceively high eory requireent and coputation/counication overhead, wherea approxiate etiation with probabilitic guarantee i a viable option. Flow cardinality etiation [ 2] i a fundaental proble in network traffic eaureent. It etiate the nuber of ditinct eleent in every flow during pre-defined eaureent period. Each flow i uniquely identified by one or ultiple field in the packet header, called flow label, which can be flexibly defined baed on application need. A exaple, the flow under eaureent ay be per-ource flow with flow label being the ource addre, per-detination flow, TCP flow, WWW flow, or application-pecific flow. The eleent under eaureent can be detination addree, ource addree, port, value in other header field, or even keyword that appear in packet payload. For exaple, for each per-ource flow, if detination addree are treated a eleent, then a flow cardinality i the nuber of ditinct detination addree that the flow ource ha contacted, which can be ued for can detection. Exiting reearch on flow cardinality etiation ainly focue on analying traffic ketche fro one eaureent period, which i the uary of the raw traffic data in that tie period. Since online torage can only hold liited inforation, the ketche are uually offloaded to a erver after each eaureent period for long-ter torage and offline query. Thi paper tudie an under-invetigated proble of analyzing ketche acro ultiple period a hown in Figure. In particular, we are intereted in eauring the peritent pread of each flow, which i defined a the nuber of ditinct eleent that how up in a network flow during a certain nuber of conecutive eaureent period. Practical Iportance: Peritent pread eaureent ha any practical application. Traditional uper-preader detection i to identify the elephant flow whoe cardinalitie are abnorally large, and can be applied to onitoring network anoalie. For intance, canner ay be identified if they end probe to too any detination addree, i.e., the cardinalitie of per-ource flow are large. But there are practical cenario where flow cardinality alone i inadequate a tealthy canner ay intentionally reduce it probing rate to reduce it flow cardinality in order to evade detection. Even with a reduced probing rate, after ufficient tie, the canner can till dicover yte with vulnerability to exploit. In thi cae, eauring peritent pread can help identify uch tealthy canner. A a canner probe different detination addree over tie, it peritent pread i zero or low; if a canner deliberately

3 Peritent Spread Meaureent for Big Network Data Baed on Regiter Interection 4:3 Internet Attacker Server End-Uer Noral Connection Stealthy DDoS Fig. 2. Stealthy DDoS attack. repeated any of the ae detination, it would ignificantly low down the already all canning rate. Therefore, odet flow cardinality but uually low peritent pread help ignal a low-rate canner that wander in the detination addre pace. In the econd exaple, DDoS attack ay be identified if unuually any client end requet to a erver, i.e., the cardinality of a per-detination flow i too high. However, a illutrated in Figure 2, with a aller nuber of available attacking achine, a tealthy DDoS attack doe not attept to overwhel the target erver with exceive requet, but to degrade it perforance [2]. If the nuber of attacking achine i iilar to the nuber of legitiate uer, we will not oberve unuual flow cardinalitie. Again, eauring peritent flow cardinality ay help. According to the tudy [22] of real-world network trace fro CAIDA [23], the continuou interaction between legitiate uer and their target erver i norally horter than twenty inute. For attacker, ince their goal i to degrade the perforance of the target erver over a long period, thee hotile achine will end requet peritently to the target erver, reulting in a ignificant peritent cardinality over tie that i higher than uual. Peritent pread eaureent alo ha application at the network edge e.g., web earch and ocial edia. Take Google trend a an exaple. If Google treat all client IP that query a keyword a a flow, the cardinality of the flow ugget the popularity of the keyword being earched. However, a ignificant nuber of colluding achine with different IP addree can periodically query the ae keyword, and ake thee keyword popular in Google trend a they wih. Since noral uer typically do not query the ae keyword periodically for a long tie, peritent pread eaureent can help detect uch long-ter earch pattern, where a large et of IP keep querying the ae keyword over ultiple period. Beide detecting faked popularity, our work ay erve a a generalized priitive tool for detecting hidden activitie that anifet only over long tie. Prior Art and Challenge: Mot previou work focue on traffic ketche of one eaureent period. To deal with a large nuber of flow, a erie of ketche were developed to reduce aive raw data to a uary of per-flow cardinalitie during online eaureent. Thee olution include PCSA [6], Multi-Reolution Bitap [], LogLog [7], and HyperLogLog HLL [8]. The principle i to allocate a eparate data tructure, containing a certain nuber of bitap, regiter or other eleentary data tructure, to each flow for recording it eleent. Over the pat decade, a ajor reearch thrut i to reduce the ketche eory footprint. But it ha been a difficult undertaking with low progre. For

4 4:4 Y. Zhou et al. intance, per-flow eory requireent for cardinality eaureent wa reduced fro thouand of bit to hundred of bit by HLL [8], which enure a large eaureent range with good accuracy. However, a the Internet enter the big-data era, hundred of bit per flow can till be too uch when there are too any flow. An exaple i odern high-peed router, which forward packet fro incoing port to outgoing port via witching fabric at the extraordinary peed. To utain high throughput, online odule for packet cheduling, acce control, quality of ervice and traffic eaureent are often ipleented on network proceor, bypaing ain eory and CPU alot entirely. The on-die eory uch a SRAM in a network proceor i fat but all, and ay have to be hared by ultiple function. Therefore, it i highly deirable to ipleent thee function a copact a poible. A thi paper focue on peritent cardinality eaureent, we want to puh it eory uage to an unprecedented low level, in order to ave pace for other function on the ae chip. In another exaple, uppoe a web-earch analyt want to profile, for each keyword phrae, quetion or entence, the nuber of ditinct uer that have earched the keyword. Thi inforation i ueful in online ocial/econoical/opinion trend tudie or optiizing earch perforance [24]. A we have dicued earlier, peritent pread eaureent can be ued to detect faked popularity. However, ince the nuber of flow one flow per keyword, phrae, quetion or entence can be in any billion, it preent a challenge in coputational reource, and eory in particular. Intead of uing an expenive and powerful erver, if we can dratically reduce the reource requireent, we ay be able to run uch analyi on a cheap coodity coputer, which i a welcoe reult when high-end achine are not readily available. To u up, there are practical cenario with great diparity between eory deand and availability, which require online cardinality eaureent to be ipleented a copact a poible. Moreover, the deign of a eaureent function hould alo enure reaonable accuracy with a large eaureent range that upport elephant flow with very high peritent cardinalitie. To the bet of our knowledge, little reearch work on peritent pread eaureent exit in literature. Chen et al. [] propoe a continuou variant of Flajolet-Martin ketche adapted fro [6], which however cannot give accurate reult when the available eory pace i tight [22]. Xiao et al. [22] deign a bit haring architecture called ulti-virtual bitap, which tore a flow inforation in a virtual bitap during each eaureent period and analyze the bitap fro ultiple period to find peritent cardinality. The ajor drawback i that the eaureent range of bitap i very all and no ore than a few thouand for a typical ipleentation. Our Contribution: The objective of our reearch i to iprove the eory efficiency and enlarge the range of peritent pread eaureent, while keeping good accuracy. Our ain contribution are uarized below. Firt, we deign a highly efficient peritent pread etiator called Interection HLL I-HLL that work over ultiple eaureent period. Every flow i allocated a eparate HLL ketch of regiter to record it cardinality in a eaureent period. We apply regiter interection over the erie of HLL ketche produced for a flow during a given nuber of eaureent period. We then eploy axiu likelihood etiation to develop the forula of the I-HLL etiator that copute an etiate of the flow peritent pread. We forally analyze the accuracy of the etiation, and how I-HLL ha a large etiation range. Second, to further iprove eory efficiency, we introduce regiter haring on top of I-HLL and propoe a highly copact Virtual Interection HLL VI-HLL architecture to eaure the peritent pread of a large nuber of flow iultaneouly. Siilar to [2], each flow i allocated a virtual HLL ketch of ultiple regiter, and the virtual HLL ketche of all flow hare a coon pool of phyical regiter. But unlike [2] that eaure flow cardinality in one period, our VI-HLL deal with peritent

5 Peritent Spread Meaureent for Big Network Data Baed on Regiter Interection 4: cardinality over ultiple period. VI-HLL achieve far better eory efficiency and uch larger range than the bet exiting work V-Bitap [22] on peritent cardinality eaureent. Finally, not only do we atheatically analyze the etiation accuracy of VI-HLL, but alo perfor extenive experient to copare it with V-Bitap. The experiental reult deontrate the uperior perforance of VI-HLL. Interetingly, it etiation accuracy iprove when the nuber of eaureent period increae. The ret of thi paper i organized a follow. Section 2 forulate the peritent pread eaureent proble. Section 3 dicue the preliinarie. Section 4 propoe our I-HLL etiator. Section preent the enhanced VI-HLL architecture. Section 6 evaluate the perforance by iulation. Section 7 draw the concluion. 2 PROBLEM STATEMENT Conider the packet trea arriving at a router or firewall inide a high-peed network or the application record produced by a erver e.g., web earch at the network edge. We odel both type of network data a a equence of flow label, eleent pair in our abtraction. Baed on the flow label, the equence of pair are claified into different flow. For the packet trea a exaple, if we want to eaure the nuber of ditinct ource that have contacted each detination, we abtract every packet a a pair of detination addre and ource addre, which can both be extracted fro the packet header. All pair i.e., packet with the ae detination addre i.e., flow label contitute a flow. In the exaple of web earch, each earch record i abtracted a a pair of keyword and ource addre fro which the earch requet i received. All pair with the ae keyword are treated a a flow. We are intereted in eauring eleent that keep howing up over tie in each flow. The iue i how to quantitatively define the peritency of keep howing up over tie. Conider the traditional definition of flow cardinality or pread eaureent [ 8], which i to find the nuber of ditinct eleent in each flow during a certain tie frae [,T]. Thi definition doe not capture the property of peritency. We illutrate it through an exaple of eauring the nuber of ditinct ource that have contacted each detination, where all packet to the ae detination for a per-detination flow. Suppoe one illion different ource contacted a detination during a day. The cardinality of thi per-detination flow i one illion. But if all the ource contacted the detination in the firt inute and no contact wa ade for the ret of the day, we cannot ay thee ource kept contacting the detination for the day. The peritent pread i zero in thi cae. To forulating peritency, one way i to divide the day into eaureent period of inute each. If we find that ource out of the illion were preent in each period, they were the peritent eleent that we want to eaure. The reaining eleent that howed up only in the firt period were not peritent. So the peritent pread i. Generally, we pecify peritency by dividing tie into eaureent period and eaure thoe eleent that are preent peritently in a pre-et nuber t of conecutive period under conideration. We give a ore foral definition a follow: Conider an arbitrary flow and t conecutive eaureent period. Let S j be the et of ditinct eleent in the flow oberved during the jth eaureent period, j t. Let S be the ubet of coon eleent oberved in all t period, i.e., S = S S 2... S t. The proble of peritent pread eaureent i to find the ize of S, denoted a n = S, which i called the peritent pread of the flow. The eleent in S are called peritent eleent. The eleent in S j S, j t, are called tranient eleent. The propoed architecture for etiating the flow peritent pread i intended to be generic, while it paraeter hould be et by yte adin baed on their application need. In particular, the length of each period and the nuber t of period ued are application-dependent. A an analogy, a yte adin will configure the threhold for can detection i.e., the triggering nuber of different detination

6 4:6 Y. Zhou et al. Peritent Spread Period= econd Nuber of Period Peritent Spread Period= inute Nuber of Period Fig. 3. Peritent pread of a packet flow to detination , with repect to different period length in the two plot and different nuber t of period on the horizontal axi. 2 4 Peritent Spread 3 29 Period= econd Nuber of Period Peritent Spread Period= inute Nuber of Period Fig. 4. Peritent pread of a packet flow to detination , with repect to different period length in the two plot and different nuber t of period on the horizontal axi. that a ource contact over a period to be ore than the eaured nuber of ot noral ource, which ay vary fro network to network. Siilarly, the paraeter of peritent pread eaureent hould alo be et baed on application-pecific and yte-pecific noral traffic tatitic. Conider the exaple in the introduction on detecting tealthy DDoS attack by eauring peritent pread of per-detination erver flow. If we et the eaureent period to be a day, we ay find ignificant peritent pread for erver in noral traffic, becaue legitiate uer ay regularly acce their eail, web and other ervice on a daily bai. If we et the eaureent period to be a few econd, we ay till find ignificant peritent pread in noral traffic becaue any ingle connection to a ervice ay lat for any conecutive period. However, if we chooe a period length in-between and ue a ufficient nuber of period, it becoe unlikely for any noral uer to exhibit the ae peritency in acceing the erver a the attacking hot [22]. The above analyi i confired by our experient uing a real network traffic trace fro CAIDA, containing 39,46 per-detination flow in an hour. We vary the length of eaureent period and the nuber t of period when eauring the peritent pread of the flow. The eaureent reult for two

7 Peritent Spread Meaureent for Big Network Data Baed on Regiter Interection 4:7 Nuber of Flow t = Nuber of Flow t = e2 e3 e3 e4 Flow Cardinality e2 e3 e3 e4 Peritent Spread Nuber of Flow t = 3 Nuber of Flow t = e2 e3 e3 e4 Peritent Spread 7 7 e2 e3 e3 e4 Peritent Spread Fig.. Flow ditribution with repect to peritent pread under different t value. randoly-elected large flow are hown in Fig. 3-4, and the tatitic of all flow are hown in Fig.. Conider the flow in Fig. 3. Both plot how that it peritent pread drop quickly when we increae the nuber of period. However, in the left plot where the period i hort econd, if the nuber t of period ued i too all e.g., 2 or 3, the peritent pread of thi noral traffic can be ignificant. For exaple, when t = 2, the peritent pread i 668, which i 36% of the pread when t =, i.e., the nuber of active ource in one period. Siilar obervation can be ade in Fig. 4. On the other hand, a we increae the period length to inute in the right plot, when t = 2, the peritent pread i jut 3.8% of the pread when t =. Be aware that a period of inute ha any ore packet thu eleent than a period of econd; therefore, the relative percentage 36% v.. 3.8% i a better indicator for the ipact of period length on peritent pread. By chooing a period length of inute and letting t = 6, we oberve jut 3 peritent eleent ource in the flow during an hour. In contrat, when the period length i econd and t = 6, we oberve 98 peritent eleent in an hour. Fig. preent the flow ditribution with repect to the pread or cardinality value when t =,2,3 and 6 in the four plot, repectively. We put flow in bin with pread range of [, ],, ],, ],... The length of each period i inute. The figure how that ot flow in thi noral traffic trace have all pread. When we increae the nuber of period, the nuber of flow in bin of large pread decreae quickly, uggeting that the peritent pread of thoe flow are reduced to all value. Thi property help in anoaly detection: When we ee the peritent pread of a per-detination flow uddenly jup

8 4:8 Y. Zhou et al. fro a uually all value to a large one, it ignal a poible DDoS attack a we explain earlier in the introduction. The objective of thi paper i to deign a peritent pread etiation architecture that conit of an online coponent and an offline coponent, where the forer record all eleent fro all flow in real tie uing highly-copact data tructure which keep only ketche of the raw traffic data and are offloaded to a erver after each eaureent period, and the latter perfor peritent pread etiation baed on the ketche fro ultiple period. We will evaluate the perforance of our deign baed on the following two etric. Meory overhead: The diparity in eory deand and upply for practical traffic eaureent cenario explained in the introduction otivate u to ake the online coponent of peritent pread eaureent a copact a poible. Etiation accuracy: Let ˆn be the etiation reult of the actual peritent pread n of a flow. The etiation accuracy i evaluated baed on the relative bia, Biaˆn n, and the relative tandard error, StdErrˆn n, which are defined below. Bia ˆn n = E ˆn StdErr ˆn n =, n Var ˆn = n Varˆn n. Clearly, aller value of relative bia and relative tandard error ean ore accurate eaureent reult. Given a certain available eory pace, we want to ake peritent-pread etiation a accurate a poible. We ake two auption, which are needed by our tatitical analyi. The firt auption i that there are a large nuber of flow in each period and the nuber of ditinct eleent/peritent eleent in any flow i negligibly all when coparing with the total nuber of ditinct eleent/peritent eleent in all flow. The econd auption i that tranient eleent can be approxiately treated a being independent aong different period. The ae auption are needed in [22], which provide network traffic analyi to upport the auption. The obervation i that when the length of each period i ufficiently long, ot tranient eleent will tay in one period becaue ot uer connection do not take that long. For exaple, when the period i et to 7 inute with a gap of 3 inute between conecutive period, traffic analyi in [22] how that le than % of all HTTP connection overlap with ore than one period. The percentage will be lower if the period i et longer. 3 PRELIMINARIES In thi ection, we firt introduce the HyperLogLog HLL algorith [8], and then preent a traightforward regiter-union approach for peritent pread etiation baed on HLL, which further otivate a ore accurate regiter-interection approach. 3. HyperLogLog HLL Algorith The HLL algorith ha ade ipact on IT indutry [9]. It i deigned to etiate the nuber of ditinct eleent in a ingle trea flow during a ingle eaureent period. HLL enure a large etiation range and a good etiation accuracy. An incoing trea i odeled a a ulti-et S, whoe eleent are in the doain D. An HLL ketch M of regiter are allocated to tore the cardinality inforation. Without lo of generality, let = 2 b,b N. The ith regiter in M i denoted by M[i], i [,. The ize

9 Peritent Spread Meaureent for Big Network Data Baed on Regiter Interection 4:9 ALGORITHM : HLL Sketch for a trea S Initialize a regiter array M of ize = 2 b with all zero; for e S do x := he; p := x x 2...x b ; q := x b+ x b+2... ; M[p] := axm[p],ρq; end return M at the end of a eaureent period. of regiter i et baed on the axiu range of the cardinalitie to be etiated. Specifically, a regiter with bit can eaure cardinalitie up to Algorith uarize how to generate an HLL ketch for trea S. Firt, we initialize all M[i] to zero, i [,. Let h : [D] [,] {,} L be a uitable hah function that ap an eleent in doain D uniforly at rando to the binary range of L bit long. Let ρq be the poition of the leftot for a binary tring q {,} L, i.e., it equal one plu the length of leading zero in q. For exaple, if q =..., then ρq = 4. For an incoing eleent e in trea S, let x be the binary repreentation of hah value he, where p i the leading b bit in x, and q i the reaining bit. Then the eleent e i apped to M[p], and M[p] i updated by M[p] := axm[p],ρq. 2 In other word, the trea S i plit into ubtrea, each of which i encoded in a regiter baed on the firt b bit of hahed value he. Each regiter i et to the axiu value of ρq aong all eleent e in the correponding ubtrea. If no eleent i encoded by a regiter, the regiter reain zero. At the end of a period, HLL etiate the nuber of ditinct eleent encoded by it ketch M = {M[], M[],..., M[ ]} through noralized haronic ean [8]: ˆn S = α 2 2 M[i], 3 where α i the bia correction contant that i 2+u du α = log u Pre-coputed value of α ay be ued in practice: α 6 =.673, α 32 =.697, α 64 =.79, and α =.723/+.79/ for 28. According to [8], the etiation tandard error i ˆnS StdErr = O. n S It ha been hown that etiation by 3 i everely biaed when the cardinality i aller than 2.. Hence, when the etiated cardinality fro 3 i aller than 2., we treat M a a bitap of bit, with each regiter M[i] converted to one bit, whoe value i one when M[i] > or zero otherwie. The etiation forula for all cardinality i where V i the fraction of bit in the bitap whoe value reain zero. i= ˆn S = lnv, 6

10 4: Y. Zhou et al. 3.2 HLL-Baed Peritent Spread Etiation The HLL ketche can be adopted for peritent pread etiation. To ake technical dicuion ore concrete, we conider per-detination flow paing a router and eaure the nuber of ditinct ource addree in each flow. For an arbitrary flow, we allocate an HLL ketch M of regiter to record the flow ource addree in each period. Denote the HLL ketch of the jth period by M j. At the beginning of the jth period, all regiter of HLL ketch M j are initialized to zero. When the router receive a packet, it extract the flow label i.e., detination addre dt fro the packet header, and record the eleent i.e., ource addre rc in M j by Algorith. By the end of the period, the router ha recorded the et S j of eleent in M j. It offload M j to a erver for long-ter torage and offline query. After t conecutive period, we have a equence of HLL ketche M, M 2,..., M t. The proble i how to ue thee HLL ketche to etiate the peritent pread n = S = S S 2... S t, which i the nuber of ditinct eleent that are preent peritently through the t period. We propoe two approache, regiter union and regiter interection, to olve thi proble Regiter-Union Approach. According to the incluion-excluion rule, the cardinality of an arbitrary et interection, including n, can be expreed a u/difference of the cardinalitie of et union. The cardinality of any et union can be etiated uing the HLL etiator 3 after perforing regiter-wie union on the correponding ketche. For exaple, the cardinality of et interection S S 2 i S S 2 = S + S 2 S S 2. 7 Naely, S S 2 i repreented a the u/difference of three cardinalitie, S, S 2 and S S 2, where S and S 2 can be etiated fro M and M 2 uing the HLL etiator, repectively. Moreover, given the ketche M and M 2 for S and S 2, the HLL ketch for the et union S S 2 i iply regiter-wie union M = M M2, where operator i defined to be M [i] = axm [i],m 2 [i], i <. After that, S S 2 can be etiated by applying the HLL etiator on M. Generalizing the above analyi to interection over ore than two period i.e., t > 2 i traightforward. Depite it atheatical iplicity, regiter-union etiate i very inaccurate ince it doe not fully explore the correlation aong the t HLL ketche. Let n be the cardinality of et union S S 2... S t, n be the cardinality of et interection S, and ˆn be it etiate uing the regiter-union approach. According to [, 8], the etiation tandard error of n i ˆn n StdErr n = O. 8 n The etiation accuracy depend on n n, and StdErr ˆn n increae a n n becoe larger. When t i et large, n ay becoe large due to addition of ore tranient eleent, wherea n ay tay ore or le the ae if the et of peritent eleent doe not change uch, which drive up n n and thu inaccuracy in etiation. The accuracy lo a t grow can prohibit a network adin fro configuring a large value for t Regiter-Interection Approach. By contrat, the regiter-interection approach calculate the interection of HLL ketche, M = M M 2... M t, where operator on two arbitrary HLL ketche i defined a M j M j2 [i] = inm j [i],m j2 [i], i <. Therefore, the value of the ith regiter in the interection ketch M i the inial value of all correponding regiter in the t original HLL ketche, M [i] = in j [,t]{ Mj [i] }, i <. 9

11 Peritent Spread Meaureent for Big Network Data Baed on Regiter Interection 4: Let M be the iaginary HLL ketch M that record the true interection et S. We derive the relationhip between M and M. A tated earlier, each flow eleent peudo-randoly pick a regiter in M j by the hah function h, and update the choen regiter accordingly. Moreover, in different period, a peritent eleent e in S alway trie to update the ae regiter M j [p] uing the ae value ρq a uggeted by 2. Therefore, the ae ketch M contructed with the peritent eleent fro S i ebedded in M j, j t, which can be conidered a M T j, where T j i the ketch contructed with the tranient eleent, S j = S j S, in the jth period. T j change fro period to period a tranient eleent vary over tie. Hence, we have M = M M 2... M t = M T M T 2... M T t = M T T 2... T t. The value of the ith regiter in M i, for i <, M [i] = ax{m [i],in{t [i],t 2 [i],...,t t [i]}}. If we ue M to approxiate M, the HLL etiator will produce a reult with poitive bia becaue it ay happen that tranient eleent et the ith regiter in all t period higher than M [i], cauing overetiation of the peritent pread, although the probability for thi to happen decreae a the nuber t of period increae. We will addre thi overetiation iue by axiu likelihood etiation. 4 INTERSECTION HLL ESTIMATOR In thi ection, we preent an Interection HLL I-HLL etiator baed on regiter interection M and axiu likelihood etiation MLE to eaure the peritent cardinality of any flow. 4. The Interection Regiter Value We firt analyze the probabilitic ditribution for an arbitrary regiter in the I-HLL ketch M, which will be ued to contruct an MLE etiator. Suppoe we eaure a flow over t period, and obtain a equence of HLL ketche, M, M 2,..., M t, which record eleent et, S, S 2,..., S t, repectively. Let n j be the cardinality of S j. The nuber of eleent in the tranient ubet S j i n j = n j n. We know M j = M T j, i.e., the HLL ketch M j of the jth period i the cobination of M for the peritent eleent and T j for the tranient eleent. In other word, M j [i] = axm [i],t j [i], i <. Applying it to 9, we can expre the interection ketch M a M [i] = in {ax M [i], T j [i] } j [,t] = ax M { [i], in Tj [i] } 2. j [,t] Suppoe the value of the ith regiter in M i k, i.e., M [i] = k, k. There are two cae: I. The peritent eleent in S et the value of regiter M [i] to be k, and the tranient eleent in at{ leat one period et the value of M [i] no larger than k. Naely, M [i] = k and T[i] = Tj [i] } k. in j [,t] II. The peritent eleent in S et the value of regiter M [i] aller than k. Of the t HLL ketche of tranient{ eleent, the iniu value in thi regiter i exactly k. Naely, M [i] < k and T[i] = in Tj [i] } = k. j [,t]

12 4:2 Y. Zhou et al. To calculate the probability for M [i] = k, we hould firt analyze the probabilitic ditribution for M [i] and T[i] to carry a particular value. Let n [i] be the total nuber of peritent eleent recorded in the ith regiter of the ketch M, and n j [i] be the nuber of tranient eleent in the ith regiter of the ketch T j, j [,t]. Since each peritent eleent in S randoly elect a regiter in M, it ha a probability of to ap into M [i]. Hence n [i] approxiately follow a binoial ditribution, n [i] Binon,, and the probability for M [i] to record ν peritent eleent i Pn n [i] = ν = ν ν n ν. 3 According to the HLL algorith, the rando variable M [i] i the axiu value of ν rando variable that are independently and geoetrically ditributed according to PY > k = 2 k,k,ν >. Thu, the cuulative ditribution function of M [i] under the condition n [i] = ν,ν > i PM [i] k n [i] = ν, ν > = ν. Since = and PM [i] k n [i] = ν, ν = =, the above 2 k conditional cuulative ditribution function i alo atified if ν =. Cobining thee two cae, we have PM [i] k n [i] = ν = 2 k ν, ν, k. 4 Therefore, baed on 3 and 4, the cuulative ditribution function F M [i]k of M [i] i F M [i]k = PM [i] k = PM [i] k n [i] = ν Pn [i] = ν n ν= n = ν n ν= ν n ν ν. 2 k In ot ituation, the peritent pread n i at leat 2 and / i aller than or equal to. 2. Hence, Poion ditribution can be ued to approxiate the binoial ditribution for efficient calculation, Binon,/ Poiλ = n. Thereby, we have F M [i]k n ν= λ ν e λ ν! 2 k ν = e λ n ν= e λ e λ 2 k = e λ 2 k = e n 2 k. λ 2 k ν ν! 6 Siilarly, we calculate the cuulative ditribution function F Tj[i]k n j F Tj[i]k = ν= n j ν e n j 2 k = e n j n ν n j ν ν 2 k 2 k. 7

13 Peritent Spread Meaureent for Big Network Data Baed on Regiter Interection 4:3 A we aue that tranient eleent in different period are approxiately independent, the cuulative ditribution function F T[i] k of T[i] i F T[i] k = PT[i] k = PT[i] > k = P in { T j [i] } t > k PT j [,t] j [i] > k j= j= j= t = PTj [i] k t = FTj[i]k. 8 Therefore, the probability P i for the firt cae that M [i] = k i { P i = PM F M [i]k F T[i] k k =, [i] = k PT[i] k = [ FM [i]k F M [i]k ] F T[i] k k. The probability P i for the econd cae that M [i] = k i P i = PM [i] < k PT[i] = k = { k =, F M [i]k [F T[i] k F T[i] k ] k. To u up, the probability for M [i] = k i PM [i] = k = P i +P i = { F M [i]k F T[i] k k =, F M [i]k F T[i] k F M [i]k F T[i] k k. 9 Let a generation function G n,n 2,...,n t,n,k repreent the expreion F M [i]k F T[i] k. Cobining 6, 7 and 8, then we have G n,n 2,...,n t,n,k = F M [i]k F T[i] k t e n 2 k e n j n 2 k. j= 2 Note that the n,n 2,...,n t can be etiated uing the HLL algorith on HLL ketche M, M 2,..., M t, repectively. Thu, they can be treated a contant in the generation function G, o we can iplify G n,n 2,...,n t,n,k to G n,k. We poition a the ubcript of function G becaue the nuber of regiter in each period i deterined by the available eory and i typically a fixed value. Therefore, the probability that the ith regiter in the interection ketch M ha the value k i { G n,k k =, PM [i] = k = G n,k G n 2,k k. In practice, a regiter can only carry a value in a pecific range due to the liited eory ize e.g., bit per regiter. Let H be the threhold, which i the axiu value upper bound that a regiter capacity can record. For intance, if the ize of a regiter i bit, it recording range i fro to 2 = 32 excluive, and the threhold H i 3. Let h be the ize of a regiter, then H = 2 h.

14 4:4 Y. Zhou et al. Conidering the liited regiter ize, we need to odify probability for M H. Aue the regiter i aigned to H when it value i out of bound. Hence, we have H PM [i] = H = PM [i] = k k= = G n,h. Therefore, the probability ditribution function for M [i] to carry a value k in 2 becoe G n,k k =, G n,k G n,k < k < H, PM [i] = k = G n,k k = H, k > H I-HLL Etiator We provide the I-HLL etiator for peritent pread n baed on MLE. To etablih the likelihood function, we firt eaure the nuber of regiter aong the regiter in M that carry the value k, which i denoted by N k. The reaon why we ue N k intead of k a the oberving factor i that the oberving pace ize of N k i equal to H, which i far le than k oberving pace ize. The probability for oberving N k regiter in M carrying the value k i PM [i] = k N k, auing thee regiter are approxiately independent. Hence, the cobined probability for oberving N, N,..., N H under the condition that there are n eleent in the peritent et i H PN,N,...,N H n = α PM [i] = k N k, 24 where α i a contant that equal! N!N!...N H!. The likelihood function for oberving N, N,..., N H with repect to n i Ln N,N,...,N H = α k= H PM [i] = k N k. 2 Taking the logarith on both ize of the likelihood function, we obtain the log baed likelihood function a follow: lnl = lnα+ k= H N k lnpm [i] = k. 26 Taking the partial derivative on log baed likelihood function with repect to n, we obtain lnl n = n lnα+ = H k= H k= k= N k lnpm [i] = k N k lnpm [i] = k n = H k= N k n PM [i] = k. PM [i] = k 27

15 Peritent Spread Meaureent for Big Network Data Baed on Regiter Interection 4: The derivative of PM [i] = k with repect to n for an arbitrary value k [,H] i given a follow, PM [i] = k n G n,k k =, n = n G n,k G n,k < k < H, 28 n G n,k k = H, where the partial derivative of G n,k over n i n G n,k n 2 k e 2 k + t j= n j n t e 2 k j= e n j n 2 k. 29 The calculation of n G n,k i given in the Appendix A. The axiu likelihood etiation i to find an etiated peritent pread ˆn that axiize the log likelihood function lnl. Therefore, we obtain an etiator for n : ˆn { } { = arg ax lnl = n n n lnl = }. 3 A a uary, we define a unified function f t to give a foral I-HLL etiator ˆn to eaure the peritent pread n over an arbitrary nuber t of tie period, which i equivalent to 3. Definition 4.. [I-HLL Peritent Spread Etiator] For t 2, a unified function to etiate the peritent pread of a flow i ˆn = f t,m,{m j } j [,t], 3 where i the nuber of regiter in each HLL ketch, M j i the HLL ketch in the jth period j [,t], and M i the interection HLL ketch that equal M M 2 M t. 4.3 Accuracy Analyi We analyze the relative bia and relative tandard error of I-HLL etiator. We denote the value of the ith regiter of M by a rando variable X i, thereby P Xi k = PM [i] = k. Then the expected value and variance of lnp X i k n are µ = E lnp Xi k n σ 2 = Var lnp Xi k n = H k= = H lnp Xi k PXi n k, k= lnp Xi k n 2 PXi k µ 2. Moreover, the new likelihood function for preerving X = k, X = k,, X = k can be written a Ln k,k 2,,k = P Xi k. 33 Note that the above likelihood function i only the original likelihood function ultiplied by a contant value α. Hence, we can till ue the notation L without confuion. Taking the logarith of the likelihood function and the derivative with repect to n, we have E lnl 2 n = E i= i= 32 lnp Xi k i n 2. 34

16 4:6 Y. Zhou et al. Since X,X,,X are roughly independent, auing ψ 2 = n σ 2, then we have E lnl 2 n lnp Xi k i 2 E + n = E i= i,j [, i j E lnp Xi k i n E lnp Xj k j n lnp Xi k i 2 + E lnp Xi k i lnp Xj k j E n n n = σ 2 +µ 2 + µ 2 = µ 2 +σ 2 = µ 2 + ψ n 2, where µ = and ψ 2 i 2 ψ 2 = n 2 G 3 n n,+ 2 G 2 n,h 3 2 2H G n,h + H n G 2 n,k 2G n,k k= 3 2 2k G n,k G n.,k The calculation of µ and ψ 2 can be found in the Appendix B. Hence, the fiher inforation [2] i Iˆn = E lnl n 2 = ψ n 2. According to the ayptotic propertie of axiu likelihood etiation, our etiator i ayptotically unbiaed, and it achieve the Craer-Rao lower bound: Therefore, the relative tandard error i and the ǫ confidence interval for n i VIRTUAL I-HLL ARCHITECTURE. Motivation ˆn d Noraln, Iˆn = Noraln, n 2. 3 ψ2 StdErr ˆn n ψ, 36 ˆn n ±Z ǫ ψ In the deign of our I-HLL etiator, all flow are allocated with eparated and equal-ized HLL ketche to record their eleent in each eaureent period, which bet fit when the flow cardinality i uniforly ditributed. However, any tudie oberve a coon fact that the ditribution of flow cardinalitie i extreely unbalanced in real network, and all percentage of large flow account for a ajority of the Internet traffic alo known a the heavy-tailed ditribution. Without lo of generality, we ue the real network trace captured by the ain gateway of our univerity a an exaple. The ditribution of per-ource flow and per-detination flow are illutrated in Figure 6a and Figure 6b, repectively. Clearly, the vat ajority of flow have all cardinalitie, while only a all nuber of flow have large cardinalitie. The ae trend i oberved in the traffic trace fro CAIDA [23]. Under thi coon obervation of unbalanced ditribution in network traffic data, aintaining one HLL ketch for each flow i not applicable due to the liited ize of on-chip SRAM. The reaon i that, when we don t know which flow are elephant flow in advance, the ize of all HLL ketche for I-HLL etiator hould be configured according to the larget flow cardinalitie in order to achieve reaonably accurate eaureent. Therefore, we have to allocate all HLL ketche with the ae ize that are large enough to accoodate the elephant flow. Hence, for the ajority of flow with all cardinalitie, the high-order bit in their regiter are actually under-utilized a any or even ot of the reain zero,

17 Peritent Spread Meaureent for Big Network Data Baed on Regiter Interection 4: Nuber of Flow Nuber of Flow e3 e4 e e6 Flow Cardinality a Per-ource flow e3 e4 e e6 Flow Cardinality b Per-detination flow Fig. 6. Flow cardinality ditribution for different flow which caue a ignificant wate of eory. To reduce the eory wate caued by the uneven flow cardinality ditribution, regiter haring hould be enabled aong the flow to utilize thee unued bit..2 Regiter Sharing and Virtual HLL Sketch Our idea i to enable regiter haring aong HLL ketche of all flow. An exaple i illutrated in Figure 7, where each cell repreent a regiter. The HLL ketche of all flow are no longer eparated. Intead, they hare regiter fro a coon regiter pool, called phyical regiter array A. Each flow peudo-randoly pick a nuber of regiter fro the phyical regiter array A to for it logical data tructure called virtual HLL ketch. Since virtual HLL ketche of all flow hare the ae regiter pool A, elephant flow can borrow eory fro all flow to utilize the unued pace. Virtual HLL Sketch Virtual HLL Sketch 2 Virtual HLL Sketch 3 Size Phyical Regiter array Size Fig. 7. Regiter haring and virtual HLL ketch. Fro above, we deign a novel peritent pread etiation architecture baed on virtual HLL ketche on top of regiter haring, called Virtual Interection HyperLogLog etiator VI-HLL, where each flow i allocated with a virtual HLL ketch of ultiple regiter in each eaureent period. Suppoe the total eory ize of A i M bit, and the ize of each regiter i h bit. So the nuber of regiter in A i = M h. Each virtual HLL ketch i configured a unified ize that i large enough to accoodate all flow. For each flow dt, we randoly elect regiter fro A to for it virtual HLL ketch A dt. The

18 4:8 Y. Zhou et al. Table. Notation A a phyical array of regiter A j a phyical regiter array of period j nuber of regiter in phyical regiter array A dt virtual HLL ketch of flow dt nuber of regiter ued by virtual HLL ketch H i dt hah function that ap the ith regiter of A dt to A n nuber of peritent eleent of flow dt ˆn an etiation of n n nuber of peritent eleent in A dt ˆn an etiation of n n u nuber of peritent eleent in A ˆn u an etiation of n u ith regiter in A dt, denoted by A dt [i], can be elected fro A a follow, A dt [i] = A[H i dt], i <, 38 where H i i a hah function whoe range i [,. The hah function H i can be ipleented uing one ater hah function H, H i dt = Hdt R[i], i <, 39 where H i a hah function whoe range i [,, i the XOR operator, and R i an array of rando eed. In the next ubection, we will introduce our VI-HLL architecture to etiate the peritent pread iultaneouly for ultiple flow. The architecture include two coponent; one for recording flow eleent in A, and the other for etiating the peritent pread for an arbitrary flow dt. The frequently ued notation are uarized in Table for quick reference..3 Record Flow Eleent in A In each tie period, a regiter array A of regiter i ued to record eleent inforation of all flow. At the beginning of each period, all regiter of A are initialized to zero. In technical dicuion below, we again conider per-detination flow through a router that eaure the ditinct nuber of ource addree in each flow. When a packet arrive, the router extract it flow label dt and treat the ource addre rc a an eleent of flow dt. The router record the eleent in the flow virtual HLL ketch A dt. To do o, it firt perfor a hah Hrc, whoe binary repreentation i denoted a x. Let p i the leading b b = log 2 bit in x, and q i the reaining bit: p = x x 2...x b, q = x b+ x b Uing the value of p, the router ap the eleent rc of flow dt peudorandoly to a regiter of it virtual HLL ketch A dt [p], and update the value A dt [p] if it current value i aller than ρq, A dt [p] = ax A dt [p],ρq. 4

19 Peritent Spread Meaureent for Big Network Data Baed on Regiter Interection 4:9 ALGORITHM 2: Online recording odule for one tie period Initialize a regiter array A of ize with all zero; for package rc, dt do x := Hrc; p := x x 2...x b ; q := x b+ x b+2... ; i := Hdt R[p]; A[i] = ax A[i],ρq ; end return A at the end of the eaureent period Applying 38 and 39, we have A[Hdt R[p]] = ax A[Hdt R[p]],ρq. 4 The online recording odule for one tie period i uarized in Algorith 2. At the end of each eaureent period, the phyical regiter array A will be offloaded fro on-chip SRAM to ain eory of a erver for long-ter torage and offline query. Aue that we have eaured t conecutive tie period, thereby we have t phyical regiter array, which are denoted by A,A 2,...,A t..4 VI-HLL Etiator We decribe our VI-HLL etiator, which ue the equence of phyical regiter array A,A 2,...,A t, to etiate the peritent pread for an arbitrary flow. Conider a flow dt under query, we recontruct it virtual HLL ketch M fro an arbitrary phyical regiter array A, where the ith regiter in virtual HLL ketch ha been apped to the regiter A[H i dt] in A, M[i] = A dt [i] = A[H i dt], i <. So M = A dt [],A dt [],...,A dt [ ] = A[H dt],a[h 2 dt],...,a[h dt]. Since we have t phyical regiter array A,A 2,...,A t, we can recontruct t virtual HLL ketche, denoted a M,M 2,...,M t. Then we have the virtual interection HLL ketch M : M = M M 2 M t. An intuitive ethod i that we can apply our previou I-HLL etiator on M,M 2,...,M t and M in Definition 4., to filter the tranient eleent and etiate the cardinality of peritent eleent in the virtual HLL ketch. However, thi intuitive ethod will caue overetiating proble for the peritent pread n of flow dt. Thi i becaue the virtual HLL ketch of flow dt not only record the peritent eleent belonging to flow dt, but alo contain the peritent eleent coing fro other flow due to the echani of regiter haring. Specifically, if oe of regiter hared with other flow happen to be et by oe peritent eleent of thee flow, then thee regiter will be updated in all virtual HLL ketche M,M 2,...,M t in all t period uch that they are recorded in M. Therefore, the peritent eleent introduced by regiter haring, called noie, caue the overetiation proble when etiating the peritent pread of the flow dt with I-HLL etiator. Our VI-HLL etiator i to reove the noie that coe fro other flow, and give unbiaed peritent pread etiation. Let n be the nuber of peritent eleent of flow dt, n be the nuber of peritent eleent recorded in virtual interection HLL ketch M of flow dt, and n u be the nuber of peritent eleent in phyical interection regiter array A = A A 2 A t. Due to regiter haring, we know that n i the peritent pread n of flow dt plu the noie peritent pread introduced by other flow. Let Y be a rando variable for the nuber of noie peritent pread recorded by the virtual

20 4:2 Y. Zhou et al. ALGORITHM 3: VI-HLL peritent pread etiator for flow dt Input:,, {M j} j [,t] and {A j} j [,t]. Step : Obtain the virtual interection HLL ketch of flow dt: M M M 2 M t. Etiate n in M by Definition 4.: ˆn := f t,m,{m j} j [,t] ; Step 2: Obtain the interection HLL ketch of all flow: A A A 2 A t. Etiate n u in A by Definition 4.: ˆn u := f t,a,{a j} j [,t] ; Step 3: Reove noie and obtain etiation of n by 4: ˆn ˆn u ; ˆn := return the etiated peritent pread ˆn. interection HLL ketch M, then we have Y = n n. 42 To recover n fro virtual HLL ketche of flow dt, we reove uch noie a follow. The total nuber of peritent eleent coing fro other flow i n u n. Fro the view of the flow dt, thee eleent are noie. A we aue that there are any flow and n n u, each noie eleent fro other flow ha approxiately the ae probability to ap into M. Thi probability i equal to due to the rando election of regiter by the virtual HLL ketch fro A regiter. Hence, Y follow a binoial ditribution, Y Binon n,. The expected nuber of noie eleent apped to M i EY = n u n. Therefore, we have En n = EY = n u n. 43 By the law of large nuber in probability theory, if the nuber of i large, the relative variance Var n n En n approache to zero. In thi cae, the expected value En n can be approxiated by an intance value, n n. Hence, we have n n n u n n n n u. 44 Baed on Definition 4., we can obtain accurate etiation ˆn and ˆn u over M and A, repectively. ˆn = f t,m,{m j } j [,t], ˆn u = f t,a,{a j } j [,t]. Therefore, we obtain the etiate for peritent pread n : ˆn = ˆn ˆn u. 4 The VI-HLL etiator for flow dt i uarized in Algorith 3.. Accuracy Analyi We now analyze the relative bia and relative tandard error of our VI-HLL etiator. According to analyi of I-HLL etiator in Section 4.3, we have the following theore.

21 Peritent Spread Meaureent for Big Network Data Baed on Regiter Interection 4:2 Theore.. Let n be the nuber of peritent eleent that are apped to the virtual HLL ketch. Suppoe the nuber of regiter i large enough. Then, where ψ i a variance related to and n. Eˆn n Varˆn n 2 StdErr ˆn n ψ 2 ψ 46.. Relative Bia. According to 42, we know that n = n +Y, and Y follow a binoial ditribution of Binon u n,. Cobing Theore., under the condition of Y = l, l [,n u n ], we have and By 47 and 48, we can calculate Eˆn = Eˆn Y = l n = n +l, 47 n PY = l = u n l n u n l= Eˆn Y = l PY = l n u n l= n +l n u n l n u n l. 48 l l n u n l = n +EY = n + n u n. 49 The value of ˆn u i etiated baed on the phyical regiter array A. Therefore, Eˆn u n u. Fro the definition in and etiation forula 4, the relative bia of ˆn i Bia ˆn n = E ˆn Eˆn = n n + n u n n Hence, the VI-HLL etiator ˆn i approxiately unbiaed for n. n Eˆn u n ˆn u n =...2 Relative Standard Error. Next we derive the relative tandard error of ˆn. Under the condition of Y = l, by Theore., we have Siilarly, we have Varˆn Y = l n +l 2 ψ 2 Varˆn u n u 2 ψ 2, 2