Distinct Random Sampling from a Distributed Stream

Transcription

1 Ditinct Random Sampling from a Ditributed Stream Yung-Yu Chung and Srikanta Tirthapura 2215 Coover Hall, Iowa State Univerity, Ame, IA, USA {ychung,nt}@iatate.edu Abtract We conider continuou maintenance of a random ample of ditinct element from a maive data tream, whoe input element are oberved at multiple ditributed ite that communicate via a central coordinator. At any point, when a query i received at the coordinator, it repond with a random ample from the et of all ditinct element oberved at the different ite o far. We preent the firt algorithm for ditinct random ampling from a ditributed tream. We alo preent a lower bound on the expected number of meage that mut be tranmitted by any ditributed algorithm, howing that our algorithm i meage optimal to within a factor of four. We preent extenion to liding window, and experimental reult howing the performance of our algorithm on real-world data et. Keyword-Ditinct Sampling, Random Sampling, Ditributed Stream Mining, Ditinct Element I. INTRODUCTION A random ample i a flexible and general ynopi of data, and can be ued to anwer aggregate querie on data with provable probabilitic guarantee on the quality of the anwer. A ditinct random ample (commonly known a a ditinct ample ) i choen from the et of all ditinct element in data. A ditinct ample ha the property that the probability of an element incluion in the ample i independent of the frequency of the element, a long a it occur within the data. Thi property i ueful in computing an important cla of aggregate that depend on the et of ditinct element in data. For intance, a ditinct ample can be ued in anwering the querie: what i the number of ditinct IP addree oberved within a tream of IP packet during a given hour, or how many ditinct viitor have ued a web ervice, who come from a particular country?, or what i the average age of the ditinct uer acceing thi webite? In contrat with a ditinct ample, a imple random ample i choen from the et of all occurrence within the data. The greater the frequency of an element in data, the more likely it incluion within a imple random ample. A imple random ample i not uited to etimate anwer to the above querie, while a ditinct ample i. A imple random ample favour the heavy-hitter within data, while a ditinct ample doe not. In cae where the ditribution of element i kewed, i.e. a few ditinct element account for a majority of the data element, a imple random ample will likely ignore element with low frequency and only contain element of high frequency (the heavy hitter ). In contrat, a ditinct random ample i equally likely to contain low frequency a well a high frequency element. Due to thi property, a ditinct ample of a data tream ha found application in databae query proceing [1], network monitoring [2], and in detecting anomalou network traffic [3]. Ditinct ampling i a fundamental primitive in current day big data ytem. Query optimizer in commercial relational databae ytem maintain a ditinct ample of individual databae column to help in chooing efficient query proceing trategie [4]. In tream databae ytem, ampling play a fundamental role a a data filtering method to reduce proceing load, and typically a variety of ampling method are implemented, including ditinct ampling [5]. Sytem uch a BlinkDB [6] ue ampling to proce querie quickly, in exchange for approximate accuracy guarantee, and have been ucceful in providing dramatic peedup for common aggregate querie. Thee ytem operate on ten of terabyte of data, pread over hundred of machine [6], and ditributed ampling i a key component of their query execution trategy. We conider the maintenance of a ditinct ample from a tream whoe element are oberved at multiple ditributed ite that communicate via a network. We preent a memory and communication-efficient olution uited to today tream proceing framework uch a IBM Infophere Stream [7], Apache Spark [8] and Apache Storm [9]. For the purpoe of decribing and analyzing the algorithm, we work in the continuou ditributed monitoring model [10] [13], where a coordinator node i required to continuouly maintain an aggregate function over the union of all tream oberved at the different ditributed ite. For ditributed proceing of large tream, the bottleneck i often the network, rather than proceor peed, and hence we focu on minimizing the meage complexity of the treaming algorithm. Sampling from a tream ha a long hitory of reearch, tarting from the popular reervoir ampling algorithm due to Waterman (ee Algorithm R from [14]) that ha been known ince the Thi include work on peeding up reervoir ampling [14], weighted reervoir ampling [15], ampling over a liding window and tream evolution [16] [19], and ditributed random ampling [10], [12]. While there i prior work on ditinct ampling from a ingle tream [1], [20] [22], thee do not extend to continuou ditributed monitoring.

2 A. Contribution We make the following contribution. Let k denote the number of ditributed ite, the deired ample ize, and d the total number of ditinct element in the entire ditributed tream. We preent a ditributed algorithm for continuouly maintaining a ditinct ample from a ditributed tream whoe expected meage complexity i O ( ) k ln d. When a query i poed at the central coordinator, it repond with a ditinct ample of ize min{, d} from the et of all element een o far in the tream. We preent a matching lower bound howing that any algorithm for continuouly maintaining a ditinct ample from the ditributed tream mut have a meage complexity of Ω ( ) k ln d. In other word, the meage complexity of our algorithm i optimal to within a contant factor. We preent an extenion to upport querie over a liding window of the ditributed data tream. We how thi algorithm i alo near-optimal in it meage complexity through demontrating an appropriate lower bound. We implemented our algorithm and evaluated it performance over real-world dataet. We found that our algorithm are eay to implement and provide a meage complexity that i typically much better than the wort cae bound that we derive. Comparion with Simple Random Sampling: It i intereting to compare the meage complexity of ditributed ditinct ampling (DDS) to that of ditributed imple random ampling (DRS). From previou work [10], [12], it i known that the meage complexity of DRS for a ample ize for n total element ditributed over k ite increae a approximately max{k, } log(n/) 1 In contrat, the meage cot of DDS increae a k log(d/), where k i the number of ite, d the total number of ditinct element, and the ample ize. The meage complexity of DDS i inherently larger than that of DRS if d (the number of ditinct element in the tream) i comparable to n (the total number of element in the tream). To undertand the reaon for thi difference, note that in DRS, when n element have already been received in the ytem, the probability of a new element being elected into the ample decreae a /n. In cae of DDS, the probability of a new element being elected into the ample decreae a /d, where d i the number of ditinct element received by the ytem. Due to the poibility of the ame element appearing at different ite at a imilar time, many ite may be fooled into ending meage to the coordinator after imultaneouly oberving the ame element, and hence more ( ) 1 k log(n/) The exact expreion i Θ if < k/8 and Θ( log(n/)) log(k/) if k/8. Thi meage complexity i tight up to a contant factor, due to matching lower bound. meage may be communicated between the ite and the coordinator in DDS. Our lower bound how that thi i not an artifact of an inferior algorithm for DDS. Intead, greater coordination i inherently neceary for DDS than for DRS. B. Prior and Related Work There i a growing literature on algorithm for continuou ditributed monitoring, tarting from the baic countdown problem [23], frequency moment [11], [23], entropy [24], heavy-hitter and quantile [25], geometric method for monitoring [26], [27], and often, matching lower bound [11]. For a recent urvey on reult in thi model, ee [13]. A urvey of ampling on tream appear in [28]. The cloet prior work i the work on algorithm and lower bound for ditributed random ampling [10], [12]. While the problem are different, our algorithm for DDS on an infinite window i eentially an adaptation of the DRS algorithm from [12] with a few important difference: one i that our algorithm ue ampling baed on a randomly choen hah function while the algorithm for DRS ample baed on independent random bit. Further, our algorithm employ additional data tructure to prevent duplicate meage from a ite to the coordinator due to oberving the ame element, thi i not neceary for imple random ampling. The analyi of the meage complexity i different for the DRS and the DDS algorithm. The lower bound argument for the two problem (DRS and DDS) are very different from each other. Our analyi of the lower bound on meage cot of any algorithm for DDS ue method, that to our knowledge, have not been applied to the continuou ditributed treaming model. A related model of ditributed tream wa conidered in [20], [29]. In thi model, the coordinator wa not required to continuouly maintain an aggregate, but intead, when the query wa poed to the coordinator, the ite would be contacted and the query reult would be contructed. In their model, the coordinator could be aid to be reactive, wherea in the model conidered in thi paper (the continuou ditributed treaming model), the coordinator i pro-active. Roadmap. We firt preent the model and problem definition in Section II, the algorithm and lower bound for infinite window in Section III, an overview of the algorithm for liding window in Section IV, and experimental reult in Section V. Due to pace contraint, we omit ome of the proof, and alo omit detail of the liding window algorithm and experimental reult. Thee detail, including all the proof, can be found in the full verion of the paper [30]. II. MODEL We conider a ytem with k ite, numbered from 1 to k. Each ite i monitor a local tream of element, which

3 are not all necearily ditinct. There i an integer time aociated with each obervation, and time i non-decreaing within a tream. At time t, let S i (t) denote the tream oberved by ite i o far, and let S(t) = k i=1 S i(t) denote the tream oberved by the ytem o far. Let D(t) denote the et of ditinct element in S(t), n(t) the number of element in S(t), and d(t) the number of ditinct element in S(t). There i a coordinator node, different from the other k ite, to whom all querie are poed. The different ite a well a the coordinator are aumed to be ynchronized in time, and we aume that the meage delay from the ite to the coordinator are all equal, a in a ynchronou ditributed ytem. Thee aumption, which have alo been ued in previou work in the continuou ditributed treaming model [12], [23], [31], allow u to focu on communication efficiency. We conider two verion of ditinct ampling, baed on the cope of the data that the query addree. In the infinite window cae, at every time intant t, the coordinator mut maintain a random ample of ize min{, d} from D(t). In the liding window cae [32], we are given a window ize w a a parameter, and the uer i intereted in all element that have arrived in the mot recent w time interval. In particular, let Si w (t) denote the tream that ha arrived at ite i at time t w + 1, t w + 2,..., t, and let S w (t) = k i=1 S i(t). Let D w (t) denote the et of ditinct element in in S w (t). The goal i for the coordinator to maintain at all time t, a random ample of ize min{, d} from the element in D w (t). Our meaure of performance i the total number of meage ent between the coordinator and the ite, and we aim to minimize thi number. Since each meage in our algorithm i of a mall ize, thi i alo a meaure of the number of byte tranmitted, in our cae. The ize 2 of local data tream S i, the order of arrival, the number of ditinct element in the local tream, and the interleaving of the tream at different ite, can all be arbitrary, and the algorithm cannot make any aumption about thee. III. INFINITE WINDOW We firt conider the cae of infinite window, when the ample ha to be choen from the et of all ditinct element een o far in the tream. Let h : U [0, 1] be a hah function that aign a real number in the range [0, 1] to each element in U. For different input, it i aumed that the output of h are mutually independent random variable. For et S, let h(s) denote {h(x) x S}. Note that for any time t, h(s(t)) = h(d(s(t))). The baic ampling trategy i a follow. The ditinct ample at time t i the et of element from S(t) that yield the mallet element in h(s(t)). It i clear that the above yield a ditinct ample from S(t). To ee thi, take any ubet T D(S(t)) of ize 2 The meage ize in our algorithm i contant, auming that each tream element can be tored in a contant number of byte. ; the probability that the element in T will yield the mallet value in h(d(s(t))) i exactly the probability that in a random permutation of h(d(s(t))), the element in T are ordered before the ret, which i 1/ ( ) D(t). Our ditributed algorithm for maintaining the above ample i a follow. Let u(t) denote the value of the th mallet element in h(s(t)). The coordinator alway ha the current value of u(t). Each ite i maintain a tate variable u i (t), which i it local view of u(t). u i (t) i initialized to 1 and i updated a follow. Whenever ite i oberve an item e uch that h(e) < u i (t), e and h(e) are ent to the coordinator, who update u(t). If h(e) indeed changed the value of u(t), then e i elected into the ample at the coordinator, replacing a current element in the ample (unle fewer than ditinct element have been een o far). In turn, the coordinator end a meage back to i to refreh the value of u i (t). The algorithm at ite i i preented in Algorithm 1 and at the coordinator i in Algorithm 2. Algorithm 1: Infinite Window: Algorithm at Site i Initialization: Receive hah function h from the coordinator; u i 1, P i ; repeat if receive element e from S i then if h(e) < u i and e P i then Inert e into P i ; Send e to the Coordinator; if receive element u from the coordinator then u i u; Dicard all element e in P i uch that h(e ) > u i ; until Forever; Lemma 1. When queried at time t, the coordinator return a random ample of ize min{, d} elected without replacement from D(t). Proof: Firt note that at time t, variable u i track u i (t) at ite i, and u track u(t) at the coordinator. For each ite i we claim u i u at all time. Thi can be proved uing induction. For the bae cae, note that initially, u i = u. There are two type of event to conider for the inductive tep: (1) when u u updated, and (2) when u i i updated. Each time u i updated, it only become maller (i.e. u i non-increaing), which maintain the invariant. Each time u i i updated, it i et to the current value of u, which alo maintain the invariant. Auming that hahe of different element are ditinct, we claim that the value of u equal the lth mallet hah value een o far, where l = min{, d}. We claim that every

4 Algorithm 2: Infinite Window: Algorithm at the Coordinator /* P i the random ample, and variable u ha the current value of u(t). */ Initialization: P, u 1; repeat if receive e from ite i then if h(e) < u then If e P, then inert e into P ; if P > then Dicard element e = argmax{h(e) e P } from P ; Update u max{h(e) e P }; Send u to ite i; if a query for a random ample arrive then Send P ; until Forever; time a new element e arrive at ite i uch that h(e) < u, e will be ent to the coordinator. To ee thi, conider an element e uch that h(e) < u, oberved at ite i. We have u u i, and h(e) < u; thee together imply that h(e) < u i. From the algorithm at the ite, it i clear that e will be ent to the coordinator. Thu, the coordinator can inductively maintain the l element from D(t) with the mallet hah value. It i clear thi contitute a random ample of ize l choen without replacement from D(t). A. Infinite Window: Analyi We preent an analyi of the meage and pace complexitie of our algorithm. In Section III-A1 we preent an upper bound on the meage complexity of our algorithm and in Section III-A2 we preent a lower bound. 1) Upper Bound: Conider the execution of the algorithm until the end of time tep t. Let d = D(t) be the total number of ditinct element that were oberved in the tream. Let d i denote the total number of ditinct element in S i (t). Clearly, d i d. Let Y i denote the total number of meage that are ent by node i (note that the number of meage ent by ite i equal the number of meage received). Let Y denote the total number of meage ent in the ytem. In the following, when the context i clear, we ue S i to mean S i (t), D i to mean D i (t), and o on. Lemma 2. An element e oberved at ite i multiple time will not lead to more than one meage tranmiion from i to the coordinator, and thi tranmiion can only happen the firt time ite i oberve e. Proof: Suppoe that e i oberved by ite i at time t 1, t 2,... where t 1 < t 2 <.... The firt time it i oberved at time t 1, uppoe that a meage wa not ent to the coordinator. In thi cae, h(e) u i (t 1 ). Since u i i nonincreaing, when e i oberved at a future time, ay t 2, it will alo be true that h(e) u i (t 2 ), and no further meage are ent to the coordinator. Next, uppoe that a meage wa ent to the coordinator when e wa oberved at time t 1. In thi cae e i inerted into P i at time t 1. When e i oberved again at time t 2 (or later), either e i till in P i, or it mut be true that u i (t 2 ) < h(e), ince otherwie, e would not have been dicarded from P i. Hence, e will not be ent to the coordinator in thi cae, according to Algorithm 1. For j = 1... d i, let e j i be the jth new ditinct element in the local tream S i. For j = 1... d i, let Y j i be a random variable equal to 1 if ite i communicated with the coordinator upon receiving e j i, and 0 otherwie. We have Y i = d i j=1 Y j i. [ ] Lemma 3. For ite i, and j = 1... d i, Pr Y j i = 1 j, if j >. Proof: Let Z j i denote the event that h(e j i ) i among the mallet element in {h(e q i ) q = 1... j}. Note that if Y j i = 1, then Z j i mut be true. Note that the convere i not necearily true; it i poible that Z j i i true, but Y j i = 0, for example, e j i may have already been oberved by a ite other than i, and it value may have been incorporated [ ] into u, and hence into u i. It i eay to ee that Pr Z j i = [ ] [ ] j for j >. Since Pr Y j i = 1 Pr Z j i, the lemma follow. Lemma 4. E [Y i ] + (H di H ) Proof: Uing linearity of expectation on Y i, we get: d i E [Y i ] = = j=1 j=1 + [ E Y j i ] d i = j=1 [ ] Pr Y j i = 1 d i j=+1 where we have ued Lemma 3. + [ ] Pr Y j i = 1 d i j=+1 [ ] Pr Y j i = 1 j = + (H d i H ) Lemma 5. Let Y denote the number of meage tranmitted by the ditributed algorithm during an execution when d ditinct element are oberved overall. E [Y ] 2k + 2k (H d H ) 2k ( 1 + ln ( )) d Proof: Note that for each meage ent by a ite, there i exactly one meage ent by the coordinator, and hence we have Y = 2 k i=1 Y i. The proof follow from Lemma 4 combined with the obervation d i d.

5 We remark that uing Lemma 4, it i poible to get a tighter upper bound for E [Y ] in cae when the number of ditinct element oberved at individual ite i much maller than the number of ditinct element overall. Obervation 1. E [Y ] 2k + 2 k (H di H ) 2k + 2 i=1 k ( ) di ln i=1 The following theorem ummarize the performance of the algorithm for infinite window. Theorem 1. Let d,, and k repectively denote the total number of ditinct element in the ditributed tream, ample ize, and the number of ite. There i an algorithm that continuouly maintain a ditinct ample of a ditributed tream, whoe expected total number of meage i no more than 2k ln ( ) de, memory conumption per ite i O(), memory conumption at the coordinator i O(), and proceing time per element i O(1). Proof: The meage complexity follow from Lemma 5. The memory at the coordinator i alo clear from an inpection of Algorithm 2. For the memory at ite i, note that a oon a an element e i inerted into P i, e i alo ent to the coordinator, and in turn the ite receive the current value of u, et u i u, and dicard all element e from P i uch that h(e) > u. Globally, there are no more than ditinct element e uch that h(e) u, hence even within ite i, there are no more than element in P i. 2) Lower Bound: Given any ditributed algorithm A that continuouly maintain a ditinct ample of the tream of ize, we contruct an input which caue A to end at leat a certain minimum number of meage, in expectation. Suppoe that the element were all choen from the et [m] = {1, 2,..., m} for m >> k. Note that the probability pace here i the one from which the random ample i choen; every algorithm need acce to random bit that define thi probability pace, uch that the ample choen i uniformly choen from the et of all ditinct element oberved o far. Lemma 6. Suppoe a et D of ditinct element of ize d ha already been oberved by the ytem o far, after ome round of computation. For any algorithm A, and any ite i = 1... k, there i an element e D i uch that upon receiving any element e [m] e D i D in the next round, ite i will end a meage to the coordinator with probability at leat 2(d+1). Proof: We ue proof by contradiction. Note that et D ha already been oberved by the ytem o far. Suppoe that there were two ditinct element e i, e i [m] D uch that upon e i, the probability that i ent a meage to the coordinator wa le than 2(d+1), and alo upon e i, the probability that i ent a meage to the coordinator wa le than 2(d+1). Conider the following two input in the next round. In one input I 1, ite i i given e i and the other ite do not receive any element. In the other input I 2, ite i i given e i and the other ite do not receive an element. In I1, at the end of thi round, there i a probability of d+1 that e i belong to the random ample. In I 2, at the end of thi round, there i a probability of d+1 that e i belong to the random ample. Thu, with probability at leat d+1, the ample at the coordinator after oberving I 1 i different from the ample after oberving I 2. However, in thi round, the coordinator oberve a change in execution between I 1 or I 2 only if either (1) ite i end a meage to the coordinator in I 1, or (2) ite i end a meage to the coordinator in I 2. The probability that at leat one of the above event happen i le than d+1, uing the union bound. Further, the behavior of the other ite ha an identical ditribution on both input, ince they did not receive any element. Thu, we have that the content of the random ample at the end of the round are different with probability at leat, but the probability that the meage oberved d+1 d+1 by the coordinator are different i le than. Thi lead to a non-zero probability that the random ample at the coordinator i incorrect at the end of thi round, and hence a contradiction. Hence, there can be at mot one element e i uch that upon receiving an element e [m] e i D, the probability of ending a meage to the coordinator i at leat 2(d+1), and the lemma follow. Lemma 7. Suppoe the et of ditinct element oberved o far i D, and let d = D, and d. For any algorithm A, there exit another round of input I(D) uch that after oberving I(D), the ite will end at leat an expected k 2(d+1) element to the coordinator. Proof: The input I(D) i contructed a follow. Let e be any element uch that e ( D ( k i=1 {ed i })). Element e i given to every ite in thi round. Uing Lemma 6, we get that each ite i = 1... k will end a meage to the coordinator with probability at leat follow. 2(d+1). The lemma Theorem 2. For any algorithm A, there exit an input ditributed tream, I A with d ditinct element uch that the expected number of meage ent by the algorithm upon receiving I A i at leat k 2 (H d H + 1) k 2 ln ( ) de. Proof: Input I A i contructed a follow. Let D 0 =. The input in the firt round i I(D 0 ). For i 0, let the et of all ditinct element oberved till (and including) round i be D i. Note that the ize of D i i exactly i. For i = 0... (d 1), the input in round i + 1 i I(D i ). From Lemma 7, in round i >, the expected number of k meage ent to the coordinator i at leat 2(i+1). Summing

6 thi over all round + 1,..., d, we get the number of k(h meage to be at leat d H ) 2. For round 1 till, imilar method yield that the expected number of meage ent in each round mut be at leat k 2, for the above input. Thu the expected total number of meage ent by thi algorithm i at leat k 2 (H d H + 1). B. Sampling With Replacement Thu far, we have conidered ditinct ampling without replacement. In ampling with replacement, the different ample are all choen independently and randomly from the et of ditinct element oberved o far, D(t). A poible olution to ditinct ampling with replacement i to repeat parallel copie of the ingle element ampling algorithm, each copy uing a different hah function. The correctne of thi cheme i trivial, and the meage cot i time the cot of a ingle element ampling algorithm, which i O(k log de). We can do better, uing the following obervation. When a node i receive an element e, it end only a ingle copy of e to the coordinator if it wa ampled by one or more of the parallel copie (of the ingle element ampling algorithm). We ketch the analyi of thi algorithm. For ite i and the jth ditinct element received by the ite, let Y j i be a random variable equal to 1 if the ite communicated with the coordinator upon receiving thi element, and 0 otherwie. The probability that the element i ampled ( by ) any of the. different copie of the algorithm i j Hence, ). E[Y j i (1 ] = 1 1 j If Yi denote the total number of meage ent by ite i during execution, we have uing linearity of expectation that E[Y i ] = ( ( ) ) d i j= j. Simplifying thi expreion (detail can be found in the full verion of the paper), we get E[Y i ] ln(d i e/). Thu the expected total number of meage in the ytem i bounded by O(k log(de/)), even for the algorithm for ampling with replacement. We alo note the following reduction from ditinct ampling without replacement to ditinct ampling with replacement. From a ditinct ample with replacement of ize lightly greater than, we elect a random ample without replacement. It i poible to prove that the reult i indeed a ditinct ample without replacement from the original dataet. Thu, the lower bound of Ω(k log( d )) applie to both ampling with and without replacement, and our method lead to an algorithm for ampling with replacement with optimal meage complexity. IV. SLIDING WINDOW In the liding window etting, a ample i deired over only a window of mot recent element of the tream. We aume that time i divided into lot that are numbered conecutively in an increaing equence. It i aumed that time i ynchronized acro the ite o that when ite 1 i oberving element in lot t, other ite are alo oberving element in the ame lot. Let w > 0 be an integer denoting the ize of the window. The problem i a follow: when a query i iued to the coordinator at lot t, it return a random ample of ize choen without replacement from the et of all ditinct element oberved in lot (t w + 1) till t, both endpoint incluive. We ue the term lot and time interchangeably in the following dicuion. Algorithm Idea: At time t, let S i (t, w) denote the element that have arrived in lot (t w +1) till t at ite i. Let S(t, w) denote the (multiet) union of S i (t, w), i = 1... k. Let D i (t, w) denote the et of ditinct element in S i (t, w), and D(t, w) the et of ditinct element in S(t, w). For a et of element E, let h(e) = {h(e) e E}. The highlevel algorithm idea i to chooe the element with the mallet hah value from D(t, w). Let u(t, w) denote the -th mallet hah value from h(d(t, w)). Implementation of the Idea: The challenge in implementing the liding window cenario, when compared with the infinite window cenario, i that u(t, w) i not monotonically decreaing. A element expire from the window, the value of u(t, w) may increae, and an algorithm ha to keep track of thi at both the coordinator and the ite. One poible method i a follow. Each ite i, at time t, keep track of the et of element with the mallet hah value, choen from S i (t, w). Note that thi alo form a ditinct ample from S i (t, w). Whenever thi ample change, the coordinator i informed. Since the coordinator ha all element with the mallet hah value from each ite, it can maintain element with the globally mallet hah value from D(t, w), and hence a ditinct ample from S(t, w). The meage complexity of thi algorithm depend on how often the local ample at the individual ite change. We can reduce the communication cot of the above algorithm by incorporating feedback from the coordinator to the ite a follow. Similar to infinite window, the coordinator maintain a variable u, which ha the value of u(t, w) at all time. When the coordinator replie back to a ite, it convey the current value of u. However, note that u may increae at the coordinator, due to element expiring from the window, and thi need alo to be conveyed to the ite (note that thi wa not neceary for the infinite window cae). One way to achieve thi i that each time u increae, the coordinator broadcat the new value of u to all node. While thi algorithm i correct, a broadcat baed method can be expenive, ince it communicate with node which may not be actively receiving meage. We intead ue an alternate approach, where the coordinator replie back with the value of u to a ite only when the ite communicate with the coordinator. Each ite i alo need to maintain a local liding window ample, coniting of the mallet hah value within D i (t, w). It i known that in general, maintaining the kth mallet element within a liding window require pace

7 linear in the window ize in the wort cae [32]. However, in our cae, we do not need to maintain the minima over arbitrary number, but need to do o over a equence of random number. Hence, we can ue the idea from priority ampling imilar to [17] to ignificantly reduce the pace conumption at each ite. For element e, e and time t, t, we ay that tuple (e, t) dominate tuple (e, t ) at ite i if t > t and h(e) < h(e ). Let τ i (e) denote the mot recent time when e wa oberved at ite i. For element e, e, we ay e dominate e at ite i if (e, τ i (e)) dominate (e, τ i (e )). Each ite i ha a data tructure T i coniting of all element that could potentially be included within the random ample of ditinct element either now, or in the future. T i can be maintained efficiently uing a Treap [33]. We omit detailed decription of the algorithm due to pace contraint, and preent the main reult on the performance of the algorithm. Theorem 3. There i a ditributed algorithm that continuouly maintain a ditinct random ample of ize over a time-baed liding window of ize w, with k ite and T total time tep. Suppoe M = min i,t { D i (t, w) }. The expected number of meage tranmiion between the ite and the coordinator i bounded by 2kT/M. We alo prove the following lower bound on the meage complexity of any algorithm for continuou ditinct ampling over a liding window. In particular, we how that the linear dependence of the meage complexity on T i neceary. Lemma 8. Suppoe there i only one ite and a coordinator. There exit an input tream S uch that for any ditributed algorithm A that can continuouly maintain a ditinct random ample at the coordinator over T tep, the expected number of meage ent to the coordinator i at leat Ω ( T w + log ) w. Lemma 9. The expected pace complexity of the liding window algorithm at ite i at time t i O( log D i (t, w) ) word. V. EXPERIMENTS We preent an experimental evaluation of our propoed algorithm. We ued two dataet. The firt i an OC48 Internet Trace Dataet [34], which ha anonymou traffic trace taken at a US wet coat OC48 peering link for a large ISP in 2002 and To generate an element, we conider the concatenation of the ender IP addre and the receiver IP addre. The other i the Enron Dataet [35], where an element i contructed by concatenating the ender addre and the receiver addre. A ummary of the data i hown in Table I. Each data point preented i the average of 20 independent run. We implemented the algorithm in Java, uing the MurmurHah [36] hah function. Due to pace contraint, we only preent the # Element # Ditinct OC48 42,268,510 4,337,768 Enron 1,557, ,330 Table I THE NUMBER OF ELEMENTS AND DISTINCT ELEMENTS IN OC48 IP AND ENRON DATASETS (a) OC48 IP Dataet Figure 1. The number of meage a a function of number of element under two different method of data ditribution, flooding and random, for 10 ite and a ample ize of 5. experimental reult for the OC48 dataet. The reult for the Enron dataet can be found in the full verion of the paper [30]. Our theoretical analyi wa for the wort cae, when the input data can be ditributed in an arbitrary manner by the adverary. Here we preent experimental reult for the infinite window cae, including the impact of data ditribution, of ample ize, and a comparion with an alternate, natural algorithm that i baed on a broadcat from the coordinator to the ite. Due to pace contraint, we do not preent the reult of the liding window experiment here. They can be found in the full verion. Impact of Data Ditribution: We examine the performance of our algorithm under different ditribution method. In the firt method, called flooding, each incoming element i aigned to every ite. In the econd method, called random, an incoming element i ent to a ingle ite, choen uniformly at random. In the third method, roundrobin, each element i ent to a ingle ite, and the element are aigned to ite in a round-robin manner, i.e. the j-th element i monitored by ite (j mod k)+1. The reult for random and round-robin turned out to be nearly identical, o we only preent the reult for random ditribution. Figure 1 how the number of meage a the function of the number of element for 10 ite and a ample ize of 5. From Figure 1, we oberve that at the beginning, the number of meage increae quickly, ince the ample i changing often. A more element are oberved, the rate of change of the ample decreae and fewer meage are ent. It i clear

8 Figure ite. (a) OC48 IP Dataet The number of meage a a function of the ample ize for (a) OC48 IP Dataet Figure 4. The number of meage ent by Algorithm Broadcat and our propoed method for 10 ite and a ample ize of 5. (a) OC48 IP Dataet Figure 3. The number of meage a function of the number of ite k for a ample ize of 20. that the number of meage under flooding i ignificantly larger than the number of meage under random, though the total number of ditinct element een in both input i the ame. Thi ( cenario i explained by our tighter upper bound 2k 1 + ) k di i=1 ln (ee Obervation 1), which how the influence of the number of ditinct element oberved at the individual ite on the total meage ent. Impact of Sample Size: Figure 2 how the number of meage a a function of the ample ize for 50 ite; the meage complexity increae almot linearly with the ample ize, though the lope are different for different method of data ditribution. Figure 3 how the number of meage a a function of the number of ite k for a ample ize of 20. For flooding, the number of meage increae linearly with the number of ite. However, for random ditribution, the number of meage i much maller than in cae of flooding, and i almot independent of the number of ite. Comparion with Other Algorithm: To our knowledge, there are no prior publihed method for ditinct ampling on a ditributed tream. We compare the performance of our algorithm with another natural algorithm, which we call Algorithm Broadcat. The difference between Algorithm Broadcat and our propoed method i that Algorithm Broadcat will broadcat the current value of u (which ha the value of u(t) at time t) to all ite whenever there i an update to u. Thi verion ha the advantage that fewer meage are ent from the ite to the coordinator, ince the u i are alway in ync with the coordinator. However, thi ha the downide of requiring a broadcat each time u change. In Figure 4, we preent a comparion between the two algorithm for 10 ite and a ample ize of 5. It i clear that Broadcat require ignificantly more meage than our algorithm; thi ugget that typically it i not worth keeping the different ite ynchronized with repect to the value of u. A lazy approach of refrehing u when neceary reult in fewer meage. We alo note that the meage cot of Algorithm Broadcat i linear in the number of ite k, and the ample ize. However, the lope of the Broadcat algorithm i coniderably higher. We how the comparion of the two algorithm a a function of the ample ize in Figure 6. Similar reult are oberved with the dependence on the number of ite. Impact of Skew: We next conider the influence of the non-uniformity in the ize of the tream oberved at different ite. Here, we contruct a ditributed input which i kewed toward a ingle ite that dominate over the other ite in term of the number of ditinct element that it oberve. Each input element i ent to only one ite; but intead of dealing it randomly or in a round-robin fahion, we end the element to ite 1 with a probability that i a factor α time the probability that a ite other than 1 i choen. We call thi factor a the kew. For example, if the kew i 200, then ite 1 i 200 time more likely to receive an element than another ite. Figure 7 how the relation

9 between number of meage tranmiion and the kew for different algorithm with 20 ite and a ample ize of 20. The number of meage tranmitted reduce a the kew increae; thi can be anticipated through Obervation 1 that preent an analyi in term of the number of ditinct element oberved at each ite. Note that the higher the kew, the cloer thi get to centralized tream monitoring. (a) OC48 IP Dataet Figure 5. The number of meage ent by Algorithm Broadcat and our propoed method on different ample ize with 50 ite. (a) OC48 IP Dataet Figure 6. The number of meage ent by Algorithm Broadcat and our propoed method on different number of ite (k) with a ample ize of 20. VI. CONCLUSION We preent new meage-efficient algorithm for ditinct random ampling on a ditributed data tream. Our algorithm are practical, eay to implement, and have nearoptimal expected meage complexity. Surpriingly, the expected meage complexity of maintaining a ditinct random ample from a ditributed manner i inherently greater than that of maintaining a imple random ample from a ditributed tream. Our analyi of the algorithm ha ued the ynchronou model of a ditributed ytem, ince it i clearer to analyze the meage complexity of a ditributed computation within a ynchronou ytem than within an aynchronou ytem. However, our algorithm are not dependent on there being ynchronou lock-tep execution of proceor within the ytem. In particular, the paradigm of chooing the element with the mallet hah value a the random ample continue to work in the preence of timing inconitencie, even if meage are delayed or reordered during tranmiion. Some direction for future work are a follow. Our lower bound for liding window are not tight, and it i intereting to obtain an optimal algorithm for DDS on liding window either through a better lower bound or a better algorithm. It i alo intereting to derive ditributed ampling algorithm for more complex tructure on data, uch a tructure on graph. REFERENCES [1] P. B. Gibbon, Ditinct ampling for highly-accurate anwer to ditinct value querie and event report, in Proceeding of the 27th International Conference on Very Large Data Bae (VLDB), 2001, pp [2] G. Cormode and M. Garofalaki, Sketching tream through the net: ditributed approximate query tracking, in Proceeding of the 31t International Conference on Very Large Data Bae. VLDB Endowment, 2005, pp [3] S. Venkataraman, D. X. Song, P. B. Gibbon, and A. Blum, New treaming algorithm for fat detection of uperpreader, in Proceeding of the Network and Ditributed Sytem Security Sympoium, NDSS, (a) OC48 IP Dataet Figure 7. The number of meage ent by Algorithm Broadcat and our propoed method, a a function of the kew, for 20 ite and a ample ize of 20. [4] A. Poddar, One pa ditinct ampling, file.wordpre.com/2011/12/one-pa-ditinct-ampling.pdf. [5] T. Johnon, S. Muthukrihnan, and I. Rozenbaum, Sampling algorithm in a tream operator, in Proceeding of the ACM SIGMOD International Conference on Management of Data, 2005, pp

10 [6] S. Agarwal, B. Mozafari, A. Panda, H. Milner, S. Madden, and I. Stoica, Blinkdb: querie with bounded error and bounded repone time on very large data, in Eighth Euroy Conference, 2013, pp [7] IBM Corporation, Infophere tream, com/oftware/product/en/infophere-tream, acceed Jan [8] M. Zaharia, T. Da, H. Li, T. Hunter, S. Shenker, and I. Stoica, Dicretized tream: Fault-tolerant treaming computation at cale, in Proc. ACM Sympoium on Operating Sytem Principle (SOSP), 2013, pp [9] Apache Software Foundation, Storm: Ditributed and faulttolerant realtime computation, project/torm.html. [10] G. Cormode, S. Muthukrihnan, K. Yi, and Q. Zhang, Continuou ampling from ditributed tream, Journal of the ACM, vol. 59, no. 2, pp. 10:1 10:25. [11] D. P. Woodruff and Q. Zhang, Tight bound for ditributed functional monitoring, in Proceeding of the Forty-fourth Annual ACM Sympoium on Theory of Computing, er. STOC 12, pp [12] S. Tirthapura and D. P. Woodruff, Optimal random ampling from ditributed tream reviited, in Proc. International Sympoium on Ditributed Computing (DISC), 2011, pp [13] G. Cormode, The continuou ditributed monitoring model, SIGMOD Record, vol. 42, no. 1, pp. 5 14, May [14] J. S. Vitter, Random ampling with a reervoir, ACM Tranaction on Mathematical Software, vol. 11, no. 1, pp , [15] P. S. Efraimidi and P. G. Spiraki, Weighted random ampling with a reervoir, Information Proceing Letter, vol. 97, no. 5, pp , [16] V. Braverman, R. Otrovky, and C. Zaniolo, Optimal ampling from liding window, Journal of Computer and Sytem Science, vol. 78, pp , [17] B. Babcock, M. Datar, and R. Motwani, Sampling from a moving window over treaming data, in Proceeding of the Thirteenth Annual ACM-SIAM Sympoium on Dicrete Algorithm, 2002, pp [18] R. Gemulla and W. Lehner, Sampling time-baed liding window in bounded pace, in Proceeding of the 2008 ACM SIGMOD International Conference on Management of Data, er. SIGMOD 08, pp [19] C. Aggarwal, On biaed reervoir ampling in the preence of tream evolution, in Proceeding of the 32nd International Conference on Very Large Databae, 2006, pp [20] P. Gibbon and S. Tirthapura, Etimating imple function on the union of data tream, in Proc. ACM Symp. on Parallel Algorithm and Architecture (SPAA), 2001, pp [21] S. Ganguly, M. Garofalaki, and R. Ratogi, Tracking etexpreion cardinalitie over continuou update tream, The VLDB Journal, vol. 13, no. 4, pp , [22] G. Frahling, P. Indyk, and C. Sohler, Sampling in dynamic data tream and application, in Proceeding of the Twentyfirt Annual Sympoium on Computational Geometry, er. SCG 05. ACM, pp [23] G. Cormode, S. Muthukrihnan, and K. Yi, Algorithm for ditributed functional monitoring, in Proceeding of the Nineteenth Annual ACM-SIAM Sympoium on Dicrete Algorithm, 2008, pp [24] C. Arackaparambil, J. Brody, and A. Chakrabarti, Functional monitoring without monotonicity, in Proceeding of the 36th International Colloquium on Automata, Language and Programming: Part I, 2009, pp [25] K. Yi and Q. Zhang, Optimal tracking of ditributed heavy hitter and quantile, Algorithmica, vol. 65, no. 1, pp , [26] I. Sharfman, A. Schuter, and D. Keren, A geometric approach to monitoring threhold function over ditributed data tream, ACM Tranaction on Databae Sytem, vol. 32, no. 4, [27] N. Giatrako, A. Deligiannaki, M. N. Garofalaki, I. Sharfman, and A. Schuter, Prediction-baed geometric monitoring over ditributed data tream, in Proceeding of the 2012 ACM SIGMOD International Conference on Management of Data, er. SIGMOD 12, pp [28] B. Lahiri and S. Tirthapura, Stream ampling, in Encyclopedia of Databae Sytem, 2009, pp [29] P. B. Gibbon and S. Tirthapura, Ditributed tream algorithm for liding window, in Proceeding of the Fourteenth Annual ACM Sympoium on Parallel Algorithm and Architecture, er. SPAA 02, pp [30] Y.-Y. Chung and S. Tirthapura, Ditinct random ampling from a ditributed tream, nt/ pub/dd.pdf, Oct [31] G. Cormode, S. Muthukrihnan, K. Yi, and Q. Zhang, Optimal ampling from ditributed tream, in Proceeding of the Twenty-ninth ACM SIGMOD-SIGACT-SIGART Sympoium on Principle of Databae Sytem, 2010, pp [32] M. Datar, A. Gioni, P. Indyk, and R. Motwani, Maintaining tream tatitic over liding window, SIAM Journal on Computing, vol. 31, no. 6, pp , [33] R. Seidel and C. R. Aragon, Randomized earch tree, in Algorithmica, vol. 16, no Springer-Verlag, 1996, pp [34] CAIDA OC48 Trace Project, The caida ucd oc48 internet trace dataet - ( ), M=CAIDA+OC48+Trace+Project, [35] CALO Project, Enron dataet, edu/ enron/. [36] V. Holub, Murmur hah 2.0, holub/ w/javamurmurhah/murmurhah.java.