Space- Efficient Es.ma.on of Sta.s.cs over Sub- Sampled Streams

Transcription

1 Space- Efficient Es.ma.on of Sta.s.cs over Sub- Sampled Streams Andrew McGregor University of Massachuse7s A. Pavan Iowa State University Srikanta Tirthapura Iowa State University David Woodruff IBM Research Almaden

2 Sampled IP Packet Streams IP Traffic Monitoring 40Gbps 100s of Gbps Cisco NeOlow standard for monitoring Sampled NeOlow Network Monitor sees only a random sample of the original packet stream Different Types of Sampling Used IETF Working Group (psamp) EsTmaTon from Sub- Sampled Streams 2

3 Sampled Streams What can we compute over a stream by observing only a random sample of the stream? Two Constraints: Only observe a random sample Streaming, Memory Bound ComputaTon EsTmaTon from Sub- Sampled Streams 3

4 Model: Bernoulli Sampling Original Stream P a 1,a 2,...,a n a i {1, 2, 3,...,m} Bernoulli Sampler selects each a i with probability p Sampled Stream L Aggregates and Analysis of Original Stream P EsTmaTon from Sub- Sampled Streams 4

5 Aggregates The stream is a sequence of items What ma7ers is the vector frequency of i f = (a 1,a 2,...,a n ) f 1, f 2,..., f m where f i is Frequency Moments Number of DisTnct Elements Empirical Entropy Heavy Hi7ers H( f ) = F k ( f ) = m i=1 m i=1 f i k f i n lg n f i EsTmaTon from Sub- Sampled Streams 5

6 Preliminaries Let g i be the frequency of item i in the substream g i = B(f i,p) L contains the frequency vector g 1,g 2,...,g m Randomized multplicatve approximaton, for parameters 1 Pr α X X α 1 δ EsTmaTon from Sub- Sampled Streams 6

7 Results Number of DisTnct Elements (Known) Upper Bound on Result Quality Simple Streaming Algorithm that meets the bound Frequency Moments, F k, k > 0 Smaller values of p increase streaming space complexity Matching upper and lower bounds (w.r.t p) Tradeoff between processing Tme and space Entropy Matching Upper and Lower Bounds for AddiTve Error RelaTve Error Impossible in small space Heavy Hi7ers Sampling is a good fit EsTmaTon from Sub- Sampled Streams 7

8 Related Work Duffield, Lund, Thorup, ProperTes and predicton of flow statstcs from sampled packet streams, IMC 2002 Duffield, Lund, and Thorup, EsTmaTng flow distributons from sampled flow statstcs, SIGCOMM 2003 Rusu and Dobra, Sketching sampled data streams ICDE 2009 Bar- Yossef, Sampling Lower Bounds via InformaTon Theory, STOC 2003 EsTmaTon from Sub- Sampled Streams 8

9 Results: Number of DisTnct Elements For constant p, estmate F 0 (P) from observing L Any algorithm must have relatve error Ω 1 (Charikar et al. PODS 2000) p A simple streaming algorithm has relatve error Ο 1 p Other EsTmators, such as GEE (Generalized Error EsTmator) also possible in single pass EsTmaTon from Sub- Sampled Streams 9

10 Frequency Moments F k Theorem (Upper Bound): There is a one pass streaming algorithm which observes L and outputs a (1 +, δ) - estmator to F k (P ) where using space. assuming k 2 Õ 1 p m1 2/k p = Ω(min(m, n) 1/k ) EsTmaTon from Sub- Sampled Streams 10

11 CompuTng F 2 : Why is This Hard? n n sampler F 2 (P ) n + n =2n p n pn F 2 (L) =p 2 n + pn =(p 2 + p)n EsTmaTon from Sub- Sampled Streams 11

12 Algorithm 1 for F 2 Z = F 2(L) F 1 (L) p 2 E[Z] =F 2 (P ) Algorithm F 2 (L) 1. EsTmate using streaming algorithm on L 2. Use in above formula for Z EsTmaTon from Sub- Sampled Streams 12

13 Algorithm 1 for F 2 (contd) To get needed is (1 +, δ) Õ - approximaton for Z, space Issue: Need to estmate F 2 (L) with very high accuracy to get good relatve error for F 2 (L) F 1 (L) p 2 1 p 2 EsTmaTon from Sub- Sampled Streams 13

14 Algorithm 2 for F 2 Collisions. The number of 2- wise collisions in P m fi C 2 (P )= 2 i=1 ObservaTon: F 2 (P )=2C 2 (P )+F 1 (P ) Algorithm: C 2 (P ),F 1 (P ) EsTmate and with mult. error (1 + ) EsTmaTon from Sub- Sampled Streams 14

15 EsTmaTng C 2 (P ) ObservaTon: E[C 2 (L)] = p 2 C 2 (P ) Var(C 2 (L)) = O(p 3 F ) If C 2 (L) estmated accurately, we are done But this is hard in general, in small space EsTmaTon from Sub- Sampled Streams 15

16 EsTmaTng C 2 (L) F 2 (P )=2C 2 (P )+F 1 (P ) Case I: Case 2: C 2 (L) =Ω(p 2 F 2 (P )) C 2 (L) =O(p 2 F 2 (P )) C 2 (L) C 2 (P ) EsTmate with good relatve error hence C 2 (P ) F 2 (P ) Accurate estmate of not needed Get an estmate within a multplicatve error of 3 C 2 (L) EsTmaTon from Sub- Sampled Streams 16

17 EsTmaTng C 2 (L) EsTmate with good relatve error when C 2 (L) =Ω(p 2 F 2 (P )) Technique due to Indyk & Woodruff (STOC 2005) Divide items {1,2..,m} into classes based on frequency EsTmate the size of different classes that contribute to the final result EsTmaTon from Sub- Sampled Streams 17

18 Tight Lower Bound for F k Theorem: Any constant- pass streaming algo. that (1 +, δ) - approximates F k, for a sufficiently small constants δ by observing a sampled stream, in the Bernoulli sampling model, requires Ω m 1 2/k /p bits of space EsTmaTon from Sub- Sampled Streams 18

19 F k Time- Space Tradeoff With sampled stream at probability p Processing Time: Streaming Space: Õ(pn) Õ m 1 2/k p Product: Õ(nm 1 2/k ) EsTmaTon from Sub- Sampled Streams 19

20 Entropy No multplicatve error approximaton possible with probability 9/10, even if p > ½ The entropy of sampled stream could be zero, while that of the original stream non- zero If p = Ω(n 1/3 ) there is an approximaton H to H(f) such that with prob. at least 99/100 H 100H(f) H H(f)/2 o(1) EsTmaTon from Sub- Sampled Streams 20

21 Heavy Hi7ers The frequency of heavy hi7ers are (approximately) proportonately maintained in the sampled stream Precise upper and lower bounds in paper EsTmaTon from Sub- Sampled Streams 21

22 Conclusions Extreme Volume Data Streams F k Upper and Lower Bounds for F k, k > 0 Smooth Space- Time tradeoff Number of distnct Elements, Entropy Sampling harms, streaming does not Heavy Hi7ers: Sampling and Streaming are both ok EsTmaTon from Sub- Sampled Streams 22