Tight Bounds for Distributed Functional Monitoring

Transcription

1 Tight Bounds for Distributed Functional Monitoring Qin Zhang MADALGO, Aarhus University Joint with David Woodruff, IBM Almaden NII Shonan meeting, Japan Jan

2 The distributed streaming model (a.k.a. distributed functional/continuous monitoring) coordinator C sites S 1 S 2 S 3 S k A(t) : set of elements received up to time t from all sites. a a Assume 1 item comes at each time unit. time 2-1

3 The distributed streaming model (a.k.a. distributed functional/continuous monitoring) coordinator C The coordinator needs to maintain f(a(t)) for all t. sites S 1 S 2 S 3 S k A(t) : set of elements received up to time t from all sites. a a Assume 1 item comes at each time unit. time 2-2

4 The distributed streaming model (a.k.a. distributed functional/continuous monitoring) coordinator C The coordinator needs to maintain f(a(t)) for all t. sites S 1 S 2 S 3 S k A(t) : set of elements received up to time t from all sites. a a Assume 1 item comes at each time unit. time Goal: minimize communication cost 2-3

5 Problems coordinator C Static case (a one-shot/static computation at the end) Top-k sites S 1 S 2 S 3 S k Heavy-hitter... Dynamic case Samplings time The Distributed Streaming Model Frequent moments (F 0, F 1, F 2,...) Heavy-hitter Quantile Entropy Non-linear functions

6 This talk What you would like to see: Efficient algorithms/protocols Practical heuristics 4-1

7 This talk What you would like to see: Efficient algorithms/protocols Practical heuristics What you (probably) do not want to see: Useless impossibility results Complicated proofs 4-2

8 This talk What you would like to see: Efficient algorithms/protocols Practical heuristics What you (probably) do not want to see: Useless impossibility results Complicated proofs Unfortunately, in the next 30 minutes

9 The multiparty communication model A model for lower bounds x 1 = x 2 = x k = x 3 = We want to compute f(x 1, x 2,..., x k ) f can be bit-wise XOR, OR, AND, MAJ

10 The multiparty communication model A model for lower bounds x 1 = x 2 = Blackboard: One speaks, everyone else hears. Message passing: If x 1 talks to x 2, others cannot hear. x k = x 3 = We want to compute f(x 1, x 2,..., x k ) f can be bit-wise XOR, OR, AND, MAJ

11 The multiparty communication model A model for lower bounds x 1 = x 2 = Blackboard: One speaks, everyone else hears. Message passing: If x 1 talks to x 2, others cannot hear. x k = x 3 = = coordinator C We want to compute f(x 1, x 2,..., x k ) f can be bit-wise XOR, OR, AND, MAJ... S 1 S 2 S 3 S k sites 5-3

12 Previously Some works in the blackboard model. Almost nothing in the message-passing model. 6-1

13 Previously Some works in the blackboard model. Almost nothing in the message-passing model. This SODA, with Jeff Phillips and Elad Verbin we proposed a general and elegant technique called symmetrization which works in both variants. In particular, we obtained (in the message-passing model) 1. Ω(nk) for the bitwise-xor/or/and/maj. 2. Ω(nk) for connectivity. 6-2

14 Previously Some works in the blackboard model. Almost nothing in the message-passing model. This SODA, with Jeff Phillips and Elad Verbin we proposed a general and elegant technique called symmetrization which works in both variants. In particular, we obtained (in the message-passing model) 1. Ω(nk) for the bitwise-xor/or/and/maj. 2. Ω(nk) for connectivity. Artificial? Well

15 Previously Some works in the blackboard model. Almost nothing in the message-passing model. This SODA, with Jeff Phillips and Elad Verbin we proposed a general and elegant technique called symmetrization which works in both variants. In particular, we obtained (in the message-passing model) 1. Ω(nk) for the bitwise-xor/or/and/maj. 2. Ω(nk) for connectivity. Artificial? Well... In any case, let s look at real important problems. 6-4

16 Now, important problems Samplings Frequent moments (F 0, F 1, F 2,...) Heavy-hitter Quantile Entropy

17 Now, important problems Samplings Solved Frequent moments (F 0, F 1, F 2,...) Heavy-hitter Quantile Entropy

18 Now, important problems Samplings Solved Frequent moments (F 0, F 1, F 2,...) Heavy-hitter Quantile Our work Entropy

19 8-1 Results

20 Results (Almost) tight bounds for all these questions Static lower bounds (almost) match dynamic upper bounds. (up to polylog factors) 8-2

21 Results Today (Almost) tight bounds for all these questions Static lower bounds (almost) match dynamic upper bounds. (up to polylog factors) 8-3

22 F 0 upper bound (Cormode, Muthu and Yi 2008) 9-1

23 The (1 + ε)-approximation F 0 problem We have k sites S 1, S 2,..., S k. S i holds a set X i. Our goal: compute F 0 ( i k X i ) up to (1 + ε)-approximation How many distinct items? A fundamental problem in data analysis. 10-1

24 The (1 + ε)-approximation F 0 problem We have k sites S 1, S 2,..., S k. S i holds a set X i. Our goal: compute F 0 ( i k X i ) up to (1 + ε)-approximation How many distinct items? A fundamental problem in data analysis. Current best UB: Õ(k/ε2 ) (Cormode, Muthu, Yi 2008) Holds in the dynamic case. 10-2

25 General idea for the one-shot computation Each site generates a sketch via small-space streaming algorithms. The coordinator combines (via communication) the sketches from the k sites to obtain a global sketch, from which we can extract the answer. 11-1

26 The FM sketch Take a pair-wise independent random hash function h : {1,..., n} {1,..., 2 d }, where 2 d > n 12-1

27 The FM sketch Take a pair-wise independent random hash function h : {1,..., n} {1,..., 2 d }, where 2 d > n For each incoming element x, compute h(x) e.g., h(5) = Count how many trailing zeros Remember the max # trailing zeroes in any h(x) 12-2

28 The FM sketch Take a pair-wise independent random hash function h : {1,..., n} {1,..., 2 d }, where 2 d > n For each incoming element x, compute h(x) e.g., h(5) = Count how many trailing zeros Remember the max # trailing zeroes in any h(x) Let Y be the max # trailing zeroes Can show E[2 Y ] = #distinct elements 12-3

29 One-shot case, the FM sketch (cont.) So 2 Y is an unbiased estimator for # distinct elements 13-1

30 One-shot case, the FM sketch (cont.) So 2 Y is an unbiased estimator for # distinct elements However, has a large variance Some techniques [Bar-Yossef et. al. 2002] can produce a good estimator that has probability 1 δ to be within relative error ε. Space increased to Õ(1/ε2 ) 13-2

31 One-shot case, the FM sketch (cont.) So 2 Y is an unbiased estimator for # distinct elements However, has a large variance Some techniques [Bar-Yossef et. al. 2002] can produce a good estimator that has probability 1 δ to be within relative error ε. Space increased to Õ(1/ε2 ) FM sketch has linearity Y 1 from A, Y 2 from B, then 2 max{y 1,Y 2 } estimates # distinct items in A B. 13-3

32 One-shot case, the FM sketch (cont.) So 2 Y is an unbiased estimator for # distinct elements However, has a large variance Some techniques [Bar-Yossef et. al. 2002] can produce a good estimator that has probability 1 δ to be within relative error ε. Space increased to Õ(1/ε2 ) FM sketch has linearity Y 1 from A, Y 2 from B, then 2 max{y 1,Y 2 } estimates # distinct items in A B. Thus, we can use it to design a one-shot algorithm with communication Õ(k/ε2 ) 13-4

33 14-1 F 0 lower bound

34 The F 0 problem We have k sites S 1, S 2,..., S k. S i holds a set X i. Our goal: compute F 0 ( i k X i ) up to (1 + ε)-approximation How many distinct items? A fundamental problem in data analysis. 15-1

35 The F 0 problem We have k sites S 1, S 2,..., S k. S i holds a set X i. Our goal: compute F 0 ( i k X i ) up to (1 + ε)-approximation How many distinct items? A fundamental problem in data analysis Current best UB: Õ(k/ε 2 ) (Cormode, Muthu, Yi, 2008) Holds in the dynamic case. Previous LB: Ω(k) (Cormode, Muthu, Yi, 2008) Ω(1/ε 2 ) (reduction from Gap-Hamming) Our LB: Ω(k/ε 2 ). Holds in the static and message-passing case.

36 The F 0 problem We have k sites S 1, S 2,..., S k. S i holds a set X i. Our goal: compute F 0 ( i k X i ) up to (1 + ε)-approximation How many distinct items? A fundamental problem in data analysis Tight! Current best UB: Õ(k/ε 2 ) (Cormode, Muthu, Yi, 2008) Holds in the dynamic case. Previous LB: Ω(k) (Cormode, Muthu, Yi, 2008) Ω(1/ε 2 ) (reduction from Gap-Hamming) Our LB: Ω(k/ε 2 ). Holds in the static and message-passing case.

37 The proof framework Step 1: We first introduce a simpler problem called k-gap-maj Step 2: We compose k-gap-maj with the Set Disjointness problem using information cost to prove a lower bound for F

38 k-gap-maj We have k sites S 1, S 2,..., S k. S i holds a bit Z i which is 1 w.p. β and 0 w.p. 1 β where ω(1/k) β 1/2 is a prefixed value. Our goal: compute the following function. GM(Z 1, Z 2,..., Z k ) = 0, 1, if i [k] Z i βk βk, if i [k] Z i βk + βk,, otherwise, where means that the answer can be arbitrary. 17-1

39 k-gap-maj We have k sites S 1, S 2,..., S k. S i holds a bit Z i which is 1 w.p. β and 0 w.p. 1 β where ω(1/k) β 1/2 is a prefixed value. Our goal: compute the following function. GM(Z 1, Z 2,..., Z k ) = 0, 1, if i [k] Z i βk βk, if i [k] Z i βk + βk,, otherwise, where means that the answer can be arbitrary. Lemma 1: If a protocol P computes k-gap-maj correctly w.p , then w.p. Ω(1), the protocol has to learn at least Ω(k) of Z i each with Ω(1) bit (that is, H(Z i Π) H b (0.01β)). 17-2

40 k-gap-maj We have k sites S 1, S 2,..., S k. S i holds a bit Z i which is 1 w.p. β and 0 w.p. 1 β where ω(1/k) β 1/2 is a prefixed value. Our goal: compute the following function. GM(Z 1, Z 2,..., Z k ) = 0, 1, if i [k] Z i βk βk, if i [k] Z i βk + βk,, otherwise, where means that the answer can be arbitrary. Lemma 1: If a protocol P computes k-gap-maj correctly w.p , then w.p. Ω(1), the protocol has to learn at least Ω(k) of Z i each with Ω(1) bit (that is, H(Z i Π) H b (0.01β)). Alternatively: I(Z 1, Z 2,..., Z k ; Π) = Ω(k) 17-3

41 Set disjointness (2-DISJ) Alice x {0, 1} n x y =? Bob y {0, 1} n 18-1

42 Set disjointness (2-DISJ) Alice x {0, 1} n x y =? Bob y {0, 1} n A classical hard instance: Distribution µ: X and Y are both random subsets of size l = (n+1)/4 from [n] such that X Y = 1 w.p. β and X Y = 0 w.p. 1 β. Razborov [1990] shows an Ω(n) for this hard distribution and error β/

43 Next step: Compose k-gap-maj with 2-DISJ n = Θ(1/ε 2 ) l = (n + 1)/4 β = 1/kε 2 coordinator C sites S 1 S 2 S 3 S k 19-1

44 Next step: Compose k-gap-maj with 2-DISJ n = Θ(1/ε 2 ) l = (n + 1)/4 β = 1/kε 2 coordinator C Y Step 1: Pick Y = y [n] of size l uniformly at random sites S 1 S 2 S 3 S k 19-2

45 Next step: Compose k-gap-maj with 2-DISJ n = Θ(1/ε 2 ) l = (n + 1)/4 β = 1/kε 2 coordinator C Y Step 1: Pick Y = y [n] of size l uniformly at random sites S 1 S 2 S 3 S k X 1 X 2 X 3 X k Step 2: Pick X 1,..., X k [n] indepedently and randomly from µ Y =y 19-3

46 Next step: Compose k-gap-maj with 2-DISJ n = Θ(1/ε 2 ) l = (n + 1)/4 β = 1/kε 2 coordinator C Y Step 1: Pick Y = y [n] of size l uniformly at random F 0 (X 1, X 2,..., X k )? sites S 1 S 2 S 3 S k X 1 X 2 X 3 X k Step 2: Pick X 1,..., X k [n] indepedently and randomly from µ Y =y 19-4

47 The proof coordinator C Y Z i = X i Y { 1 w.p. β 0 w.p. 1 β sites S 1 S 2 S 3 S k X 1 X 2 X 3 X k F 0 (X 1, X 2,..., X k ) k-gap-maj(z 1, Z 2,..., Z k ) (Z i = X i Y ) learn Ω(k) Z i s well (by Lemma 1) need Ω(k/ε 2 ) bits (learning each Z i = X i Y well needs Ω(n) = Ω(1/ε 2 ) bits, by 2-DISJ) 20-1

48 The proof coordinator C Y Z i = X i Y { 1 w.p. β 0 w.p. 1 β sites S 1 S 2 S 3 S k X 1 X 2 X 3 X k F 0 (X 1, X 2,..., X k ) k-gap-maj(z 1, Z 2,..., Z k ) Q.E.D. (Z i = X i Y ) learn Ω(k) Z i s well (by Lemma 1) need Ω(k/ε 2 ) bits (learning each Z i = X i Y well needs Ω(n) = Ω(1/ε 2 ) bits, by 2-DISJ) 20-2

49 Proof sketch of Lemma 1 Lemma 1: If a protocol P computes k-gap-maj correctly w.p , then w.p. Ω(1), for Ω(k) Z i s, we have H(Z i Π) H b (0.01β). Proof: 1. Suppose Π does not satisfy this. 2. Since the Z i are independent given Π, k i=1 Z i Π is a sum of independent Bernoulli random variables. 3. Since most H(Z i Π) are large, by anti-concentration, both of the following events occur with constant probability: k i=1 Z i Π > βk + βk, k i=1 Z i Π < βk βk. 4. So P can t succeed with large probability. 21-1

50 22-1 F 2 lower bound

51 The F 2 problem We have k sites S 1, S 2,..., S k. S i holds a set X i. Our goal: compute F 2 ( i k X i ) up to (1 + ε)-approximation Join What s the size of self-join? Another fundamental problem in data analysis. 23-1

52 The F 2 problem We have k sites S 1, S 2,..., S k. S i holds a set X i. Our goal: compute F 2 ( i k X i ) up to (1 + ε)-approximation Join What s the size of self-join? Another fundamental problem in data analysis. Previous UB: Õ(k 2 /ε + k 1.5 /ε 3 ) (Cormode, Muthu, Yi 2008) Our UB: Õ(k/poly(ε)), one way protocol Holds in the dynamic case. Previous LB: Ω(k) (Cormode, Muthu, Yi, 2008) Our LB: Ω(k/ε 2 ). Holds in the static and blackboard case. 23-2

53 The F 2 problem We have k sites S 1, S 2,..., S k. S i holds a set X i. Our goal: compute F 2 ( i k X i ) up to (1 + ε)-approximation Join Almost Tight! What s the size of self-join? Another fundamental problem in data analysis. Previous UB: Õ(k 2 /ε + k 1.5 /ε 3 ) (Cormode, Muthu, Yi 2008) Our UB: Õ(k/poly(ε)), one way protocol Holds in the dynamic case. Previous LB: Ω(k) (Cormode, Muthu, Yi, 2008) Our LB: Ω(k/ε 2 ). Holds in the static and blackboard case. 23-3

54 A quick glance: (1 + ε)-approximation F 2 2-party gap-hamming: Alice has X = {X 1, X 2,..., X 1/ε 2}, Bob has Y = {Y 1, Y 2,..., Y 1/ε 2}. They want to compute: GHD(X, Y ) = 0, 1, if i [1/ε 2 ] X i Y i 1/2ε 2 1/ε, if i [1/ε 2 ] X i Y i 1/2ε 2 + 1/ε,, otherwise, where means that the answer can be arbitrary. 24-1

55 A quick glance: (1 + ε)-approximation F 2 2-party gap-hamming: Alice has X = {X 1, X 2,..., X 1/ε 2}, Bob has Y = {Y 1, Y 2,..., Y 1/ε 2}. They want to compute: GHD(X, Y ) = 0, 1, if i [1/ε 2 ] X i Y i 1/2ε 2 1/ε, if i [1/ε 2 ] X i Y i 1/2ε 2 + 1/ε,, otherwise, where means that the answer can be arbitrary. k-disj: We have k sites S 1, S 2,..., S k. S i holds a set Z i. We promise that either Z i (i = 1,..., k) are all disjoint, or they intersect on one element and the rest are all disjoint (sun-flower). The goal is to find out which is the case. 24-2

56 A quick glance: (1 + ε)-approximation F 2 2-party gap-hamming: Alice has X = {X 1, X 2,..., X 1/ε 2}, Bob has Y = {Y 1, Y 2,..., Y 1/ε 2}. They want to compute: GHD(X, Y ) = 0, 1, if i [1/ε 2 ] X i Y i 1/2ε 2 1/ε, if i [1/ε 2 ] X i Y i 1/2ε 2 + 1/ε,, otherwise, where means that the answer can be arbitrary. k-xor 2 copies k-disj: We have k sites S 1, S 2,..., S k. S i holds a set Z i. We promise that either Z i (i = 1,..., k) are all disjoint, or they intersect on one element and the rest are all disjoint (sun-flower). The goal is to find out which is the case. 24-3

57 A quick glance: (1 + ε)-approximation F 2 2-party gap-hamming: Alice has X = {X 1, X 2,..., X 1/ε 2}, Bob has Y = {Y 1, Y 2,..., Y 1/ε 2}. They want to compute: GHD(X, Y ) = 0, 1, if i [1/ε 2 ] X i Y i 1/2ε 2 1/ε, if i [1/ε 2 ] X i Y i 1/2ε 2 + 1/ε,, otherwise, where means that the answer can be arbitrary. k-xor 2 copies compose via information cost k-bta k-disj: We have k sites S 1, S 2,..., S k. S i holds a set Z i. We promise that either Z i (i = 1,..., k) are all disjoint, or they intersect on one element and the rest are all disjoint (sun-flower). The goal is to find out which is the case. 24-4

58 A quick glance: (1 + ε)-approximation F 2 2-party gap-hamming: Alice has X = {X 1, X 2,..., X 1/ε 2}, Bob has Y = {Y 1, Y 2,..., Y 1/ε 2}. They want to compute: GHD(X, Y ) = 0, 1, if i [1/ε 2 ] X i Y i 1/2ε 2 1/ε, if i [1/ε 2 ] X i Y i 1/2ε 2 + 1/ε,, otherwise, where means that the answer can be arbitrary. k-xor 2 copies compose via information cost k-bta CC(k-BTA) = Ω(k/ε 2 ) k-disj: We have k sites S 1, S 2,..., S k. S i holds a set Z i. We promise that either Z i (i = 1,..., k) are all disjoint, or they intersect on one element and the rest are all disjoint (sun-flower). The goal is to find out which is the case. 24-5

59 A quick glance: (1 + ε)-approximation F 2 2-party gap-hamming: Alice has X = {X 1, X 2,..., X 1/ε 2}, Bob has Y = {Y 1, Y 2,..., Y 1/ε 2}. They want to compute: GHD(X, Y ) = 0, 1, if i [1/ε 2 ] X i Y i 1/2ε 2 1/ε, if i [1/ε 2 ] X i Y i 1/2ε 2 + 1/ε,, otherwise, where means that the answer can be arbitrary. k-xor 2 copies compose via information cost k-bta CC(k-BTA) = Ω(k/ε 2 ) k-disj: We have k sites S 1, S 2,..., S k. S i holds a set Z i. We promise that either Z i (i = 1,..., k) are all disjoint, or they intersect on one element and the rest are all disjoint (sun-flower). The goal is to find out which is the case. Finally, we reduce F 2 to k-bta. 24-6

60 The end T HAN K YOU Q and A 25-1