The complexity of enumeration and counting for acyclic conjunctive queries

Transcription

1 The complexity of enumeration and counting for acyclic conjunctive queries Cambridge, march 2012

2 Counting and enumeration Counting Output the number of solutions Example : sat, Perfect Matching, Permanent, problems on graph, knots polynomials Enumeration Generate all solutions one by one Output can be large Natural algorithmic task This talk: look at these two tasks for fragments of conjunctive (i.e. {, }) queries (mainly ACQ)

3 Outline Enumeration A tour on complexity measure for enumeration Results for query problems Complexity characterization for ACQ Weighted Counting Polynomial representation of query Counting through polynomial evaluation Characterization of the tractability frontier for ACQ Common point : role of free variables in formulas.

4 Enumeration: measure of complexity Intractability NP-completeness of the existence of one solution (hard to start) hard after generation of a part of the solution set (hard to continue) Tractability Polynomial total time (succeed but don t know how) Incremental Polynomial Time (harder as the number of generated solutions increases) Polynomial delay (regular process)

5 Example Generate all models of a propositional formula (hum... intractable) Generate all independant sets of maximum size of a graph (intractable) Generate all maximal (for inclusion) independant sets polynomial delay for lexicographic ordering intractable for reverse lexicographic ordering Generate all models of a 2-CNF propositional formula (polynomial delay)

6 Complexity of enumeration of satisfiability problems Let ϕ be a formula with variables x 1,..., x n. If ϕ x 1 SAT : enumerate all solutions of ϕ x 1 If ϕ x 1 SAT : enumerate all solutions of ϕ x 1 Theorem (Creignou, Hebrard 97) In the Boolean case, there is no other efficient algorithm than the one described above. Consequently, easy (i.e. polynomial delay) cases are Horn, Anti-Horn, 2-CNF and affine. Schnoor & Schnoor 07 The situation is more complex for CSP on arbitrary finite domain See also Bulatov, Dalmau, Grohe & Marx 10

7 Enumeration algorithm: some precision { Precomputation step Algorithms in two steps: + Enumeration step(s) Computation models (in this talk) : RAM with uniform cost (but with bounded values in registers) Let A be an enumeration algorithm for Enum(R) and x I. time i (x) denotes the time when the algorithm finishes to write the ith solution if it exists. delayi (x) = time i+1 (x) time i (x). Delay may be : sub-exponential, polynomial, linear, constant (?) Warning: space is an important factor!

8 Small classes of enumeration problems An enumeration algorithm A is constant delay if there is a constant c such that for any x I and i, delay i (x) c. Enum(R) is computable with constant delay (Enum(R) Constant-Delay) if there is a constant delay algorithm A which computes Enum(R). Enum(R) Constant-Delay lin if it is reducible in linear time (in the input size) to a problem in Constant-Delay precomputation reduction

9 Query problems Query L = set of formulas S = set of (finite) structures Enumeration problem Enum(L, S) input: φ L and A S output: enumerate φ(a) = {ā D k (A, ā) = φ( x)} Counting problem Count(L, S) input: φ L and A S output: φ(a) = {ā D k (A, ā) = φ( x)}

10 Complexity Measures Framework for complexity analysis of multi-parameterized problems Parameters A, ϕ (inputs) ϕ(a) But also: number of free variables, of quantified variables, alternation, arity... A, ϕ and ϕ(a) : Combined Complexity. A and ϕ(a) : Data Complexity ϕ and ϕ(a) : Expression Complexity Parameterized complexity: express the complexity in terms of A, ϕ and ϕ(a) but consider ϕ as a parameter.

11 Some tractable cases for enumeration Data complexity deg τ d : the class of τ-structures of maximal degree d. Enum(fo, deg τ d ) constant-delay lin. Delay : triply exponential in the size of formula (Kazana, Segoufin 10) (D. Grandjean 07, Lindell 08) tw τ k : the class of τ-structures of treewidth at most k. Enum(mso, tw τ k ) linear-delay lin. (Bagan 06, Courcelle 07, Frick,..)

12 Conjunctive Queries Conjunctive Queries (CQ) L: {, }-fragment of first-order logic i.e. formulas ϕ( x) ȳψ( x, ȳ) where ψ is a conjunction of atoms. Combined complexity of model checking: NP-complete. No hope for enumeration. Tractable fragments for decision by restricting the formula (acyclicity, hypergraph decomposition methods).

13 Hypergraph associated to formulas The hypergraph of a formula ϕ is the hypergraph H = (V, E) such that V is the set of variables of ϕ for each atom R( v) of ϕ, we associate a hyperedge var(r( v)) t u x v y z

14 Acyclic Hypergraph Most general notion: α-acyclicity. An hypergraph is α-acyclic if the application of the following rules as long as possible outputs the empty hypergraph: 1. Remove hyperedges contained in other hyperedges; 2. Remove vertices that appear in at most one hyperedge. A conjunctive query is acyclic if its associated hypergraph is acyclic.

15 Acyclic hypergraph/query: examples Acyclic Cyclic Acyclic Large class of hypergraph/queries. Base case for more evolved decomposition methods. Remark: Adding an edge to a cyclic hypergraph may result in an acyclic hypergraph.

16 Acyclic Hypergraph: alternative definition A join tree of a hypergraph H = (V, E) is a pair (T, λ) where T = (V T, T ) is a tree and λ is a function from V T to E such that: For each e E, there is a t V T such that λ(t) = e (each vertex of T is a bag containing one edge) For each v V, the set {t V T : v λ(t)} is a connected subtree of T. Equivalent characterization : H is acyclic iff it has a join tree.

17 Acyclic Hypergraph: alternative definition e a, c, e, g f d g a b c a, b, c a, e, f c, d, e I

18 Complexity of acyclic conjunctive queries ACQ Acyclic Conjunctive Queries. ACQ Acyclic Conjunctive Queries with inequalities Example: ϕ(x, y) z t u(rxyz Ryzt Rztu Syt x u) Known results ACQ solvable in O( ϕ A ϕ(a) ) times (Yannakakis 81) ACQ solvable in O(2 O( ϕ ) A ϕ(a) log 2 ( A )) times (Papadimitriou & Yannakakis-99) or in O(( ϕ!) A ϕ(a) ) (Bagan-D.-Grandjean-07)

19 Yannakakis algorithm Input: (A, ϕ), a tree decomposition T - Take a leaf t V T, let R( x, z) the associated atom. - Take its father associated to S( x, ȳ) ({ȳ} { z} = ) Filter relation S in A by relation R : S := {(ā, b) S and c(ā, c) R} S(x,y) R(x,z)

20 Yannakakis algorithm Input: (A, ϕ), a tree decomposition T - Take a leaf t V T, let R( x, z) the associated atom. - Take its father associated to S( x, ȳ) ({ȳ} { z} = ) Filter relation S in A by relation R : S := {(ā, b) S and c(ā, c) R} S(x,y) Continue bottom up with new data and drop R( x, z) from the list of atoms

21 Enumeration of ACQ Enumeration Can we enumerate the results efficiently? Easy but not that fast! let ϕ (x 1 ) x 2... x k ȳϕ( x, ȳ) For a ENUM(ϕ, A) let ϕa ϕ(a, x 2,..., x k, ȳ) For b ENUM(ϕ a, A) : output (a, b) Linear O( ϕ A ) delay. Similarities with the SAT enumeration in easy cases.

22 Enumeration of ACQ Easy and fast if all variables are free (i.e. participate to the result). Compute a tree decomposition Filter each parent with its children so that only tuples that can be extended to a solution remain ( local consistency) run through the tree and output the solutions Linear O( ϕ A ) precomputation and constant O( ϕ ) delay. Works also with disequalities (but delay exponential in ϕ )

23 What s the point with enumeration Tree decomposition T for ϕ - Blue boxes : quantified variables only - Yellow boxes : free variables only - In real life : green boxes also... Good situation

24 What s the point with enumeration Tree decomposition T for ϕ - Blue boxes : quantified variables only - Yellow boxes : free variables only - In real life : green boxes also... Bad situation

25 Further question Can we do better than linear delay (with linear precomputation)? Can we find other fragments with constant delay? Can we fully characterize the enumeration complexity of acyclic conjunctive queries?

26 Free-connex acyclic formulas Let H = (V, E) be an acyclic hypergraph and S V. H is S-connex acyclic if there is an acyclic hypergraph H = (V, E ) and a tree-structure T of H such that E E e E e E : e e (H inclusive extension of H) there is a connex subset A of vertices of T such that t A λ(t) = S A formula ϕ is free-connex acyclic or CCQ if its hypergraph is free(ϕ)-connex acyclic.

27 Example of S-connex acyclic hypergraph Let H = (V, E) with V = {a, b, c, d, e, f } and E = {{a, b, c}, {c, d, e}, {c, d, f }} and S = {b, c, d} a, b, c b, c c, d, e c, d c, d, f

28 Other characterisation of free-connex acyclic formulas Let H = (V, E) be a hypergraph and S V. An S-path is a path (x, y 1,... y n, z) of size at least 2 such that x S, z S for any i, yi / S there is no hyperedge e E such that x, z e Lemma H is S-connex acyclic iff H is acyclic and doesn t admit any S-path. This property can be checked in polynomial time. Free-connex acyclicity: being grouped up to hypergraph padding

29 Matrix Multiplication Matrix Product Enumeration Given two matrices A and B (with coefficients in some space), - enumerate all C i,j or - enumerate the indices (i, j) of C = A B such that C i,j 0. Boolean case: solutions of Φ(x, z) y E(x, y) E(y, z).

30 Dichotomy result (Bagan-D.-Grandjean 07) Theorem: Let ϕ be a simple ACQ (resp. ACQ ) formula and assume that the boolean matrix product cannot be done in linear time. Then one of the two following cases hold: or ϕ is free-connex acyclic Enum(ϕ) Constant-Delay lin Eval(ϕ) can be evaluated in time O( A + ϕ(a) ) ϕ is not free-connex acyclic Enum(ϕ) / Constant-Delay lin Eval(ϕ) cannot be evaluated in time O( A + ϕ(a) )

31 Proof of hardness ϕ ACQ CCQ i.e., acyclic but not free(ϕ)-connex. Main steps : Find a S-path (recall : S = free(ϕ)) From two boolean matrices, A and B construct a structure A, in linear time, such that: A encodes A and B. Data for A and B are on each side of the S-path indices of non zero coefficients of A B are (pairs of) vertices related in the path. Some problems to solve : the path may be arbitrary, there are other parts in the query,... Skip rest of proof

32 Proof of hardness ϕ ACQ CCQ i.e., acyclic but not free-connex. H φ admits a chordless S-path P = (x, z 1,..., z k, y) with k 1, so that P is a path: there are k + 1 hyperedges e 0, e 1,..., e k 1, e k E that contain e 0 = {x, z 1}, e 1 = {z 1, z 2 },..., e k 1 = {z k 1, z k }, e k = {z k, y} P is an S-path: x, y S and z 1,..., z k 1 / S P is chordless: for each e E, e P 1 or e P = e i for some i, (0 i k) Skip rest of proof

33 Proof of hardness ϕ simple formula ϕ of the form: ϕ(x, y, t) z 1... z k ūψ(x, y, z, t, ū) with ψ conjunction of atoms A e, e E of the following form ( v { t, ū}): 1. R e (x, z 1, v) 2. R e (z k, y, v) 3. R e (z i, z i+1, v) where 1 i < k 4. R e (w, v) where w {x, y, z 1,..., z k } 5. R e ( v)

34 Proof of hardness Transformation r of the structure: to each σ AB -structure A = (D, A, B) associates the following A : A = (D, (R e ) e E ) D = D { } : here is a new special padding symbol For each e E define the relation of arity p, R e according to the form of its (unique) occurrence in ϕ 1. R e = A { } p 2 2. R e = B { } p 2 3. R e = I { } p 2 where I is the identity (equality) relation of D (I = {(a, a) : a D}) 4. R e = D { } p 1 5. R e = { } p

35 Proof of hardness Π(A) = A B Fact 1: The map r : A A is computable in time O( A ) Fact 2: ϕ (A ) = Π(A) { } m Fact 3: The projection (a, b, c) (a, b) is a one-one mapping from ϕ(a ) onto Π(A)

36 Conclusion for enumeration Preceeding result shows some dichotomy in the complexity of enumeration (not in the same sense as usual dichotomy results in complexity). Dichotomy distinguishes tractable and very tractable cases Dichotomy based on positions of free variables in the formula

37 #ACQ Here, situation differs more drastically if quantified variables are allowed. Theorem (Pichler, Skritek 11) If only quantifier free formulas are authorized as inputs, then #ACQ is computable in polynomial time. #ACQ is #P-complete even for formulas with only one existentially quantified variable. Hardness result (interpretation) Given bipartite graph G, one construct a structure A G and an acyclic conjunctive formula ϕ G ( x) = yφ( x, y) such that : The number of perfect matching in G = ϕ G (A G )

38 #ACQ End of the story for #ACQ? What about weighted counting for ACQ? Hardness or tractability for counting the disjunction (or conjunction) of, say, two ACQ? One quantifier is enough to design hard instance but is that all we can say? Can we isolate island of tractability for #ACQ with quantified variables? or better chart the tractability frontier for quantified #ACQ?

39 Weighted counting F: ring with operations + and S: finite structure of domain D F-weight function for S: mapping w : D F If ā D k, the weight of ā is # F CQ (resp. # F ACQ) w(ā) = k w(a i ). Input: A conjunctive query (resp. acyclic) Φ = (S, φ) and an F-weighted function w Output: ā φ(s) w(ā). When w is the constant function 1, this value is equal to φ(s). i=1

40 Weighted counting What is the complexity of # F ACQ for quantifier free formulas? Either in terms of number of symbolic operation or in number of bits (for spaces where iterated multiplication and addition is polynomial time) Strategy Consider query instances as polynomials Construct efficiently arithmetic circuits that recognize these polynomials Evaluate the polynomial to solve the weighted counting problem

41 Arithmetic circuits Arithmetic circuit over F Labeled directed acyclic graph (DAG) with vertices (gates) with indegree (fanin) 0 or 2. Input gates have fanin 0 and are labeled with constants from F or variables X 1, X 2,..., X n. Computation gates have fanin 2 and are labeled with or +. One gate with fanout 0: the output gate The size of the circuit is the number of gates The depth is the length of the longest path form an input gate to the output gate

42 Arithmetic circuit: example x 1 x 2 P(X 1, X 2 ) = 2X 1 X 2 + (X 1 + X 2 )X 2 = 3X 1 X 2 + X

43 Multiplicatively disjoint circuits x 1 x 2 x 2 A circuit is multiplicatively disjoint if, for each -gate, its two input subcircuits are disjoint

44 Arithmetic circuits Valiant s setting for computation of families of polynomials by families of circuits. Same computational power polynomial size circuits of polynomial degree multiplicatively disjoint circuits logarithmic depth semi-unbounded circuits. Algebraic equivalent of LOGCFL... See: Valiant, Burgisser, Koiran, Malod,...

45 Polynomials representing queries Q-Polynomials Φ = (S, φ) : conjunctive query with domain D. The following polynomial Q(Φ) in the variables {X d d D} is assigned to Φ: Q(Φ) := a φ(s) x var(φ) X a(x) = a φ(s) d D X µ d (a) d, where µ d (a) = {x var(φ) a(x) = d} is the number of variables mapped to d by a. Example Φ = (S, φ) such that φ(s) = {(1, 3, 2, 5), (1, 2, 1, 3), (5, 2, 3, 1), (1, 1, 2, 1), (4, 2, 1, 2)}. Then: Q(Φ) := 2X 1 X 2 X 3 X 5 + X 2 1 X 2 X 3 + X 3 1 X 2 + X 2 2 X 1 X 4.

46 Polynomials representing queries The number of variables in Q(Φ) is D, the size of the domain of S Q(Φ) is homogeneous of degree free(φ) No bijective correspondence between solutions of the query and monomials Existential quantification in formulas does not correspond to projection on some polynomial variable.

47 # F ACQ for quantifier free acyclic formulas Here and after: joint work with Stefan Mengel. Φ = S + φ Theorem Given an acyclic quantifier free conjunctive query Φ = (S, φ), we can in time polynomial in Φ compute a multiplicatively disjoint arithmetic circuit C that computes Q(Φ). Corollary The problem # F ACQ is computable in polynomial time Skip proof

48 Computation of Q(Φ) for Φ acyclic: proof By induction, adapting Yannakakis algorithms for ACQ Φ = (S, φ): the input (T, λ): the join tree associated with φ For t V T, φ t : conjunction of constraints corresponding to the subtree T t rooted at t Set e t = var(φ t ) = t T t var(λ(t )) Skip rest of proof

49 Computation of Q(Φ) for Φ acyclic: proof Compute the more general polynomial: f t,ā,c = ᾱ φ t(s) ᾱ ā X α(x). Where c λ(t), ā is an assignment of some variable of φ and ᾱ ā means ᾱ agrees with ā on common variables. Strategy and Problems Compute inductively from some f ti,ā i,c i where t i is a child of t. If two children share a variable there is a risk of overcounting so... x c Partition the variables in λ(t) and assign parts to children Skip rest of proof

50 Computation of Q(Φ) for Φ acyclic: proof The case of a leaf t V T is immediate. Let t V T and t 1,..., t k its children in V T c 0, c 1,..., c k : partition of c into disjoint sets such that c i e i c, for i = 1,..., k and c 0 c\ k i=1 e t i f t,ā,c = = ᾱ φ t(s) x c ᾱ ā ᾱ φ t(s) ᾱ ā X α(x) Xᾱ(x) x c 1 x c k Xᾱ(x) x c 0 Xᾱ(x)

51 Computation of Q(Φ) for Φ acyclic: proof Last remark Let A t = ((λ t (S) φ t1 (S)) φ t2 (S))... φ tk (S). Each solution ᾱ φ t (S) can be uniquely expressed as the natural join of a β A t with some ᾱ i φ ti (S), i = 1,..., k, compatible with β (converse also true) f t,ā,c = ᾱ φ t(s) ᾱ ā = β A t β ā = β A t β ā Xᾱ(x) x c 1 ᾱ 1 φ t1 (S) ᾱ 1 β x c k Xᾱ(x) ᾱ k φ tk (S) ᾱ k β f t1,β,c 1 f tk,β,c k x c 0 Xᾱ(x) Xᾱ1 (x) x c 1 x c 0 X β(x) Xᾱk (x) X β(x) x c k x c 0

52 Union and intersection of acyclic queries Theorem Computing the size of the union and the intersection of query results to two quantifier free #ACQ-instances are both #P-complete. This result remains true for #ACQ on boolean domain and arity at most 3. Proof by chain of reductions #circuitsat #( - -grid)-circuitsat: circuit sat with and gate on a grid. #CQ on a grid

53 Pause to ponder Bad news The conjunction (or disjunction) of two ACQ makes counting hard Idem for ACQ for one existentially quantified variable. But... formula in Pichler and Skritek s hardness proof looks like that: 9 Is it a sign of hardness?

54 S-components A hypergraph H = (V, E) and S V. Let E S = {e E : e S}. S-component The S-component of e E S is the hypergraph H[E ] where E is the set of all edges e E S such that there is a path from e S to e S in H[V S]. A subhypergraph H of H is an S-component if there is an edge e E S such that H is the S-component of e.

55 Decomposition into S-components: example S vertices in red

56 Decomposition into S-components: example S vertices in red

57 S-star size Definition (S-k-star, S-star size) Let H = (V, E) be a hypergraph, S V and k N. The subhypergraph H = (V, E ) of H is a S-k-star if: H is an S-component of H. there exist y 1,..., y k V S such that there is no edge e E that contains more than one of the y i. y 1,..., y k are independant and form the S-k-star. The S-star size of H is the maximum k such that there is a S-k-star in H.

58 S-star size : example S-star size is 4

59 quantified-star size Quantified star size The quantified star size of an acyclic conjunctive formula φ( x) is the S-star size of the hypergraph H associated to φ( x), where S is the set of free variables in φ( x).

60 quantified-star size: example Formula φ(x, y) t zr(x, y, t) S(x, z, t) Quantified star size = 1 Path formulas (of arbitrary length), e.g. φ(x, y, z) t 1 t 2 R(x, t 1 ) R(t 1, z) R(z, t 2 ) R(t 2, y) Quantified star size = 2. Star formulas, e.g. φ(x, y, z, t) ur(u, x) R(u, y) R(u, z) R(u, t) Quantified star size = degree of the center of the star (here 4).

61 S-star size: main results I Query evaluation is easy Theorem There is an algorithm that given an acyclic conjunctive query Φ computes an arithmetic circuit C that computes Q(Φ). The runtime of the algorithm is Φ O(k) where k is the quantified star size of Φ. Corollary There is an algorithm for the problem #ACQ that runs in time Φ O(k) where k is the quantified star size of the input query Φ.

62 Some remarks on the proof Star size expresses how pieces of the result are spread Computation of the result is made component by component For each component, if quantified-star size is k and R 1 ( z 1 ),..., R k ( z k ) atoms of φ containing all free variables of φ, one needs to pre-compute all k-tuples ā 1 R 1,..., ā k R k and check if they lead to a solution of φ(s) (by the method for Q(Φ) when Φ acyclic and quantified free)

63 S-star size: main results II Computing S-star is easy Theorem There is a polynomial time algorithm that, given a hypergraph H = (V, E) and S V, computes the S-star size of H. Proof Equivalent to finding a maximal independant set (and a minimal edge cover) in an acyclic hypergraph. Ad hoc algorithm. Conclusion: Classes of #ACQ-instances of bounded quantified star size are efficiently decidable. Go to: star size needed

64 Parametrizations of #ACQ p-star-#acq: counting parameterized by the quantified star size, p-var-#acq: counting parameterized by the number of free variables, p-#acq: counting parameterized by the size of the conjunctive formula. Lemma p-star-#acq, p-var-#acq and p-#acq are all #W[1]-hard.

65 Bounded quantified star size is necessary S-hypergraph: pair (H, S) where H = (V, E) is a hypergraph and S V. Definition #ACQ is tractable for a class G of S-hypergraphs if for all #ACQ instances Φ with the associated hypergraph H of Φ and the set S of free variables of Φ with (H, S) G we can solve #ACQ in polynomial time

66 Bounded quantified star size is necessary Theorem Assume FPT #W[1], and let G be a recursively enumerable class of acyclic S-hypergraphs. Then #ACQ is polynomial time solvable for G if and only if G is of bounded S-star size.

67 Proof sketch First prove that #ACQ is hard on G S the class of k-stars (in the graph sense) Consider an instance Φ = (A, ϕ( x)) with ϕ( x) = y k i=1 R i(y, x i ) Suppose there is a class G of unbounded S-star size which is tractable. Embed efficiently Φ into a suitable instance (with big enough star) Ψ whose associated hypergraph is in G. Deduce that #ACQ is not hard on G S.

68 Conclusion Complete characterization of tractability frontier for #ACQ Easy evaluation for counting on bounded S-star size extend to generalized hypertree width S-k-star size recognition seem to depend on k for some decomposition