Structural Tractability of Counting of Solutions to Conjunctive Queries

Transcription

1 Structural Tractability of Counting of Solutions to Conjunctive Queries Arnaud Durand 1 1 University Paris Diderot Journées DAG, june 2013 Joint work with Stefan Mengel (JCSS (to appear) + ICDT 2013) 1 / 30

2 Introduction Counting solutions of queries Counting is a fundamental algorithmic task It appears in many fields of computer science, maths, statistical physics,... However: Its complexity theory is well developed Counting is one of the natural aggregate functions for database queries Counting is harder than deciding the existence of a solution Very few natural polynomial time decision problems are also easy to count 2 / 30

3 Introduction Counting solutions of queries Investigate counting complexity for fragments of conjunctive queries Goals Exhibit islands of tractability Determine source of hardness for the counting of solutions Fully characterize the frontier between tractable and intractable for large classes of CQ. 3 / 30

4 Introduction Conjunctive Query Problem L = class of {, }-FO (conjunctive) formulas Counting Problems for Queries #CQ(L ) input: ϕ( x) L and a database/structure A output: ϕ(a) = {ā A k (A, ā) = ϕ( x)} Alternative problems: Decision: exists ā A k such that (A, ā) = ϕ? Enumeration/Generation: compute ϕ(a). 4 / 30

5 Introduction Example SELECT count(*) FROM EMP E WHERE E.dept= info AND E.job= manager AND E.salary > 40000; ϕ(number, name, dept, job, salary) = E(number, name, info, manager, salary) salary > / 30

6 Introduction Example: paths in graphs A = V, E graph, consider ϕ(v, w): x 1 x 2... x k 1 x i x j E(v, x 1 ) E(x i, x i+1 ) E(x k 1, w) i j i [k 2] Count the number of pairs of vertices (v, w) with length k paths between them. If projection (i.e. ) is disallowed: count the number of length k paths. Different meaning. 6 / 30

7 Introduction How to count: example ϕ(v, x 1, x 2, w) E(v, x 1 ) E(x 1, x 2 ) E(x 2, w) To be evaluated on the following structure A = (V, E): / 30

8 Introduction How to count: example ϕ(v, x 1, x 2, w) E(v, x 1 ) E(x 1, x 2 ) E(x 2, w) To be evaluated on the following structure A = (V, E): 0 One associates the polynomial Q(Φ)(X 0,..., X 9 ) below : / 30

9 Introduction How to count: example ϕ(v, x 1, x 2, w) E(v, x 1 ) E(x 1, x 2 ) E(x 2, w) To be evaluated on the following structure A = (V, E): One associates the polynomial Q(Φ)(X 0,..., X 9 ) below : X X 3 0 X 1 + X 3 0 X 2+ X 2 0 X 1X 3 + X 2 0 X 1X 4 + X 2 0 X 2X 4 + X 2 0 X 2X 5 + X 0 X 1 X 3 X 6 + X 0 X 1 X 3 X 7 + X 0 X 1 X 4 X 7 + X 0 X 2 X 4 X 7 + X 0 X 2 X 5 X 6 + X 1 X 3 X 6 X 8 + X 1 X 3 X 6 X 9 + X 1 X 3 X 7 X 9 + X 1 X 4 X 7 X 9 + X 2 X 4 X 7 X 9 + X 2 X 5 X 6 X 8 + X 2 X 5 X 6 X / 30

10 Introduction How to count: example ϕ(v, x 1, x 2, w) E(v, x 1 ) E(x 1, x 2 ) E(x 2, w) To be evaluated on the following structure A = (V, E): One associates the polynomial Q(Φ)(X 0,..., X 9 ) below : X X 3 0 X 1 + X 3 0 X 2+ X 2 0 X 1X 3 + X 2 0 X 1X 4 + X 2 0 X 2X 4 + X 2 0 X 2X 5 + X 0 X 1 X 3 X 6 + X 0 X 1 X 3 X 7 + X 0 X 1 X 4 X 7 + X 0 X 2 X 4 X 7 + X 0 X 2 X 5 X 6 + X 1 X 3 X 6 X 8 + X 1 X 3 X 6 X 9 + X 1 X 3 X 7 X 9 + X 1 X 4 X 7 X 9 + X 2 X 4 X 7 X 9 + X 2 X 5 X 6 X 8 + X 2 X 5 X 6 X 9 It holds that : Q(Φ)(1,..., 1) = ϕ(a) Nice but too long... needs to know all the solutions 7 / 30

11 Introduction Example But Q(Φ)(X 0,..., X 9 ) can be factorized knowing ϕ and A: Q(Φ)(X 0,..., X 9 ) = X X 3 0 X 1 + X 3 0 X 2+ X 2 0 X 1X 3 + X 2 0 X 1X 4 + X 2 0 X 2X 4 + X 2 0 X 2X 5 + X 0 X 1 X 3 X 6 + X 0 X 1 X 3 X 7 + X 0 X 1 X 4 X 7 + X 0 X 2 X 4 X 7 + X 0 X 2 X 5 X 6 + X 1 X 3 X 6 X 8 + X 1 X 3 X 6 X 9 + X 1 X 3 X 7 X 9 + X 1 X 4 X 7 X 9 + X 2 X 4 X 7 X 9 + X 2 X 5 X 6 X 8 + X 2 X 5 X 6 X 9 = (i,j) E X ix j ( k:(j,k) E X k( h:(h,i) E X h)) This polynomial can be succinctly represented (by this expression i.e. an arithmetic circuit). Then... It can be easily evaluated on any (reasonable) set of points. 8 / 30

12 Introduction Arithmetization of queries Given Φ = (A, ϕ), the polynomial Q(Φ) is defined as : Q(Φ)(X 1,.., X n ) := k X ai. ā φ(a) i=1 where n = A, size of the domain of the database. Counting : evaluating Q(Φ)(1,.., 1) Weighted counting by evaluating on particular values Weights can be put directly on tuples too. But... Evaluation is feasible when Q(Φ) admits a succint representation... So... Which queries have succintly representable Q-polynomials? 9 / 30

13 Known results Complexity of conjunctive queries: hardness Let s go back to the complexity of conjunctive queries... CQ is NP-complete for Combined complexity (i.e. database and formulas as inputs) So counting is even worst. Tractable fragments by either: considering restriction on the data considering restriction on the formulas : decomposition techniques. 10 / 30

14 Known results Formulas and their hypergraphs The hypergraph of a formula ϕ is the hypergraph H = (V, E) such that V is the set of variables of ϕ for each atom R( v) of ϕ, we associate a hyperedge var(r( v)) Q(a, b, c) R(a, e, f ) R(c, d, e) P(a, c, e, g) e f g d a b c Alternative: transform each hyperedge into a clique (Gaifman graph) 11 / 30

15 Known results Tractability of decision Tractability characterized through decomposition properties (and associated width measures) of the query. Principle Vertices and/or edges are grouped into clusters of fixed constant size Clusters are arranged into a tree Size of the clusters: width of a decomposition Specific additional conditions give different decompostions. Technics coming from : Graph theory : Robertson, Seymour Database Theory and logic : Gottlob, Courcelle, Leone, Scarcello, Grohe, Marx,... CSP : Jeavons, Cohen, Gyssens, / 30

16 Known results Tractability of decision {v 3, v 4, v 5, u 3, u 4, u 5 } {v 4, v 5, v 6, v 8 }, {v 7, v 8, u 5, u 6 } v 3 u 3 u 4 u 6 u 5 v 7 v 3, v 4, v 5, v 6, v 7, v 8, u 3, u 4, u 5, u 6 v 4 v 5 v 8 v 2 u 2 u 1 v 1 v 6 u 7 u 8 v 9 {v 1, u 1 }, {v 2, u 1, u 2 }, {v 2, v 4, u 2, u 3 } v 1, v 2, v 4, u 1, u 2, u 3 {v 4, v 5, v 6, v 8 }, {v 6, v 9, u 7 }, {v 8, v 9, u 8 } v 4, v 5, v 6, v 8, v 9, u 7, u 8 Measures for being "nearly" acyclic For graph representation: bounded tree-width For hypergraph representation, long list of decomposition notions: biconnected component, cycle cutset, cycle hypercutset, hinge-tree, hypertreewidth, generalized hypertreewidth, / 30

17 Known results Tractable fragments for decision: known results For all class L of decomposable hypergraphs above, given ϕ L and a database A, deciding if ϕ(a) = i.e. if the query has a solution can be done in polynomial time. If L ="acyclic hypergraph" : Yannakakis algorithms For all cases: rely strongly on the underlying tree structure given by the decomposition Warning: sometimes deciding if the formulas is decomposable (for some decomposition method) is... hard. 14 / 30

18 Known results Tractable fragments for counting Can not do better than for decision... but that s not an answer Tractability is obtained when the two properties below are satisfied... Desirable properties For every k the class of queries of width k should be tractable, i.e. Boolean CQ (resp. #CQ) should be solvable in polynomial time. Given an instance it should be possible to decide if there is a decomposition of width k and construct one if it exists. In other words, one wants a catalog of classes: for which the counting problem is easy to solve for which is is easy to decide if a given query belongs to the class 15 / 30

19 Known results Efficient Counting: informal result For all classes L above, there is a polynomial time algorithm s.t. given a projection free (i.e. without existential quantification) ϕ( x) L, a database A computes ϕ(a). How? (or why?) Based on Yannakakis algorithms, it is possible to build a succint representation of the polynomial Q(Φ). Will be a "small" alternation of and.. Not true anymore for ϕ 1 ( x) ϕ 2 ( x) or ϕ 1 ( x) ϕ 2 ( x) for "nice" formulas ϕ 1, ϕ 2 And why assuming projection free? 16 / 30

20 Known results Back to the example: introducing projections ϕ (v, w) x 1 x 2 E(v, x 1 ) E(x 1, x 2 ) E(x 2, w) / 30

21 Known results Back to the example: introducing projections ϕ (v, w) x 1 x 2 E(v, x 1 ) E(x 1, x 2 ) E(x 2, w) The polynomial Q(Φ)(X 0,..., X 9 ) was: X X 3 0 X 1 + X 3 0 X 2+ X 2 0 X 1X 3 + X 2 0 X 1X 4 + X 2 0 X 2X 4 + X 2 0 X 2X 5 + X 0 X 1 X 3 X 6 + X 0 X 1 X 3 X 7 + X 0 X 1 X 4 X 7 + X 0 X 2 X 4 X 7 + X 0 X 2 X 5 X 6 + X 1 X 3 X 6 X 8 + X 1 X 3 X 6 X 9 + X 1 X 3 X 7 X 9 + X 1 X 4 X 7 X 9 + X 2 X 4 X 7 X 9 + X 2 X 5 X 6 X 8 + X 2 X 5 X 6 X / 30

22 Known results Back to the example: introducing projections ϕ (v, w) x 1 x 2 E(v, x 1 ) E(x 1, x 2 ) E(x 2, w) The polynomial Q(Φ)(X 0,..., X 9 ) was: X X 3 0 X 1 + X 3 0 X 2+ X 2 0 X 1X 3 + X 2 0 X 1X 4 + X 2 0 X 2X 4 + X 2 0 X 2X 5 + X 0 X 1 X 3 X 6 + X 0 X 1 X 3 X 7 + X 0 X 1 X 4 X 7 + X 0 X 2 X 4 X 7 + X 0 X 2 X 5 X 6 + X 1 X 3 X 6 X 8 + X 1 X 3 X 6 X 9 + X 1 X 3 X 7 X 9 + X 1 X 4 X 7 X 9 + X 2 X 4 X 7 X 9 + X 2 X 5 X 6 X 8 + X 2 X 5 X 6 X 9 The new polynomial Q(Φ )(X 0,..., X 9 ) is now : X X 0X 1 + X 0 X 2 + X 0 X 3 + X 0 X 4 + X 0 X 5 + X 0 X 6 + X 0 X 7 + X 1 X 8 + X 1 X 9 + X 2 X 9 + X 2 X 8 17 / 30

23 Known results Back to the example: introducing projections ϕ (v, w) x 1 x 2 E(v, x 1 ) E(x 1, x 2 ) E(x 2, w) The polynomial Q(Φ ) is shorter (but can generaly be exponential bigger than A ) It is not a priori obtained by an algebraic operation from Q(Φ) (but in this case can be obtained from Q(Φ)) Is it hopeless? / 30

24 Optimal tractability result The hardness result revisited Pichler, Skritek 11: #CQ is #P-hard even for formulas whose hypergraphs are trees (acyclic) and with only a single -quantifier. Proof (idea): Given graph G = (V, E) and k N counting cliques of size k in G is #P-hard. Reduce this (undirectly) to #CQ for acyclic formulas. End of the story? 19 / 30

25 Optimal tractability result The hardness result revisited Pichler, Skritek 11: #CQ is #P-hard even for formulas whose hypergraphs are trees (acyclic) and with only a single -quantifier. Proof (idea): Given graph G = (V, E) and k N counting cliques of size k in G is #P-hard. Reduce this (undirectly) to #CQ for acyclic formulas. End of the story? The resulting formula has this hypergraph: 9 19 / 30

26 Optimal tractability result S-hypergraphs and components S-hypergraph of a formula : associated hypergraph and set S to mark "free variables". Idea 1: quantified variables can only interact if they are connected through other quantified variables look at the non interacting parts independently S-component An S-component of an S-hypergraph (H, S) is a maximal set of edges that is connected by vertices in V \ S. 20 / 30

27 Optimal tractability result Decomposition into S-components: example S vertices : 21 / 30

28 Optimal tractability result Decomposition into S-components: example S vertices : 22 / 30

29 Optimal tractability result S-star size Idea: measure how free variables are spread in each piece. The S-star size of H is the size of the biggest independent set in a S-component. S-vertices : S-star size is 4 23 / 30

30 Optimal tractability result quantified-star size Quantified star size of a formula: S-star size of its associated hypergraph. Formula ϕ(x, y) t zr(x, y, t) S(x, z, t) Quantified star size = 1 Path formulas (of arbitrary length), e.g. ϕ(x, y) t 1 t 2 t 3 R(x, t 1 ) R(t 1, t 2 ) R(t 2, t 3 ) R(t 3, y) Quantified star size = 2 Star formulas, e.g. ϕ(x, y, z, t) ur(u, x) R(u, y) R(u, z) R(u, t) Quantified star size = degree of the center of the star (here 4). 24 / 30

31 Optimal tractability result Tractable counting for S-star size Let L be any nicely decomposable class seen so far. Theorem There is an algorithm that given: a #CQ-instance Φ = (A, ϕ) of quantified starsize l and ϕ L a L-decomposition of Φ of width k counts the solutions of ϕ(a) in time Φ p(k,l) for a fixed polynomial p. Works also for weighted queries: ā=(a 1,...,a k ) φ(s) k i=1 w(a i) 25 / 30

32 Optimal tractability result Tractable counting for S-star size : proof idea Step 1: Reduce to quantifier free queries (with same generalized hypertree width). Technical. Step 2: Reduce to acyclic formulas (while preserving the number of solutions). Standard Step 3: Solve the acyclic case by arithmetization of the query. Fun 26 / 30

33 Optimal tractability result Optimality of the result FPT : f (k) n c, with fixed c, f is a function #W[1] : power of computing k-cliques, O(n k ) Let L be any nicely decomposable class seen so far. Theorem If FPT #W[1], for any (recursively enumerable) subclass C of L, the following statements are equivalent: #CQ for instances in C can be solved in polynomial time C is of bounded quantified star size. Tractability... but for fixed decomposition 27 / 30

34 Optimal tractability result Optimality of the result: bounded arity FPT : f (k) n c, with fixed c, f is a function W[1] : power of deciding the existence of a k-cliques, O(n k ) Theorem Let G be a recursively enumerable class of S-hypergraphs of bounded arity. Assume that W[1] FPT. Then the following statements are equivalent: 1 #CQ for all instances whose S-hypergraph is in G is solvable in polynomial time. 2 There is a constant c such that for each S-hypergraph (H, S) in G the treewidth of H and the S-star size are at most c. 28 / 30

35 Tractable discovery Tractability of Discovery Star size of an hypergraph/formula need to be computed efficiently... in order to claim that tractable cases are really tractable... Theorem (non-strict) Let β be any decomposition technique for hypergraphs commonly considered in the literature. Then maximum independent sets on hypergraphs of β-width at most k can be computed in time n O(k). some decomposition techniques allow FPT-algorithms (tree decompositions, hinge decompositions). Tricky... for most other decompositions W[1]-hard and thus probably not FPT for most decomposition techniques also polynomial time k-approximation algorithm 29 / 30

36 Tractable discovery Open questions Push arithmetization technics for decision in different contexts (bag semantics) change representation of relations: negative representation, SAT, mixed representation,... allow nesting of queries, universal quantification, / 30