Efficient query evaluation

Transcription

1 Efficient query evaluation Maria Luisa Sapino Set of values E.g. select * from EMPLOYEES where SALARY = 1500; Result of a query Sorted list E.g. select * from CAR-IMAGE where color = red ; 2 Queries as boolean combinations How do we combine sets and sorted lists? : example: database containing information about CDs query: return the names of all albums whose artist is the Queen, and whose cover is mostly black. (i) traditional database query, asking for the names of all albums whose artist is Queen --> set (ii) multimedia query asking for album covers which are black --> sorted list (Artist = Queen ) and (AlbumColor = black ) 3

2 Queries as boolean combinations What if we replace and with or? Is the answer a set, a list, or a combination of the two? How about if we combine two multimedia queries? (color = black ) and ( title contains kind ) the answer is a sorted list sorted according to which criterion? 4 Graded sets sets of pairs (x, g), where x is an object (such as a tuple) g is a real number in [0,1] graded sets as sorted lists, where the objects are sorted by their grades graded sets of generalization of both sets and sorted lists 5 Assumptions All data in all the subsystems deal with the attributes of a specific set of objects of some fixed types Atomic queries are of the form X = t, where X is the name of an attribute, and t is a target Queries are boolean combination of atomic queries For each atomic query, a grade is assigned to each object 6

3 Dealing with boolean combinations There are a number of aggregation functions that assign a grade to a fuzzy conjunction, as a function of the grades assigned to the conjuncts Standard rules of fuzzy logic (Zadeh, 1965) use min as the aggregation function: conjunction rule: µ A B (x) = min{ µ A (x), µ B (x)} disjunction rule : µ A B (x) = max{ µ A (x), µ B (x)} negation rule: µ not A (x) = 1- µ A (x) 7 Standard fuzzy rules: pros They are a conservative extension of the standard propositional semantics Theorem of Bellman and Giertz: The unique aggregation functions for evaluating AND and OR that preserve logical equivalence of queries involving only conjunction and disjunction and that are monotonic in their arguments are min and max. 8 Logical equivalence preservation If Q 1 and Q 2 are logically equivalent queries involving only conjunction and disjunction (not negation), then µ Q1 (x) = µ Q2 (x), for every x Examples: µ A A (x) = µ A (x) µ A (B C) (x) = µ (A B) (A C) (x) 9

4 Monotonicity µ A (x) <= µ A (x ) and µ B (x) <= µ B (x ) => µ A B (x) <= µ A B (x ) 10 Algorithms for query evaluation Different subsystems are involved in query answering, and information coming from them is pieced together by the middleware system (e.g., Garlic [Fagin]) (Artist = Queen ) (AlbumColor = black ) Assumption: there are not too many objects that satisfy the first conjunct Intuitive algorithm: Compute S = set of objects that do satisfy (Artist = Queen ) using random access, obtain grades (e.g. from QBIC) for the objects in S sort objects in S according to the returned grades 11 Algorithms for query evaluation (AlbumColor = black ) (Shape = circle ) assumption: one subsystem deals with colors, and another one deals with shapes. The grade of any object x under the query is the minimum of the grades of x under the subqueries A1= (AlbumColor = black ), and A2= (Shape = circle ) how to get TOP k elements? 12

5 TOP k naive algorithm Have the subsystem dealing with color to output explicitely the graded set containing all pairs (x, µ A1 (x) ), for each x Have the subsystem dealing with shape to output explicitely the graded set containing all pairs (x, µ A2 (x) ), for each x Compute µ A1 A2 (x) = min{(µ A1 (x), µ A2 (x) } for every x Return the top k objects, braking arbitrarily the ties. Note: The constant k plays a role only at the 4th step improving efficiency (Fagin) Assumption: Sorted access the subsystems can output the graded set consisting of all objects one by one, in sorted order based on grade, until the middleware system tells the subsystem to stop the middleware system can later tell the subsystem to resume outputting the graded set where it left off. (alternatively, the sorted objects can be output in sets of cardinality k, for any constant k, instead of one by one) Random access the middleware could ask the subsystem the grade, wrt a query, of any given object 14 Evaluating conjunctions of atomic queries Query: Q= F t (A 1, A 2,, A m ), where Q is monotonic t is the aggregation function each A i is an atomic query, evaluated by the subsystem i. Assumption: there are at least k objects (so that top k answers make sense) 15

6 Algorithm Sorted Access Phase: for each i, give the query A i under sorted access. Stop when there are at least k matches, that is, when there is a set L of at least k objects such that all the subsystems have output all the members of L Random Access Phase: for each object x that has been seen, do random access to each subsystem j to find µ Aj (x) Computation Phase: Compute the grade µ Q (x) =t(µ A1 (x), µ A2 (x),, µ Aj (x) ) for each object x that has been seen. Return the graded set Y containing the objects with the k highest grades 16?? X X X Assumptions: Q is monotonic Predicates provide sorted_access Predicates provide random_access X3 X1 X4 17 Sorted Access Phase (k=3)?? X X X X3 X1 X4 Assumptions: Q is monotonic Predicates provide sorted_access Predicates provide random_access 18

7 Sorted Access Phase (k=3)?? X X X X3 X1 X4 Assumptions: Q is monotonic Predicates provide sorted_access Predicates provide random_access 19 Sorted Access Phase (k=3)?? 0.90? X X X X3 X1 X4 Assumptions: Q is monotonic Predicates provide sorted_access Predicates provide random_access 20 Random Access Phase (k=3)?? X X X X3 X1 X4 Assumptions: Q is monotonic Predicates provide sorted_access Predicates provide random_access 21

8 Result (k=3) X X X X3 X1 X4 Assumptions: Q is monotonic Predicates provide sorted_access Predicates provide random_access 22 Advantage!! X X X X3 X1 X4 X1 and X4 have never been accessed! 23 Properties of the algorithm Correctness hold for monotone queries After finding the top k answers, in order to find the next k best answer we can continue where we left off. If the t function is min, the algorithm can be made more efficient. How??? 24

9 Use only one pred. for sorted access (k=3)?? X X X X3 X1 X4 25 Sorted+Random Access (k=3)?? X X X X3 X1 X4 Stop when the next value is smaller than the third candidate 26 Sorted+Random Access (k=3) X X X X3 X1 X4 X1, X3, and X4 have never been accessed! 27

10 Evaluating disjunctions of atomic queries Query: Q = A 1 A 2 A m Q : standard fuzzy disjunction Sorted Access Phase: for each i, give the query A i and collect the top k answers to this query Computation Phase: for each object x that has been returned by any of the m subsystems, compute h(x) = max{ µ Ai (x) }, for all i. Return the graded set Y containing the objects with the k highest grades h(x) 28 Weighting the Importance of Subqueries Q= (AlbumColor = black ) and (Shape = circle ) What if the user cares twice as much about colors as shape? Intuitively, we wish to assign twice as much weight to color as to shape in the user interface, sliders are one mechanism to convey information about the weights need to define the weighted version of aggregation functions 29 Weighting the Importance of Subqueries (ctd) The query Q is A 1 A 2 A m. g i is the score of conjunct A i f: function whose domain is the set of all tuples, of all sizes, over [0,1], and with range [0,1] f(g 1,,g m ) is the overall score. θ 1,, θ m >=0, θ i = 1 θ i is the weight of attribute i Θ= (θ 1,, θ m ) is a weighting 30

11 Weighting the Importance of Subqueries (ctd) For each weighting Θ= (θ 1,, θ m ), a function f Θ is derived, whose domain is the set of m- tuples f Θ (g 1,,g m ) is the overall score, when the weights are given by the weighting Θ ifθ 1 >= θ 2 >= >= θ m, then Θ= (θ 1,, θ m ) is said ordered. 31 Desiderata... f (1/m,, 1/m ) (g 1,,g m ) = f (g 1,,g m ) if all the weights are equal, the weighted function f Θ coincides with the corresponding unweighted one f (θ 1,., θm 1,0) (g 1,,g m ) = f (θ 1,., θm 1) (g 1,,g m ) if a particular argument has 0 weight, it can be dropped f (θ 1,., θm) (g 1,,g m ) is a continuous function of θ 1,, θ m 32 Weighting formula When θ 1 >= θ 2 >= >= θ m, f Θ (g 1, g m ) = (θ 1 -θ 2 ). f(g 1 ) + 2. (θ 2 -θ 3 ). f(g 1,g 2 )+ 3. (θ 3 -θ 4 ). f(g 1, g 2, g 3 )+ + m. θ m. f(g 1,...,g m ) Note: the weighting formula is well defined even when some θ i are equal. 33

12 Weighting formula (ctd) Monotonicity and strictness of the (unweighted) f is inherited by the weighted functions f Θ. In particular, the optimal algorithm to evaluate conjunctions of atomic queries preserves correctness and optimality also in the weighted case. 34 Adding filter conditions (Chauduri and Gravano) 35 Query model requirements Account for the grade of match between the value of an attribute of a multimedia object, and a given constant Grade (attr, value) (o) Real number in [0,1] attribute object 36

13 Query model requirements (2) Allow the user to specify thresholds on the grade of match of the acceptable objects (filter conditions) Atomic filter condition:» Grade (attr, value) (o)>=grade Compound conditions» conjunction/disjunction of conditions 37 Query model requirements(3): Enable the user to ask for only a few topmatching objects (ranking expressions) to compute a composite grade for an object from individual grades of match and the composition functions Min, Max 38 SQL-like syntax SQL-like syntax Select oid from Repository where Filter_condition Order[k] by Ranking_expression 39

14 example The repository contains information about criminals A record on every person on file consists of A textual description p (profile) A scanned fingerprint f A recording of a voice sample v. Query: Select oid from Repository where (Grade (v, V)>=0.5 and Grade(p, american citizen )>=0.9) or Grade(f,F) >= order[10] by max (grade(f,f), Grade(v,V)) Expressivity of the query model Do we really need both filter condition (F) and ranking condition (R)??? Can we embed the filter condition in a new ranking expression R F so that the top objects of R F are the top objects for R that satisfy F? 41 Expressivity of the query model Do we really need both filter condition (F) and ranking condition (R)??? Can we embed the filter condition in a new ranking expression R F so that the top objects of R F are the top objects for R that satisfy F? NO (counterexample!!!!!) 42

15 counterexample Let e1 = Grade(a1,v1) e2=grade(a2,v2) Filter condition F = e1 >= 0.2 Ranking expression R =e2 By contradiction, exists RF that satisfies both F and R RF equivalent to one of the following: e1, e2, min(e1,e2), max (e1, e2) 43 database (example) object e1 e2 Min(e1,e2) Max(e1,e2) o o o Storage-level access interface GradeSearch(attribute, value, min_grade) TopSearch (attribute, value, count) Probe(attribute, value, {oid}) Not all the repositories have to support all of these interfaces 45 at the physical level.

16 Filter conditions Assumption: the filter conditions are independent A filter condition f is independent if: Every atomic filter condition occurs at most once in f p(e 1 e n ) = Π (i=1..n) p(e i ), being any e i an atomic filter condition, and p(e i ) the probability that the filter condition e i is true. The repository requires the use of an index to evaluate every filter condition. 46 Possible evaluation strategies Use one GRADESEARCH for EACH atomic condition in the filter condition, and then merge the returned sets of object ids through a sequence of union/intersections Use GRADESEARCH only for SOME atomic condition, and for the rest of the conditions use PROBE The key optimization problem becomes to determine the set of filter conditions that are evaluated using Gradesearch. 47 Search minimal condition Goal: given a filter condition f, characterize the smallest sets of atomic conditions such that by searching the conditions in any of these sets we retrieve all the objects that satisfy f Example f =a 4 ((a 1 a 2 ) a 3 ) 48

17 Search minimal condition By searching on a condition using Grade Search we obtain a set of objects Some additional probes might be needed, to determine the set of objects that satisfy the rest of the condition as well residue of f for a Res(a,f) Boolean condition that the object retrieved using a must satisfy, in order to satisfy f. 49 residue examples f =a 4 ((a 1 a 2 ) a 3 ) Res( a 2, f) = a 1 a 4 Res(a 4,f) = (a 1 a 2 ) a 3 50 Search minimal conditions GOAL: to characterize the smallest sets of atomic conditions such that, by searching the conditions in any of these sets, we retrieve all the objects that satisfy f (plus some extra ones that are pruned out with probing) Example f =a 4 ((a 1 a 2 ) a 3 ) Search minimal condition sets: {a 4 } {a 1, a 3 } {a 2, a 3 } 51

18 Example (ctd) If we decide to search on {a 2, a 3 } Search on a 2, and probe the retrieved objects with Res( a 2, f) = a 1 a 4. Keep the objects that satisfy Res( a 2, f) Search on a 3, and probe the retrieved objects with Res( a 3, f) = a 4 Keep the object that satisfy Res( a 3, f) Return the objects kept. 52 Search minimal execution Execution like the one given in the previous example, based on any search minimal condition set. 53 Search minimal execution Execution like the one given in the previous example, based on any search minimal condition set. How do we pick a plan from the space of search minimal executions? 54

19 The cost model Statistics associated with each atomic condition a: Selectivity Factor Sel(a) Fraction of the objects in the repository that satisfy the condition a Search Cost SC(a) Cost of retrieving the ids of the objects that satisfy the condition a using Grade Search Probe Cost PC(a,p) Cost of checking the condition a for p objects, using the probe access method. 55 Cost of the Search minimal executions m: search minimal condition, for the filter condition f w: algorithm for probing conditions O a : number of objects that satisfy the condition a, that is, the product of the number of objects by the selectivity of a. R(a,f): residue of f for a (boolean condition that the objects retrieved using a should satisfy to satisfy the entire condition f) C w (f,m) = Σ a m (SC(a) + PC w (R(a,f), O a )) 56 Optimal search minimal condition for f Let f be a filter condition, and f a subexpression of f Inductive search-minimal condition for f wrt f: If f = a (atomic condition), then SM f (f ) ={a} If f = f 1 f n, then SM f (f ) = SM f ( f n, ), where C(f, SM f (f n, )) = min { C(f, SM f ( f 1, )),, C(f, SM f ( f n, )) } (break the ties arbitrarily) If f = f 1 f n, then SM f (f ) = SM f ( f 1, ) SM f ( f n, ) If f is independent, then SM f ( f ) is optimal 57

20 Theorem: The problem of determining an optimal search-minimal condition set for ARBITRARY filter conditions is NP-hard 58