Query Evaluation on Probabilistic Databases. CSE 544: Wednesday, May 24, 2006

Transcription

1 Query Evaluation on Probabilistic Databases CSE 544: Wednesday, May 24, 2006

2 Problem Setting Queries: Tables: Review A(x,y) :- Review(x,y), Movie(x,z), z > 1991 name rating p Movie Monkey Love good.5 title year p Twelve Monkeys Monkey Love fair.2 fair.6 poor.9 Answers: Top k Monkey Love title rating p Monkey Love Pl Twelve Monkeys fair.53 Problem: complexity Monkey Love good.42 Monkey Love Pl fair.15 of query evaluation 2

3 Two Problems Fixed schema S, conjunctive query Q(x,y) Query evaluation problem Fix answer tuple (a,b) Given database I, compute Pr(Q(a,b)) Top-k answering problem Fix k > 0 Given database I, find k answer tuples with highest probabilities 3

4 Related Work: DB Cavallo&Pitarelli:1987 Barbara,Garcia-Molina, Porter:1992 Lakshmanan,Leone,Ross&Subrahmanian:1997 Fuhr&Roellke:1997 Dalvi&S:2004 Widom:2005 4

5 Related Work: Logic Query reliability [Gradel,Gurevitch,Hirsch 98] Degrees of belief [Bacchus,Grove,Halpern,Koller 96] Probabilistic Logic [Nielson] Probabilistic model checking [Kwiatkowska 02] Probabilistic Relational Model [Taskar,Abbeel,Koller 02] 5

6 Probabilistic Database Schema S, Domain D, Set of instances Inst Definition Probabilistic database is a probability distribution Pr : Inst [0,1], I Pr[I] = 1 If Pr[I] > 0 then I is called possible world 6

7 Probabilistic Database Representation: Independent tuples: I-database DB over some schema S i Independent and disjoint tuples: ID-database DB over some schema S id Semantics: DB means probability distribution Pr over schema S 7

8 Independent Events A tuple is in the database with probability p Any two tuples are independent events 8

9 Representation I-Databases Reviews i (M,S,p) Movie Score P m42 good p 1 m99 good p 2 Reviews(M,S) m76 poor p 3 Mov Scor Mov Scor Mov Scor Mov Scor Mov Scor Mov Scor Mov Scor Mov Scor m42 good m99 good m42 good m76 poor m42 good m m42 good Pr[I 1 ]= (1-p 1 )*(1-p 2 )*(1-p 3 ) Pr[I 4 ]= m99 good m76 poor p 1 *p 2 *(1-p 3 ) Pr[I 1 ] + Pr[I 2 ] Pr[I 8 ] = 1 m76 poor Pr[I 8 ]= m99 good m76 poor p 1 *p 2 *p 3 Possible worlds semantics 9

10 Disjoint Events Needed in Many-to-1 matchings Possible values for attributes [Barbara 92] Name Age Name Age P John Mary (0.3) 43 (0.7) John John Mary

11 ID-Databases Activities id Time d Activity P t walk p 1 t run p 2 Activities t+1 walk p 3 Time Act Time Act Time Act Pr[I 1 ]= t (1-p 1 -p 2 )*(1-p 3 ) walk Pr[I 3 ]= t run p 2 *(1-p 3 ) Time Act t+1 walk Pr[I 5 ]= Time Pr[I 1 ] + Pr[I 2 ] Pr[I 6 ] = 1 11 t Act walk t+1 walk p 1 *p 3 Time Act t run t+1 walk

12 ID subsumes I Reviews id Movie d Score d P m42 good p 1 m99 good p 2 m76 poor p 3 = Reviews i Movie Score P m42 good p 1 m99 good p 2 m76 poor p 3 Note: Movie Score P Reviews id m42 good p 1 m99 good p 2 means all tuples are m76 poor p 3 disjoint 12

13 Queries Syntax: conjunctive queries over schema S Q(y) :- Movie(x,y), Review(x,z), z >= 3 Movie i id year P m m m m Review i mid rating p m m m m m

14 Two Query Semantics Possible answer sets Given set A: Pr[{t I Q(t)} = A] Used for views This Possible tuples talk Given tuple t: Pr[I Q(t)] Used for query evaluation and top-k 14

15 Query Semantics id year p 1 id year p 2 id year p 3 id year p 4 m m m m m m m m m m Q(y) :- Movie(x,y), Review(x,z) year p Tuple probabilities 1935 p 2 + p 3 = p 1 + p 3 = p 3 = top k 15

16 Summary on Data Model Data Model: Semantics = possible worlds Syntax = I-databases or ID-databases Queries: Syntax = unchanged (conjunctive queries) Semantics = tuple probabilities 16

17 Problem Definition Fix schema S, query Q, answer tuple t Problem: given I/ID-database DB, compute Pr[I Q(t)] notation: Pr[Q(t)] Conventions: For upper bounds (P or #P): probabilities are rationals For lower bounds (#P): probabilities are 1/2 17

18 Query Evaluation on I-Databases Outline Intuition Extensional plans: PTIME case Hard queries: #P-complete case Dichotomy Theorem 18

19 Q(y) :- Movie(x,y), Review(x,z) Movie i Intuition Review i Answer id year p m p 1 m p 2 m p 3 m p 4 mid rate p m42 4 q 1 m42 2 q 2 m42 3 q 3 m99 1 q 4 m99 3 q 5 m76 5 q 6 Year 1995 p p 1 (1 - (1 - q 1 ) (1 - q 2 ) (1 - q 3 )) (1 - p 2 (1 - (1 - q 4 ) (1 - q 5 ))) (1 - p 3 q 6 ) 19

20 I-Extensional Plans [Barbara92,Lakshmanan97] Add p Join p = p 1 * p 2 Projection p = 1-(1-p 1 )(1-p 2 )...(1-p n ) Selection σ p = p Note: data complexity is PTIME 20

21 Extensional Query Plans x x pq x 1-(1-p1)(1-p2)(1-p3) x p σ x p x q x x x p1 p2 p3 x p 21

22 Extensional Query Plans Each tuple t has a probability t.p Algebra operators compute t.p Data complexity: PTIME 22

23 (1-pq 1 )(1-pq 2 )(1-pq 3 ) Q(y) :- Movie(x,y), Review(x,z) 1995 m1 pq m1 p(1-(1-q 1 )(1-q 2 )(1-q 3 )) 1995 m1 pq m1 pq 3 m1 1 - (1-q 1 )(1-q 2 )(1-q 3 ) Movie 1995 p Review m1 q 1 Movie Review m1 q p m1 q 1 m1 q 3 INCORRECT! 23 m1 q 2 m1 q 3 CORRECT

24 #P-Complete Queries R i S T i A p A B B p p 1 q 1 p 2 q 2 p 3 q 3 p 4 q 4 Q bad :- R i (x), S(x,y), T i (y) Theorem: Data complexity is #P-complete

25 Proof: Theorem [Provan&Ball83] Counting the number of satisfying assignments for bipartite DNF is #P-complete R i S T i Reduction: A p x 1 1/2 A B x 2 y 3 B p y 1 1/2 x 2 y 3 V x 1 y 2 V x 4 y 3 V x 3 y 1 x 2 1/2 x 1 y 2 y 2 1/2 x 3 1/2 x 4 y 3 y 3 1/2 x 4 1/2 x 3 y 1 Q bad :- R i (x), S(x,y), T i (y) 25

26 I-Dichotomy Q = boolean conjunctive query Definition 1. For each variable x: goals(x) = set of goals that contain x Definition 2. Q is hierarchical if forall x, y: (a) goals(x) goals(y) =, or (b) goals(x) goals(y), or (c) goals(y) goals(x)

27 Q :- R(x),S(x,y),T(x,y,z),K(x,v) x y z S R T v K hierarchical Q :- R(x), S(x,y), T(y) x y non-hierarchical R S T 27

28 [Dalvi&S. 04] I-Dichotomy Schema S i = {R 1i, R 2i,..., R mi } Theorem Let Q = conjunctive query w/o self-joins. Then one of the following holds: Q is in PTIME Q has a correct extensional plan or: Q is hierarchical Q is #P-complete Q has subgoals R(x,...),S(x,y,...),T(y,...) 28

29 Proof Lemma 1. If Q is non-hierarchical, then #P-complete Proof: v x y R S T K z Q :- R i (v,x), S i (x,y), T i (y,z), K i (z) rest is like for Q bad 29

30 Proof Lemma 2. If Q is hierarchical, then PTIME Proof: Case 1: has no root Pr(Q) = Pr(Q 1 ) Pr(Q 2 ) Pr(Q 3 ) This is extensional join 30

31 Proof Case 2: has root x x Dom={a 1, a 2,..., a n } Pr(Q) = 1 - (1-Pr(Q(a 1 /x))(1-pr(q(a 2 /x))...(1-pr(q(a n /x))) This is an extensional projection: QED 31

32 Query Evaluation on ID-Databases ID-extensional plans #P-complete queries Dichotomoy Theorem 32

33 Extensional Plans for ID-DBs Only difference: two kinds of projections: independent 1-(1-p 1 )...(1-p n ) disjoint p p n 33

34 #P-Complete Queries Q 1 :- R i (x), S i (x,y), T i (y) Q 2 :- R d (x d,y), S d (y d ) Q 3 :- R d (x d,y), S d (z d,y) 34

35 [Dalvi&S. 04] I-DB Dichotomy Schema S id s.t. each table is either R i or R id Theorem Let Q = conjunctive query w/o self-joins. Then one of the following holds: Q is in PTIME Q has a correct extensional plan or: Q is #P-complete Q has one of Q 1, Q 2, Q 3 as subqueries 35

36 Extensions Extensions of the dichotomoy theorem exists for: Mixed schemas (some relations are deterministic) Functional dependencies 36

37 Summary on Query Evaluation Extensional plans: popular, efficient, BUT Equivalent plans lead to different results Some queries admit correct plans Some simple queries: #P-complete complexity Dichotomy theorem Future work: remove no-self-join restriction 37

38 Conclusions Strong motivation from practical applications Merge query and search technologies Probabilistic DB s are hard! Hacks don t work (yet). Need principled approach. 38

39 Thank you! Questions?