Database design and implementation CMPSCI 645

Transcription

1 Database design and implementation CMPSCI 645 Lectures 20: Probabilistic Databases *based on a tutorial by Dan Suciu

2 Have we seen uncertainty in DB yet? } NULL values Age Height Weight 20 NULL 200 NULL Denotes uncertainty / lack of informagon 2

3 Databases today are deterministic } An item either is in the database or is not } A tuple either is in the query answer or is not } This applies to all variety of data models: } RelaGonal, E/R, NF2, hierarchical, XML, 3

4 What is a probabilistic database? } An item belongs to the database is a probabilisgc event } A tuple is an answer to the query is a probabilisgc event } Can be extended to all data models; we discuss only probabilisgc rela%onal data 4

5 Example: Information extraction...52 A Goregaon West Mumbai... [Gupta&Sarawagi 2006] ID House-No Street City P 1 52 Goregaon West Mumbai A Goregaon West Mumbai Goregaon West Mumbai A Goregaon West Mumbai Here probabiliges are meaningful 20% of such extracgons are correct 7

6 Example: RFID ecosystem at UW [Welbourne 2007] 8

7 Two types of probabilistic data } Database is determinisgc Query answers are probabilisgc } Database is probabilisgc Query answers are probabilisgc 10

8 Motivating applications } Text extracgon / record linking } Inconsistent data } Ranking query answers 11

9 Ranking query answers Database is determinisgc The query returns a ranked list of tuples } User interested in top-k answers. 12

10 [Agrawal,Chaudhuri,Das,Gionis 2003] The empty answers problem Query is over-specified: no answers Example: try to buy a house in Amherst SELECT * FROM Houses WHERE bedrooms = 4 AND style = crafsman AND district = downtown AND price <

11 [Agrawal,Chaudhuri,Das,Gionis 2003] Ranking: Compute a similarity score between a tuple and the query Q = SELECT * FROM R WHERE A 1 =v 1 AND AND A m =v m Query is a vector: Tuple is a vector: Q = (v 1,, v m ) T = (u 1,, u m ) Rank tuples by their TF/IDF similarity to the query Q Includes pargal matches 14

12 [Motro:1988,Dalvi&S:2004] Similarity predicates in SQL Beyond a single table: Find the good deals in a neighborhood SELECT * FROM Houses x WHERE x.bedrooms ~ 4 AND x.style ~ crafsman AND x.price ~ 600k AND NOT EXISTS (SELECT * FROM Houses y WHERE x.district = y.district AND x.id!= y.id AND y.bedrooms ~ 4 AND y.style ~ crafsman AND y.price ~ 600k Users specify similarity predicates with ~ System combines atomic similariges using probabiliges 15

13 [HrisGdis&PapakonstanGnou 2002,BhaloGa et al.2002] Keyword search in databases Goal: } Users want to search via keywords } Do not know the schema Techniques: } Matching objects may be scarered across physical tables due to normalizagon; need on the fly joins } Score of a tuple = number of joins, plus presgge based on in-degree 16

14 [HrisGdis,PapakonstanGnou 2002] Q = Abiteboul and Widom Join sequences (tuple trees): Paper Author Author Abiteboul Person Paper In Conference Paper Paper Person Widom Conference Author Editor Author Author Person Abiteboul Person Widom Person Paper Author Conference Person Widom Person Abiteboul 17

15 Record linkage Determine if two data records describe same object Scenarios: } Join/merge two relagons } Remove duplicates from a single relagon } Validate incoming tuples against a reference 18

16 [Bertosi&Chomicki:2003] Inconsistent data Goal: consistent query answers from inconsistent databases ApplicaGons: } IntegraGon of autonomous data sources } Un-enforced integrity constraints } Temporary inconsistencies 23

17 [Bertosi&Chomicki:2003] The repair semantics Key (?!?) Name Affilia*on State Area Miklau UW WA Data security Dalvi UW WA Prob. Data Balazinska UW WA Data streams Balazinska MIT MA Data streams Miklau UMass MA Data security Find people in State=WA Dalvi Find people in State=MA ; Consider all repairs High precision, but low recall 24

18 Alternative probabilistic semantics Name Affilia*on State Area P Miklau UW WA Data security 0.5 Dalvi UW WA Prob. Data 1 Balazinska UW WA Data streams 0.5 Balazinska MIT MA Data streams 0.5 Miklau Umass MA Data security 0.5 State=WA Dalvi, Balazinska(0.5), Miklau(0.5) State=MA Balazinska(0.5), Miklau(0.5) Lower precision, but berer recall 25

19 [Barbara et al.1992] What is a Probabilistic Database (PDB)? HasObject p Keys Probability Non-keys Object Time Person P Laptop77 9:07 John 0.62 Jim 0.34 Mary 0.45 Book302 9:18 John 0.33 Fred 0.11 What does it mean? 27

20 Possible world semantics Object Time Person P Laptop77 9:07 Book302 9:18 John p 1 Jim p 2 Mary p 3 John p 4 Fred p 5 PDB Ω={ Object Time Person Object Time Person Laptop77 9:07 John } Object Time Person Laptop77 9:07 John Book302 9:18 Object Mary Time Person Laptop77 Object 9:07 John Time Person Book302 Laptop77 9:18 John Object 9:07 Jim Time Person Book302 Laptop77 9:18 Fred 9:07 Jim Book302 Laptop77 9:18 Object Mary 9:07 Time Jim Person p 1 p 3 Book302 9:18 Object John Time Person p 1 p 4 Book302 Laptop77 9:18 Object 9:07 Fred John Time Person Laptop77 Object 9:07 Jim Time Person Book302 Object 9:18 Mary Time Person Book302 Object 9:18 John Time Person p 1 (1- p 3 -p 4 -p 5 ) Book302 9:18 Fred Possible worlds 29

21 HasObject(Object, Time, Person, P) Object Time Person P Laptop77 9:07 John p 1 Jim p 2 Disjoint Book302 9:18 Mary p 3 John p 4 Disjoint Independent Fred p 5 Meets(Person1, Person2, Time, P) Person1 Person2 Time P John Jim 9.12 p 1 Mary Sue 9:20 p 2 Independent John Mary 9:20 p 3 31

22 Tuple correlation Disjoint NegaGvely correlated Independent PosiGvely correlated IdenGcal Pr(t 1 t 2 ) = 0 Pr(t 1 t 2 ) < Pr(t 1 ) Pr(t 2 ) Pr(t 1 t 2 ) = Pr(t 1 ) Pr(t 2 ) Pr(t 1 t 2 ) > Pr(t 1 ) Pr(t 2 ) Pr(t 1 t 2 ) = Pr(t 1 ) = Pr(t 2 ) 34

23 Query semantics Given a query Q and a probabilisgc database I p, what is the meaning of Q(I p )? 35

24 Query semantics Seman*cs 1: Possible Answers A probability distribugons on sets of tuples Pr(Q = A) = I INST. Q(I) = A Pr(I) Seman*cs 2: Possible Tuples A probability funcgon on tuples Pr(t Q) = I INST. T Q(I) Pr(I) 36

25 Purchase p Name City Product John Seattle Gizmo John Seattle Camera Sue Denver Gizmo Sue Denver Camera Name City Product John Boston Gizmo Sue Denver Gizmo Sue Seattle Gadget Name City Product John Seattle Gizmo John Seattle Camera Sue Seattle Camera Name City Product John Boston Camera Sue Seattle Camera Pr(I 1 ) = 1/3 Pr(I 2 ) = 1/12 Pr(I 3 ) = 1/2 Pr(I 4 ) = 1/12 SELECT DISTINCT x.product FROM Purchase p x, Purchase p y WHERE x.name = 'John' and x.product = y.product and y.name = 'Sue' Possible answers semangcs: Answer set Probability Gizmo, Camera 1/3 Pr(I 1 ) Gizmo 1/12 Pr(I 2 ) Camera 7/12 P(I 3 ) + P(I 4 ) Possible tuples semangcs: Tuple Probability Camera 11/12 Pr(I 1 )+P(I 3 ) + P(I 4 ) Gizmo 5/12 Pr(I 1 )+Pr(I 2 ) 37

26 How do we evaluate queries? 38

27 Probability of Boolean Expressions Φ = X 1 X 2 v X 1 X 3 v X 2 X 3 P(X 1 )= p 1, P(X 2 )= p 2, P(X 3 )= p 3 Compute P(Φ) Ω=. X 1 X 2 X 3 P Φ (1-p 1 )p 2 p p 1 (1-p 2 )p p 1 p 2 (1-p 3 ) p 1 p 2 p 3 1 Pr(Φ)=(1-p 1 )p 2 p 3 + p 1 (1-p 2 )p 3 + p 1 p 2 (1-p 3 ) + p 1 p 2 p 3 39

28 Complexity of Boolean expression probability [Valiant:1979] Theorem: For a Boolean expression E, compugng Pr(E) is #P-complete NP: class of problems of the form is there a witness (SAT) #P: class of problems of the form how many witnesses (#SAT) 40

29 Query q + Database PDB è Φ q= R(x, y), S(x, z) R p PDB= A B P a 1 b 1 p 1 X 1 a 2 b 2 p 2 X 2 S p A C P a 1 c 1 q 1 Y 1 a 1 c 2 q 2 Y 2 a 2 c 3 q 3 Y 3 a 2 c 4 q 4 Y 4 Φ = ê X 1 Y 1 v X 1 Y 2 v X 2 Y 3 v X 2 Y 4 v X 2 Y 5 a 2 c 5 q 5 Y 5 41

30 Application to query evaluation Corollary Fix FO query q Exact evaluagon of Pr(q) on input PDB is in #P Corollary Fix a conjuncgve query q. ApproximaGon of Pr(q) on input PDB is in PTIME (FPTRAS) [Graedel,Gurevitch,Hirsch:1998] 42

31 Query complexity Data complexity of a query Q: } Compute Q(I p ), for probabilisgc database I p Simplest scenario only: } Possible tuples semangcs for Q } Independent tuples for I p 43

32 Challenges } Query opgmizagon } A #P query with subqueries in PTIME } Combine safe plans and inference } TheoreGcal analysis of self-joins } Complex probabilisgc models } Capture complex correlagons } Hard and sof constraints on query evaluagon } 50