Representing and Querying Correlated Tuples in Probabilistic Databases

Transcription

1 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen Amol Deshpande Department of Computer Science University of Maryland, College Park. International Conference on Data Engineering, 2007 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 1/26

2 Introduction Abundance of uncertain data. Numerous approaches proposed to handle uncertainty. However, most models make assumptions about data uncertainty that restricts applicability. Need for a Database model with a model of uncertainty that can capture correlations simple and intuitive semantics that are readily understood and defines precise answers to every query Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 2/26

3 Introduction Abundance of uncertain data. Numerous approaches proposed to handle uncertainty. However, most models make assumptions about data uncertainty that restricts applicability. Need for a Database model with a model of uncertainty that can capture correlations simple and intuitive semantics that are readily understood and defines precise answers to every query Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 2/26

4 Motivations for correlated data Applications: Dirty databases [CP87, DS96, AFM06]: Arise while trying to integrate data from various sources. DB1: John $1200 DB2: John $1600 Name Salary John $1200 } mutually John $1600 exclusive Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 3/26

5 Motivations for correlated data Applications: Dirty databases [CP87, DS96, AFM06]: Arise while trying to integrate data from various sources. Sensor Networks [DGM + 04]: Often show strong spatial correlations, e.g., nearby sensors report similiar values Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 3/26

6 Motivations for correlated data Applications: Dirty databases [CP87, DS96, AFM06]: Arise while trying to integrate data from various sources. Sensor Networks [DGM + 04]: Often show strong spatial correlations, e.g., nearby sensors report similiar values. More applications: Pervasive computing, approximate string matching in DB systems [DS04] etc. Additional motivation: Correlated tuples arise while evaluating queries [DS04]. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 3/26

7 Outline 1 Motivation. 2 Probabilistic databases with correlations. 3 Query evaluation. 4 Qualitative comparison with other approaches. 5 Experiments. 6 Conclusion and future work. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 4/26

8 Representing correlated tuples: Ingredients Which theory to use? Probability theory, Dempster-Schafer theory, fuzzy Logic, logic. At what level do we represent uncertainty? Tuple level, attribute level. Approach to representing correlations? Probabilistic graphical models (include as special cases: Bayesian networks and Markov networks) Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 5/26

9 Representing correlated tuples: Ingredients Which theory to use? Probability theory, Dempster-Schafer theory, fuzzy Logic, logic. At what level do we represent uncertainty? Tuple level, attribute level. Approach to representing correlations? Probabilistic graphical models (include as special cases: Bayesian networks and Markov networks) Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 5/26

10 Review: Independent tuple-based probabilistic databases possible worlds semantics [DS04] Example from Dalvi and Suciu [DS04]: S A B prob s 1 m s 2 n T C D prob t 1 1 p 0.4 possible worlds instance probability {s 1, s 2, t 1 } 0.12 {s 1, s 2 } 0.18 {s 1, t 1 } 0.12 {s 1 } 0.18 {s 2, t 1 } 0.08 {s 2 } 0.12 {t 1 } Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 6/26

11 Representing correlations: Basic ideas Two concepts from probabilistic graphical models literature [Pea88]: Tuple-based Random Variables Associate every tuple t with a boolean valued random variable X t. Factors f (X) is a function of a (small) set of random variables X. 0 f (X) 1 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 7/26

12 Representing correlations: Basic ideas Two concepts from probabilistic graphical models literature [Pea88]: Tuple-based Random Variables Associate every tuple t with a boolean valued random variable X t. Factors f (X) is a function of a (small) set of random variables X. 0 f (X) 1 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 7/26

13 Representing correlations: Basic ideas Two concepts from probabilistic graphical models literature [Pea88]: Tuple-based Random Variables Associate every tuple t with a boolean valued random variable X t. Factors f (X) is a function of a (small) set of random variables X. 0 f (X) 1 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 7/26

14 Representing correlations: Basic ideas - contd. Associate with each tuple in the probabilistic database a random variable. Define factors on (sub)sets of tuple-based random variables to encode correlations. The probability of an instantiation of the database is given by the product of all the factors. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 8/26

15 Representing correlations: Mutual exclusivity example Suppose we want to represent mutual exclusivity between tuples s 1 and t 1. In particular, let us try to represent the possible worlds: S A B prob s 1 m s 2 n T C D prob t 1 1 p 0.4 possible worlds instance probability {s 1, s 2, t 1 } 0 {s 1, s 2 } 0.3 {s 1, t 1 } 0 {s 1 } 0.3 {s 2, t 1 } 0.2 {s 2 } 0 {t 1 } X t1 X s1 f X s2 f Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 9/26

16 Representing correlations: Mutual exclusivity example Suppose we want to represent mutual exclusivity between tuples s 1 and t 1. In particular, let us try to represent the possible worlds: S A B prob s 1 m s 2 n T C D prob t 1 1 p 0.4 possible worlds instance probability {s 1, s 2, t 1 } 0 {s 1, s 2 } 0.3 {s 1, t 1 } 0 {s 1 } 0.3 {s 2, t 1 } 0.2 {s 2 } 0 {t 1 } X t1 X s1 f X s2 f Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 9/26

17 Representing correlations: Mutual exclusivity example Suppose we want to represent mutual exclusivity between tuples s 1 and t 1. In particular, let us try to represent the possible worlds: S A B prob s 1 m s 2 n T C D prob t 1 1 p 0.4 possible worlds instance probability {s 1, s 2, t 1 } 0 {s 1, s 2 } 0.3 {s 1, t 1 } 0 {s 1 } 0.3 {s 2, t 1 } 0.2 {s 2 } 0 {t 1 } X t1 X s1 f X s2 f Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 9/26

18 Representing correlations: Mutual exclusivity example Suppose we want to represent mutual exclusivity between tuples s 1 and t 1. In particular, let us try to represent the possible worlds: S A B prob s 1 m s 2 n T C D prob t 1 1 p 0.4 possible worlds instance probability {s 1, s 2, t 1 } 0 {s 1, s 2 } 0.3 {s 1, t 1 } 0 {s 1 } 0.3 {s 2, t 1 } 0.2 {s 2 } 0 {t 1 } X t1 X s1 f X s2 f Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 9/26

19 Representing correlations: Mutual exclusivity example Suppose we want to represent mutual exclusivity between tuples s 1 and t 1. In particular, let us try to represent the possible worlds: S A B prob s 1 m s 2 n T C D prob t 1 1 p 0.4 possible worlds instance probability {s 1, s 2, t 1 } 0 {s 1, s 2 } 0.3 {s 1, t 1 } 0 {s 1 } 0.3 {s 2, t 1 } 0.2 {s 2 } 0 {t 1 } X t1 X s1 f X s2 f Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 9/26

20 Representing correlations: Mutual exclusivity example Suppose we want to represent mutual exclusivity between tuples s 1 and t 1. In particular, let us try to represent the possible worlds: S A B prob s 1 m s 2 n T C D prob t 1 1 p 0.4 possible worlds instance probability {s 1, s 2, t 1 } 0 {s 1, s 2 } 0.3 {s 1, t 1 } 0 {s 1 } 0.3 {s 2, t 1 } 0.2 {s 2 } 0 {t 1 } X t1 X s1 f X s2 f Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 9/26

21 Representing correlations: Mutual exclusivity example Suppose we want to represent mutual exclusivity between tuples s 1 and t 1. In particular, let us try to represent the possible worlds: S A B prob s 1 m s 2 n T C D prob t 1 1 p 0.4 possible worlds instance probability {s 1, s 2, t 1 } 0 {s 1, s 2 } 0.3 {s 1, t 1 } 0 {s 1 } 0.3 {s 2, t 1 } 0.2 {s 2 } 0 {t 1 } X t1 X s1 f X s2 f Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 9/26

22 Representing correlations: Positive correlation example Suppose we want to represent positive correlation between t 1 and s 1. In particular, let us try to represent the possible worlds: S A B s 1 m 1 s 2 n 1 T C D t 1 1 p possible worlds instance probability {s 1, s 2, t 1 } 0.2 {s 1, s 2 } 0.1 {s 1, t 1 } 0.2 {s 1 } 0.1 {s 2, t 1 } 0 {s 2 } 0.2 {t 1 } X t1 X s1 f X s2 f Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 10/26

23 Representing correlations: Positive correlation example Suppose we want to represent positive correlation between t 1 and s 1. In particular, let us try to represent the possible worlds: S A B s 1 m 1 s 2 n 1 T C D t 1 1 p possible worlds instance probability {s 1, s 2, t 1 } 0.2 {s 1, s 2 } 0.1 {s 1, t 1 } 0.2 {s 1 } 0.1 {s 2, t 1 } 0 {s 2 } 0.2 {t 1 } X t1 X s1 f X s2 f Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 10/26

24 Representing correlations: Positive correlation example Suppose we want to represent positive correlation between t 1 and s 1. In particular, let us try to represent the possible worlds: S A B s 1 m 1 s 2 n 1 T C D t 1 1 p possible worlds instance probability {s 1, s 2, t 1 } 0.2 {s 1, s 2 } 0.1 {s 1, t 1 } 0.2 {s 1 } 0.1 {s 2, t 1 } 0 {s 2 } 0.2 {t 1 } X t1 X s1 f X s2 f Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 10/26

25 Representing correlations: Positive correlation example Suppose we want to represent positive correlation between t 1 and s 1. In particular, let us try to represent the possible worlds: S A B s 1 m 1 s 2 n 1 T C D t 1 1 p possible worlds instance probability {s 1, s 2, t 1 } 0.2 {s 1, s 2 } 0.1 {s 1, t 1 } 0.2 {s 1 } 0.1 {s 2, t 1 } 0 {s 2 } 0.2 {t 1 } X t1 X s1 f X s2 f Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 10/26

26 Representing correlations: Positive correlation example Suppose we want to represent positive correlation between t 1 and s 1. In particular, let us try to represent the possible worlds: S A B s 1 m 1 s 2 n 1 T C D t 1 1 p possible worlds instance probability {s 1, s 2, t 1 } 0.2 {s 1, s 2 } 0.1 {s 1, t 1 } 0.2 {s 1 } 0.1 {s 2, t 1 } 0 {s 2 } 0.2 {t 1 } X t1 X s1 f X s2 f Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 10/26

27 Probabilistic graphical model representation Definition A probabilistic graphical model is graph whose nodes represent random variables and edges represent correlations [Pea88]. X t1 X s1 X t1 X s1 X t1 X s1 X s2 Complete Ind. Example X s2 Mutual Exclusivity Example X s2 Positive Correlation Example Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 11/26

28 Outline 1 Motivation. 2 Probabilistic databases with correlations. 3 Query evaluation. 4 Qualitative comparison with other approaches. 5 Experiments. 6 Conclusion and future work. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 12/26

29 Query evaluation: Basic ideas Treat intermediate tuples as regular tuples. Carefully represent correlations between intermediate tuples, base tuples and result tuples to construct a probabilistic graphical model. Cast the probability computations resulting from query evaluation to inference in probabilistic graphical models. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 13/26

30 Query evaluation example Compute D (S B=C T ) S: A B s 1 m 1 s 2 n 1 f s1, f s2 T : C D t 1 1 p S B=C T f AND i 1,s 1,t 1, f AND i 2,s 2,t 1 A B C D i 1 m 1 1 p i 2 n 1 1 p D (S B=CT ) r 1 D p f t1 f OR r 1,i 1,i 2 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 14/26

31 Query evaluation example Compute D (S B=C T ) S: A B s 1 m 1 s 2 n 1 f s1, f s2 T : C D t 1 1 p S B=C T f AND i 1,s 1,t 1, f AND i 2,s 2,t 1 A B C D i 1 m 1 1 p i 2 n 1 1 p D (S B=CT ) r 1 D p f t1 f OR r 1,i 1,i 2 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 14/26

32 Query evaluation example Compute D (S B=C T ) S: A B s 1 m 1 s 2 n 1 f s1, f s2 T : C D t 1 1 p S B=C T f AND i 1,s 1,t 1, f AND i 2,s 2,t 1 A B C D i 1 m 1 1 p i 2 n 1 1 p D (S B=CT ) r 1 D p f t1 f OR r 1,i 1,i 2 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 14/26

33 Query evaluation example Compute D (S B=C T ) S: A B s 1 m 1 s 2 n 1 f s1, f s2 T : C D t 1 1 p S B=C T f AND i 1,s 1,t 1, f AND i 2,s 2,t 1 A B C D i 1 m 1 1 p i 2 n 1 1 p D (S B=CT ) r 1 D p f t1 f OR r 1,i 1,i 2 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 14/26

34 Query evaluation example Compute D (S B=C T ) S: A B s 1 m 1 s 2 n 1 f s1, f s2 T : C D t 1 1 p S B=C T f AND i 1,s 1,t 1, f AND i 2,s 2,t 1 A B C D i 1 m 1 1 p i 2 n 1 1 p D (S B=CT ) r 1 D p f t1 f OR r 1,i 1,i 2 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 14/26

35 Query evaluation example Compute D (S B=C T ) S: A B s 1 m 1 s 2 n 1 f s1, f s2 T : C D t 1 1 p S B=C T f AND i 1,s 1,t 1, f AND i 2,s 2,t 1 A B C D i 1 m 1 1 p i 2 n 1 1 p D (S B=CT ) r 1 D p f t1 f OR r 1,i 1,i 2 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 14/26

36 Query evaluation example: Probabilistic graphical model Query evaluation problem in Prob. Databases: Compute the probability of the result tuple summed over all possible worlds of the database [DS04]. Equivalent problem in prob. graph. models: marginal probability computation. Thus we can use inference algorithms (e.g., VE [ZP94]). X s X t X s X s1 X t1 X s2 X i1 X i2 X r1 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 15/26

37 Query evaluation example: Probabilistic graphical model Query evaluation problem in Prob. Databases: Compute the probability of the result tuple summed over all possible worlds of the database [DS04]. Equivalent problem in prob. graph. models: marginal probability computation. Thus we can use inference algorithms (e.g., VE [ZP94]). X s X t X s X s1 X t1 X s2 X i1 X i2 X s1, X s2, X i1, X i2, X t1 X r1 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 15/26

38 Query evaluation example: Probabilistic graphical model Query evaluation problem in Prob. Databases: Compute the probability of the result tuple summed over all possible worlds of the database [DS04]. Equivalent problem in prob. graph. models: marginal probability computation. Thus we can use inference algorithms (e.g., VE [ZP94]). X t1 X s2 X i1 X t X t X s2 X i1 X i2 X s1, X s2, X i1, X i2, X t1 X r1 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 15/26

39 Outline 1 Motivation. 2 Probabilistic databases with correlations. 3 Query evaluation. 4 Qualitative comparison with other approaches. 5 Experiments. 6 Conclusion and future work. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 16/26

40 Qualitative comparison with other probability computation approaches Compute {}(L J R): L A B l 1 m 1 J B C j 1 1 p j 2 1 q X r1 X j1 X l1 X j2 X i1 X i2 X r2 R C D r 1 p 1 r 2 q 1 X i3 X res X i4 Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 17/26

41 Qualitative comparison with extensional semantics [FR97, DS04] X j1 X l1 X j2 X r1 X i1 X i2 X r2 Is not guaranteed to match possible world semantics. X i3 X i4 Safe plans [DS04] return tree-structured graphical models. X res Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 18/26

42 Qualitative comparison with extensional semantics [FR97, DS04] X j1 X l1 X j X r1 X i1 X i2 X r2 Is not guaranteed to match possible world semantics. X i3 X i4 Safe plans [DS04] return tree-structured graphical models. X res Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 18/26

43 Qualitative comparison with extensional semantics [FR97, DS04] X j1 X l1 X j2 0.5 X r1 X i X i2 0.5 X r2 Is not guaranteed to match possible world semantics X i3 X i4 Safe plans [DS04] return tree-structured graphical models X res Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 18/26

44 Qualitative comparison with intensional semantics [FR97, DS04] X j1 X l1 X j2 X r1 X i1 X i2 X r2 Work with a subgraph of the graph. model. X i3 X i4 Inference algorithms can exploit graph structure better. X res (X r1 X j1 X l1 ) (X l1 X j2 X r2 ) Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 19/26

45 Outline 1 Motivation. 2 Probabilistic databases with correlations. 3 Query evaluation. 4 Qualitative comparison with other approaches. 5 Experiments. 6 Conclusion and future work. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 20/26

46 Experiments with correlated bibliographic data Recall N Database contains 860 publications from CiteSeer [GBL98]. Searched for publications for given (misspelt) author name. Naturally involves mutual exclusivity correlations. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 21/26

47 Experiments with TPC-H data Run-times (sec) Full Query Bare Query 10 0 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Ran experiments on randomly generated TPC-H dataset of size 10MB. Q7 and Q8 are queries without safe plans [DS04]. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 22/26

48 Aggregate Computations probability average Computes the average of an attribute of 500 synthetic tuples. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 23/26

49 Outline 1 Motivation. 2 Probabilistic databases with correlations. 3 Query evaluation. 4 Qualitative comparison with other approaches. 5 Experiments. 6 Conclusion and future work. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 24/26

50 Conclusion & future work Introduced probabilistic databases with correlated tuples. Utilized ideas from probabilistic graphical models to represent such correlations. Cast the query evaluation problem as an inference problem. Future Work: Add more representation power: Attribute uncertainty and correlated attributes. Ways to restructure graphical model to speed up inference. Share/reuse computation to speed up inference. Explore the use of approximate inference methods [GRS96, JGJS99]. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 25/26

51 Thank You. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 26/26

52 References Periklis Andritsos, Ariel Fuxman, and Renee J. Miller. Clean answers over dirty databases. In International Conference on Data Engineering, Roger Cavallo and Michael Pittarelli. The theory of probabilistic databases. In International Conference on Very Large Data Bases, Amol Deshpande, Carlos Guestrin, Sam Madden, Joseph M. Hellerstein, and Wei Hong. Model-driven data acquisition in sensor networks. In International Conference on Very Large Data Bases, Debabrata Dey and Sumit Sarkar. A probabilistic relational model and algebra. ACM Transactions on Database Systems., Nilesh Dalvi and Dan Suciu. Efficient query evaluation on probabilistic databases. In International Conference on Very Large Data Bases, Norbert Fuhr and Thomas Rolleke. Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 26/26

53 A probabilistic relational algebra for the integration of information retrieval and database systems. ACM Transactions on Information Systems, C. Lee Giles, Kurt Bollacker, and Steve Lawrence. Citeseer: An automatic citation indexing system. In Conference on Digital Libraries, Walter R. Gilks, Sylvia Richardson, and David J. Spiegelhalter. Markov Chain Monte Carlo in Practice. Chapman & Hall, Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Machine Learning, Judaea Pearl. Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann, Nevin Lianwen Zhang and David Poole. A simple approach to bayesian network computations. In Canadian Conference on Artificial Intelligence, Representing and Querying Correlated Tuples in Probabilistic Databases Prithviraj Sen and Amol Deshpande 26/26