Topics in Probabilistic and Statistical Databases. Lecture 2: Representation of Probabilistic Databases. Dan Suciu University of Washington

Transcription

1 Topics in Probabilistic and Statistical Databases Lecture 2: Representation of Probabilistic Databases Dan Suciu University of Washington 1

2 Review: Definition The set of all possible database instances: Inst = {I 1, I 2, I 3,..., I N } Sample space (Ω) Definition A probabilistic database PDB = (Inst, Pr) is a discrete probability distribution: Pr : Inst! [0,1] s.t. i=1,n Pr(I i ) = 1 Definition A possible world is I s.t. Pr(I) > 0 A possible tuple is a tuple t 2 I, for a possible world I 2

3 Representation System Informally: it is a syntax + semantics that allows us to represent a probabilistic database concisely Definition A representation system for ProbDB is (S,Rep), where S is a set of representations and Rep: S (set of PDBs) assigns a probabilistic database to each representation 3

4 Review: Disjoint-Independent Databases Definition A PDB is disjoint-independent if for any set T of possible tuples one of the following holds: T is an independent set, or T contains two disjoint tuples A disjoint-independent database can be fully specified by: all marginal tuple probabilities an indication of which tuples are disjoint or independent 4

5 Representations Systems for D/I- Databases MystiQ s representation Trio s xor-, maybe- tuples Attribute-level probabilities 5

6 D/I Relations in MystiQ At the schema level: Possible worlds key = set of attributes Probability = expression on attributes Constraint at the instance level: For each key value, sum of probabilities 1 6

7 Possible worlds key =? Attribute expression =? Constraint =? S = Object Time Person P LaptopX77 9:07 John p 1 LaptopX77 9:07 Jim p 2 Book302 9:18 Mary p 3 Book302 9:18 John p 4 Rep(S) = Book302 9:18 Fred p 5 { Object Time Person Object Time Person LaptopX77 9:07 John } Object Time Person LaptopX77 9:07 John Book302 9:18 Object Mary Time Person LaptopX77 Object 9:07 John Time Person Book302 LaptopX77 9:18 John Object 9:07 Jim Time Person Book302 LaptopX77 9:18 Fred 9:07 Jim Book302 LaptopX77 9:18 Object Mary 9:07 Time Jim Person Book302 9:18 Object John Time Person Book302 LaptopX77 9:18 Object 9:07 Fred John Time Person LaptopX77 Object 9:07 Jim Time Person Book302 Object 9:18 Mary Time Person Book302 Object 9:18 John Time Person p 1 (1- p 3 -p 4 -p 5 ) Book302 9:18 Fred p 1 p 3 p 1 p 4 7

8 X-Relations in Trio Maybe tuple is t?, meaning it may be missing X-tuple is <t1, t2, > meaning it may be any of t1, t2, An X-relation is a collection of X-tuples and maybe-tuples 8

9 Maybe- and X-tuples in Trio S = Rep(S) =? 9

10 Attribute-level Uncertainty For some attribute A, give a probability distribution on possible values Note: sum of probabilities must be 1 More generally, for a set of attributes A1, A2, give a probability distribution on possible values 10

11 [Barbara 92] Attribute-Level Uncertainty S = Rep(S) =? 11

12 Review Query Semantics Semantics 1: Possible Sets of Answers A probability distributions on sets of tuples 8 A. Pr(Q = A) = I 2 Inst. Q(I) = A Pr(I) Semantics 2: Possible Tuples A probability function on tuples 8 t. Pr(t 2 Q) = I 2 Inst. t2 Q(I) Pr(I) 12

13 The Representation Problem How do we represent correlations between tuples in a probdb? How do we represent query answers (views)? 13

14 Main Techniques Incomplete databases Concerned with representing possible worlds, i.e. no probabilities Probabilistic Networks Concerned with representing correlations, i.e. no databases 14

15 Incomplete Databases 15

16 [Imilelinski&Lipsi 1984, Green&Tannen 2006] Incomplete Databases Let Inst = the set of all possible instances over domain D Definition An incomplete database is a set of possible worlds I = {I 1, I 2, I 3,...} Inst Definition A representation system is (S, Rep) where S is a set of representations described by some syntax, and: Rep : S! 2 Inst 16

17 Discussion We often denote an incomplete database {I 1, I 2, } with <I 1, I 2, > This is called an OR-set: the true state can be any I <I 1, I 2, > 17

18 Rule of Thumb #1 A ProbDB = an Incomplete DB + probabilities 18

19 Discussion Question: what does < > mean (the empty set of worlds)? 19

20 Discussion Question: what does < > mean (the empty set of worlds)? Answer: inconsistency! We do not allow < > 20

21 Examples Every traditional database instance I is also an incomplete database I = <I> The no-information, or zero-information incomplete database is: N = < I I Inst> (in other words: N = Inst) 21

22 Four Representation Systems Codd-tables V-tables C-tables OR-sets 22

23 Codd-Tables T = NULL or Rep(T) =? 23

24 V-Tables T = Rep(T) =? marked nulls : 1, 2, 24

25 C-Tables T = Any Boolean condition Rep(T) =? 25

26 Boolean C-Tables: Vars only in Cond T = A B Cond a1 b1 (x=1) (y=2) (z=1) a1 b2 x=2 a2 b2 z=1 We often state explicitly the domain of each variable: Dom(X) = {1, 2} Dom(Y) = {1,2,3} Dom(Z) = {0,1}... Rep(T) =? In G&T s definition all vars are Boolean; minor distinction. 26

27 OR-Sets Review of Nested Relational Algebra NRA: Types: T ::= basetype T T { T } Operations (in class): 27

28 OR-Sets Types: T ::= basetype T T { T } < T > Operations (in class): 28

29 OR-Sets Examples What are their types? What do they mean? {[Gizmo, <99,110>], [Camera, <10,12,19>]} {<3,5>,<3,7>,<5,8,12>} {<{<1,2>,<3>},{<4>}>,<{<1>},{<2,3>,<4>}>} 29

30 Discussion Which concepts from incomplete databases where borrowed by the three simple representation systems for ProbDB? MystiQ s disjoint/independent tables Trio s X-tuples Attribute level probabilities 30

31 Two Important Properties Completeness: can a representation system represent all incomplete databases? Closure: can the result of a query also be represented in the same system? 31

32 Closure and Completeness Assume the domain D is finite Why? Definition A representation system is complete if for any incomplete database I there exists a representation S s.t. Rep(S) = I. 32

33 Which Are Complete? Why? Codd-Tables? V-Tables? C-Tables? OR-Sets? 33

34 Which Are Complete? Why? Codd-Tables? NO: constant cardinality V-Tables? NO: constant cardinality C-Tables? YES OR-Sets? YES 34

35 Closure Definition A representation system is closed w.r.t. a query language Q, if for every representation S and query q in Q, there exists S s.t. Rep(S ) = q(rep(s) q S S Rep Rep(S)={I 1, I 2, } q Rep Rep(S )={q(i 1 ), q(i 2 ), } 35

36 Which Are Closed w.r.t. RA? Codd-Tables? V-Tables? C-Tables? OR-Sets? 36

37 Which Are Closed w.r.t. RA? Codd-Tables? NO in class V-Tables? NO in class C-Tables? YES in class OR-Sets? YES in class 37

38 Completeness Closure Fact: every complete system is closed w.r.t. the Relational Algebra Why? 38

39 Completeness Closure? Challenge: give an example of a system that is closed w.r.t. Relational Algebra but is not complete! 39

40 [Das Sarma (unpublished) 2008] Completeness Closure? Consider a representation system S s.t. S can represent any deterministic instance <I> S can represent <{0}, {1}> S is closed under {Π,,σ} Then S is complete PROOF: in class 40

41 Lineage Lineage = a Boolean expression annotating a tuple that explains why the tuple is there Technically: lineage = condition in a Boolean C-table A.k.a provenance 41

42 [Benjelloun, VLDBJ 2008] Example Start from Trio s X-relations: ID Saw(Witness, Car) X1 <[Amy, Mazda], [Amy, Toyota]>? X2 <[Betty, Honda]> ID Drives(Person, Car) Y1 <[Jimmy, Mazda], [Jimmy, Toyota]> Y2 <[Billy, Mazda], [Billy, Honda]> 42

43 [Benjelloun, VLDBJ 2008] Example Convert to Boolean C-Tables Saw Drives Person Car Cond Witness Car Cond Jimmy Mazda Y1=1 Amy Mazda X1=1 Jimmy Toyota Y1=2 Amy Toyota X1=2 Billy Mazda Y2=1 Betty Honda X2=1 Billy Honda Y2=2 Q: How do we say Amy is a maybe tuple, Betty is a certain tuple? 43

44 Example Answer: Dom(X1) = {0,1,2} Dom(X2) = {1} Dom(Y1) = {1,2} Dom(Y2) = {1,2} 44

45 [Benjelloun, VLDBJ 2008] Saw Example Compute the query q(w,p) :- Saw(w,c),Drives(c,p) Drives Person Car Cond Witness Car Cond Jimmy Mazda Y1=1 Amy Mazda X1=1 Jimmy Toyota Y1=2 Amy Toyota X1=2 Billy Mazda Y2=1 Betty Honda X2=1 Billy Honda Y2=2 Witness Person Cond Amy Jimmy X1=1 Y1=1 X1=2 Y1=2 Betty Billy X2=1 Y2=2 45

46 Uniform Lineage Call a lineage expression uniform if: It is in DNF All conjuncts have the same number k of literals Each literal is of the form X=v Representation of uniform lineage: add 2k columns! 46

47 Saw Example Compute the query q(w,p) :- Saw(w,c),Drives(c,p) Drives Person Car Y W Witness Car X V Jimmy Mazda Y1 1 Amy Mazda X1 1 Jimmy Toyota Y1 2 Amy Toyota X1 2 Billy Mazda Y2 1 Betty Honda X2 1 Billy Honda Y2 2 Witness Person X V Y W Amy Jimmy X1 1 Y1 1 Amy Jimmy X1 2 Y1 2 Betty Billy X2 1 Y2 2 X1=1 Y1=1 X1=2 Y1=2 47

48 Summary of Incomplete DBs Main goal: specify a set of possible worlds Very relevant to ProbDBs! Key concepts: closure and completeness Turn out to be equivalent, except trivial cases Boolean C-tables closed&complete; others not Lineage: important tool in ProbDB, is derived from C-tables Some open questions next Want to tell & show? by Wed. 48

49 Open Questions in Incomplete/ Probabilistic Databases Variables OWA v.s. CWA Certain tuples, possible tuples Partial information order Strong v.s. weak representation systems Homework: pick one and apply to probdbs. Tell us next time your thoughts 49

50 Variables as Attribute Values T = A B b1 Rep(T) = A large, or infinite set a2 b1 ProbDBs don t use variables as attribute values What if we allow ProbDBs to have variables? 50

51 OWA v.s. CWA CWA: if R(a,b,c) is not mentioned in the knowledge/data base, then R(a,b,c) is assumed OWA: R(a,b,c) is inferred only if it is explicitly stated in the knowledge/data base Which one is standard semantics in DB? And in KR? 51

52 OWA v.s. CWA in Incomplete DB T = A B b1 a2 b1 CWA: Rep(T) = A B a1 b1 A B a1 b1 A B a3 b1 A B a4 b1... a2 b1 a2 b1 a2 b1 OWA: Rep(T) = A B a1 b1 a2 b1 A B... A B A B a1 b1 a1 b1 a1 b1 a2 b What would OWA mean for ProbDBs?... 52

53 Certain v.s. Possible Tuples Consider an incomplete database I = {I 1, I 2, I 3,...} Definition The certain tuples I =I 1 I 2 I 3 Definition The possible tuples I =I 1 I 2 I 3 What do certain/possible tuples correspond to in ProbDB? 53

54 Certain v.s. Possible Tuples Two incomplete databases I, J are equivalent if I = J Two incomplete databases I, J are equivalent w.r.t. a query language Q if forall q in Q, q(i) = q(j) These notions are used to define weak representation systems (see [AHV]).. Are there similar notions of equivalences between ProbDBs 54?

55 Partial Information Order Let (D, ) be an ordered set. Consider two sets A={a 1,,a m }, B={b 1,,b n }. When can we say A B? Definition [Smythe or upper] A B if b j a i : a i b j Definition [Hoare or lower] A B if a i b j : a i b j Definition [Plotkin or convex] A B if A B and A B 55

56 Partial Information Order Let s order OR-sets Values of base type: a a and a Records: [x,y] [u,v] when? OR-sets <a1,a2,a3> <b1,b2> when? Sets: {a1,a2,a3} {b1, b2} when? 56

57 Partial Information Order Let s order OR-sets Values of base type: a a and a Records: [x,y] [u,v] when? When x u and y v OR-sets <a1,a2,a3> <b1,b2> when? Smythe Sets: {a1,a2,a3} {b1, b2} when? Hoare (for OWA), or equality (for CWA) 57

58 Partial Information Order What is the partial information order on ProbDBs? 58

59 Probabilistic Networks 59

60 Probabilistic Models Problem setting: Given m random variables V 1,, V m Each with domain D (same for all V i ) A probability space over D m : Pr: D m [0,1], st v1,,vm in D Pr(v 1,,v m ) = 1 Called the joint probability distribution Problem: give a compact representation of Pr 60

61 Example V1 V2 P A disjoint probabilistic database Here m=2, but in general m is large, need compact rep. 61

62 Background Marginal probability: Pr(V i =a) = v1,,vm in D, vi = a Pr(v 1,,v m ) What does this mean? Pr(V i ), Pr(V i V j ) Conditional prob: Pr(E F) = Pr(EF)/Pr(F) Independence: Pr(EF) = Pr(E) * Pr(F) Equivalently: Pr(E F) = Pr(E) Conditional indep: Pr(E,F G)= Pr(E G)*Pr(F G) 62

63 Independence V1 V2 P V1 P V2 P Marginal probabilities Are V1, V2 independent, i.e. Pr(V1,V2) = Pr(V1)*Pr(V2)? Note: need to check 4 equalities (why?) 63

64 Factored Representation More general: if P(V1=a,V2=b) = p(a) * q(b) forall a,b then V1, V2 are independent V1 V2 P a 1 b 1 p 1 q 1 V1 P V2 P a 1 b 2 p 1 q 2 a 2 b 1 p 2 q 1 a 2 b 2 p 2 q = a 1 p 1 a 2 p 2... Factors b 1 q 1 b 2 q

65 1 st Connection to ProbDBs: Variable Attribute Every joint distribution over variables V 1,, V m corresponds to a trivial probabilistic table with attributes V 1,, V m What s trivial about it? 65

66 1 st Connection to ProbDBs: Variable Attribute Conversely: each block in a disjoint/ independent relation defines a joint distribution on the values of its attribute Object Time Person P LaptopX77 9:07 John 0.5 Jim 0.5 Mary 0.2 Book302 9:18 John 0.4 Fred

67 Independence 67

68 Independence Employee Department Quality Bonus Sales P John Smith Toy Great Yes 30k-34k 0.4*0.3 John Smith Toy Good Yes 30k-34k 0.5*0.3 John Smith Toy Fair No 30k-34k 0.1*0.3 John Smith Toy Great Yes 35k-39k 0.4*0.7 John Smith Toy Good Yes 35k-39k 0.5*0.7 John Smith Toy Fair No 35k-39k 0.1*0.7 Fre Jones Houseware Good Yes 20k-24k 0.5 Fre Jones Houseware Good Yes 24k-29k

69 Independence Factors are disjoint/indep. Tables! Employee Department Quality Bonus P John Smith Toy Great Yes 0.4 John Smith Toy Good Yes 0.5 John Smith Toy Fair No 0.1 Fre Jones Houseware Good Yes 1 Employee Department Sales P John Smith Toy 30k-34k 0.3 John Smith Toy 35k-39k 0.7 Fre Jones Houseware 20k-24k 0.5 Fre Jones Houseware 24k-29k

70 Rule of Thumb #2 Correlated table = Independent tables + join Same principle as in traditional schema normalization Decompose big table into small tables with independent attributes Big table = a view (single join!) over the small tables 70

71 Database 101 Assume that (Quality, Bonus) is independent from (Sales) Drop the P column a traditional table (no more probabilities) Question What functional dependency holds in this table? 71

72 Database 101 Assume that (Quality, Bonus) is independent from (Sales) Drop the P column a traditional table (no more probabilities) Question What functional dependency holds in this table? A Multivalued FD!: Emp, Dept Quality, Bonus Sales 72

73 Rule of Thumb #3 Factor decomposition = MVD decomposition + probability identities 73

74 Conditional Independence V1 V2 W P 1 1 a a a a b b b b Are V1, V2 independent given W, i.e. Pr(V1,V2 W) = Pr(V1 W) * Pr(V2 W)? 74

75 Factors Conditional Independence V1 V2 W P 1 1 a a a a b b b b They: they are conditional independent P(W=a)=0.5; P(-- W=a): V1 V2 P P(W=b)=0.5; P(-- W=b): V1 V2 P

76 Conditional Independence V1 V2 W P 1 1 a a a a 0.28 = W P a 0.5 b b b b b V1 W P 1 a a b b 0.5 V2 W P 1 a a b b

77 Representing Conditional Independent Vars More general: if P(V1=a,V2=b,W=c) = p(a,c) * q(b,c) forall a,b,c then V1, V2 are independent given c V1 V2 W P = W P c 1 r 1 a 1 b 1 c 1 p 11 q 11 c 2 r 2 a 1 b 2 c 1 p 11 q a 2 b 1 c 1 p 21 q 11 V1 W P V2 W P a 2 b 2 c 1 p 21 q c 2 P 12 q 12 a 1 c 1 p 11 /P 1 b 1 c 1 q 11 /Q a 2 c 1 p 21 /P 1.. c 2 p 12 /P 2 b 2 c 1 q 21 /Q 1.. c 2 q 12 /Q

78 Summary of Prob Networks (Conditional) indep. = MVDs + prob ident. Factored decomposition = base tables + joins Next time: Discuss the networks in probabilistic networks; WSdecomposition, U-relations Partial/approximate representations Begin query evaluation Send by Wed. if want to show&tell 78