Database Design and Implementation

Transcription

1 Database Design and Implementation CS 645 Data provenance

2 Provenance provenance, n. The fact of coming from some particular source or quarter; origin, derivation [Oxford English Dictionary] Data provenance / lineage [BunemanKhannaTan 01]: aims to explain how a particular result was derived. Data-intensive science Worry about provenance

3 Motivation Data integration [WangMadnick90, LeeBressanMadnick98] Data Warehousing [CuiWidonWiener00] Scientific Data Management [BunemanKhannaTan01] Determines trust on results Ensure reliability, quality of data Repeatability/verifiability Avoid effort duplication Understanding transport of annotations

4 Example of data provenance A typical question: For a given database query Q, a database D and a tuple t in the output of Q(D), which parts of D contribute to t? R Emp John Susan Anna Dept D01 D02 D04 S Did D01 D02 D03 Mgr Mary Ken Ed Q Q = select r.a, r.b, s.c from R r, S s where r.b = s.b Emp Dept Mgr John D01 Mary Susan D02 Ken The question can be applied to attribute values, tables, etc.

5 Two approaches Eager or annotation-based Changes the transformation from Q to Q to carry extra information Source data not needed after transformation Annotation-based Q Q Extra information Lazy or non-annotation based Q is unchanged Good when extra storage is an issue Recomputation and access to source required

6 Types of provenance Why What DB tuples contribute to the presence of each result tuple? How By what process is each output tuple produced from the DB instance? Where Where (from what attribute of what tuple) does each output tuple value come from?

7 Why-provenance example a.name, DISTINCT a.phone a.name, a.phone

8 Lineage for an output tuple t is a subset of the input tuples which are relevant to the output tuple DISTINCT a.name, a.phone Lineage: {t1, t5, t6} Problem: Not very precise. e.g., lineage above does not specify that t5 and t6 do not both need to exist.

9 Why provenance DISTINCT a.name, a.phone Witness of t: Any subset of the database sufficient to reconstruct tuple t in the query result. Witness basis: Leaves of the proof tree showing how result tuple t is generated Lineage: {t1, t5, t6} {t1, t5} {t1, t6} {t1, t2, t6, t8} {{t1, t5}, {t1, t6}}

10 Why: query rewriting t1 t t2 t3 Why(Q, I, t): {{t 1 }} Why(Q, I, t): {{t 1 }, {t 1, t 2 }} Minimal witness basis: Minimal witnesses in the witness basis

11 The view deletion problem D a database instance and V=Q(D) a view defined over D. Find a set of tuples ΔD to remove from D so that a specific tuple t is removed from the view Minimize the number of side-effects in the view View side-effect problem Hard: queries with joins and projection or union PTIME: the rest Minimize the number of tuples deleted from D Source side-effect problem Same dichotomy [BunemanKhannaTan. PODS 2002]

12 How provenance Identifies witness tuples and the operations performed on them to produce each result tuple Expresses operations using provenance semirings MERGE (+): union or projection JOIN ( ): joins

13 Propagating annotations (1) R A B C a b c S D B E d b e Join (on B) R S A B C D E a b c d e The annotation means joint use of the data annotated by p and the data annotated by r

14 Propagating annotations (2) R A B C a b c Union R S A B C a b c p + r S A B C a b c The annotation p + r means alternative use of the data annotated by p and the data annotated by r

15 Propagating annotations (3) R A B C a b c 1 a b c 2 p r Project π AB R A B a b p + r + s a b c 3 s + denotes alternative use of data

16 An example (SPJU) R A B C a b c d b e f g e p r s Q = σ C=e π AC (π AB R π BC R π AC R π BC R) A C a c a e d c d e f e For selection, multiply with annotation 0 and 1.

17 Example

18 Example

19 Example

20 Example

21 Example

22 Example

23 Example

24 Example

25 Example

26 Example

27 Back to example R A B C a b c d b e f g e p r s Q A C a c a e d c d e f e

28 Applying the laws: polynomials R A B C a b c p Q A C a e pr d b e r d e 2r 2 + rs f g e s f e rs + 2s 2 Polynomials with coefficients in and annotation tokens as indeterminates p, r, s capture a very general form of provenance

29 How to read this provenance R A B C a b c p Q A C a e pr d b e r d e 2r 2 + rs f g e s f e rs + 2s 2 3 ways to derive (d e) 2 of the ways use only r, but they use it twice the 3 rd uses r once and s once

30 Deletion Propagation R A B C a b c p Q A C a e pr Q A C a e 0 Q A C f e 2s 2 d b e r d e 2r 2 + rs d e 0 f g e s f e rs + 2s 2 f e 2s 2 Delete (d b e) from R Set r to 0!

31 Some useful commutative semirings Set Semantics Bag Semantics Probabilistic events Access Control Public Top Secret

32 Example: access control where a c 2p 2 a b c p d b e r f g e s p=p, r=s, s=t q a d d f e c e e pr pr 2r 2 +rs 2s 2 +rs a b c d b e f g e P S T q a a d d c e c e P S S S Evaluate with p=p, r=s, s=t using min for +, max for f e T User with secret clearance

33 Where provenance Identifies witness cells Important for annotations SELECT * FROM R WHERE A <> 5 UNION SELECT A, 7 AS B FROM R WHERE A= 5 UPDATE R SET B=7 WHERE A=5 R A B ? A B

34 Color algebra [Geerts, Kementsietsidis, Milano 06] A B P[Q] A B Q = SELECT * FROM R WHERE A <> 5 UNION SELECT A, 7 AS B FROM R WHERE A= 5

35 Color algebra A B P[Q] A B Q = UPDATE R SET B=7 WHERE A=5

36 Where provenance and semirings R u A x B y C 1 a 1 b 1 c 1 S v B 1 C 1 b z c 1 m π AC (π AB R (π BC R S)) A 1 C 1 a 1 c 1 u 2 p 2 xy 2 + uvpmxyz 1 is a neutral annotation, used when we don t bother to track data

37 Different annotations à Different tuples R A B C a b c d b e z f g e w p r s π C σ C=e π AC (π AB R π BC R) C e z e w pr+r 2 s 2

38 Wrap up: issues and directions Archiving Compression Generalizations Program Slicing [Cheney07] Negative Provenance Why Not? [SIGMOD09], Artemis [PVLDB09] Causality