Integra(ng and Ranking Uncertain Scien(fic Data

Transcription

1 Jan 19, Biomedical and Health Informatics 2 Computer Science and Engineering University of Washington Integra(ng and Ranking Uncertain Scien(fic Data Wolfgang Ga*erbauer 2 Based on joint work with: Todd Detwiler 1, Abhay Jha 2, Brent Louie 1, Dan Suciu 2, and Peter Tarczy- Hornoch 1

2 Mo1va1on: Retrieving relevant infos across several DBs Keyword ABCC8 DB2 B C DB1 A B ABCC8 AGTTC... DB3 AGTTC... xxx AGTTC... xyz AGTCC... xyz DB4 B xxx xyy C a c Results a c e B C xyz e... AGTTC... xyy xzz e AGTTC... xyz AGTCC... xzz Problem: mul(ple expansions across different data- bases can quickly lead to many less relevant results. Ques1on: how can prune or rank those results? 2

3 Agenda How to model uncertainty in data integra(on? How do we rank? How well, how fast, how robust on real data? A short database research point of view 3

4 4 Probabilis1c Metrics in UII Granularity Schema Instance E/R En(ty p s p r Rela(onship q s q r 4

5 Example Belief Metrics Schema graph p s0 Q p s1 p s4 q q s1-4 s0-1 p s3 q s0-2 q s2-3 R q s3-5 Instance graph p s2 E 0 q r0-1 q r0-2 p r1 q E r1-5 1 p r2 q r2-3 E 2 E 3 q r2-4 p r3 p r4 E 4 qr3-5 q r4-6 E 5 E 6 p r5 p r6 Final Scores p = p s p r q = q s q r 5

6 Transla1on of Uncertain1es into Probabilis1c Weights We use domain experts to quan(fy and transform data uncertain(es into the 4 types of probabilis(c weights Example transforma(ons:

7 How can we assign some score (here the color)... Source: Todd

8 ... that allows ranking? Source: Todd

9 Agenda How to model uncertainty in data integra(on? How do we rank? How well, how fast, how robust on real data? A short database research point of view 9

10 Network Reliability Theory ( source- target reachability ) Source- target- reachability: probability that a node is reachable from the start (query) node Query node Func(on # Func(on #2 Func(on #2 = = 0.81 Func(on #1 = 1 Prob(all paths failed) = 1 ( )( ) =

11 Incorpora1ng Uncertainty: Network Reliability Theory score = probability that an answer node is reachable from the start (query) node. s q q p q p Problem: Compu(ng U2 score is #P. p q q p q t 11

12 Why is reliability = reachability hard? The following graph is nasty = hard! Can come in different forms: Wheatstone Bridge :n n:m n:1 Reachability score: = :n 1:n n:1 n:1 12

13 Closed solu1on is possible some1mes Detail: gene ABCC8, upstream node

14 Techniques to perform probabilis1c scoring Naive Monte Carlo simula(on Improved Monte Carlo simula(on Analyze the necessary number of simula(ons Graph reduc(ons (Parallel- serial reduc(ons) Closed solu(on for subgraphs Propaga1on score Deterministc counterparts 14

15 Ignoring correla1ons: the relevance propaga1on model Ignoring correla(ons leads to a local point of view. One equa(on for relevance r for each node n i and each arc a i,j Solve simple equa(on system (closed or itera(vely) ARC a i,j NODE e i,j p i q i,j r i1,j p j n i a i,j n j r i2,j n j r i r i,j r i3,j r j r i,j = r i q i,j r j = (1- i (1- r i,j )) p j 15

16 Example: reliability vs. propaga1on Reliability Propaga(on Reliability = Propaga(on s 0.5 s s u r = u r = u r =

17 Comparing reliability and propaga1on: complexity Reliability Propaga(on global measure combinatorial problem P# = hard Mone Carlo es(mates local measure con(nuous state space P = not hard Itera(ve algorithm 17

18 Agenda How to model uncertainty in data integra(on? How do we rank? How well, how fast, how robust on real data? A short database research point of view 18

19 Experiments: Func1onal gene annota1on 3 ques1ons 1) How well do different approaches perform? [Average precision (AP)] 2) How fast is probabilis(c query evalua(on? [Focus on reliability] 3) Where do you get the probabili(es from? How robust is our system to varia(ons in the input probabili(es? [Sensi(vity analysis] 6 data sources: Pfam, TIGRFAM, NCBIBlast, EntrezProtein, EntrezGen, AmiGo 3 scenarios 1) Well- known func(ons for well- studied proteins (306/20) 2) Less- known func(ons for well- studied proteins (7/3) 3) Unknown func(ons for less- studied proteins (11/11) 19

20 1. How well (1/3): Average Precision Assume 4 out of 10 items are relevant Rank Ranking method 1 relevant precision@k x 1.00 (=1/1) x 1.00 (=2/2) x x 0.75 (=3/4) 0.57 (=4/7) Ranking method 2 relevant precision@k x 1.00 (=1/1) x x x 0.67 (=2/3) 0.75 (=3/4) 0.50 (=4/8) Random AP Averaged over all 10 4 permutabons AP AP as measure for the quality of the ranking seman(cs with regard to ground truth 20

21 1. How well (2/3): Scoring func1ons Scoring func1on Example graph Example score Reliability s t Propaga(on s t InEdge s t 2 incoming edges PathCount s t 3 paths (1 shown) Random AP no score: AP averaged over all ranking permutabons 21

22 1. How well (3/3): AP across 3 scenarios Scenario 1: 306 well- known func(on, 20 well- studied proteins Scenario 2 7 less- known func(ons, 3 well- studied proteins Scenario 1: 11 unknown func(ons, 11 less- studied proteins Observa(on 1: Probabilis(c methods perform berer for predic(ng less- known or previously unknown func(ons! 22

23 2. How fast: Several techniques for speeding up reliability Techniques (not discussed in detail): naive Monte Carlo (N), efficient Monte Carlo (M), instead of simulabons (e4, e5), graph reducbons (R), closed solubon (C) Observa(on 2: Several techniques allowed us to evaluate the reliability seman(cs in ~20ms (propaga(on ~5ms, InEdge and Pathcount ~1ms) 23

24 3. How robust: sensi1vity analysis Our approach depends on transforming uncertainty into probabilisbc weights. How robust is the performance to systemabc variabons in these input parameters? Idea: mulb- way sensibvity analysis p = Lo 1 Lo(p)+ε ε = N(0, σ 2 ) Lo(p) = log( p 1 p ) Observa(on 3: Small random perturba(ons to the ini(al parameters do not nega(vely affect the quality of rankings. The approach is robust! 24

25 Take- way from experiments on real data Uncertainty of informa(on Unknown informa1on Less- known informa1on Well- known informa1on Determinis1c Probabilis1c Informa(on integra(on approach Explicit modeling of uncertain(es as probabili(es increases our ability to predict less- known or previously unknown protein func(ons. This suggests that uncertainty models offer u(lity for knowledge discovery. Small perturba(ons in the input probabili(es (parameters) tend to produce only minor changes in the quality of our result rankings. This suggests that probabilis(c methods are robust against varia(ons in the way uncertain(es are transformed into probabili(es. Several techniques allow us to evaluate probabilis(c rankings efficiently. This suggests that probabilis(c query evalua(on is not as hard for real- world problems as theory indicates. 25

26 Agenda How to model uncertainty in data integra(on? How do we rank? How well, how fast, how robust on real data? A short database research point of view 26

27 Short database background (1/2) Schema ATTEND(student,class) TEACH(class,prof)! DEP(prof,department)! SQL query select!a.student, T.department! from!attend A, TEACH T, DEP D! where!attend.class=teach.class! and!teach.prof=dep.prof! 27

28 Short database background (2/2) Schema R(A,B) S(B,C)! T(C,D)! SQL query select!r.a, T.D! from!r, S, T! where!r.b=s.b! and!s.c=t.c! Datalog q(x,u):-r(x,y),s(y,z),t(z,u)! Conjunc(ve queries: very efficient! 28

29 Probabilis)c databases (1/3) q(x,u):-r(x,y), S(y,z), T(z,u)! R S T A B B C C D a y 1 y 1 z 1 z 1 d a y 2 y 1 z 2 z 2 d y 2 z 2 Which tuples? q(a,d)! 29

30 Probabilis)c databases (2/3) q(x,u):-r p (x,y), S(y,z), T p (z,u)! Which tuples & how likely?! R p S T p A B a y 1 a y 2 a p 1 p 2 y 1 B C p 1 y 1 z 1 z 1 d p 3 p 2 y 1 z 2 z 2 d p 4 y 1 y 2 y 1 y 2 y 2 z 2 C D Nasty graph! Not efficient! p 3 p 4 d Can propagadon help? P[q(a,d)] = p 1 p 3 p 1 p 4 p 2 p 4 = reachability a d z 1 z 2 z 2 30

31 Probabilis)c databases (3/3) q(x,u):-r p (x,y), S(y,z), T p (y,u)! R p S T p A B B C C D a y 1 p 1 y 1 z 1 y 1 d p 3 a y 2 p 2 y 1 z 2 y 2 d p 4 y 2 z 2 q(x,y):-r p (x,y), R p (x,z), T p (z,u)! Non- linear chain queries / self joins. How to define a propagadon semandcs? 31

32 Which ranking seman1cs is appropriate for real data? Input (probabilis1c) data? Ouput ranked results R p 1 1. Hard in general A B a a p 1 a e p 2 Possible world seman(cs ~ reliability SensiBvity of ranking with respect to accur- acy of input probabilit b c p 3 d c p 4 d a p 5 e a p 6 e c p 7 e d p 8 4. Hidden dependencies in the input data in the first place Alterna(ve ranking seman(cs ~ propagabon Decrease in ranking quality due to approximabon Can we get good ranking results for arbitrary queries on real data or at least a good trade- off speed / ranking accuracy? 32

33 Further informa.on PAPER L. Detwiler, W. Ga4erbauer, B. Louie, D. Suciu and P. Tarczy- Hornoch. IntegraEng and Ranking Uncertain ScienEfic Data. In Proceedings of the 25th InternaEonal Conference on Data Engineering, PROJECT WEB PAGE h4p:// DATABASE RESEARCH GROUP h4p://db.cs.washington.edu/ CONTACT Wolfgang Ga4erbauer: THANKS! 33