cient and E ective Similarity Search based on Earth Mover s Distance

Transcription

1 36th International Conference on Very Large Data Bases E cient and E ective Similarity Search over Probabilistic Data based on Earth Mover s Distance Jia Xu 1, Zhenjie Zhang 2, Anthony K.H. Tung 2, Ge Yu 1 1 Northeastern University, China 2 National University of Singapore, Republic of Singapore

2 Agenda Motivation Rltd Related works Definitions Index construction Algorithms for EMD based similarity il i search Experimental results Conclusion

3 Motivation Probabilistic data management Where do probabilities come from? Discrete events? Car ID Color Brand Probability NEU2010 Red Mazda 0.7 NEU2010 Silver Honda 0.3 SGD2010 Black BMW 0.5 Not easy to get the probabilities in real scenario

4 Motivation Probabilistic data management Where do probabilities come from? Discrete events? Car ID Color Brand Probability NEU2010 Red Mazda 0.7 NEU2010 Silver Honda 0.3 SGD2010 Black BMW 0.5 Not easy to get the probabilities in real scenario

5 Motivation Where do probabilities come from? Probability histograms are naturally available

6 Motivation Where do probabilities come from? Probability histograms are naturally available Samples of the pixels Pixel R G B P P P P

7 Motivation Where do probabilities come from? Probability histograms are naturally available Samples of the pixels Pixel R G B P P P P D color histogram

8 Motivation Where do probabilities come from? Probability histograms are naturally available Samples of the pixels Pixel R G B P P P P More examples on sensor network, RFID, etc. 3D color histogram

9 Motivation Queries on probabilistic histograms Accumulated range query Top k query Similarity search over histograms is more important

10 Motivation Queries on probabilistic histograms Accumulated range query Top k query Similarity search over histograms is more important

11 Motivation Distance metric on probabilistic domain? Euclidean distance? L p distance? H H1 H H

12 Motivation Distance metric on probabilistic domain? Euclidean distance? L p distance? H H1 H H

13 Motivation Distance metric on probabilistic domain? Euclidean distance? L p distance? H H1 H H

14 Motivation Distance metric on probabilistic domain? Euclidean distance? L p distance? Against human intuition H H1 L 1 (H 1, H 2 )=2 > L 1 (H 1, H 3 )=1.15 H H

15 Motivation Intuition of the Earth Mover s Distance (EMD) Work flow moved between bins H1 H1 H2 H3

16 Motivation Intuition of the Earth Mover s Distance (EMD) Work flow moved between bins Distance moved betweenbins bins H1 H1 H2 H3

17 Motivation Intuition of the Earth Mover s Distance (EMD) Work flow moved between bins Distance moved betweenbins bins H1 H1 EMD(H 1, H 2 ) < EMD(H 1, H 3 ) H2 H3

18 Motivation Intuition of the Earth Mover s Distance (EMD) Work flow moved between bins Distance moved betweenbins bins H1 H1 Better reflect the human intuition H2 H3

19 Motivation In this paper, we probabilistic records are represented by histograms discuss the problem of similarity search based on the Earth Mover s Distance (EMD) present a new indexing i scheme to answer similarity il it queries based on EMD with efficiency and effectiveness 19

20 Agenda Motivation Rltd Related works Definitions Index construction Algorithms for EMD based similarity il i search Experimental results Conclusion 20

21 RltdW Related Works Scan And Filter framework Ira Assent et al, Approximation techniques for indexing i the earth mover's distance in multimedia databases ICDE 2006 Marc Wichterich et al, Efficient EMD-based similarity search in multimedia databases via flexible dimensionality reduction SIGMOD 2008 Preprocessing: dimensionality reduction Scan: the histogram records are scan one by one and filtered by two lower bound filters on EMD Refinement: if the lower bound filters cannot prune the record, run complete EMD computation 21

22 RltdW Related Works Our work Does not scan the whole data set Index all records with a forest of B+ trees Easy to be implemented in commercial relational dtb database Achieves high scalability and concurrency 22

23 Agenda Motivation Rltd Related works Definitions Index construction Algorithms for EMD based similarity il i search Experimental results Conclusion 23

24 Dfiiti Definitions Formal definition of EMD Linear programming problem P Q

25 Dfiiti Definitions Formal definition of EMD Linear programming problem P Q

26 Dfiiti Definitions Formal definition of EMD Linear programming problem P f 2g3 =0.5 f 56 =0.2 f 78 =0.3 Q

27 Dfiiti Definitions Formal definition of EMD Linear programming problem Constraint 1: cannot send more earth than there is

28 Dfiiti Definitions Formal definition of EMD Linear programming problem Constraint 1: cannot send more earth than there is Constraint 2: cannot receive more earth than it can hold

29 Dfiiti Definitions Formal definition of EMD Linear programming problem Constraint 1: cannot send more earth than there is Constraint 2: cannot receive more earth than it can hold Constraint 3: The amount of earth moved should not be a negative number

30 Dfiiti Definitions Formal definition of EMD Linear programming problem Constraint 1: cannot send more earth than there is Constraint 2: cannot receive more earth than it can hold Constraint 3: The amount of earth moved should not be a negative number Time complexity: O(N 3 logn) negative number

31 Agenda Motivation Rltd Related works Preliminaries Index construction Algorithms for EMD based similarity il i search Experimental results Conclusion 31

32 Pi Primal Dual ld ltransformation Primal problem (EMD)

33 Pi Primal Dual ld ltransformation Primal problem (EMD) Dual problem

34 Pi Primal Dual ld ltransformation Primal problem (EMD) Dual problem

35 Pi Primal Dual ld ltransformation Primal problem (EMD) Dual problem

36 Pi Primal Dual ld ltransformation Feasible solution F = {f ij } in the primal problem Ф= {π i, Ф j } in the dual problem

37 Pi Primal Dual ld ltransformation Feasible solution F = {f ij } in the primal problem Ф= {π i, Ф j } in the dual problem Ф j p[i]+ π i q[j] EMD(p,q) f ij d ij

38 Pi Primal Dual ld ltransformation Feasible solution F = {f ij } in the primal problem Ф= {π i, Ф j } in the dual problem Ф j p[i]+ π i q[j] EMD(p,q) f ij d ij Dual Space Lower bound

39 Pi Primal Dual ld ltransformation Feasible solution F = {f ij } in the primal problem Ф= {π i, Ф j } in the dual problem Ф j p[i]+ π i q[j] EMD(p,q) f ij d ij Dual Space Lower bound Primal Space(EMD) Upper bound

40 Pi Primal Dual ld ltransformation Two good properties of dual space Independency constrains are independent to those histograms involved (i.e., p and q ) can derive any feasible solution Ф ={Ф i, π j } regardless of p and q 40

41 Pi Primal Dual ld ltransformation Two good properties of dual space Independency constrains are independent to those histograms involved (i.e., p and q ) can derive any feasible solution Ф ={Ф i, π j} regardless of p and q Lower bounding a feasible solution Ф ={Ф i, π j }, derives a lower bound (i.e., φ i pi [] + π i q [ j ] ) for EMD(p,q) i i j j can use lower bound to filter out those no hit records 41

42 Index Construction ti Dual space:

43 Index Construction ti Dual space: Ф π

44 Index Construction ti Dual space: P Ф π

45 Index Construction ti Dual space: P Ф π key = Ф i p i

46 Index Construction ti Dual space: P Ф Q π key = Ф i p i

47 Index Construction ti Dual space: P Ф Q π key = counterkey = Ф i p i π i q i

48 Index Construction ti Dual space: P Ф Q π key = counterkey = Ф i p i π i q i key + counterkey = Ф i p i + π i q i

49 Index Construction ti Dual space: P Ф Q π key = counterkey = Ф i p i π i q i EMD(P, Q) key + counterkey Lower bound = Ф i p i + π i q i

50 Index Construction ti Dual space: P Ф Q π S 5 key = counterkey = Ф i p i π i q i EMD(P, Q) key + counterkey Lower bound = Ф i p i + π i q i

51 Index Construction ti Dual space: P Ф Q π S 5 key = Ф i p i counterkey = π i q i EMD(P, Q) key + counterkey Lower bound = Ф i p i + π i q i

52 Index Construction ti Dual space: P Ф Q π S 5 key = Ф i p i counterkey = π i q i EMD(P, Q) key + counterkey Lower bound = Ф i p i + π i q i

53 Index Construction ti B+ tree filter S 5

54 Index Construction ti B+ tree filter S 5

55 Index Construction ti B+ tree filter S 5

56 Index Construction ti A forest of B+ trees

57 Index Construction ti A forest of B+ trees Ф 1 = <Ф i, π j > Ф 2 = <Ф i, π j >

58 Index Construction ti A forest of B+ trees Ф 1 = <Ф i, π j > Ф 2 = <Ф i, π j > S1 S2 S3 S4 S5 S6 S7 S8 S15 S9 S13 S5 S14 S12 S7

59 Agenda Motivation Rltd Related works Definitions Index Structure Construction Algorithms for EMD based Similarity il i Search Experimental results Conclusion 59

60 Algo. Range Query T 1 T 2 S1 S2 S9 S4 S5 S3 S6 S7 S8 S15 S9 S3 S5 S14 S12 S7

61 Algo. Range Query T 1 T 2 S1 S2 S9 S4 S5 S3 S6 S7 S8 S15 S9 S3 S5 S14 S12 S7

62 Algo. Range Query T 1 T 2 S1 S2 S9 S4 S5 S3 S6 S7 S8 S15 S9 S3 S5 S14 S12 S7 S3 S5

63 Algo. Range Query T 1 T 2 S1 S2 S9 S4 S5 S3 S6 S7 S8 S15 S9 S3 S5 S14 S12 S7 S3 S5 R EMD Filter (Sigmod 08)

64 Algo. Range Query T 1 T 2 S1 S2 S9 S4 S5 S3 S6 S7 S8 S15 S9 S3 S5 S14 S12 S7 S3 S5 R EMD Filter (Sigmod 08) LB IM Filter (ICDE 06)

65 Algo. Range Query T 1 T 2 S1 S2 S9 S4 S5 S3 S6 S7 S8 S15 S9 S3 S5 S14 S12 S7 S3 S5 R EMD Filter (Sigmod 08) LB IM Filter (ICDE 06) UB p Filter

66 Algo. Range Query T 1 T 2 S1 S2 S9 S4 S5 S3 S6 S7 S8 S15 S9 S3 S5 S14 S12 S7 S3 S5 R EMD Filter (Sigmod 08) LB IM Filter (ICDE 06) UB p Filter Exact EMD cal.

67 Algo. knn Query (k=2) T 1 T 2 S1 S2 S3 S4 S5 S6 S7 S8 S15 S9 S13 S5 S14 S12 S7

68 Algo. knn Query (k=2) T 1 T 2 S1 S2 S3 S4 S5 S6 S7 S8 S15 S9 S13 S5 S14 S12 S7 Key(q,Ф 1 ) Key(q,Ф 2 )

69 Algo. knn Query (k=2) T 1 T 2 S1 S2 S3 S4 S5 S6 S7 S8 S15 S9 S13 S5 S14 S12 S7 CR1 CL1 Key(q,Ф 1 ) Key(q,Ф 2 ) CR2 S6 S7 S8 S14 S12 S7 CL2 S5 S4 S3 S2 S1 S5 S13 S9 S15

70 Algo. knn Query (k=2) T 1 T 2 S1 S2 S3 S4 S5 S6 S7 S8 S15 S9 S13 S5 S14 S12 S7 CR1 CL1 CR2 S6 S7 S8 S14 S12 S7 CL2 S5 S4 S3 S2 S1 S5 S13 S9 S15 Buffer

71 Algo. knn Query (k=2) T 1 T 2 S1 S2 S3 S4 S5 S6 S7 S8 S15 S9 S13 S5 S14 S12 S7 CR1 CL1 CR2 S6 S7 S8 S14 S12 S7 CL2 S5 S4 S3 S2 S1 S5 S13 S9 S15 Buffer S5 S5 2 S6 1 S14 1

72 Algo. knn Query (k=2) T 1 T 2 S1 S2 S3 S4 S5 S6 S7 S8 S15 S9 S13 S5 S14 S12 S7 CR1 CL1 CR2 S6 S7 S8 S14 S12 S7 CL2 S5 S4 S3 S2 S1 S5 S13 S9 S15 Buffer S5 S5 S5 2 S6 1 S14 1 S6 1 S14 1 S7 1 S4 1 S12 1 S13 1

73 Algo. knn Query (k=2) T 1 T 2 S1 S2 S3 S4 S5 S6 S7 S8 S15 S9 S13 S5 S14 S12 S7 CR1 CL1 CR2 S6 S7 S8 S14 S12 S7 CL2 S5 S4 S3 S2 S1 S5 S13 S9 S15 Buffer S5 S5 S5 S7 S5 2 S6 1 S14 1 S6 1 S14 1 S7 1 S4 1 S12 1 S13 1 S7 2 S14 1 S6 1 S4 1 S12 1 S13 1 S8 1 S1 1 S9 1

74 Algo. knn Query (k=2) T 1 T 2 S1 S2 S3 S4 S5 S6 S7 S8 S15 S9 S13 S5 S14 S12 S7 CR1 CL1 CR2 S6 S7 S8 S14 S12 S7 CL2 S5 S4 S3 S2 S1 S5 S13 S9 S15 Buffer S5 S5 S5 S7 S5 2 S6 1 S14 1 S6 1 S14 1 S7 1 S4 1 S12 1 S13 1 S7 2 S14 1 S6 1 S4 1 S12 1 S13 1 S8 1 S1 1 S9 1

75 Agenda Motivation Rltd Related works Definitions Index structure construction Algorithms for EMD based similarity il i search Experimental results Conclusion

76 Experimental lresults Data set RETINA1 Dimension: 96 Cardinality = 3,932 IRMA Dimension: 199 Cardinality = 10,000 DBLP Dimension: 8 Cardinality = 250,000000

77 Experimental lresults Comparison algorithms TBI (Tree Based Index) TBI C: Using clustering based method to find feasible solution TBI R: Using random sampling based method to find feasible solution SAR (Scan And Refinement) Measurement CPU time, # of EMD refinement 77

78 Experimental lresults: Range Queries

79 Experimental lresults: k NN Queries

80 Conclusions We present a new B + tree based indexing scheme for the general purposesofsimilarityof similarity search on Earth Mover's Distance Our index method relies on the primal dual ltheory to construct mapping functions from the original probabilistic space to one dimensional i domain Our B+ tree based index framework is High scalability High efficiency

81 For more information Introduction.html ti

82 36th International Conference on Very Large Data Bases Thank you! contact: ed